{"id":23197,"date":"2024-03-16T00:57:24","date_gmt":"2024-03-16T00:57:24","guid":{"rendered":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/"},"modified":"2024-03-22T00:41:33","modified_gmt":"2024-03-22T00:41:33","slug":"how-to-import-hive-data-into-elasticsearch","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/","title":{"rendered":"How to import hive data into Elasticsearch?"},"content":{"rendered":"

To import Hive data into Elasticsearch (ES), you can follow these steps:<\/p>\n

    \n
  1. Export the results of a Hive query to a CSV file. You can use the Hive command INSERT OVERWRITE LOCAL DIRECTORY to export the query results to a local directory, and set the output file format to CSV. For example:<\/li>\n<\/ol>\n
    INSERT<\/span> OVERWRITE LOCAL<\/span> DIRECTORY '\/tmp\/hive_data'<\/span>\r\nROW<\/span> FORMAT DELIMITED\r\nFIELDS TERMINATED BY<\/span> ','<\/span>\r\nSELECT<\/span> *<\/span> FROM<\/span> your_table;\r\n<\/code><\/pre>\n
    This will store the query results in a CSV file under the directory \/tmp\/hive_data.<\/p>\n
      \n
    1. Please provide your input.<\/li>\n
    2. Screen<\/li>\n
    3. the result<\/li>\n<\/ol>\n
      input {\r\n  file {\r\n    path => \"\/tmp\/hive_data\/*.csv\"\r\n    start_position => \"beginning\"\r\n    sincedb_path => \"\/dev\/null\"\r\n  }\r\n}\r\n\r\nfilter {\r\n  csv {\r\n    separator => \",\"\r\n    columns => [\"col1\", \"col2\", \"col3\"]  # \u4e0eCSV\u6587\u4ef6\u7684\u5217\u5bf9\u5e94\r\n  }\r\n}\r\n\r\noutput {\r\n  elasticsearch {\r\n    hosts => [\"localhost:9200\"]\r\n    index => \"your_index\"\r\n  }\r\n}\r\n<\/code><\/pre>\n
        \n
      1. To run Logstash, use the command “logstash -f your_config.conf” where “your_config.conf” is the path to the configuration file of Logstash.<\/li>\n
      2. Wait for Logstash to complete the import. Logstash will automatically read the CSV file and import the data into Elasticsearch.<\/li>\n<\/ol>\n
        Before executing these steps, make sure that Hive, Elasticsearch, and Logstash are installed and configured properly, and ensure that the CSV file containing Hive query results matches the path and column names specified in Logstash’s configuration file.<\/p>\n","protected":false},"excerpt":{"rendered":"
        To import Hive data into Elasticsearch (ES), you can follow these steps: Export the results of a Hive query to a CSV file. You can use the Hive command INSERT OVERWRITE LOCAL DIRECTORY to export the query results to a local directory, and set the output file format to CSV. For example: INSERT OVERWRITE LOCAL […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-23197","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\nHow to import hive data into Elasticsearch? - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to import hive data into Elasticsearch?\" \/>\n<meta property=\"og:description\" content=\"To import Hive data into Elasticsearch (ES), you can follow these steps: Export the results of a Hive query to a CSV file. You can use the Hive command INSERT OVERWRITE LOCAL DIRECTORY to export the query results to a local directory, and set the output file format to CSV. For example: INSERT OVERWRITE LOCAL […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-16T00:57:24+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-22T00:41:33+00:00\" \/>\n<meta name=\"author\" content=\"William Carter\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:site\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"William Carter\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/\"},\"author\":{\"name\":\"William Carter\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/f697031891aacefc4b681d139781d3c0\"},\"headline\":\"How to import hive data into Elasticsearch?\",\"datePublished\":\"2024-03-16T00:57:24+00:00\",\"dateModified\":\"2024-03-22T00:41:33+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/\"},\"wordCount\":164,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/\",\"name\":\"How to import hive data into Elasticsearch? - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\"},\"datePublished\":\"2024-03-16T00:57:24+00:00\",\"dateModified\":\"2024-03-22T00:41:33+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.silicloud.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to import hive data into Elasticsearch?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"name\":\"Silicon Cloud Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\",\"name\":\"Silicon Cloud Blog\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"contentUrl\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"width\":1024,\"height\":1024,\"caption\":\"Silicon Cloud Blog\"},\"image\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\",\"https:\/\/twitter.com\/SiliCloudGlobal\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/f697031891aacefc4b681d139781d3c0\",\"name\":\"William Carter\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/1786698071dd8d74bec894b512f9e3c610c3a2a32985f67e688976cee3c8bbef?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/1786698071dd8d74bec894b512f9e3c610c3a2a32985f67e688976cee3c8bbef?s=96&d=mm&r=g\",\"caption\":\"William Carter\"},\"url\":\"https:\/\/www.silicloud.com\/blog\/author\/williamcarter\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How to import hive data into Elasticsearch? - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/","og_locale":"en_US","og_type":"article","og_title":"How to import hive data into Elasticsearch?","og_description":"To import Hive data into Elasticsearch (ES), you can follow these steps: Export the results of a Hive query to a CSV file. You can use the Hive command INSERT OVERWRITE LOCAL DIRECTORY to export the query results to a local directory, and set the output file format to CSV. For example: INSERT OVERWRITE LOCAL […]","og_url":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/","og_site_name":"Blog - Silicon Cloud","article_publisher":"https:\/\/www.facebook.com\/SiliCloudGlobal\/","article_published_time":"2024-03-16T00:57:24+00:00","article_modified_time":"2024-03-22T00:41:33+00:00","author":"William Carter","twitter_card":"summary_large_image","twitter_creator":"@SiliCloudGlobal","twitter_site":"@SiliCloudGlobal","twitter_misc":{"Written by":"William Carter","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/#article","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/"},"author":{"name":"William Carter","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/f697031891aacefc4b681d139781d3c0"},"headline":"How to import hive data into Elasticsearch?","datePublished":"2024-03-16T00:57:24+00:00","dateModified":"2024-03-22T00:41:33+00:00","mainEntityOfPage":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/"},"wordCount":164,"commentCount":0,"publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/","url":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/","name":"How to import hive data into Elasticsearch? - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/#website"},"datePublished":"2024-03-16T00:57:24+00:00","dateModified":"2024-03-22T00:41:33+00:00","breadcrumb":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/blog\/how-to-import-hive-data-into-elasticsearch\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.silicloud.com\/blog\/"},{"@type":"ListItem","position":2,"name":"How to import hive data into Elasticsearch?"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/blog\/#website","url":"https:\/\/www.silicloud.com\/blog\/","name":"Silicon Cloud Blog","description":"","publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.silicloud.com\/blog\/#organization","name":"Silicon Cloud Blog","url":"https:\/\/www.silicloud.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","contentUrl":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","width":1024,"height":1024,"caption":"Silicon Cloud Blog"},"image":{"@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/SiliCloudGlobal\/","https:\/\/twitter.com\/SiliCloudGlobal"]},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/f697031891aacefc4b681d139781d3c0","name":"William Carter","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/1786698071dd8d74bec894b512f9e3c610c3a2a32985f67e688976cee3c8bbef?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1786698071dd8d74bec894b512f9e3c610c3a2a32985f67e688976cee3c8bbef?s=96&d=mm&r=g","caption":"William Carter"},"url":"https:\/\/www.silicloud.com\/blog\/author\/williamcarter\/"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/23197","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/comments?post=23197"}],"version-history":[{"count":1,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/23197\/revisions"}],"predecessor-version":[{"id":57160,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/23197\/revisions\/57160"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/media?parent=23197"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/categories?post=23197"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/tags?post=23197"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}