{"id":2911,"date":"2024-03-13T05:38:26","date_gmt":"2024-03-13T05:38:26","guid":{"rendered":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/"},"modified":"2024-03-19T14:54:55","modified_gmt":"2024-03-19T14:54:55","slug":"how-is-metadata-managed-in-hadoop","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/","title":{"rendered":"How is metadata managed in Hadoop?"},"content":{"rendered":"

Metadata in Hadoop is typically managed by the Hadoop Distributed File System (HDFS) and the ResourceManager of Hadoop YARN.<\/p>\n

    \n
  1. Metadata Management in HDFS: HDFS maintains metadata about files and directories, including filenames, file sizes, creation time, and access permissions. This metadata is stored in the NameNode, which regularly saves the metadata to the EditLog and FsImage files on disk to prevent data loss. Users can manage metadata in HDFS by using Hadoop’s command-line tools or API, such as creating, deleting, moving files or directories.<\/li>\n
  2. YARN Metadata Management: YARN is responsible for managing resources in the cluster and maintaining metadata information about jobs and tasks. The YARN ResourceManager tracks resource usage, job status, and task status in the cluster, and allocates and schedules resources as needed. Users can manage job and task metadata using YARN’s command line tools or API, such as submitting jobs, checking job status, and terminating jobs.<\/li>\n<\/ol>\n

    Overall, the metadata management in Hadoop is performed collaboratively by HDFS and YARN, allowing users to manage and operate metadata information in the cluster using the respective tools and APIs.<\/p>\n","protected":false},"excerpt":{"rendered":"

    Metadata in Hadoop is typically managed by the Hadoop Distributed File System (HDFS) and the ResourceManager of Hadoop YARN. Metadata Management in HDFS: HDFS maintains metadata about files and directories, including filenames, file sizes, creation time, and access permissions. This metadata is stored in the NameNode, which regularly saves the metadata to the EditLog and […]<\/p>\n","protected":false},"author":11,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-2911","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\nHow is metadata managed in Hadoop? - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How is metadata managed in Hadoop?\" \/>\n<meta property=\"og:description\" content=\"Metadata in Hadoop is typically managed by the Hadoop Distributed File System (HDFS) and the ResourceManager of Hadoop YARN. Metadata Management in HDFS: HDFS maintains metadata about files and directories, including filenames, file sizes, creation time, and access permissions. This metadata is stored in the NameNode, which regularly saves the metadata to the EditLog and […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-13T05:38:26+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-19T14:54:55+00:00\" \/>\n<meta name=\"author\" content=\"Olivia Parker\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:site\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Olivia Parker\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/\"},\"author\":{\"name\":\"Olivia Parker\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/3ff7b3da0e45ac5dbbef2502f3cea8d9\"},\"headline\":\"How is metadata managed in Hadoop?\",\"datePublished\":\"2024-03-13T05:38:26+00:00\",\"dateModified\":\"2024-03-19T14:54:55+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/\"},\"wordCount\":187,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/\",\"name\":\"How is metadata managed in Hadoop? - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\"},\"datePublished\":\"2024-03-13T05:38:26+00:00\",\"dateModified\":\"2024-03-19T14:54:55+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.silicloud.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How is metadata managed in Hadoop?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"name\":\"Silicon Cloud Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\",\"name\":\"Silicon Cloud Blog\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"contentUrl\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"width\":1024,\"height\":1024,\"caption\":\"Silicon Cloud Blog\"},\"image\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\",\"https:\/\/twitter.com\/SiliCloudGlobal\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/3ff7b3da0e45ac5dbbef2502f3cea8d9\",\"name\":\"Olivia Parker\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/56c66f189ba32a6f9eb50f31a38fe774e2a725c213d4070835ccc51b8fbbc54b?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/56c66f189ba32a6f9eb50f31a38fe774e2a725c213d4070835ccc51b8fbbc54b?s=96&d=mm&r=g\",\"caption\":\"Olivia Parker\"},\"url\":\"https:\/\/www.silicloud.com\/blog\/author\/oliviaparker\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How is metadata managed in Hadoop? - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/","og_locale":"en_US","og_type":"article","og_title":"How is metadata managed in Hadoop?","og_description":"Metadata in Hadoop is typically managed by the Hadoop Distributed File System (HDFS) and the ResourceManager of Hadoop YARN. Metadata Management in HDFS: HDFS maintains metadata about files and directories, including filenames, file sizes, creation time, and access permissions. This metadata is stored in the NameNode, which regularly saves the metadata to the EditLog and […]","og_url":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/","og_site_name":"Blog - Silicon Cloud","article_publisher":"https:\/\/www.facebook.com\/SiliCloudGlobal\/","article_published_time":"2024-03-13T05:38:26+00:00","article_modified_time":"2024-03-19T14:54:55+00:00","author":"Olivia Parker","twitter_card":"summary_large_image","twitter_creator":"@SiliCloudGlobal","twitter_site":"@SiliCloudGlobal","twitter_misc":{"Written by":"Olivia Parker","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/#article","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/"},"author":{"name":"Olivia Parker","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/3ff7b3da0e45ac5dbbef2502f3cea8d9"},"headline":"How is metadata managed in Hadoop?","datePublished":"2024-03-13T05:38:26+00:00","dateModified":"2024-03-19T14:54:55+00:00","mainEntityOfPage":{"@id":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/"},"wordCount":187,"commentCount":0,"publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/","url":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/","name":"How is metadata managed in Hadoop? - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/#website"},"datePublished":"2024-03-13T05:38:26+00:00","dateModified":"2024-03-19T14:54:55+00:00","breadcrumb":{"@id":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/blog\/how-is-metadata-managed-in-hadoop\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.silicloud.com\/blog\/"},{"@type":"ListItem","position":2,"name":"How is metadata managed in Hadoop?"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/blog\/#website","url":"https:\/\/www.silicloud.com\/blog\/","name":"Silicon Cloud Blog","description":"","publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.silicloud.com\/blog\/#organization","name":"Silicon Cloud Blog","url":"https:\/\/www.silicloud.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","contentUrl":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","width":1024,"height":1024,"caption":"Silicon Cloud Blog"},"image":{"@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/SiliCloudGlobal\/","https:\/\/twitter.com\/SiliCloudGlobal"]},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/3ff7b3da0e45ac5dbbef2502f3cea8d9","name":"Olivia Parker","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/56c66f189ba32a6f9eb50f31a38fe774e2a725c213d4070835ccc51b8fbbc54b?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/56c66f189ba32a6f9eb50f31a38fe774e2a725c213d4070835ccc51b8fbbc54b?s=96&d=mm&r=g","caption":"Olivia Parker"},"url":"https:\/\/www.silicloud.com\/blog\/author\/oliviaparker\/"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/2911","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/users\/11"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/comments?post=2911"}],"version-history":[{"count":1,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/2911\/revisions"}],"predecessor-version":[{"id":35766,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/2911\/revisions\/35766"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/media?parent=2911"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/categories?post=2911"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/tags?post=2911"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}