{"id":2921,"date":"2024-03-13T05:39:09","date_gmt":"2024-03-13T05:39:09","guid":{"rendered":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/"},"modified":"2024-03-19T14:56:32","modified_gmt":"2024-03-19T14:56:32","slug":"how-does-distributed-computing-work-in-hadoop","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/","title":{"rendered":"How does distributed computing work in Hadoop?"},"content":{"rendered":"

In Hadoop, distributed computing is achieved through the Hadoop Distributed File System (HDFS) and the MapReduce computing model.<\/p>\n

    \n
  1. HDFS is a distributed file system designed for storing large amounts of data. The data is divided into multiple blocks and replicated across different nodes in the cluster to ensure high reliability and availability of data.<\/li>\n
  2. MapReduce is a programming model used for parallel computation of large-scale datasets, dividing the task into Map and Reduce phases. During the Map phase, data is split into smaller chunks and processed in parallel by different nodes. The results from the Map phase are then combined and summarized during the Reduce phase to obtain the final computation result.<\/li>\n<\/ol>\n

    The workflow of distributed computing in Hadoop is as follows:<\/p>\n

      \n
    1. The client stores data in HDFS and submits MapReduce jobs to the ResourceManager (YARN).<\/li>\n
    2. The resource manager assigns tasks to different nodes in the cluster, with each node running Map and Reduce tasks.<\/li>\n
    3. The Map task processes data in parallel on data blocks and generates intermediate results.<\/li>\n
    4. The Reduce task involves aggregating and merging intermediate results to obtain the final outcome.<\/li>\n
    5. The final result is written back to HDFS for clients to read from.<\/li>\n<\/ol>\n

      Hadoop enables distributed computing through the HDFS and MapReduce models, allowing for efficient processing of large datasets.<\/p>\n","protected":false},"excerpt":{"rendered":"

      In Hadoop, distributed computing is achieved through the Hadoop Distributed File System (HDFS) and the MapReduce computing model. HDFS is a distributed file system designed for storing large amounts of data. The data is divided into multiple blocks and replicated across different nodes in the cluster to ensure high reliability and availability of data. MapReduce […]<\/p>\n","protected":false},"author":8,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[1],"tags":[],"class_list":["post-2921","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"yoast_head":"\nHow does distributed computing work in Hadoop? - Blog - Silicon Cloud<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How does distributed computing work in Hadoop?\" \/>\n<meta property=\"og:description\" content=\"In Hadoop, distributed computing is achieved through the Hadoop Distributed File System (HDFS) and the MapReduce computing model. HDFS is a distributed file system designed for storing large amounts of data. The data is divided into multiple blocks and replicated across different nodes in the cluster to ensure high reliability and availability of data. MapReduce […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-13T05:39:09+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-03-19T14:56:32+00:00\" \/>\n<meta name=\"author\" content=\"William Carter\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:site\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"William Carter\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/\"},\"author\":{\"name\":\"William Carter\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/f697031891aacefc4b681d139781d3c0\"},\"headline\":\"How does distributed computing work in Hadoop?\",\"datePublished\":\"2024-03-13T05:39:09+00:00\",\"dateModified\":\"2024-03-19T14:56:32+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/\"},\"wordCount\":220,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/\",\"name\":\"How does distributed computing work in Hadoop? - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\"},\"datePublished\":\"2024-03-13T05:39:09+00:00\",\"dateModified\":\"2024-03-19T14:56:32+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.silicloud.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How does distributed computing work in Hadoop?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"name\":\"Silicon Cloud Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\",\"name\":\"Silicon Cloud Blog\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"contentUrl\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"width\":1024,\"height\":1024,\"caption\":\"Silicon Cloud Blog\"},\"image\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\",\"https:\/\/twitter.com\/SiliCloudGlobal\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/f697031891aacefc4b681d139781d3c0\",\"name\":\"William Carter\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/1786698071dd8d74bec894b512f9e3c610c3a2a32985f67e688976cee3c8bbef?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/1786698071dd8d74bec894b512f9e3c610c3a2a32985f67e688976cee3c8bbef?s=96&d=mm&r=g\",\"caption\":\"William Carter\"},\"url\":\"https:\/\/www.silicloud.com\/blog\/author\/williamcarter\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"How does distributed computing work in Hadoop? - Blog - Silicon Cloud","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/","og_locale":"en_US","og_type":"article","og_title":"How does distributed computing work in Hadoop?","og_description":"In Hadoop, distributed computing is achieved through the Hadoop Distributed File System (HDFS) and the MapReduce computing model. HDFS is a distributed file system designed for storing large amounts of data. The data is divided into multiple blocks and replicated across different nodes in the cluster to ensure high reliability and availability of data. MapReduce […]","og_url":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/","og_site_name":"Blog - Silicon Cloud","article_publisher":"https:\/\/www.facebook.com\/SiliCloudGlobal\/","article_published_time":"2024-03-13T05:39:09+00:00","article_modified_time":"2024-03-19T14:56:32+00:00","author":"William Carter","twitter_card":"summary_large_image","twitter_creator":"@SiliCloudGlobal","twitter_site":"@SiliCloudGlobal","twitter_misc":{"Written by":"William Carter","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/#article","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/"},"author":{"name":"William Carter","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/f697031891aacefc4b681d139781d3c0"},"headline":"How does distributed computing work in Hadoop?","datePublished":"2024-03-13T05:39:09+00:00","dateModified":"2024-03-19T14:56:32+00:00","mainEntityOfPage":{"@id":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/"},"wordCount":220,"commentCount":0,"publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/","url":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/","name":"How does distributed computing work in Hadoop? - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/#website"},"datePublished":"2024-03-13T05:39:09+00:00","dateModified":"2024-03-19T14:56:32+00:00","breadcrumb":{"@id":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/blog\/how-does-distributed-computing-work-in-hadoop\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.silicloud.com\/blog\/"},{"@type":"ListItem","position":2,"name":"How does distributed computing work in Hadoop?"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/blog\/#website","url":"https:\/\/www.silicloud.com\/blog\/","name":"Silicon Cloud Blog","description":"","publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.silicloud.com\/blog\/#organization","name":"Silicon Cloud Blog","url":"https:\/\/www.silicloud.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","contentUrl":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","width":1024,"height":1024,"caption":"Silicon Cloud Blog"},"image":{"@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/SiliCloudGlobal\/","https:\/\/twitter.com\/SiliCloudGlobal"]},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/f697031891aacefc4b681d139781d3c0","name":"William Carter","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/1786698071dd8d74bec894b512f9e3c610c3a2a32985f67e688976cee3c8bbef?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/1786698071dd8d74bec894b512f9e3c610c3a2a32985f67e688976cee3c8bbef?s=96&d=mm&r=g","caption":"William Carter"},"url":"https:\/\/www.silicloud.com\/blog\/author\/williamcarter\/"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/2921","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/users\/8"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/comments?post=2921"}],"version-history":[{"count":1,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/2921\/revisions"}],"predecessor-version":[{"id":35777,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/2921\/revisions\/35777"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/media?parent=2921"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/categories?post=2921"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/tags?post=2921"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}