{"id":3680,"date":"2024-03-13T07:18:09","date_gmt":"2024-03-13T07:18:09","guid":{"rendered":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/"},"modified":"2025-07-30T19:42:47","modified_gmt":"2025-07-30T19:42:47","slug":"how-to-perform-model-distillation-in-keras","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/","title":{"rendered":"Keras Model Distillation: Step-by-Step Guide"},"content":{"rendered":"<p>Model distillation is a method of training a larger, complex model, and then approximating it with a smaller model. In Keras, model distillation can be done through the following steps:<\/p>\n<ol>\n<li>Define the original model and the smaller model: Firstly establish a larger, complex model as the original model, then define a smaller model as the distilled model.<\/li>\n<li>Prepare the dataset: Get the dataset ready for training, typically the dataset used to train the original model.<\/li>\n<li>Train the base model: Train the base model with the dataset and save the weights of the base model.<\/li>\n<li>Generate soft labels using the original model: Predict the dataset using the original model to obtain soft labels.<\/li>\n<li>Train a distilled model: Train using a distilled model and soft labels to make the distilled model as close as possible to the original model.<\/li>\n<\/ol>\n<p>Here is a simple example code demonstrating how to perform model distillation in Keras.<\/p>\n<pre class=\"post-pre\"><code><span class=\"hljs-keyword\">from<\/span> keras.models <span class=\"hljs-keyword\">import<\/span> Sequential\r\n<span class=\"hljs-keyword\">from<\/span> keras.layers <span class=\"hljs-keyword\">import<\/span> Dense\r\n<span class=\"hljs-keyword\">from<\/span> keras.optimizers <span class=\"hljs-keyword\">import<\/span> Adam\r\n\r\n<span class=\"hljs-comment\"># \u5b9a\u4e49\u539f\u59cb\u6a21\u578b<\/span>\r\noriginal_model = Sequential()\r\noriginal_model.add(Dense(<span class=\"hljs-number\">64<\/span>, activation=<span class=\"hljs-string\">'relu'<\/span>, input_shape=(<span class=\"hljs-number\">100<\/span>,)))\r\noriginal_model.add(Dense(<span class=\"hljs-number\">64<\/span>, activation=<span class=\"hljs-string\">'relu'<\/span>))\r\noriginal_model.add(Dense(<span class=\"hljs-number\">10<\/span>, activation=<span class=\"hljs-string\">'softmax'<\/span>))\r\n\r\n<span class=\"hljs-comment\"># \u7f16\u8bd1\u539f\u59cb\u6a21\u578b<\/span>\r\noriginal_model.<span class=\"hljs-built_in\">compile<\/span>(optimizer=<span class=\"hljs-string\">'adam'<\/span>, loss=<span class=\"hljs-string\">'categorical_crossentropy'<\/span>, metrics=[<span class=\"hljs-string\">'accuracy'<\/span>])\r\n\r\n<span class=\"hljs-comment\"># \u8bad\u7ec3\u539f\u59cb\u6a21\u578b<\/span>\r\noriginal_model.fit(X_train, y_train, epochs=<span class=\"hljs-number\">10<\/span>, batch_size=<span class=\"hljs-number\">32<\/span>)\r\n\r\n<span class=\"hljs-comment\"># \u4f7f\u7528\u539f\u59cb\u6a21\u578b\u9884\u6d4b\u751f\u6210\u8f6f\u6807\u7b7e<\/span>\r\nsoft_labels = original_model.predict(X_train)\r\n\r\n<span class=\"hljs-comment\"># \u5b9a\u4e49\u84b8\u998f\u6a21\u578b<\/span>\r\ndistilled_model = Sequential()\r\ndistilled_model.add(Dense(<span class=\"hljs-number\">32<\/span>, activation=<span class=\"hljs-string\">'relu'<\/span>, input_shape=(<span class=\"hljs-number\">100<\/span>,)))\r\ndistilled_model.add(Dense(<span class=\"hljs-number\">32<\/span>, activation=<span class=\"hljs-string\">'relu'<\/span>))\r\ndistilled_model.add(Dense(<span class=\"hljs-number\">10<\/span>, activation=<span class=\"hljs-string\">'softmax'<\/span>))\r\n\r\n<span class=\"hljs-comment\"># \u7f16\u8bd1\u84b8\u998f\u6a21\u578b<\/span>\r\ndistilled_model.<span class=\"hljs-built_in\">compile<\/span>(optimizer=<span class=\"hljs-string\">'adam'<\/span>, loss=<span class=\"hljs-string\">'categorical_crossentropy'<\/span>, metrics=[<span class=\"hljs-string\">'accuracy'<\/span>])\r\n\r\n<span class=\"hljs-comment\"># \u8bad\u7ec3\u84b8\u998f\u6a21\u578b<\/span>\r\ndistilled_model.fit(X_train, soft_labels, epochs=<span class=\"hljs-number\">10<\/span>, batch_size=<span class=\"hljs-number\">32<\/span>)\r\n<\/code><\/pre>\n<p>In the example above, a base model and a distilled model are first defined. The base model is then trained and used to predict soft labels. Finally, the distilled model is trained using the soft labels to approximate the base model as closely as possible.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Model distillation is a method of training a larger, complex model, and then approximating it with a smaller model. In Keras, model distillation can be done through the following steps: Define the original model and the smaller model: Firstly establish a larger, complex model as the original model, then define a smaller model as the [&hellip;]<\/p>\n","protected":false},"author":13,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[1],"tags":[960,1269,75,2355,944],"class_list":["post-3680","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-deep-learning","tag-keras-tutorial","tag-machine-learning","tag-model-distillation","tag-neural-networks"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>Keras Model Distillation: Step-by-Step Guide - Blog - Silicon Cloud<\/title>\n<meta name=\"description\" content=\"Learn how to perform model distillation in Keras: define models, prepare data, train base model &amp; create efficient distilled models.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Keras Model Distillation: Step-by-Step Guide\" \/>\n<meta property=\"og:description\" content=\"Learn how to perform model distillation in Keras: define models, prepare data, train base model &amp; create efficient distilled models.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-13T07:18:09+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-07-30T19:42:47+00:00\" \/>\n<meta name=\"author\" content=\"Isabella Edwards\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:site\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Isabella Edwards\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/\"},\"author\":{\"name\":\"Isabella Edwards\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/5579144e23c225c8188167f3e3f888dd\"},\"headline\":\"Keras Model Distillation: Step-by-Step Guide\",\"datePublished\":\"2024-03-13T07:18:09+00:00\",\"dateModified\":\"2025-07-30T19:42:47+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/\"},\"wordCount\":201,\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"keywords\":[\"Deep Learning\",\"Keras Tutorial\",\"machine learning\",\"Model Distillation\",\"Neural Networks\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/\",\"name\":\"Keras Model Distillation: Step-by-Step Guide - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\"},\"datePublished\":\"2024-03-13T07:18:09+00:00\",\"dateModified\":\"2025-07-30T19:42:47+00:00\",\"description\":\"Learn how to perform model distillation in Keras: define models, prepare data, train base model & create efficient distilled models.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.silicloud.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Keras Model Distillation: Step-by-Step Guide\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"name\":\"Silicon Cloud Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\",\"name\":\"Silicon Cloud Blog\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"contentUrl\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"width\":1024,\"height\":1024,\"caption\":\"Silicon Cloud Blog\"},\"image\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\",\"https:\/\/twitter.com\/SiliCloudGlobal\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/5579144e23c225c8188167f3e3f888dd\",\"name\":\"Isabella Edwards\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/d4d4dec47f553ac7961d9fa4cc9bdcdcf5b7ce5106594330b6d25c5694fdbaec?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/d4d4dec47f553ac7961d9fa4cc9bdcdcf5b7ce5106594330b6d25c5694fdbaec?s=96&d=mm&r=g\",\"caption\":\"Isabella Edwards\"},\"url\":\"https:\/\/www.silicloud.com\/blog\/author\/isabellaedwards\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Keras Model Distillation: Step-by-Step Guide - Blog - Silicon Cloud","description":"Learn how to perform model distillation in Keras: define models, prepare data, train base model & create efficient distilled models.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/","og_locale":"en_US","og_type":"article","og_title":"Keras Model Distillation: Step-by-Step Guide","og_description":"Learn how to perform model distillation in Keras: define models, prepare data, train base model & create efficient distilled models.","og_url":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/","og_site_name":"Blog - Silicon Cloud","article_publisher":"https:\/\/www.facebook.com\/SiliCloudGlobal\/","article_published_time":"2024-03-13T07:18:09+00:00","article_modified_time":"2025-07-30T19:42:47+00:00","author":"Isabella Edwards","twitter_card":"summary_large_image","twitter_creator":"@SiliCloudGlobal","twitter_site":"@SiliCloudGlobal","twitter_misc":{"Written by":"Isabella Edwards","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/#article","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/"},"author":{"name":"Isabella Edwards","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/5579144e23c225c8188167f3e3f888dd"},"headline":"Keras Model Distillation: Step-by-Step Guide","datePublished":"2024-03-13T07:18:09+00:00","dateModified":"2025-07-30T19:42:47+00:00","mainEntityOfPage":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/"},"wordCount":201,"publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"keywords":["Deep Learning","Keras Tutorial","machine learning","Model Distillation","Neural Networks"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/","url":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/","name":"Keras Model Distillation: Step-by-Step Guide - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/#website"},"datePublished":"2024-03-13T07:18:09+00:00","dateModified":"2025-07-30T19:42:47+00:00","description":"Learn how to perform model distillation in Keras: define models, prepare data, train base model & create efficient distilled models.","breadcrumb":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-model-distillation-in-keras\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.silicloud.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Keras Model Distillation: Step-by-Step Guide"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/blog\/#website","url":"https:\/\/www.silicloud.com\/blog\/","name":"Silicon Cloud Blog","description":"","publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.silicloud.com\/blog\/#organization","name":"Silicon Cloud Blog","url":"https:\/\/www.silicloud.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","contentUrl":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","width":1024,"height":1024,"caption":"Silicon Cloud Blog"},"image":{"@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/SiliCloudGlobal\/","https:\/\/twitter.com\/SiliCloudGlobal"]},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/5579144e23c225c8188167f3e3f888dd","name":"Isabella Edwards","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/d4d4dec47f553ac7961d9fa4cc9bdcdcf5b7ce5106594330b6d25c5694fdbaec?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/d4d4dec47f553ac7961d9fa4cc9bdcdcf5b7ce5106594330b6d25c5694fdbaec?s=96&d=mm&r=g","caption":"Isabella Edwards"},"url":"https:\/\/www.silicloud.com\/blog\/author\/isabellaedwards\/"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/3680","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/users\/13"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/comments?post=3680"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/3680\/revisions"}],"predecessor-version":[{"id":148341,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/3680\/revisions\/148341"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/media?parent=3680"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/categories?post=3680"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/tags?post=3680"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}