{"id":5285,"date":"2024-03-14T02:37:24","date_gmt":"2024-03-14T02:37:24","guid":{"rendered":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/"},"modified":"2025-08-01T13:05:29","modified_gmt":"2025-08-01T13:05:29","slug":"how-to-perform-regularization-on-a-model-in-pytorch","status":"publish","type":"post","link":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/","title":{"rendered":"PyTorch Regularization: L1 &#038; L2 Guide"},"content":{"rendered":"<p>In PyTorch, we can regularize our model by using regularization methods in the optimizer. Common regularization methods include L1 regularization and L2 regularization.<\/p>\n<p>For L1 regularization, we can specify the regularization coefficient by passing the weight_decay parameter when defining the optimizer, for example:<\/p>\n<pre class=\"post-pre\"><code>optimizer = torch.optim.SGD(model.parameters(), lr=<span class=\"hljs-number\">0.01<\/span>, weight_decay=<span class=\"hljs-number\">0.001<\/span>)\r\n<\/code><\/pre>\n<p>For L2 regularization, we can also specify the regularization coefficient by passing the weight_decay parameter when defining the optimizer, for example:<\/p>\n<pre class=\"post-pre\"><code>optimizer = torch.optim.SGD(model.parameters(), lr=<span class=\"hljs-number\">0.01<\/span>, weight_decay=<span class=\"hljs-number\">0.001<\/span>)\r\n<\/code><\/pre>\n<p>In addition to defining regularization in the optimizer, we can also manually calculate and add regularization terms during the training process of the model, for example:<\/p>\n<pre class=\"post-pre\"><code><span class=\"hljs-comment\"># \u5b9a\u4e49L2\u6b63\u5219\u5316\u9879<\/span>\r\nl2_reg = torch.tensor(<span class=\"hljs-number\">0.<\/span>, requires_grad=<span class=\"hljs-literal\">True<\/span>)\r\n<span class=\"hljs-keyword\">for<\/span> param <span class=\"hljs-keyword\">in<\/span> model.parameters():\r\n    l2_reg += torch.norm(param)\r\n    \r\n<span class=\"hljs-comment\"># \u5b9a\u4e49\u635f\u5931\u51fd\u6570\uff0c\u5e76\u52a0\u5165L2\u6b63\u5219\u5316\u9879<\/span>\r\ncriterion = nn.CrossEntropyLoss()\r\nloss = criterion(output, target) + <span class=\"hljs-keyword\">lambda<\/span> * l2_reg\r\n<\/code><\/pre>\n<p>This way, regularization of the model can be achieved.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>In PyTorch, we can regularize our model by using regularization methods in the optimizer. Common regularization methods include L1 regularization and L2 regularization. For L1 regularization, we can specify the regularization coefficient by passing the weight_decay parameter when defining the optimizer, for example: optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.001) For L2 regularization, we can also specify [&hellip;]<\/p>\n","protected":false},"author":7,"featured_media":0,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_import_markdown_pro_load_document_selector":0,"_import_markdown_pro_submit_text_textarea":"","footnotes":""},"categories":[1],"tags":[5726,2862,1239,2371,5727],"class_list":["post-5285","post","type-post","status-publish","format-standard","hentry","category-uncategorized","tag-l1-regularization","tag-l2-regularization","tag-pytorch","tag-regularization","tag-weight_decay"],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v21.5 (Yoast SEO v21.5) - https:\/\/yoast.com\/wordpress\/plugins\/seo\/ -->\n<title>PyTorch Regularization: L1 &amp; L2 Guide - Blog - Silicon Cloud<\/title>\n<meta name=\"description\" content=\"Learn L1\/L2 regularization in PyTorch using weight_decay parameter. Prevent overfitting with optimizer techniques.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"PyTorch Regularization: L1 &amp; L2 Guide\" \/>\n<meta property=\"og:description\" content=\"Learn L1\/L2 regularization in PyTorch using weight_decay parameter. Prevent overfitting with optimizer techniques.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/\" \/>\n<meta property=\"og:site_name\" content=\"Blog - Silicon Cloud\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\" \/>\n<meta property=\"article:published_time\" content=\"2024-03-14T02:37:24+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2025-08-01T13:05:29+00:00\" \/>\n<meta name=\"author\" content=\"Sophia Anderson\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:site\" content=\"@SiliCloudGlobal\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Sophia Anderson\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"1 minute\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/#article\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/\"},\"author\":{\"name\":\"Sophia Anderson\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/19a24313de9c988db3d69226b4a40a30\"},\"headline\":\"PyTorch Regularization: L1 &#038; L2 Guide\",\"datePublished\":\"2024-03-14T02:37:24+00:00\",\"dateModified\":\"2025-08-01T13:05:29+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/\"},\"wordCount\":106,\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"keywords\":[\"L1 Regularization\",\"L2 regularization\",\"PyTorch\",\"regularization\",\"weight_decay\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/\",\"name\":\"PyTorch Regularization: L1 & L2 Guide - Blog - Silicon Cloud\",\"isPartOf\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\"},\"datePublished\":\"2024-03-14T02:37:24+00:00\",\"dateModified\":\"2025-08-01T13:05:29+00:00\",\"description\":\"Learn L1\/L2 regularization in PyTorch using weight_decay parameter. Prevent overfitting with optimizer techniques.\",\"breadcrumb\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/www.silicloud.com\/blog\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"PyTorch Regularization: L1 &#038; L2 Guide\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#website\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"name\":\"Silicon Cloud Blog\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\"},\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#organization\",\"name\":\"Silicon Cloud Blog\",\"url\":\"https:\/\/www.silicloud.com\/blog\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"contentUrl\":\"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png\",\"width\":1024,\"height\":1024,\"caption\":\"Silicon Cloud Blog\"},\"image\":{\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/www.facebook.com\/SiliCloudGlobal\/\",\"https:\/\/twitter.com\/SiliCloudGlobal\"]},{\"@type\":\"Person\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/19a24313de9c988db3d69226b4a40a30\",\"name\":\"Sophia Anderson\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/c726c09aa40e37115fb5c62d0c3ed62c16ca255d3763e2e3ae83a70ddf8c2175?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/c726c09aa40e37115fb5c62d0c3ed62c16ca255d3763e2e3ae83a70ddf8c2175?s=96&d=mm&r=g\",\"caption\":\"Sophia Anderson\"},\"url\":\"https:\/\/www.silicloud.com\/blog\/author\/sophiaanderson\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"PyTorch Regularization: L1 & L2 Guide - Blog - Silicon Cloud","description":"Learn L1\/L2 regularization in PyTorch using weight_decay parameter. Prevent overfitting with optimizer techniques.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/","og_locale":"en_US","og_type":"article","og_title":"PyTorch Regularization: L1 & L2 Guide","og_description":"Learn L1\/L2 regularization in PyTorch using weight_decay parameter. Prevent overfitting with optimizer techniques.","og_url":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/","og_site_name":"Blog - Silicon Cloud","article_publisher":"https:\/\/www.facebook.com\/SiliCloudGlobal\/","article_published_time":"2024-03-14T02:37:24+00:00","article_modified_time":"2025-08-01T13:05:29+00:00","author":"Sophia Anderson","twitter_card":"summary_large_image","twitter_creator":"@SiliCloudGlobal","twitter_site":"@SiliCloudGlobal","twitter_misc":{"Written by":"Sophia Anderson","Est. reading time":"1 minute"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/#article","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/"},"author":{"name":"Sophia Anderson","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/19a24313de9c988db3d69226b4a40a30"},"headline":"PyTorch Regularization: L1 &#038; L2 Guide","datePublished":"2024-03-14T02:37:24+00:00","dateModified":"2025-08-01T13:05:29+00:00","mainEntityOfPage":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/"},"wordCount":106,"publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"keywords":["L1 Regularization","L2 regularization","PyTorch","regularization","weight_decay"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/","url":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/","name":"PyTorch Regularization: L1 & L2 Guide - Blog - Silicon Cloud","isPartOf":{"@id":"https:\/\/www.silicloud.com\/blog\/#website"},"datePublished":"2024-03-14T02:37:24+00:00","dateModified":"2025-08-01T13:05:29+00:00","description":"Learn L1\/L2 regularization in PyTorch using weight_decay parameter. Prevent overfitting with optimizer techniques.","breadcrumb":{"@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/www.silicloud.com\/blog\/how-to-perform-regularization-on-a-model-in-pytorch\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/www.silicloud.com\/blog\/"},{"@type":"ListItem","position":2,"name":"PyTorch Regularization: L1 &#038; L2 Guide"}]},{"@type":"WebSite","@id":"https:\/\/www.silicloud.com\/blog\/#website","url":"https:\/\/www.silicloud.com\/blog\/","name":"Silicon Cloud Blog","description":"","publisher":{"@id":"https:\/\/www.silicloud.com\/blog\/#organization"},"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.silicloud.com\/blog\/#organization","name":"Silicon Cloud Blog","url":"https:\/\/www.silicloud.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","contentUrl":"https:\/\/www.silicloud.com\/blog\/wp-content\/uploads\/2023\/11\/EN-SILICON-Full.png","width":1024,"height":1024,"caption":"Silicon Cloud Blog"},"image":{"@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/SiliCloudGlobal\/","https:\/\/twitter.com\/SiliCloudGlobal"]},{"@type":"Person","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/19a24313de9c988db3d69226b4a40a30","name":"Sophia Anderson","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.silicloud.com\/blog\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/c726c09aa40e37115fb5c62d0c3ed62c16ca255d3763e2e3ae83a70ddf8c2175?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/c726c09aa40e37115fb5c62d0c3ed62c16ca255d3763e2e3ae83a70ddf8c2175?s=96&d=mm&r=g","caption":"Sophia Anderson"},"url":"https:\/\/www.silicloud.com\/blog\/author\/sophiaanderson\/"}]}},"_links":{"self":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/5285","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/users\/7"}],"replies":[{"embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/comments?post=5285"}],"version-history":[{"count":2,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/5285\/revisions"}],"predecessor-version":[{"id":150027,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/posts\/5285\/revisions\/150027"}],"wp:attachment":[{"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/media?parent=5285"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/categories?post=5285"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.silicloud.com\/blog\/wp-json\/wp\/v2\/tags?post=5285"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}