How to perform regularization on a model in PyTorch?

In PyTorch, we can regularize our model by using regularization methods in the optimizer. Common regularization methods include L1 regularization and L2 regularization.

For L1 regularization, we can specify the regularization coefficient by passing the weight_decay parameter when defining the optimizer, for example:

optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.001)

For L2 regularization, we can also specify the regularization coefficient by passing the weight_decay parameter when defining the optimizer, for example:

optimizer = torch.optim.SGD(model.parameters(), lr=0.01, weight_decay=0.001)

In addition to defining regularization in the optimizer, we can also manually calculate and add regularization terms during the training process of the model, for example:

# 定义L2正则化项
l2_reg = torch.tensor(0., requires_grad=True)
for param in model.parameters():
    l2_reg += torch.norm(param)
    
# 定义损失函数，并加入L2正则化项
criterion = nn.CrossEntropyLoss()
loss = criterion(output, target) + lambda * l2_reg

This way, regularization of the model can be achieved.