PyTorch Learning Rate Schedulers

There are several types of learning rate schedulers in PyTorch.

1. StepLR: The learning rate decreases by a factor of gamma at every given step size.
2. MultiStepLR: Define a list where the learning rate decreases by a factor of gamma at each step size in the list.
3. ExponentialLR: The learning rate decays exponentially.
4. CosineAnnealingLR: Cosine annealing learning rate scheduling.
5. ReduceLROnPlateau: Decrease the learning rate when a metric stops improving.
6. LambdaLR: Implement a learning rate scheduler by utilizing a designated function.
7. CyclicLR: Periodic adjustment of the learning rate within a cycle range.
8. OneCycleLR: a learning rate scheduler that accelerates model convergence by using varying learning rates during training.
9. Cosine Annealing with warm restarts is a learning rate scheduler that incorporates the functionality of cosine annealing.
10. MultiplicativeLR: multiply the learning rate by a given factor at each step.