Transfer learning

Downloading and reusing the trained versions of milestone models with “minor adjustments” to be reused on any datasets

For example, using ResNet models for MNIST dataset:

1. Replace the initial layer with MNIST images - untrained layer taking images with 1 channel instead of 3
2. Replace final layer to match number of MNIST classes - adding Linear layer to reduce size of output vector from 1D vector of 1000 to 1D vector of 10 elements
3. Freeze all layers except new ones - prevents parameters from being changed during training

# Freeze all layers except the new first and last layers
for param in resnet.parameters():
    param.requiresGrad = False
resnet.conv1.requiresGrad = True
resnet.fc[1].requiresGrad = True

4. Train new layers that have random values in trainable parameters