Transfer Learning and Fine Tuning are both topics we’ll cover in much more detail in our Intermediate Series. For now, we’ll provide you with a quick primer, so you’ll be familiar with the concepts when you hear them out in the wild.
Transfer Learning
Transfer learning is a machine learning method where a model developed for one task is repurposed as the starting point for a model on a second task. It is a popular approach in deep learning where pre-trained models are used as the basis for performing new tasks. Technically, this involves taking layers from a model (their architecture, hyperparameters and trained parameters) already trained on a dataset and leveraging that learned feature map for a new task with potentially less data. An example of this you may have already used is Google’s Teachable Machine (TM), which uses Google’s MobileNet model as the base. When you provide new images to TM, it adds a layer to the top of the pre-trained model that is trained on your new images. This is how TM can recognize images with just a few hundred examples, rather than the tens of thousands (or even more) typically required for a computer vision model.
Fine-Tuning
Fine-tuning a model refers to the process of taking a pre-trained model (like those used in transfer learning) and continuing the training process to adjust the weights for a specific task. It usually involves unfreezing the entire model or a portion of it, and training it again with a smaller learning rate, to not destroy the previously learned features. The fine-tuning adjusts the specialized features to make the model more relevant for the specific task at hand. This process typically requires less data and computation than training a model from scratch. An example would be taking a pre-trained large language model and fine-tuning it with legal documents to specialize the model to reproduce legal-sounding text.
