Citation, DOI & article data
The concept of transfer learning in artificial neural networks is taking knowledge acquired from training on one particular domain and applying it in learning a separate task.
In recent years, a well-established paradigm has been to pre-train models using large-scale data (e.g., ImageNet) and then to fine-tune the models on target tasks that often have less training data 3. For example, a neural network that has previously been trained to recognize pictures of animals may more effectively learn how to categorize pathology on a chest x-ray. In this example, the initial training of the network in animal image recognition is known as “pre-training”, while training on the subsequent data set of chest x-rays is known as “fine tuning”. This tool is most useful when the number of training examples in the pre-training data set is relatively large (e.g. 100,000 animal images) while the fine-tuning data set is relatively small (e.g. 200 chest x-rays).
The most popular dataset used for pre-training is the ImageNet dataset 5, a very large dataset containing more than 14 million annotated images 4.
The initial layers in a neural network for most image recognition tasks are involved in recognizing simple features such as edges and curves. As such, a network which has been pre-trained on an unrelated image recognition task has already learned to see these lower level features. A network already pre-trained on images of animals does not need to re-learn such features, and is, therefore, able to train for the task of recognizing chest x-ray pathology with fewer training examples.
- 1. NG A. "Machine Learning | Coursera". Coursera, 2019. [Link].
- 2. Ian Goodfellow, Yoshua Bengio, Aaron Courville. Deep Learning. (2016) ISBN: 9780262035613
- 3.K. He, R. Girshick and P. Dollar, "Rethinking ImageNet Pre-Training," 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, Korea (South), 2019, pp. 4917-4926, doi: 10.1109/ICCV.2019.00502.
- 4.O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. Bernstein, A. C. Berg, and L. Fei-Fei. ImageNet Large Scale Visual Recognition Challenge. International Journal of Computer Vision (IJCV), 2015
- 5.What makes ImageNet good for transfer learning? M Huh, P Agrawal, AA Efros - arXiv preprint arXiv:1608.08614, 2016 - arxiv.org