Essentially, autoencoding is a data compression algorithm where the compression and decompression functions are learned automatically from examples via the use of neural networks.
The main components of an autoencoder are:
- encoder: through which the model learns how to reduce the input dimensions and compress the input data into an encoded representation
- bottleneck: which is the layer that contains the compressed representation of the input data; this is the lowest possible dimensions of the input data
- decoder: through which the model learns how to reconstruct the data from the encoded representation to be as close to the original input as possible
- reconstruction loss: this is the loss function that measures how well the decoder is performing and how close the output is to the original input
The key characteristic of autoencoders’ architecture is a bottleneck which limits the amount of information that can flow through the network, forcing a learned compression of the input data. The simplest way for implementing a bottleneck is to constrain the number of nodes present in the hidden layers of the network. By then penalizing the network according to the reconstruction error, the model can learn the most important/representative attributes of the input data and how to best reconstruct the original.
Autoencoders generally are data-specific and lossy. "Data-specific" means that they are only able to compress data similar to what they have been trained on. For example, an autoencoder trained on photos of faces would do a rather poor job of compressing photos of flowers, because the features it would learn would be face-specific 4. "Lossy" means that the decompressed outputs will be degraded compared to the original inputs; this differs from lossless compression.
There are several different types of autoencoders 6:
- denoising autoencoders
- sparse autoencoders
- deep autoencoders
- contractive autoencoders
- undercomplete autoencoders
- convolutional autoencoders
- variational autoencoders
Autoencoders in radiology
Uses of autoencoders relevant to radiological AI include noise reduction and anomaly detection 2.
History and etymology
The idea of autoencoders has been in the neural network literature for decades (LeCun, 1987; Bourlard and Kamp, 1988; Hinton and Zemel,1994) 3. Traditionally autoencoders were used for dimensionality reduction or feature learning, in other words, to reduce the complexity of data and reveal its internal structure.
- 1. Goodfellow, Ian; Bengio, Yoshua; Courville, Aaron (2016). Deep Learning. MIT Press. ISBN 978-0262035613.
- 2. Daisuke Sato M.D., Shouhei Hanaoka M.D., Yukihiro Nomura, Tomomi Takenaga, Soichiro Miki M.D., Takeharu Yoshikawa M.D., Naoto Hayashi M.D., Osamu Abe M.D.. A primitive study on unsupervised anomaly detection with an autoencoder in emergency head CT volumes. (2018) 10575: 105751P. doi:10.1117/12.2292276
- 3. Hinton, G. E., & Zemel, R. S. (1994). Autoencoders, minimum description length and Helmholtz free energy. In Advances in Neural Information Processing Systems 6, J. D. Cowan, G. Tesauro, and J. Alspector, Eds., Morgan Kaufmann, 3–10. [Available online at http://papers.nips.cc/paper/798-autoencoders-minimum-description-length-and-helmholtz-free-energy.]
- 4. Zeng, Nianyin; Zhang, Hong; Song, Baoye; Liu, Weibo; Li, Yurong; Dobaie, Abdullah M. (2018-01-17). "Facial expression recognition via learning deep sparse autoencoders". Neurocomputing. 273: 643–649. ISSN 0925-2312.doi:10.1016/j.neucom.2017.08.043
- 5. Diederik P. Kingma, Max Welling. An Introduction to Variational Autoencoders. (2019) Foundations and Trends in Machine Learning. 12 (4): 307. doi:10.1561/2200000056
- 6. Zhai, J., Zhang, S., Chen, J., & He, Q. (2018). Autoencoder and Its Various Variants. 2018 IEEE International Conference on Systems, Man, and Cybernetics (SMC). doi:10.1109/smc.2018.00080