Curse of dimensionality
Citation, DOI & article data
The curse of dimensionality can refer to a number of phenomenon related to high-dimensional data in several fields. In terms of machine learning for radiology, it generally refers to the phenomenon that as the number of image features employed to train an algorithm increases there is a geometric increase in the number of training examples required.
A feature is a quantity or trait based on which machine learning algorithms make predictions. Supervised machine learning algorithms take a collection of features as their inputs. Ideally, an algorithm uses the minimum number of features that can differentiate between possible answers. In the context of machine learning in clinical medicine, features may take on many different categories of variables; this may include patient demographic data (e.g. age, sex, weight), clinical characteristics (e.g. CRP, heart rate, temperature), or information derived from medical imaging (e.g. greyscale value of each individual pixel)
As the number of features increases, the number of data points (or “training examples” in machine learning) required to train the algorithm increases exponentially. The intuition behind this can be visualized by imagining a set of 10 points on a line 10 centimeters long. If we increase the dimension of this line to 2 (i.e. it becomes a square), the number of points required for a similar density is increased to 100 (or 102). This property of an exponentially-growing requirement for input data as the number of features increases is known as the curse of dimensionality.
Importance in radiology
In the context of radiology, the number of input features can grow very large, particularly in pixel-based machine learning algorithms where each pixel (or voxel) of an input image represents a distinct feature. The process of reducing the dimension size of the input into machine learning algorithms to avoid the curse of dimensionality is known as dimensionality reduction. In medical imaging, this typically involves one or more pre-processing steps applied to inputted images aimed at extracting the most salient features of the images.
History and etymology
The term "curse of dimensionality" was originally used by Richard Bellman, a mathematician, in 1957 to describe some mathematical phenomenon associated with additional dimensions in mathematical spaces 3. One implication of this "curse" was exponentially increasing computational heaviness, an extremely important issue before graphical processing units (GPUs) became popular. Over half a century later, the term is now used by many AI practitioners to imply problems with datasets with many features (which almost all radiology datasets are).
- 1. Ian Goodfellow, Yoshua Bengio, Aaron Courville. Deep Learning. (2016) ISBN: 9780262035613
- 2. Hubbard, A., Trostle, J., Cangemi, I., & Eisenberg, J. N. S. (2019). Countering the Curse of Dimensionality. Epidemiology, 30(4), 609–614. doi:10.1097/ede.0000000000001025 doi:10.1097/EDE.0000000000001025
- 3. R. Bellman, Richard Dynamic Programming, Princeton University Press, Princeton, NJ, 1957.
- 4. Michael D. Kuo, Neema Jamshidi. Behind the Numbers: Decoding Molecular Phenotypes with Radiogenomics—Guiding Principles and Technical Considerations. (2014) Radiology. doi:10.1148/radiol.13132195