The gradient descent algorithm aims to minimize a cost function (degree of predicting error) of a model in order to produce a model that gives the most accurate predictions. Gradient descent is by far the most commonly used algorithm in machine learning, and is usually the first algorithm most people learn due to its simplicity and usefulness.
As the name suggests, gradient descent can be seen as descending to the lowest point in an n-dimensional space. The lowest point represents the lowest value, and hence the minimum value of the cost function. Gradient descent uses the derivative of the function (the gradient of the curve), and changes the parameters of the algorithm in small steps (determined by a learning rate), each time moving towards another point that has a smaller cost. Eventually, gradient descent converges to a point where the gradient is close to 0, the minimum value for the cost function. The parameters that the gradient descent algorithm yielded will form the best model as it produced the minimum error in prediction.
- 1. Marsland S. Machine Learning. Hoboken: CRC Press; 2011 ISBN:1420067184 9781420067187