How it works
Gradient descent is an optimization method used to adjust parameters of a model. It seeks to reduce a loss function, which represents how much predictions differ from the desired outcome. In simple terms, the algorithm measures in which direction the loss increases and updates the parameters in the opposite direction.
Imagine a surface with valleys and hills. The position represents the current values of the parameters, and the height represents the error. The gradient indicates the local slope. The method takes a step down, recalculates the slope and repeats the process until it reaches a stopping criterion.
The size of each step is controlled by the learning rate. If it is too high, updates can overshoot and oscillate. If it is too low, training can progress slowly. The choice depends on the model, the data and the training procedure.
With large datasets, the calculation can use all examples at once or small batches. Stochastic gradient descent updates parameters using individual examples or subsets. Modern variants combine the gradient with mechanisms that adapt step sizes per parameter.
Practical example
Suppose a program estimates delivery time for orders based on distance. The model starts with a parameter that transforms kilometers into an estimate of minutes. For each training example, it compares the prediction with the observed time and calculates a loss.
If predictions are systematically low, the gradient points out how to adjust the parameter to reduce that error. The algorithm updates the value, calculates new predictions and checks if the loss fell. This cycle continues until the defined criterion is reached.
The example is simple, but the idea also appears in training neural networks, which can have many parameters. Gradient calculation often uses backpropagation to discover how each parameter contributed to the loss.
What it is used for
The method is used in machine learning to train regression, classification and neural network models. Instead of manually choosing each weight, training adjusts these values based on the data and the specified objective.
The loss function and the data influence what the model learns. A poorly chosen loss can favor the wrong behavior, and biased or inadequate data remain a problem even when optimization converges.
Converging also does not mean the model is good for any situation. It is necessary to evaluate performance on data that was not used to adjust the parameters, observe errors by group and verify if the result meets the product objective.
Difference between gradient descent and gradient
The gradient is mathematical information about the direction of greatest local increase of a function. Gradient descent is the procedure that uses this information, with repeated steps in the opposite direction, to seek smaller values of the function.
This distinction helps understand training: gradient calculation provides a direction, while the learning rate and the update rule determine how parameters move.