Gradient Descent

Updated 04 Oct 2026

•
What it is
- •
  iteratively reduces a differentiable objective by moving opposite its gradient.
•
Formula
- •
  $\theta_{k+1}=\theta_k-\alpha_k\nabla J(\theta_k)$ .
•
Interpretation
- •
  $\alpha_k$ is the learning rate or step size.
- •
  too large a step can overshoot or diverge; very small steps can converge slowly.
•
Regression example
- •
  for $J(\beta)=|X\beta-y|_2^2/(2n)$ , $\nabla J(\beta)=X^T(X\beta-y)/n$ .
•
Distinguish from
- •
  Unavailable reference specifies an objective; gradient descent is one way to optimize an objective
- •
  the normal equations characterize least-squares stationary solutions without taking gradient-decent steps
•

Linked references 1