Optimization and Gradient Descent

Optimization is the problem of finding parameters that minimize (or maximize) an objective function, possibly subject to constraints. Gradient descent is an iterative first-order method that repeatedly moves the parameters in the direction of the negative gradient of the objective; its stochastic variants, which estimate the gradient from small random subsets of data, are the standard means of training neural networks.

Most machine learning is framed as minimizing a loss $L(\theta)$ over parameters $\theta$ (FOUND-007 Learning, Reasoning and Generalization). Whether a model learns well depends on its architecture and data, and also on how the optimizer navigates the loss landscape. In deep learning the parameter count can be very large and the loss is non-convex, so exact minimization is infeasible and the relevant questions become: how fast does the method make progress, where does it end up, and how does the end point generalize?

Gradient-based training became practical through backpropagation, which computes gradients of a network's loss efficiently (Rumelhart et al., 1986; see TECH-BACKPROP-001 Backpropagation). Optimization is thus the link between a model specification and a trained system, and choices such as learning rate, batch size and optimizer are significant engineering decisions in large-scale training (TECH-DL-001 Deep Learning).

Gradient descent is conventionally attributed to Cauchy's note to the Académie des Sciences of 18 October 1847, motivated by astronomical calculations (Lemaréchal, 2012). The attribution is taken from Lemaréchal's account; Cauchy's note itself was not retrieved. Robbins and Monro (1951) introduced stochastic approximation, the basis of stochastic gradient descent (SGD): iterating with noisy gradient estimates and decreasing step sizes converges under conditions. Polyak (1964) introduced the heavy-ball momentum method to speed convergence on ill-conditioned problems.

Rumelhart, Hinton and Williams (1986) showed that gradient descent with backpropagated gradients could train multi-layer networks to learn useful internal representations, a milestone in the connectionist revival (HIST-008 Backpropagation and the Connectionist Revival). Adaptive per-parameter methods followed, including Adam (Kingma & Ba, 2015), and Loshchilov and Hutter (2019) proposed decoupling weight decay from the gradient update (AdamW). Bottou, Curtis and Nocedal (2018) provide a comprehensive review of optimization methods for large-scale machine learning. Standard theoretical background in convex optimization is in Boyd and Vandenberghe (2004), and a deep-learning-oriented treatment is in Goodfellow, Bengio and Courville (2016).

#4.1 Gradient descent

For a differentiable objective $L(\theta)$, gradient descent iterates

where $\eta > 0$ is the learning rate. The negative gradient is the direction of steepest local decrease. For a convex function whose gradient is $\beta$-Lipschitz ($\beta$-smooth), the choice $\eta = 1/\beta$ gives $L(\theta_k) - L(\theta^*) = O(1/k)$, a standard result (Boyd & Vandenberghe, 2004; Bottou et al., 2018). For strongly convex functions the convergence is linear (geometric). Neither guarantee applies directly to neural-network losses, which are non-convex.

#4.2 Stochastic and mini-batch gradients

With a training objective $L(\theta) = \frac{1}{N}\sum_{i=1}^N \ell_i(\theta)$, computing the full gradient costs $O(N)$ per step. SGD replaces it with an unbiased estimate from a random mini-batch $B$:

Each step is cheap, and the gradient noise is often argued to help escape poor regions, although this is an empirical expectation and not a general guarantee. Convergence analyses require step sizes that decay or are small enough to control variance (Robbins & Monro, 1951; Bottou et al., 2018). Larger batches reduce gradient variance but increase cost per step.

#4.3 Momentum

Momentum accumulates a velocity:

with $\mu \in [0,1)$. It damps oscillations in directions of high curvature and accelerates progress along consistent directions (Polyak, 1964).

#4.4 Adaptive methods

Adam maintains exponential moving averages of the gradient $g_t$ and its square:

and updates $\theta_{t+1} = \theta_t - \eta\,\hat m_t / (\sqrt{\hat v_t} + \epsilon)$, where $\hat m_t$ and $\hat v_t$ are bias-corrected (Kingma & Ba, 2015). Each parameter thus has its own effective step size. AdamW applies weight decay directly to the parameters, separately from the adaptive gradient scaling (Loshchilov & Hutter, 2019).

#4.5 Learning-rate schedules and practicalities

Learning rate is typically varied over training (warm-up, decay). Gradient clipping, normalization layers and mixed-precision arithmetic address numerical stability. The choices are largely empirical; no single schedule is optimal across problems.

#4.6 The loss landscape

Neural-network losses have many parameters, saddle points and flat regions. Dauphin et al. (2014) argued, using ideas from statistical physics and empirical evidence, that in high dimensions saddle points rather than poor local minima are the principal obstacle to optimization. Choromanska et al. (2015) analysed the loss surfaces of multilayer networks under simplifying assumptions and related the structure of critical points to network size. These analyses rely on assumptions that do not fully match practical networks, and they are best regarded as suggestive.

#4.7 Regularization as optimization

Explicit penalties (for example, $L_2$ weight decay) change the objective; optimizer choice and early stopping act as implicit regularizers. Which solutions SGD-like methods select among the many that fit the training data, and why those solutions generalize, is a research question.

• First-order methods: gradient descent, SGD, momentum, Nesterov-type acceleration, adaptive methods (AdaGrad, RMSProp, Adam, AdamW).
• Second-order and quasi-Newton methods: use curvature information (Hessian approximations); costly for models with very many parameters, so used mainly at smaller scale or through approximations.
• Derivative-free and evolutionary methods: used where gradients are unavailable (black-box tuning, some reinforcement learning settings).
• Convex versus non-convex problems: convex problems have global-optimality guarantees (Boyd & Vandenberghe, 2004); neural-network training is non-convex.
• Constrained and discrete optimization: relevant to planning, scheduling and combinatorial search (FOUND-004 Computational Complexity in AI).

• Training neural networks with SGD or Adam-family optimizers is the standard practice in deep learning (Kingma & Ba, 2015; Goodfellow et al., 2016). AdamW, which decouples weight decay from the gradient update in Adam (Loshchilov & Hutter, 2019), is a widely used Adam variant; this article cites no specific training reports for its prevalence, and because recipes change, current practice should be checked against recent technical reports and the relevant Living Reference entries.
• Linear and logistic regression, support vector machines: convex problems solved by gradient or quasi-Newton methods (Bishop, 2006).
• Large-batch training. Keskar et al. (2017) reported that large-batch methods tended to converge to sharper minima and showed a generalization gap in their experiments. Subsequent work has examined whether the gap is intrinsic or can be mitigated with tuning; the conclusion in the original paper is therefore Contested in general.
• Adaptive versus non-adaptive generalization. Wilson et al. (2017) reported cases in which adaptive methods generalized worse than SGD with momentum when tuned comparably; this does not imply that adaptive methods are inferior in general.

Strengths. Gradient methods are simple, scale to billions of parameters when combined with data and model parallelism, and work with any differentiable model.

Limitations.

• No global-optimality guarantee for non-convex losses.
• Sensitivity to hyperparameters (learning rate, batch size, schedule); instability and divergence are common failure modes.
• Vanishing and exploding gradients in deep or recurrent architectures.
• Optimizing a proxy loss does not guarantee that the downstream objective is met; a model can minimize its training loss while failing on goals the loss does not capture (relevant to SAFE-ALIGN-001 AI Alignment).

Open questions. Why simple first-order methods find solutions that generalize in over-parameterized networks; the relationship between flat minima and generalization (Keskar et al., 2017, and critics); whether the saddle-point picture of Dauphin et al. (2014) fully accounts for training dynamics; and optimizer behaviour at frontier scale.

As of 2026-10-15:

• Convergence results for convex smooth objectives and for SGD under standard assumptions are Established (Boyd & Vandenberghe, 2004; Bottou et al., 2018).
• SGD and Adam-family optimizers are the dominant practical choice for training neural networks (Goodfellow et al., 2016; Kingma & Ba, 2015); specific hyperparameters used by named frontier models are mostly undisclosed or change quickly and are not tabulated here.
• The relative generalization of adaptive and non-adaptive methods, and the role of batch size and sharpness, are Contested (Wilson et al., 2017; Keskar et al., 2017).
• The picture that saddle points rather than local minima dominate difficulty in high dimensions is Preliminary to Well Supported, depending on the model class (Dauphin et al., 2014; Choromanska et al., 2015).

No registered forecast is made. Observed research directions include second-order and preconditioned methods that remain tractable at scale, optimizers designed for very large batch sizes and low-precision arithmetic, and theory of implicit regularization. Near-term change is likely to be incremental refinements of existing optimizers rather than replacement of gradient-based training. Claims that a particular new optimizer will supersede Adam-family methods should be evaluated against independent, well-tuned baselines.

• Foundational Concepts:
  • FOUND-008 Probability, Statistics and Bayesian Inference: likelihood objectives that are optimized.
  • FOUND-007 Learning, Reasoning and Generalization: empirical risk minimization and generalization.
  • FOUND-009 Information Theory: cross-entropy losses.
• Related Methods & Technologies:
  • TECH-BACKPROP-001 Backpropagation: computing the gradients.
  • TECH-DL-001 Deep Learning: the main application setting.
  • TECH-RL-001 Reinforcement Learning: policy-gradient optimization.
  • HIST-008 Backpropagation and the Connectionist Revival: the historical context.
• Advanced Implications & Future Trajectories:
  • TECH-SCALE-001 Scaling Laws: how loss behaves under optimization at scale.
  • SAFE-ALIGN-001 AI Alignment: optimization of proxy objectives.