Gradient Descent: How to Explain It in an Interview (2026)

Gradient descent is the workhorse behind most modern machine learning training loops. In an interview you need to convey three things quickly: what it does, how it does it, and why you would pick a particular flavor.

One‑sentence definition

Gradient descent is an iterative optimization algorithm that updates model parameters by moving them a small step opposite the gradient of the loss function, seeking a local minimum.

How the mechanism works

1. Compute the gradient – For the current parameters (\theta), calculate (\nabla L(\theta)), the vector of partial derivatives of the loss with respect to each parameter.
2. Choose a step size – The learning rate (\eta) scales how far you move along the negative gradient.
3. Update – Apply (\theta \leftarrow \theta - \eta \nabla L(\theta)).
4. Repeat – Iterate until the loss stops improving or a preset number of epochs is reached.

The process can be visualized as a ball rolling downhill on a surface defined by the loss. Each iteration is a small step that tries to follow the steepest descent direction.

Variants and trade‑offs

Learning rate is the most sensitive hyper‑parameter. Too large and you overshoot minima; too small and training drags on. Common strategies include:

• Constant – Simple but may stall near minima.
• Decay schedules – Reduce (\eta) over time (step decay, exponential, cosine).
• Adaptive methods – Adam, RMSProp adjust per‑parameter step sizes automatically.

Choosing a variant is a trade‑off between computational efficiency and the smoothness of the loss trajectory. Mini‑batch often hits the sweet spot for deep nets because it balances gradient accuracy and hardware throughput.

Concrete example: linear regression

Suppose you have points ((x_i, y_i)) and you want a line (y = wx + b). The loss is mean‑squared error: [L(w,b) = \frac{1}{n}\sum_{i=1}^{n} (y_i - (wx_i + b))^2] The gradients are: [\frac{\partial L}{\partial w} = -\frac{2}{n}\sum (y_i - (wx_i + b))x_i] [\frac{\partial L}{\partial b} = -\frac{2}{n}\sum (y_i - (wx_i + b))] Starting with (w=0, b=0) and a learning rate of 0.01, each iteration updates (w) and (b) using the formulas above. After a few dozen iterations the line settles near the optimal fit, and the loss plateaus.

Questions interviewers often ask

1. Why not use closed‑form solutions? – For simple models like linear regression you can solve analytically, but gradient descent scales to any differentiable loss and high‑dimensional parameters.
2. How do you detect convergence? – Common signals: loss change below a threshold, gradient norm near zero, or a validation metric stopping improvement.
3. What happens if the loss surface has many local minima? – Gradient descent may settle in a local basin; initialization and stochasticity can help escape shallow minima.
4. Explain learning‑rate schedules. – Discuss why you might start with a higher rate to make rapid progress, then decay to fine‑tune near the optimum.
5. When would you prefer Adam over plain SGD? – Adam adapts per‑parameter rates, useful when gradients vary widely or when you have sparse features.

60‑second spoken answer (template)

"Gradient descent is an iterative method that finds a local minimum of a differentiable loss by repeatedly moving the parameters opposite the gradient. You compute the gradient of the loss with respect to each parameter, multiply it by a learning rate, and subtract that product from the current parameters. The algorithm repeats until the loss stops decreasing. In practice we choose between batch, mini‑batch, and stochastic variants. Mini‑batch is the most common because it balances noisy gradient estimates with efficient GPU usage. The learning rate is crucial: too high and you overshoot, too low and training drags. Typical tricks include decaying the rate over epochs or using adaptive optimizers like Adam. For a simple linear regression example, each step updates the slope and intercept based on the mean‑squared error gradient, and after a few dozen steps the line converges to the best fit. Interviewers often probe convergence criteria, learning‑rate schedules, and why you’d pick one variant over another."

How to practice this

1. Write the core update loop in a notebook for a toy problem (e.g., fitting a line) and time yourself explaining each line.
2. Record a 60‑second run‑through and replay it, noting filler words and clarity. Use Call Assistant to capture the audio and suggest concise phrasing.
3. Mock interview with a peer who asks the typical follow‑up questions listed above. Focus on linking the answer back to a concrete project from your résumé.

---

FAQ

• Q: Can gradient descent get stuck in a bad local minimum? A: Yes, especially on highly non‑convex surfaces. Random restarts, momentum, or stochastic variants can help the optimizer escape shallow basins.
• Q: How does momentum modify the update rule? A: Momentum adds a fraction of the previous update to the current one, smoothing the trajectory and often accelerating convergence on ravines.
• Q: When is batch gradient descent preferable? A: When the dataset is small enough to fit in memory and you need deterministic, reproducible results, such as in academic experiments.
• Q: What is the main drawback of using Adam? A: Adam’s adaptive rates can sometimes lead to poorer generalization compared to plain SGD with a well‑tuned learning‑rate schedule.