Python Gradient Descent Exercises: Loss, Steps, Learning Rate
Gradient descent is easier to trust when every step is visible. These exercises isolate the loss, the gradient, the update, and the stopping rule, then show how one bad learning rate can overwhelm all of them.
Exercise: compute a squared-error loss
A loss must return one number. Here the model is a single weight through the origin, so the prediction is w times x and the error is the difference from y.
def loss(w, xs, ys):
return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)
print(loss(1, [1, 2], [2, 4]))The current model's score
Exercise: compute the gradient
Differentiate the squared error with respect to w. The factor of two and the extra x are not decoration; they determine the size and direction of the update.
def gradient(w, xs, ys):
return 2 * sum((w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
print(gradient(1, [1, 2], [2, 4]))A negative slope toward improvement
Exercise: record the loss history
Printing only the final weight hides whether learning was stable. Append the loss before or after every update and inspect the sequence. A healthy run trends downward.
def gradient(w, xs, ys):
return 2 * sum((w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
def loss(w, xs, ys):
return sum((w * x - y) ** 2 for x, y in zip(xs, ys)) / len(xs)
xs = [1, 2]
ys = [2, 4]
w = 1.0
history = []
for _ in range(3):
history.append(loss(w, xs, ys))
w -= 0.1 * gradient(w, xs, ys)
print([round(value, 2) for value in history])The loss should fall
Exercise: compare two learning rates
Run the same number of steps with a small and a large rate. Do not print a giant number just to prove divergence; test whether the final value stayed in a sane range.
def gradient(w, xs, ys):
return 2 * sum((w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
def final_weight(rate, xs, ys):
w = 1.0
for _ in range(5):
w -= rate * gradient(w, xs, ys)
return w
xs = [1, 2]
ys = [2, 4]
print(abs(final_weight(0.1, xs, ys)) < 10)
print(abs(final_weight(1.5, xs, ys)) < 10)One rate converges, one does not
Exercise: add a stopping rule
A fixed number of steps is a budget, not a convergence test. Stop when the gradient or the parameter update is below a tolerance, and keep the budget as a hard upper bound.
def gradient(w, xs, ys):
return 2 * sum((w * x - y) * x for x, y in zip(xs, ys)) / len(xs)
xs = [1, 2]
ys = [2, 4]
w = 1.0
for _ in range(100):
g = gradient(w, xs, ys)
if abs(g) < 0.01:
break
w -= 0.1 * g
print(abs(gradient(w, xs, ys)) < 0.01)Stop when the signal is small
Your practice checklist
- Define the loss before tuning the step size.
- Check the sign of the gradient with one manual update.
- Record loss history instead of only the final weight.
- Use a hard step budget and a convergence tolerance.
- Scale features when their ranges differ greatly.
Common questions
Is gradient descent guaranteed to find the best answer?
No. It can stop at a local minimum, diverge, or be slowed by poor scaling. For simple convex losses, the right step size has stronger guarantees.
How do I choose a learning rate?
Start with a small value, inspect the loss curve, and adjust by orders of magnitude. There is no universal best value.
Why is the gradient negative at the start?
The current prediction is below the target, so increasing the weight reduces the loss. A negative gradient tells the update to increase w.
Practice one update at a time
12 questions on loss, gradients, learning rates, divergence, and stopping rules.
Keep reading: