Blockwise.
Back to learning
Model guide

SGD linear models

Improve a model a little at a time. Look at one example, move the weights toward a smaller error, then repeat with another example.

Start with an example

After underestimating a delivery time, an update nudges the line upward for similar inputs. Later examples may nudge it in another direction.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Start with a model that multiplies inputs by weights and adds a starting value. Look at one training example, compare the prediction with its known answer, and adjust the weights to reduce that example's error cost.
  2. Repeat for the next example and make several passes through the data. Learning rate controls each update's size; an epoch is one complete pass. SGD is this update method, and the error cost you choose determines what it tries to improve.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

Watch a straight-line model improve through repeated small updates. An epoch is one pass through all the training examples.

Across: the horizontal axis
The input measurement x, from 0 to 10. In a delivery task, an input could be distance; here it has no unit.
Up: the vertical axis
The target: the numerical answer being predicted. In a delivery task, it could be time; here the target has no unit.

Each dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: that is the predicted target for this input.

2. Predict, change, observe

  1. Set Training epochs to 1 and Learning rate to 0.02. Notice the gaps between the prediction line and the observed dots.
  2. Predict whether 60 passes will bring the line closer to these dots. Set Training epochs to 60, keep the learning rate fixed, and compare the training error.
Interactive SGD linear models illustrationEach dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.02.557.5100246810Target valueInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 4.971

Mean squared training error: 0.948

Mean squared training error averages the squared vertical gaps from the observed dots to the line. Smaller means a closer fit to these training examples, not necessarily better predictions on new examples.

3. Explain it in your own words

Are more epochs and a larger learning rate the same way of changing training?

Compare your reasoning

No. More epochs repeat the updates; a larger learning rate makes each update bigger. Large updates can overshoot. If unsure, reset, change just one control at a time, and describe whether you changed the number of steps or their size.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

Training visits one example at a time and updates the weights. More epochs repeat those visits; the learning rate sets the size of each update.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed target
0.302.78
0.922.97
1.544.17
2.164.08
2.784.33
3.403.24
4.023.41
4.642.87
5.263.37
5.883.16
6.504.58
7.125.27
7.746.6
8.366.59
8.987.51
9.607.14

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

w←w−η∇ℓi(w)w\leftarrow w-\eta\nabla\ell_i(w)Read it like thisReplace the current weights with the current weights minus the learning rate times this example's gradient. This is one small training step.What the symbols meanw,  ℓi(w)w,\;\ell_i(w)w is the current list of model weights. The curly letter ell means loss, or mistake cost; lower i selects the current example. Evaluate that example's loss using the weights w.η\etaEta: the learning rate, which controls how strongly one training update or new model changes the current result.×,  ⋅\times,\;\cdotBoth marks mean multiply. Letters written next to each other also multiply: wx means w times x. For vectors, a dot means multiply matching entries and add the results.−-Minus: subtract the next value. Before a value, it changes that value's sign.←\leftarrowReplace the value on the left with the result on the right. It describes an update, not an equality that was already true.∇ℓi(w)\nabla\ell_i(w)Nabla is the gradient: a list showing how loss changes as each weight increases. Subtracting it moves toward lower loss when the step is small enough.

w is the coefficient vector, η is the learning rate, and ∇ℓᵢ is the gradient of the current example's loss. A gradient points toward the steepest local increase, so subtract it.

What to watch for

  • Large steps can diverge; very small steps can learn slowly.
  • Updates are noisy. Monitor validation evidence and save the best state rather than assuming the final pass is best.
Linear regressionLogistic regressionSoftmax regression

Further reading

The official documentation goes further when you are ready.