Blockwise.
Back to learning
Model guide

Linear regression

Draw one straight trend through examples. The model learns how much a prediction changes when an input increases by one unit.

Start with an example

A delivery estimate might start at 8 minutes and add 3 minutes per kilometre. A 4 km trip would then take an estimated 20 minutes.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Start with a prediction at input zero, called the intercept. Add a weight times each input: the weight tells how much that measurement changes the prediction.
  2. Training tries weights and a starting value that reduce the gaps between predictions and known answers. It squares each gap before adding them, so larger mistakes count more. One input gives a straight line.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

See how a straight line turns an input into a numerical prediction. You will separate moving the whole line from changing its steepness.

Across: the horizontal axis
The input measurement x, from 0 to 10. In a delivery task, an input could be distance; here it has no unit.
Up: the vertical axis
The target: the numerical answer being predicted. In a delivery task, it could be time; here the target has no unit.

Each dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: that is the predicted target for this input.

2. Predict, change, observe

  1. Reset the experiment. Before moving Intercept from 2 to 3, predict whether the line will rise everywhere or only at the right-hand end.
  2. Try it, then reset and raise Slope from 0.5 to 1. Watch how the prediction at x = 6 changes while the line still starts at the same height at x = 0.
Interactive Linear regression illustrationEach dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.02.557.5100246810Target valueInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 5.000

Mean squared training error: 0.896

Mean squared training error averages the squared vertical gaps from the observed dots to the line. Smaller means a closer fit to these training examples, not necessarily better predictions on new examples.

3. Explain it in your own words

You want to add 1 to every prediction, whatever the input. Which control would you change, and why?

Compare your reasoning

Increase Intercept by 1. It is the starting height added to every prediction. Slope changes how much the prediction grows per input unit, so its effect depends on x. If unsure, compare the line at x = 0 and x = 6 before and after each change.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

Move the line to bring its predictions closer to the observations. Training chooses the slope and intercept automatically.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed target
0.302.78
0.922.97
1.544.17
2.164.08
2.784.33
3.403.24
4.023.41
4.642.87
5.263.37
5.883.16
6.504.58
7.125.27
7.746.6
8.366.59
8.987.51
9.607.14

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

y^=b+∑j=1pwjxjmin⁡b,w∑i(yi−y^i)2\begin{gathered}\hat y=b+\sum_{j=1}^{p}w_jx_j\\\min_{b,w}\sum_i(y_i-\hat y_i)^2\end{gathered}Read it like thisMultiply each measurement by its weight, add the contributions, and add the starting value. Training chooses the weights and starting value to make the total squared error as small as possible.What the symbols meanx,  xjx,\;x_jx is the input: measurements supplied to the model. A small lower j selects input number j, such as distance from several measurements.yi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.b,  wjb,\;w_jb is the intercept, the starting prediction before adding input contributions. w is a learned weight; the lower j selects the weight for input j. A weight tells how strongly that input affects the prediction.p,  jp,\;jHere p is the number of input measurements and j counts through them. For three measurements, j takes the values 1, 2, and 3.×,  ⋅\times,\;\cdotBoth marks mean multiply. Letters written next to each other also multiply: wx means w times x. For vectors, a dot means multiply matching entries and add the results.==Equals: the expressions on both sides have the same value.++Plus: add the values together.−-Minus: subtract the next value. Before a value, it changes that value's sign.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.min⁡\minMinimize: choose the listed parameters to make the following error or cost as small as possible.x2x^2Squared: multiply a value by itself. For example, 3 squared is 9. Squaring an error removes its sign and gives larger mistakes more influence.

x is an input measurement, w is its learned weight, b is the intercept, y is an observed outcome, and ŷ is a prediction. The sum runs over training examples.

What to watch for

  • An unusual point can strongly influence a squared-error fit.
  • A useful association does not prove cause and effect, and extrapolating beyond the training range can fail.
Polynomial regressionRidgeK-nearest neighbors

Further reading

The official documentation goes further when you are ready.