Blockwise.
Back to learning
Model guide

Polynomial regression

Give a linear model extra input columns such as distance squared. It can then trace a curve while still learning an ordinary weighted sum.

Start with an example

A trip's time may grow faster on long routes. Adding distance² lets the fitted curve become steeper as distance increases.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Make extra measurements from the input: for example, distance squared is distance multiplied by itself. Degree sets the highest power you allow; degree 1 uses the original input alone.
  2. Learn a weight for each resulting measurement and add their contributions. The squared-input contribution can grow faster than the original input, allowing the prediction to bend instead of remaining a straight line.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

See how allowing a curve to bend changes the pattern a model can fit. Degree is a setting that controls this flexibility.

Across: the horizontal axis
The input measurement x, from 0 to 10. In a delivery task, an input could be distance; here it has no unit.
Up: the vertical axis
The target: the numerical answer being predicted. In a delivery task, it could be time; here the target has no unit.

Each dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: that is the predicted target for this input.

2. Predict, change, observe

  1. Set Polynomial degree to 1 and notice the straight line. Predict what a degree of 2 will allow it to do.
  2. Change degree to 2, then 6. Compare the bends and the gaps between the line and the dots, especially near the ends of the graph.
Interactive Polynomial regression illustrationEach dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.02.557.5100246810Target valueInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 4.344

Mean squared training error: 0.457

Mean squared training error averages the squared vertical gaps from the observed dots to the line. Smaller means a closer fit to these training examples, not necessarily better predictions on new examples.

3. Explain it in your own words

A flexible curve passes closer to the training dots. Does that establish that it will predict new examples better?

Compare your reasoning

No. It may be following accidental variation in these particular examples. The graph shows training fit; new, held-out examples are needed to judge future predictions. Compare degree 1 with 6 again and explain what extra flexibility buys and what it might risk.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

The curve is fitted by least squares after adding powers of the input. Higher degree can follow more bends, including noise.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed target
0.302.78
0.922.97
1.544.17
2.164.08
2.784.33
3.403.24
4.023.41
4.642.87
5.263.37
5.883.16
6.504.58
7.125.27
7.746.6
8.366.59
8.987.51
9.607.14

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

y^=b+w1x+w2x2+⋯+wdxd\hat y=b+w_1x+w_2x^2+\cdots+w_dx^dRead it like thisStart with the intercept, then add weighted powers of the input. For example, the squared-input term adds its weight times x times x.What the symbols meanx,  xjx,\;x_jx is the input: measurements supplied to the model. A small lower j selects input number j, such as distance from several measurements.yi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.b,  wjb,\;w_jb is the intercept, the starting prediction before adding input contributions. w is a learned weight; the lower j selects the weight for input j. A weight tells how strongly that input affects the prediction.d,  wdd,\;w_dd is the highest power of the input used by the curve. w with lower d is the learned weight multiplying that last power.×,  ⋅\times,\;\cdotBoth marks mean multiply. Letters written next to each other also multiply: wx means w times x. For vectors, a dot means multiply matching entries and add the results.==Equals: the expressions on both sides have the same value.++Plus: add the values together.x2x^2Squared: multiply a value by itself. For example, 3 squared is 9. Squaring an error removes its sign and gives larger mistakes more influence.xdx^dA small raised number or letter is a power. For a positive whole number d, multiply x by itself d times. For example, 2 to the power 3 is 8.⋯\cdotsThe dots mean continue the same pattern through the last listed item.

d is the selected degree. The curve is nonlinear in x but linear in the learned coefficients w. Feature construction itself does not learn the target relationship.

What to watch for

  • High-degree curves can chase noise and swing sharply outside the training range.
  • The number of generated columns can grow quickly when there are many original inputs.
Linear regressionRidgeLasso

Further reading

The official documentation goes further when you are ready.