Blockwise.
Back to learning
Model guide

Ridge

Fit a linear model while charging it for large weights. The model must balance matching the examples against relying on extreme coefficients.

Start with an example

A high-degree delivery curve may wiggle between examples. A Ridge penalty can reduce those swings by shrinking its coefficients.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. A linear model multiplies inputs by weights and adds them. Ridge chooses those weights by balancing prediction errors against an extra cost for large weights: add each weight multiplied by itself. This is the L2 penalty.
  2. The alpha setting controls the penalty strength. Raising it encourages smaller weights, so predictions vary less with the inputs. Put measurements on comparable scales before fitting so their units do not unfairly determine the penalty.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

See how a penalty discourages a line from relying on a large slope. This graph fits one straight line, even when the example above discusses a more complex curve.

Across: the horizontal axis
The input measurement x, from 0 to 10. In a delivery task, an input could be distance; here it has no unit.
Up: the vertical axis
The target: the numerical answer being predicted. In a delivery task, it could be time; here the target has no unit.

Each dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: that is the predicted target for this input.

2. Predict, change, observe

  1. Set Penalty strength (alpha) to 0 and compare the prediction at x = 0 with the one at x = 10.
  2. Predict what will happen to that difference when alpha rises to 20. Increase it and watch the line become flatter; the fitting procedure recomputes the line for each setting.
Interactive Ridge illustrationEach dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.02.557.5100246810Target valueInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 4.918

Mean squared training error: 0.888

Mean squared training error averages the squared vertical gaps from the observed dots to the line. Smaller means a closer fit to these training examples, not necessarily better predictions on new examples.

3. Explain it in your own words

If the line becomes flatter as the penalty grows, has the input become less important to this fitted model, or has it been proved irrelevant in the real world?

Compare your reasoning

The fitted model relies less on the input. This does not prove real-world irrelevance. The penalty trades a closer fit for smaller weights. If unsure, compare alpha = 0 and alpha = 20 and describe the change in predictions without making a claim about cause and effect.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

This one-feature fit minimizes mean squared error plus a penalty on the slope. Increasing alpha shrinks the slope; L1 can make it exactly zero. The intercept is not penalized.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed target
0.302.78
0.922.97
1.544.17
2.164.08
2.784.33
3.403.24
4.023.41
4.642.87
5.263.37
5.883.16
6.504.58
7.125.27
7.746.6
8.366.59
8.987.51
9.607.14

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

min⁡b,w∑i(yi−y^i)2+α∑jwj2\min_{b,w}\sum_i(y_i-\hat y_i)^2+\alpha\sum_j w_j^2Read it like thisChoose a line that balances two costs: squared prediction mistakes and alpha times squared weights. The intercept is not penalized.What the symbols meanyi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.b,  wjb,\;w_jb is the intercept, the starting prediction before adding input contributions. w is a learned weight; the lower j selects the weight for input j. A weight tells how strongly that input affects the prediction.α\alphaAlpha: the penalty strength you choose. Larger alpha makes large weights more costly; it is not a prediction or an input measurement.++Plus: add the values together.−-Minus: subtract the next value. Before a value, it changes that value's sign.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.min⁡\minMinimize: choose the listed parameters to make the following error or cost as small as possible.x2x^2Squared: multiply a value by itself. For example, 3 squared is 9. Squaring an error removes its sign and gives larger mistakes more influence.

α is nonnegative penalty strength and w are feature coefficients. Here the intercept is fitted separately without the penalty; objective scaling differs across estimator conventions.

What to watch for

  • Too much penalty can erase real structure and cause underfitting.
  • Shrinkage usually keeps coefficients nonzero; a small weight is not proof that an input has no real-world effect.
Polynomial regressionLassoElastic Net

Further reading

The official documentation goes further when you are ready.