Blockwise.
Back to learning
Model guide

Lasso

Fit a linear model with a penalty that can switch some weights completely off. The resulting prediction may use fewer input columns.

Start with an example

If two sensors carry similar information, Lasso may keep one and assign the other a zero coefficient in this particular fit.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. A weight controls how strongly an input contributes to a prediction. Lasso adds a cost for weight size, ignoring the sign: both a negative and a positive weight of the same size cost the same. This is the L1 penalty.
  2. Training balances prediction errors against this penalty. Increasing alpha strengthens the penalty and can make some weights exactly zero, so this particular fitted model stops using those inputs. Compare settings on examples kept separate from fitting.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

See how a penalty discourages a line from relying on a large slope. This graph fits one straight line, even when the example above discusses a more complex curve.

Across: the horizontal axis
The input measurement x, from 0 to 10. In a delivery task, an input could be distance; here it has no unit.
Up: the vertical axis
The target: the numerical answer being predicted. In a delivery task, it could be time; here the target has no unit.

Each dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: that is the predicted target for this input.

2. Predict, change, observe

  1. Set Penalty strength (alpha) to 0 and compare the prediction at x = 0 with the one at x = 10.
  2. Predict what will happen to that difference when alpha rises to 20. Increase it and watch the line become flatter; the fitting procedure recomputes the line for each setting.
Interactive Lasso illustrationEach dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.02.557.5100246810Target valueInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 4.905

Mean squared training error: 0.899

Mean squared training error averages the squared vertical gaps from the observed dots to the line. Smaller means a closer fit to these training examples, not necessarily better predictions on new examples.

3. Explain it in your own words

If the line becomes flatter as the penalty grows, has the input become less important to this fitted model, or has it been proved irrelevant in the real world?

Compare your reasoning

The fitted model relies less on the input. This does not prove real-world irrelevance. The penalty trades a closer fit for smaller weights. If unsure, compare alpha = 0 and alpha = 20 and describe the change in predictions without making a claim about cause and effect.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

This one-feature fit minimizes mean squared error plus a penalty on the slope. Increasing alpha shrinks the slope; L1 can make it exactly zero. The intercept is not penalized.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed target
0.302.78
0.922.97
1.544.17
2.164.08
2.784.33
3.403.24
4.023.41
4.642.87
5.263.37
5.883.16
6.504.58
7.125.27
7.746.6
8.366.59
8.987.51
9.607.14

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

min⁡b,w12n∑i(yi−y^i)2+α∑j∣wj∣\min_{b,w}\frac{1}{2n}\sum_i(y_i-\hat y_i)^2+\alpha\sum_j|w_j|Read it like thisChoose weights to reduce average squared error plus a charge for their absolute sizes. Divide the squared-error total by twice the training-row count. The intercept is not penalized.What the symbols meanyi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.b,  wjb,\;w_jb is the intercept, the starting prediction before adding input contributions. w is a learned weight; the lower j selects the weight for input j. A weight tells how strongly that input affects the prediction.α\alphaAlpha: the penalty strength you choose. Larger alpha makes large weights more costly; it is not a prediction or an input measurement.n,  in,\;in is the number of training examples and i selects one example. Dividing their total by n gives their average.×,  ⋅\times,\;\cdotBoth marks mean multiply. Letters written next to each other also multiply: wx means w times x. For vectors, a dot means multiply matching entries and add the results.++Plus: add the values together.−-Minus: subtract the next value. Before a value, it changes that value's sign.ab\frac{a}{b}A fraction means divide the entire top by the entire bottom. For example, 6 over 3 means 6 divided by 3, which is 2. The bottom cannot be zero.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.min⁡\minMinimize: choose the listed parameters to make the following error or cost as small as possible.x2x^2Squared: multiply a value by itself. For example, 3 squared is 9. Squaring an error removes its sign and gives larger mistakes more influence.∣w∣|w|Absolute value: the size of a number without its sign. Both -3 and 3 have absolute value 3.

n is the training-row count, α is penalty strength, and |w| is absolute weight size. The intercept is not included in this penalty.

What to watch for

  • Correlated inputs can make the selected nonzero coefficients unstable.
  • A zero coefficient does not prove irrelevance or lack of a causal effect. Excessive alpha can underfit.
RidgeElastic Net

Further reading

The official documentation goes further when you are ready.