Blockwise.
Back to learning
Model guide

Stacking

Train a second model to combine the first models' predictions. This blender learns how their answers can work together.

Start with an example

Two parcel models estimate review probabilities 0.2 and 0.7. Those two numbers become one training row for a logistic-regression blender.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. First obtain several models' predictions for each training row. To make this honest evidence, each prediction must come from a fit that did not use that row; temporarily hold groups of rows aside to do this.
  2. Use these predictions as inputs to a combining model, called the meta-model, and teach it using the known answers. Finally, refit the base models on all training rows for future predictions. Keep a separate evaluation set outside the entire training process.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

Use two models' estimates as inputs to another model. Dashed lines show the fixed base estimates; the solid purple line is the combining model's output.

Across: the horizontal axis
The input measurement x, from 0 to 10. Moving right means a larger input.
Up: the vertical axis
The estimated probability of class 1, from 0 to 1. A curve height of 0.8 means an estimated 80% chance, not an observed class of 0.8.

Dark dots are observed answers: 0 or 1. Dashed green and orange lines show base models A and B; the solid purple line shows their combined estimate.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond. Its height is the estimated probability of class 1 at that input; it is not a new observed example.

2. Predict, change, observe

  1. Set both Meta-model weights to 0. Predict whether the solid output will still follow either dashed line, then observe its constant height of 0.5.
  2. Increase only Meta-model weight A to 6. Then reset and increase only weight B to 6. Compare which member's pattern influences the combined output.
Interactive Stacking illustrationDark dots are observed answers: 0 or 1. Dashed green and orange lines show base models A and B; the solid purple line shows their combined estimate.00.250.50.7510246810Class probabilityInput feature (x)
  • Observations
  • Model A
  • Model B
  • Combined prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 0.919

Training errors: 2 of 16.

Training errors count observed labels that disagree with the decision: estimates of at least 0.5 choose class 1; lower estimates choose class 0.

3. Explain it in your own words

Does changing these weights mean this experiment has learned how best to combine the models?

Compare your reasoning

No. You are choosing weights by hand. A real stack learns its combining rule using base predictions made for rows those base fits did not train on. If unsure, separate what is fixed here (the dashed base lines) from what you control (the combining weights).

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

This illustration sends two fixed classifiers’ probabilities into a logistic meta-model. Adjust its weights by hand. A real stack learns these weights using held-out predictions; this graph does not train a stack.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed class
0.300
0.920
1.540
2.161
2.780
3.400
4.020
4.641
5.261
5.881
6.501
7.120
7.741
8.361
8.981
9.601

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

y^=g ⁣(h1(x),…,hT(x))\hat y=g\!\left(h_1(x),\ldots,h_T(x)\right)Read it like thisGet one prediction from each base model, then feed that list into the final model to make the combined prediction.What the symbols meanh1(x),  hT(x),  gh_1(x),\;h_T(x),\;gh_1 through h_T are the T base models' predictions for input x. g is the final model: the list in its parentheses supplies those predictions as its inputs.x,  xjx,\;x_jx is the input: measurements supplied to the model. A small lower j selects input number j, such as distance from several measurements.yi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.==Equals: the expressions on both sides have the same value.⋯\cdotsThe dots mean continue the same pattern through the last listed item.

h₁ through h_T are base models and g is the final blender. Its training features must be honest held-out predictions, not predictions from base models fitted on those same rows.

What to watch for

  • Training the blender on in-sample base predictions can leak overly optimistic evidence.
  • Several fold fits and prediction layers add cost; the extra complexity needs a measured validation benefit.
Voting ensemblesLogistic regressionRandom forest

Further reading

The official documentation goes further when you are ready.