Blockwise.
Back to learning
Model guide

Bagging and pasting

Train several models on different samples of the training data, then combine their answers. Sampling helps them make different mistakes.

Start with an example

Repeatedly sample parcels, fit a tree to each sample, and average their class probabilities. With bagging, a row may appear more than once in one sample.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Create several training samples and fit one model to each. Bagging allows the same row to be selected more than once in a sample; pasting selects rows without repeating them within that sample.
  2. The models can learn independently. Combine their numerical predictions by averaging, or combine class estimates or votes for classification. Different samples help the models make different predictions, which gives averaging a chance to reduce individual mistakes.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

See the combined numerical prediction from several trees. The green line shows their average, so a jump made by one tree need not dominate the result.

Across: the horizontal axis
The input measurement x, from 0 to 10. In a delivery task, an input could be distance; here it has no unit.
Up: the vertical axis
The target: the numerical answer being predicted. In a delivery task, it could be time; here the target has no unit.

Each dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: that is the predicted target for this input.

2. Predict, change, observe

  1. Set Number of trees to 1 and keep Maximum depth at 3. Notice the flat regions from a single tree.
  2. Predict whether averaging 30 trees must still look like that one tree. Increase Number of trees to 30 and compare the smaller changes in the combined line.
Interactive Bagging and pasting illustrationEach dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.02.557.5100246810Target valueInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 3.738

Mean squared training error: 0.149

Mean squared training error averages the squared vertical gaps from the observed dots to the line. Smaller means a closer fit to these training examples, not necessarily better predictions on new examples.

3. Explain it in your own words

If three trees predict 2, 5, and 8 for the same input, what does their average predict? Does adding trees guarantee better future predictions?

Compare your reasoning

Their average is 5. Averaging changes how individual answers contribute; it does not guarantee improvement on new data. If unsure, hold depth fixed while changing tree count so that you are comparing the effect of averaging rather than also changing each tree's complexity.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

Each tree learns from a bootstrap sample of the examples, then predictions are averaged. This one-feature example illustrates bagging; full Random Forests also sample features.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed target
0.302.78
0.922.97
1.544.17
2.164.08
2.784.33
3.403.24
4.023.41
4.642.87
5.263.37
5.883.16
6.504.58
7.125.27
7.746.6
8.366.59
8.987.51
9.607.14

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

y^=1T∑t=1Tht(x)\hat y=\frac{1}{T}\sum_{t=1}^{T}h_t(x)Read it like thisAdd every member model's numerical prediction and divide by the number of members to get their average.What the symbols meanx,  xjx,\;x_jx is the input: measurements supplied to the model. A small lower j selects input number j, such as distance from several measurements.yi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.T,  t,  ht(x)T,\;t,\;h_t(x)T is the number of member models, t selects one member, and h with lower t is that member's prediction for input x. A function name followed by (x) means evaluate it using x.==Equals: the expressions on both sides have the same value.ab\frac{a}{b}A fraction means divide the entire top by the entire bottom. For example, 6 over 3 means 6 divided by 3, which is 2. The bottom cannot be zero.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.

T is the member count and hₜ is member t. bootstrap chooses replacement sampling; max_samples controls sample size. The base estimator need not be a tree.

What to watch for

  • Highly similar member errors leave less room for averaging to help.
  • Adding members increases computation and does not remove systematic bias in the data or model family.
Random forestVoting ensemblesAdaBoost

Further reading

The official documentation goes further when you are ready.