Blockwise.
Back to learning
Model guide

Random forest

Ask many somewhat different decision trees, then combine their answers. One tree's unusual split matters less when the others disagree.

Start with an example

For delivery times, tree estimates of 20, 24, and 28 minutes average to 24. A classifier instead averages class-probability estimates across trees.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Build several decision trees, each asking questions that split examples into smaller groups. Usually each tree receives a random sample that can repeat some training rows and omit others.
  2. At each question, let a tree consider only a random selection of input measurements. This helps trees differ. Average their numerical predictions or their class-probability estimates to produce the forest's answer.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

See the combined numerical prediction from several trees. The green line shows their average, so a jump made by one tree need not dominate the result.

Across: the horizontal axis
The input measurement x, from 0 to 10. In a delivery task, an input could be distance; here it has no unit.
Up: the vertical axis
The target: the numerical answer being predicted. In a delivery task, it could be time; here the target has no unit.

Each dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: that is the predicted target for this input.

2. Predict, change, observe

  1. Set Number of trees to 1 and keep Maximum depth at 3. Notice the flat regions from a single tree.
  2. Predict whether averaging 30 trees must still look like that one tree. Increase Number of trees to 30 and compare the smaller changes in the combined line.
Interactive Random forest illustrationEach dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.02.557.5100246810Target valueInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 3.738

Mean squared training error: 0.149

Mean squared training error averages the squared vertical gaps from the observed dots to the line. Smaller means a closer fit to these training examples, not necessarily better predictions on new examples.

3. Explain it in your own words

If three trees predict 2, 5, and 8 for the same input, what does their average predict? Does adding trees guarantee better future predictions?

Compare your reasoning

Their average is 5. Averaging changes how individual answers contribute; it does not guarantee improvement on new data. If unsure, hold depth fixed while changing tree count so that you are comparing the effect of averaging rather than also changing each tree's complexity.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

Each tree learns from a bootstrap sample of the examples, then predictions are averaged. This one-feature example illustrates bagging; full Random Forests also sample features.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed target
0.302.78
0.922.97
1.544.17
2.164.08
2.784.33
3.403.24
4.023.41
4.642.87
5.263.37
5.883.16
6.504.58
7.125.27
7.746.6
8.366.59
8.987.51
9.607.14

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

y^=1T∑t=1Tht(x),c^=arg⁡max⁡c1T∑tpt,c(x)\hat y=\frac{1}{T}\sum_{t=1}^{T}h_t(x),\qquad \hat c=\underset{c}{\arg\max}\frac{1}{T}\sum_t p_{t,c}(x)Read it like thisFor regression, average all tree predictions. For classification, average the trees' probabilities for each class and choose the class with the largest average.What the symbols meanx,  xjx,\;x_jx is the input: measurements supplied to the model. A small lower j selects input number j, such as distance from several measurements.yi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.T,  t,  ht(x)T,\;t,\;h_t(x)T is the number of member models, t selects one member, and h with lower t is that member's prediction for input x. A function name followed by (x) means evaluate it using x.pt,c(x)p_{t,c}(x)Member t's estimated probability for class c at input x. Both lower labels select one entry: first the member, then the class.c,  c^c,\;\hat cc is one possible class, such as ordinary or urgent. c with a hat is the class the model chooses.==Equals: the expressions on both sides have the same value.ab\frac{a}{b}A fraction means divide the entire top by the entire bottom. For example, 6 over 3 means 6 divided by 3, which is 2. The bottom cannot be zero.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.arg⁡max⁡c\underset{c}{\arg\max}Arg max: return the class with the largest score, rather than the score itself. If scores for A and B are 0.2 and 0.8, choose B.

T is the number of trees and hₜ(x) is tree t's prediction. Classifier probability columns must be interpreted using the fitted class order.

What to watch for

  • Averaging cannot repair systematically wrong labels or missing predictive information.
  • Forests are harder to explain than a small tree and can cost more memory and prediction time.
Decision treeExtra-TreesBagging and pasting

Further reading

The official documentation goes further when you are ready.