Blockwise.
Back to learning
Model guide

Extra-Trees

Combine trees with extra randomness in their questions. Instead of searching every possible threshold, a tree compares randomly proposed thresholds.

Start with an example

One tree might try a mass threshold of 3.4 kg and another 5.1 kg. Averaging many such trees can smooth the effect of a single arbitrary split.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. A decision tree asks whether a measurement is below a threshold. Extra-Trees proposes threshold values at random, then chooses the best of those proposals for separating the current examples.
  2. Build many such trees and average their numerical answers or class-probability estimates. Random thresholds create differences between trees even when they see the same training rows; row resampling is a separate optional choice.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

See the combined numerical prediction from several trees. The green line shows their average, so a jump made by one tree need not dominate the result.

Across: the horizontal axis
The input measurement x, from 0 to 10. In a delivery task, an input could be distance; here it has no unit.
Up: the vertical axis
The target: the numerical answer being predicted. In a delivery task, it could be time; here the target has no unit.

Each dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: that is the predicted target for this input.

2. Predict, change, observe

  1. Set Number of trees to 1 and keep Maximum depth at 3. Notice the flat regions from a single tree.
  2. Predict whether averaging 30 trees must still look like that one tree. Increase Number of trees to 30 and compare the smaller changes in the combined line.
Interactive Extra-Trees illustrationEach dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.02.557.5100246810Target valueInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 4.098

Mean squared training error: 0.164

Mean squared training error averages the squared vertical gaps from the observed dots to the line. Smaller means a closer fit to these training examples, not necessarily better predictions on new examples.

3. Explain it in your own words

If three trees predict 2, 5, and 8 for the same input, what does their average predict? Does adding trees guarantee better future predictions?

Compare your reasoning

Their average is 5. Averaging changes how individual answers contribute; it does not guarantee improvement on new data. If unsure, hold depth fixed while changing tree count so that you are comparing the effect of averaging rather than also changing each tree's complexity.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

The trees use random split thresholds and their predictions are averaged. This one-feature example illustrates Extra Trees; full models also sample features.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed target
0.302.78
0.922.97
1.544.17
2.164.08
2.784.33
3.403.24
4.023.41
4.642.87
5.263.37
5.883.16
6.504.58
7.125.27
7.746.6
8.366.59
8.987.51
9.607.14

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

y^=1T∑t=1Tht(x),pc=1T∑t=1Tpt,c(x)\hat y=\frac{1}{T}\sum_{t=1}^{T}h_t(x),\qquad p_c=\frac{1}{T}\sum_{t=1}^{T}p_{t,c}(x)Read it like thisAverage the trees' predictions for regression. For classification, average their probability estimates separately for each class.What the symbols meanx,  xjx,\;x_jx is the input: measurements supplied to the model. A small lower j selects input number j, such as distance from several measurements.yi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.T,  t,  ht(x)T,\;t,\;h_t(x)T is the number of member models, t selects one member, and h with lower t is that member's prediction for input x. A function name followed by (x) means evaluate it using x.pt,c(x)p_{t,c}(x)Member t's estimated probability for class c at input x. Both lower labels select one entry: first the member, then the class.c,  c^c,\;\hat cc is one possible class, such as ordinary or urgent. c with a hat is the class the model chooses.==Equals: the expressions on both sides have the same value.ab\frac{a}{b}A fraction means divide the entire top by the entire bottom. For example, 6 over 3 means 6 divided by 3, which is 2. The bottom cannot be zero.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.

The final classifier chooses the largest p꜀. n_estimators sets tree count; max_features and depth settings help control diversity and complexity.

What to watch for

  • Extra randomness can help or hurt; it needs validation on the actual task.
  • It still inherits tree-ensemble memory costs and cannot turn feature importance into causal evidence.
Random forestDecision treeBagging and pasting

Further reading

The official documentation goes further when you are ready.