Blockwise.
Back to learning
Model guide

Decision tree

Follow a sequence of questions, such as whether mass is below a threshold. Each answer sends you toward a final group of similar training examples.

Start with an example

A parcel may first be split by mass, then by length. Its final group supplies a class estimate or an average delivery time.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Try questions such as whether the input is below a chosen value. Select a question that divides the examples into groups with more similar answers, then repeat inside each group.
  2. Stop at a final group, called a leaf. For a number, predict that group's mean answer. For a class, use the group's class shares. Depth limits how many successive questions the tree may ask.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

See a numerical prediction made by asking threshold questions about x. A final group of examples is called a leaf; its average answer becomes the prediction for that region.

Across: the horizontal axis
The input measurement x, from 0 to 10. In a delivery task, an input could be distance; here it has no unit.
Up: the vertical axis
The target: the numerical answer being predicted. In a delivery task, it could be time; here the target has no unit.

Each dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: that is the predicted target for this input.

2. Predict, change, observe

  1. Set Maximum depth to 0. With no questions allowed, all examples belong to one group and the graph is flat.
  2. Predict what happens when depth rises to 2, then 6. Look for flat regions separated by jumps: additional questions let the model distinguish smaller groups.
Interactive Decision tree illustrationEach dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.02.557.5100246810Target valueInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 3.438

Mean squared training error: 0.214

Mean squared training error averages the squared vertical gaps from the observed dots to the line. Smaller means a closer fit to these training examples, not necessarily better predictions on new examples.

3. Explain it in your own words

Why can two different input values receive exactly the same prediction from a tree?

Compare your reasoning

They can reach the same leaf and receive the same group average. A tree does not need a different output for every input. If unsure, move the inspection line within one flat region, then across a jump, and compare the readout.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

Each split divides the input into two regions. Leaves predict an average value, or a class fraction for a classifier. More depth allows smaller regions.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed target
0.302.78
0.922.97
1.544.17
2.164.08
2.784.33
3.403.24
4.023.41
4.642.87
5.263.37
5.883.16
6.504.58
7.125.27
7.746.6
8.366.59
8.987.51
9.607.14

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

y^=1nL∑i∈Lyi,G=1−∑cpc2\hat y=\frac{1}{n_L}\sum_{i\in L}y_i,\qquad G=1-\sum_c p_c^2Read it like thisFor regression, average the observed answers in the reached leaf. For classification, square each class's share, add the squares, and subtract from one to measure the leaf's mixture.What the symbols meanyi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.L,  nLL,\;n_LL is the group of training examples reaching this leaf. n with lower L is how many examples are in that group.G,  pcG,\;p_cG is Gini impurity: how mixed the classes are. p with lower c is class c's share of the leaf's training weight. Adding the squared shares and subtracting from 1 gives G; zero means only one class is present.c,  c^c,\;\hat cc is one possible class, such as ordinary or urgent. c with a hat is the class the model chooses.==Equals: the expressions on both sides have the same value.−-Minus: subtract the next value. Before a value, it changes that value's sign.ab\frac{a}{b}A fraction means divide the entire top by the entire bottom. For example, 6 over 3 means 6 divided by 3, which is 2. The bottom cannot be zero.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.∈\inBelongs to: the item is a member of the named group.x2x^2Squared: multiply a value by itself. For example, 3 squared is 9. Squaring an error removes its sign and gives larger mistakes more influence.

p꜀ is the fraction of a leaf's training weight belonging to class c. A pure classification leaf has only one class and Gini impurity zero.

What to watch for

  • A deep tree can memorize accidental detail and change greatly after a small data change.
  • A leaf frequency is an estimate, not certainty. Ordinary feature importance does not prove causation.
Random forestBagging and pastingPCA

Further reading

The official documentation goes further when you are ready.