Blockwise.
Back to learning
Model guide

Histogram gradient boosting

Use the same stage-by-stage correction idea as gradient boosting, but group nearby numeric values into bins so there are fewer split positions to examine.

Start with an example

Instead of considering every measured route distance separately, first group distances into a limited set of intervals, then learn useful tree questions between them.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Group nearby measurement values into intervals, called bins. For example, several similar distances can share one bin. Trees can then compare groups of values instead of examining every distinct value as a possible split.
  2. Start with an estimate, then fit small trees in sequence to correct what it gets wrong. Each correction is scaled by the learning rate. Binning changes how useful tree questions are found; it keeps the idea of gradually correcting the combined prediction.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

Build a numerical prediction through a sequence of corrections. A residual is the observed answer minus the current prediction: it tells the next small tree what still needs fixing.

Across: the horizontal axis
The input measurement x, from 0 to 10. In a delivery task, an input could be distance; here it has no unit.
Up: the vertical axis
The target: the numerical answer being predicted. In a delivery task, it could be time; here the target has no unit.

Each dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: that is the predicted target for this input.

2. Predict, change, observe

  1. Set Number of stages to 1 and keep Learning rate at 0.3. Notice where the line still falls above or below the observed dots.
  2. Predict whether 20 stages can reduce those remaining gaps. Raise Number of stages to 20, then compare the line and the mean squared training error.
Interactive Histogram gradient boosting illustrationEach dark dot pairs one input with its observed target. The green line gives the model's predicted target at each input. A vertical gap from a dot to the line is a prediction error.02.557.5100246810Target valueInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 5.302

Mean squared training error: 0.402

Mean squared training error averages the squared vertical gaps from the observed dots to the line. Smaller means a closer fit to these training examples, not necessarily better predictions on new examples.

3. Explain it in your own words

If an answer is 6 and the current prediction is 4, the remaining error is +2. How much would a perfect +2 correction add at learning rate 0.3?

Compare your reasoning

It adds 0.6, giving a new prediction of 4.6. The learning rate scales the correction instead of replacing the whole prediction. If unsure, revisit one stage at a low rate and explain why more small corrections may be needed.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

This regression example starts with the mean and repeatedly fits a small tree to the remaining residuals. The learning rate scales each correction. It illustrates the regression variant of the family.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed target
0.302.78
0.922.97
1.544.17
2.164.08
2.784.33
3.403.24
4.023.41
4.642.87
5.263.37
5.883.16
6.504.58
7.125.27
7.746.6
8.366.59
8.987.51
9.607.14

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

Ft(x)=Ft−1(x)+ηht(bin⁡(x))F_t(x)=F_{t-1}(x)+\eta h_t(\operatorname{bin}(x))Read it like thisTake the previous combined prediction and add the learning rate times a new correction model that uses binned inputs.What the symbols meanFt,  Ft−1,  ht,  bin⁡(x)F_t,\;F_{t-1},\;h_t,\;\operatorname{bin}(x)F is the combined prediction; lower t selects the current stage and t-1 the previous one. h_t is the new correction model. bin(x) groups input measurements into intervals before the tree searches for a split.η\etaEta: the learning rate, which controls how strongly one training update or new model changes the current result.x,  xjx,\;x_jx is the input: measurements supplied to the model. A small lower j selects input number j, such as distance from several measurements.×,  ⋅\times,\;\cdotBoth marks mean multiply. Letters written next to each other also multiply: wx means w times x. For vectors, a dot means multiply matching entries and add the results.==Equals: the expressions on both sides have the same value.++Plus: add the values together.−-Minus: subtract the next value. Before a value, it changes that value's sign.

bin(x) denotes the grouped representation used to search tree splits. max_bins controls numeric binning; learning_rate and max_iter control correction size and stage budget.

What to watch for

  • Coarse bins can hide useful distinctions, and categorical or missing-value behavior depends on the installed library version.
  • Fit preparation only on training rows and define how unseen categories will be handled.
Gradient boostingDecision tree

Further reading

The official documentation goes further when you are ready.