Blockwise.
Back to learning
Model guide

AdaBoost

Train a small learner, notice which examples it misses, and give those examples more attention in the next round. Combine the learners with weighted votes.

Start with an example

A depth-one tree asks one question. After it misses some parcels, the next tree trains with more relative weight on those mistakes.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Fit a small classifier, then identify which training examples it misclassifies. Give those examples more weight when fitting the next classifier, so its training pays more attention to them.
  2. Repeat and combine the classifiers' votes, giving a classifier more influence when its weighted error is lower. Learning rate scales that influence. Each stage depends on earlier mistakes, so the stages must be trained in order.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

Combine simple yes-or-no rules into a class vote. Positive scores favor class 1, negative scores favor class 0; the height is a vote score, not a probability.

Across: the horizontal axis
The input measurement x, from 0 to 10. Moving right means a larger input.
Up: the vertical axis
The weighted vote score, from -1 to 1. Negative values favor class 0; positive values favor class 1. This is not a probability.

Dark dots show observed classes: class 0 is drawn at -1 and class 1 at +1 so their labels match the score's direction. The green line shows the combined vote score.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond on the curve. Read its height on the left: the sign determines the predicted class.

2. Predict, change, observe

  1. Set Number of stages to 1 and Learning rate to 1. Find dots whose class disagrees with the sign of the prediction line.
  2. Predict what later rules will focus on. Increase Number of stages to 30: later rules give more attention to previously misclassified examples. Compare the training-error count.
Interactive AdaBoost illustrationDark dots show observed classes: class 0 is drawn at -1 and class 1 at +1 so their labels match the score's direction. The green line shows the combined vote score.-1-0.500.510246810Weighted vote scoreInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Weighted vote score at x = 6.0: 0.781

Training errors: 2 of 16.

Training errors count dots whose class disagrees with the vote score's sign. This experiment assigns a zero score to class 1.

3. Explain it in your own words

Does a weighted vote score of 0.8 mean an 80% probability of class 1?

Compare your reasoning

No. The sign chooses a class in this experiment; the score describes the balance of weighted votes. A zero score is assigned to class 1 here. If unsure, compare the vertical score axis with the logistic-regression probability axis before interpreting the number.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

Each new decision stump gives more weight to examples earlier stumps misclassified. The graph shows a weighted vote score, not a probability.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed class
0.300
0.920
1.540
2.161
2.780
3.400
4.020
4.641
5.261
5.881
6.501
7.120
7.741
8.361
8.981
9.601

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

at=ηlog⁡1−etetw~i=wiexp⁡ ⁣(at1[yi≠ht(xi)]),wi←w~i∑jw~j\begin{gathered}a_t=\eta\log\frac{1-e_t}{e_t}\\\tilde w_i=w_i\exp\!\left(a_t\mathbf{1}[y_i\ne h_t(x_i)]\right),\qquad w_i\leftarrow\frac{\tilde w_i}{\sum_j\tilde w_j}\end{gathered}Read it like thisUse the learner's weighted error to calculate its voting weight. Increase example weights where its prediction is wrong, then divide all changed weights by their total so they add to one.What the symbols meanat,  et,  wi,  w~ia_t,\;e_t,\;w_i,\;\tilde w_ia with lower t is member t's voting weight; e with lower t is its weighted error share. w with lower i is example i's current weight. A tilde over w marks the changed weight before dividing all weights by their new total. Here e_t must be between zero and one half.η\etaEta: the learning rate, which controls how strongly one training update or new model changes the current result.yi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.x,  xjx,\;x_jx is the input: measurements supplied to the model. A small lower j selects input number j, such as distance from several measurements.T,  t,  ht(x)T,\;t,\;h_t(x)T is the number of member models, t selects one member, and h with lower t is that member's prediction for input x. A function name followed by (x) means evaluate it using x.==Equals: the expressions on both sides have the same value.−-Minus: subtract the next value. Before a value, it changes that value's sign.ab\frac{a}{b}A fraction means divide the entire top by the entire bottom. For example, 6 over 3 means 6 divided by 3, which is 2. The bottom cannot be zero.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.1[condition]\mathbf{1}[\text{condition}]Indicator: use 1 when the condition in brackets is true, and 0 otherwise. Adding these values counts how often it is true.≠\neNot equal: the two values or labels differ.←\leftarrowReplace the value on the left with the result on the right. It describes an update, not an equality that was already true.log⁡\logNatural logarithm: the reverse of exp. It asks what power of e produces a positive input. For example, log(1) is 0. Here e is a fixed number about 2.718.exp⁡(z)\exp(z)Exponential: raise the fixed number e, about 2.718, to the power z. For example, exp(0) is 1. It always returns a positive value.

eₜ is the member's weighted error, η is learning_rate, aₜ is its vote weight, and wᵢ is example i's weight. This binary formula assumes 0 < eₜ < 1/2; fitting handles stopping cases separately.

What to watch for

  • Emphasizing difficult examples can also emphasize wrong labels and outliers.
  • The binary SAMME weight formula is not a universal formula for every boosting algorithm or multiclass case.
Decision treeGradient boostingBagging and pasting

Further reading

The official documentation goes further when you are ready.