Blockwise.
Back to learning
Model guide

Voting ensembles

Let several fitted models contribute to one answer. Hard voting counts their chosen labels; soft voting averages their probability estimates.

Start with an example

Labels 0, 0, and 1 give a hard-vote result of 0. But class-1 probabilities 0.49, 0.49, and 0.95 average above 0.5, giving soft-vote result 1.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Train several models to answer the same question, using matching class labels. Hard voting counts their chosen classes; soft voting averages their probability estimates for each class.
  2. Choose the class with the most votes or the largest averaged probability. A member's weight controls how much its answer contributes. For numerical predictions, average the members' numerical answers instead.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

Combine two fixed probability estimates. The dashed green and orange lines are models A and B; the solid purple line is their weighted average.

Across: the horizontal axis
The input measurement x, from 0 to 10. Moving right means a larger input.
Up: the vertical axis
The estimated probability of class 1, from 0 to 1. A curve height of 0.8 means an estimated 80% chance, not an observed class of 0.8.

Dark dots are observed answers: 0 or 1. Dashed green and orange lines show base models A and B; the solid purple line shows their combined estimate.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond. Its height is the estimated probability of class 1 at that input; it is not a new observed example.

2. Predict, change, observe

  1. Set Weight given to model A to 0. Predict which dashed line the solid combined line will match, then check.
  2. Move the weight to 1, then 0.5. At the halfway setting, compare the combined height with both member heights at the same x.
Interactive Voting ensembles illustrationDark dots are observed answers: 0 or 1. Dashed green and orange lines show base models A and B; the solid purple line shows their combined estimate.00.250.50.7510246810Class probabilityInput feature (x)
  • Observations
  • Model A
  • Model B
  • Combined prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 0.904

Training errors: 2 of 16.

Training errors count observed labels that disagree with the decision: estimates of at least 0.5 choose class 1; lower estimates choose class 0.

3. Explain it in your own words

If A estimates 0.2 and B estimates 0.8, what is the equally weighted estimate?

Compare your reasoning

It is 0.5, halfway between 0.2 and 0.8. At weight 0 only B contributes; at weight 1 only A contributes. If unsure, try those two endpoints first, then return to 0.5 and read the three lines at one input.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

Two fixed classifiers give probabilities. The combined line is their weighted average: increase model A’s weight to move the result toward its predictions.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed class
0.300
0.920
1.540
2.161
2.780
3.400
4.020
4.641
5.261
5.881
6.501
7.120
7.741
8.361
8.981
9.601

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

pc=∑tatpt,c∑tat,c^=arg⁡max⁡c  pcp_c=\frac{\sum_t a_t p_{t,c}}{\sum_t a_t},\qquad \hat c=\underset{c}{\arg\max}\;p_cRead it like thisFor each class, multiply each member's probability by its voting weight, add the products, and divide by total weight. Choose the class with the largest combined probability.What the symbols meanat,  pt,c,  pca_t,\;p_{t,c},\;p_ca_t is member t's nonnegative voting weight. p_{t,c} is its probability for class c; p_c is the combined probability. Divide the weighted total by the total weight, which must be positive.c,  c^c,\;\hat cc is one possible class, such as ordinary or urgent. c with a hat is the class the model chooses.×,  ⋅\times,\;\cdotBoth marks mean multiply. Letters written next to each other also multiply: wx means w times x. For vectors, a dot means multiply matching entries and add the results.==Equals: the expressions on both sides have the same value.ab\frac{a}{b}A fraction means divide the entire top by the entire bottom. For example, 6 over 3 means 6 divided by 3, which is 2. The bottom cannot be zero.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.arg⁡max⁡c\underset{c}{\arg\max}Arg max: return the class with the largest score, rather than the score itself. If scores for A and B are 0.2 and 0.8, choose B.

pₜ,꜀ is member t's probability estimate for class c; aₜ is a nonnegative voting weight. Equal weights give a simple average.

What to watch for

  • Confident but poorly calibrated members can distort soft voting.
  • A committee is not automatically better than its best member; correlated mistakes can persist.
StackingRandom forestLogistic regression

Further reading

The official documentation goes further when you are ready.