Blockwise.
Back to learning
Model guide

Softmax regression

Let several competing classes share one probability budget. A larger class score receives a larger share, and all shares add to one.

Start with an example

For exactly one of three delivery routes, three equal scores give probabilities of one third each. Increasing one score raises that route's share.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Learn a score for each possible class. Turn each score into a positive amount, then divide it by the total across classes. The resulting shares are probabilities that add to one.
  2. Compare the probability assigned to each training example's known class. Giving that class very little probability incurs a large cost; training adjusts the scores to reduce that cost. Prediction selects the class with the largest share.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

Compare three competing classes at the same input. Each colored line belongs to one class; softmax shares a total probability of 1 between them.

Across: the horizontal axis
The input measurement x, from 0 to 10. Compare all class lines at the same horizontal position.
Up: the vertical axis
Each class's estimated probability, from 0 to 1. A height of 0.8 means an estimate of 80% for that class.

Green, orange, and purple lines belong to classes A, B, and C. There are no observed training dots here: these class scores or contests are constructed for illustration.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line. Compare the three curve heights there: the highest identifies the winning class. The outlined diamond marks that highest value.

2. Predict, change, observe

  1. Keep Center of class B at 5. Inspect x = 2, then 5, then 8, and notice which class has the highest line at each position.
  2. Predict what raising Score sharpness will do near each class center. Increase it: the favored class gets a larger share, while the others get smaller shares.
Interactive Softmax regression illustrationGreen, orange, and purple lines belong to classes A, B, and C. There are no observed training dots here: these class scores or contests are constructed for illustration.00.250.50.7510246810Class probabilityInput feature (x)
  • Class A
  • Class B
  • Class C
  • Inspected result (diamond)
Highest class probability at x = 6.0: 0.626

Compare class lines at this input. This readout is the highest value, not the sum of the class values.

3. Explain it in your own words

At one input, two softmax probabilities are 0.2 and 0.3. What must the third be, and why?

Compare your reasoning

It must be 0.5 because the three probabilities add to 1. These hand-chosen score curves illustrate the conversion to probabilities, not a fitted linear softmax model. If unsure, read all three heights at the same x rather than comparing different inputs.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

Softmax converts three scores into probabilities that sum to 1. The toy scores favor the class whose center is closest. These scores are illustrative, not a trained model.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

pc=exp⁡(zc−m)∑jexp⁡(zj−m),ℓ=−log⁡(ptrue)p_c=\frac{\exp(z_c-m)}{\sum_j\exp(z_j-m)},\qquad \ell=-\log(p_{\mathrm{true}})Read it like thisSubtract the largest score from every class score. Apply exp, then divide each result by their total so the probabilities add to one. The loss is minus log of the observed class's probability.What the symbols meanzc,  m,  pc,  ptrue,  ℓz_c,\;m,\;p_c,\;p_{\mathrm{true}},\;\ellz with lower c is class c's score; m is the largest score in this row. p with lower c is that class's probability; p with lower true is the probability given to the observed class. Ell is the loss. The sum over j includes every competing class.c,  c^c,\;\hat cc is one possible class, such as ordinary or urgent. c with a hat is the class the model chooses.==Equals: the expressions on both sides have the same value.−-Minus: subtract the next value. Before a value, it changes that value's sign.ab\frac{a}{b}A fraction means divide the entire top by the entire bottom. For example, 6 over 3 means 6 divided by 3, which is 2. The bottom cannot be zero.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.log⁡\logNatural logarithm: the reverse of exp. It asks what power of e produces a positive input. For example, log(1) is 0. Here e is a fixed number about 2.718.exp⁡(z)\exp(z)Exponential: raise the fixed number e, about 2.718, to the power z. For example, exp(0) is 1. It always returns a positive value.

z꜀ is class c's score, m is the largest score in the row, and p_true is the probability assigned to its observed class. The sum covers competing classes.

What to watch for

  • One shared probability budget suits mutually exclusive classes; independent tags need a different output structure.
  • Large confidence is not proof of correctness or calibration.
Logistic regressionOne-versus-restOne-versus-one

Further reading

The official documentation goes further when you are ready.