Blockwise.
Back to learning
Model guide

K-nearest neighbors

Ask the most similar stored examples. Nearby deliveries suggest a duration; nearby parcels suggest a category.

Start with an example

If the three closest routes took 20, 24, and 28 minutes, uniform-weight regression predicts their average: 24 minutes.

How it learns

An input feature is a measurement about an example. The target is its known answer: a number for regression, or a category for classification. Training means learning from examples whose answers are already known.

  1. Store the training examples. For a new input, find the k closest stored inputs; k is the number of neighbors you choose to consult.
  2. To predict a number, average their known answers. To predict a class, count their class votes. Closeness uses the input measurements, not the unknown answer you are trying to predict.

Try it yourself

The example above gives the idea. The experiment below uses a separate, made-up dataset with unitless measurements so you can focus on how the model works. Read the graph guide first, then change one thing at a time.

1. Understand what you are looking at

Use nearby examples to estimate a class. This graph demonstrates classification; the delivery-time example above demonstrates the related averaging idea for regression.

Across: the horizontal axis
The input measurement x, from 0 to 10. Moving right means a larger input.
Up: the vertical axis
The estimated probability of class 1, from 0 to 1. A curve height of 0.8 means an estimated 80% chance, not an observed class of 0.8.

Dark dots are observed answers: class 0 at the bottom or class 1 at the top. The green line is the model's estimate, which can lie between 0 and 1.

How to read one result: Start at x on the bottom axis and follow the vertical dashed line to the outlined diamond. Its height is the estimated probability of class 1 at that input; it is not a new observed example.

2. Predict, change, observe

  1. Set Neighbors (k) to 1. Move Input to inspect (x) across the graph: the estimate follows the class of the closest dot, measured horizontally.
  2. Before increasing k to 15, predict whether one unusual dot will have more or less influence. Try it and compare how abruptly the line changes.
Interactive K-nearest neighbors illustrationDark dots are observed answers: class 0 at the bottom or class 1 at the top. The green line is the model's estimate, which can lie between 0 and 1.00.250.50.7510246810Class probabilityInput feature (x)
  • Observations
  • Model prediction
  • Inspected result (diamond)
Prediction at x = 6.0: 1.000

Training errors: 2 of 16.

Training errors count observed labels that disagree with the decision: estimates of at least 0.5 choose class 1; lower estimates choose class 0.

3. Explain it in your own words

If 2 of the 3 nearest examples belong to class 1, what height should the prediction have?

Compare your reasoning

The height is 2 divided by 3, about 0.67: two of the three neighbors vote for class 1. Distance here uses x, not the dot's class height. If unsure, set k to 1 first and follow how the nearest horizontal position determines the answer.

Ready to continue when you can explain the result and why it changes. If the explanation is still unclear, reset and repeat the two steps above with one control at a time.

What this experiment simplifies

Each prediction is the fraction of class-1 labels among the k nearest examples. A larger k uses a wider neighborhood.

These are small teaching implementations, not scikit-learn runs. Training error on these examples does not estimate performance on new data.

View the plotted examples
Input feature (x)Observed class
0.300
0.920
1.540
2.161
2.780
3.400
4.020
4.641
5.261
5.881
6.501
7.120
7.741
8.361
8.981
9.601

The math, one step at a time

Read the symbols after the idea makes sense. You do not need to memorize the formula.

y^=1k∑i∈Nk(x)yic^=arg⁡max⁡c∑i∈Nk(x)1[yi=c]\begin{gathered}\hat y=\frac{1}{k}\sum_{i\in N_k(x)}y_i\\\hat c=\underset{c}{\arg\max}\sum_{i\in N_k(x)}\mathbf{1}[y_i=c]\end{gathered}Read it like thisFor a numerical prediction, add the nearest neighbors' observed answers and divide by the neighbor count. For a class prediction, count each class's neighbors and choose the class with the most votes.What the symbols meanx,  xjx,\;x_jx is the input: measurements supplied to the model. A small lower j selects input number j, such as distance from several measurements.yi,  y^iy_i,\;\hat y_iy is an observed answer; y with a hat is a prediction. A small lower i selects example number i. Their difference is one prediction error.k,  Nk(x)k,\;N_k(x)k is how many neighbors to consult. N with lower k, evaluated at x, is the group of k training inputs closest to the new input x.c,  c^c,\;\hat cc is one possible class, such as ordinary or urgent. c with a hat is the class the model chooses.==Equals: the expressions on both sides have the same value.ab\frac{a}{b}A fraction means divide the entire top by the entire bottom. For example, 6 over 3 means 6 divided by 3, which is 2. The bottom cannot be zero.∑\sumCapital sigma: add one term for every indicated item. The symbol below says which item to count; limits such as j = 1 to p mean start at 1 and stop at p. For example, adding 2, 4, and 6 gives 12.arg⁡max⁡c\underset{c}{\arg\max}Arg max: return the class with the largest score, rather than the score itself. If scores for A and B are 0.2 and 0.8, choose B.1[condition]\mathbf{1}[\text{condition}]Indicator: use 1 when the condition in brackets is true, and 0 otherwise. Adding these values counts how often it is true.∈\inBelongs to: the item is a member of the named group.

Nₖ(x) is the set of k nearest training inputs to x. This formula assumes uniform weights; k is a setting, not a learned coefficient.

What to watch for

  • Feature units matter: kilometres and grams need an appropriate shared scale.
  • Very small k can follow noise; very large k can blur useful local patterns. Prediction also requires searching stored examples.
Linear regressionDecision tree

Further reading

The official documentation goes further when you are ready.