The Kolmogorov-Smirnov Test as an Overtraining Check

August 14, 2026 · 5 min read

If you have ever trained a BDT for a signal-versus-background classifier and then wondered whether your model is actually learning the underlying physics or simply memorising the training sample, you are not alone. Overtraining is one of the most common silent problems in HEP analyses, and catching it early saves a great deal of pain later. The Kolmogorov-Smirnov overtraining check — comparing the BDT score distributions on the training set and the test set — is the standard first diagnostic, and it is simpler to run than most physicists expect.

Why Overtraining Shows Up in Score Distributions

When a BDT overtrained, it has learned the statistical fluctuations in the training sample rather than the genuine separation between signal and background. The consequence is that the model assigns scores that are slightly too extreme to events it has already seen. Plot the BDT output score for training events and test events on the same canvas, and an overtrained model will show visibly different shapes — the training distribution is typically sharper or more peaked near the extremes than the test distribution.

A KS test BDT comparison formalises this visual inspection. The Kolmogorov-Smirnov statistic measures the maximum absolute difference between two empirical cumulative distribution functions. If the training and test distributions are consistent with coming from the same underlying distribution, the KS statistic should be small and the associated p-value large. If the model has overtrained, the two CDFs diverge and the p-value drops.

Running the KS Test Step by Step

Step 1 — Split your sample before any training

Reserve a held-out test set strictly before you train anything. Events used during training must never appear in the test set. This is obvious in principle but easy to violate accidentally when using k-fold cross-validation or when you reuse events for both optimisation and evaluation. The KS test is only meaningful if the two samples are truly independent.

Step 2 — Train your BDT and collect scores

Train your classifier on the training set. Then pass both the training events and the test events through the trained model and store the output score for each event. Do this separately for signal and background, so you end up with four arrays: training signal scores, test signal scores, training background scores, and test background scores.

Step 3 — Compute the KS statistic and p-value

Run a two-sample KS test comparing the training signal scores to the test signal scores, and then separately for background. In Python, scipy.stats.ks_2samp does this in one line. The function returns the KS statistic and the p-value. Perform this for both signal and background independently.

Step 4 — Interpret the result

The p-value answers the question: if the training and test distributions were drawn from the same underlying distribution, how often would you observe a KS statistic at least this large by chance alone? A large p-value means the two distributions are consistent — no evidence of Kolmogorov-Smirnov overtraining. A small p-value is a warning that the model has memorised features of the training sample.

Think of it like a goodness-of-fit test. You have a hypothesis (training and test come from the same distribution) and you are computing a test statistic under that hypothesis. A small p-value rejects the hypothesis, just as a large chi-square rejects a fit. The physics intuition maps cleanly onto statistics you already use every day.

Step 5 — Look at the distributions too

Never rely on the p-value alone. Plot the training and test score histograms overlaid on the same axes, with error bars from the finite sample size. The KS test is sensitive to the overall shape, but a visual check helps you understand where the distributions differ — at the signal peak, in the background tail, or across the whole range. If the discrepancy is localised near the score threshold you plan to cut on, that is more operationally significant than a mismatch in a region you will not use.

Common Pitfalls

Unequal sample sizes inflate sensitivity. The KS test becomes very sensitive when you have large samples, meaning even tiny, physically irrelevant differences can produce small p-values. Use your judgment alongside the plot.

Reweighted events need care. If your training or test events carry per-event weights, the standard KS test does not account for them correctly. Several weighted KS implementations exist; make sure you are using one that properly handles your weight scheme.

Running the test only once is not enough. A single train-test split is itself subject to statistical fluctuation. If resources allow, use k-fold cross-validation and aggregate the KS statistics across folds for a more stable picture. The full HEP ML course covers cross-validation strategies that integrate naturally with overtraining checks like this one.

Going Further

The KS test is a first check, not an exhaustive diagnosis. Complement it with learning curves (training and validation loss versus training sample size), early stopping, and regularisation tuning. If you are building your first classifier and want a structured path through these ideas, the complete course walks through the full workflow from feature engineering to final validation in a particle physics context.

A consistent KS test is necessary but not sufficient: it tells you the model is not memorising the training sample, but it does not guarantee the model is physically meaningful or well-calibrated.

Takeaway: Run a KS test comparing train and test BDT score distributions for signal and background separately, inspect the plots alongside the p-value, and treat a small p-value as a prompt to reduce model complexity — not just a number to report.