How to Detect Overtraining in a Boosted Decision Tree

You have trained a boosted decision tree. The ROC curve on your training sample looks spectacular. You are about to cut on the classifier output and count signal events.

Stop. The first question any reviewer will ask is: is this overtrained?

Overtraining means the classifier has memorised statistical fluctuations in the training sample rather than learning the true signal-background separation. An overtrained model reports excellent performance on the data it was trained on, then degrades — sometimes catastrophically — on independent data. Here is how to detect it.

The train-test overlay

The most basic diagnostic is a histogram of the classifier output, drawn separately for the training sample and a held-out test sample, for signal and background independently. If the model has learned the true distributions, the training and test histograms should agree within statistical fluctuations. If the training histograms are more sharply peaked (signal pushed toward 1, background pushed toward 0) than the test histograms, the model is overtrained.

This is the default diagnostic in TMVA (Hoecker et al., arXiv:physics/0703039), where it appears as the "overtraining check" plot.

import numpy as np
from sklearn.ensemble import GradientBoostingClassifier
from sklearn.model_selection import train_test_split
import matplotlib.pyplot as plt

# Assume X, y are your feature matrix and labels.
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.5, random_state=42
)

bdt = GradientBoostingClassifier(
    n_estimators=300, max_depth=3, learning_rate=0.1
)
bdt.fit(X_train, y_train)

score_train = bdt.predict_proba(X_train)[:, 1]
score_test = bdt.predict_proba(X_test)[:, 1]

bins = np.linspace(0, 1, 41)
fig, axes = plt.subplots(1, 2, figsize=(10, 4))

for ax, label, cls in zip(axes, ["Background", "Signal"], [0, 1]):
    ax.hist(score_train[y_train == cls], bins=bins, density=True,
            histtype="stepfilled", alpha=0.4, label="Train")
    ax.hist(score_test[y_test == cls], bins=bins, density=True,
            histtype="step", lw=2, label="Test")
    ax.set_xlabel("Classifier output")
    ax.set_ylabel("Normalised density")
    ax.set_title(label)
    ax.legend()

plt.tight_layout()
plt.savefig("overtraining_check.png", dpi=120)

If the "Train" and "Test" histograms visually agree, you are in reasonable shape. If the training histogram is systematically more extreme, you are overtrained.

The Kolmogorov-Smirnov test

Visual agreement is necessary but subjective. The standard quantitative follow-up is the Kolmogorov-Smirnov (KS) test, which measures the maximum distance between two empirical cumulative distributions and returns a p-value for compatibility. TMVA reports this by default for every trained classifier.

from scipy.stats import ks_2samp

for cls, name in [(0, "Background"), (1, "Signal")]:
    stat, pval = ks_2samp(
        score_train[y_train == cls],
        score_test[y_test == cls]
    )
    print(f"{name}: KS stat = {stat:.4f}, p-value = {pval:.4f}")

A p-value well above 0.05 means the training and test distributions are compatible. A very small p-value indicates a statistically significant difference — evidence of overtraining.

A warning about interpretation. A passing KS test does not guarantee the model is safe. With large samples, a p-value can be high even when the distributions differ in ways that matter for your analysis. With small samples, it can fail due to statistical noise. The KS test is a necessary sanity check, not a proof of correctness.

What to do when you detect overtraining

The standard remedies, roughly in order of how often they help:

1. Reduce model complexity. Lower n_estimators, reduce max_depth, or increase min_samples_leaf. Shallower trees with fewer boosting rounds generalise better.

2. Increase regularisation. Lower the learning_rate (and compensate with more estimators). Add subsample < 1.0 for stochastic gradient boosting.

3. Remove noisy or redundant features. Every input variable is a potential axis for the model to overfit on. If a variable adds no separation power, dropping it reduces overtraining risk.

4. Get more training data. If your MC statistics are limited, consider whether your sample size is adequate for the model complexity you need.

5. Use k-fold cross-validation to get a more stable estimate of generalisation performance, especially when statistics are tight.

The bigger picture

Overtraining is the failure mode that analysis reviewers check first because it invalidates everything downstream: your efficiency estimate, your signal yield, and the systematic uncertainties you propagate from the classifier. A model that passes the overtraining check is not guaranteed to be correct, but a model that fails it is guaranteed to be unreliable.

The full sequence — from training a boosted decision tree through overtraining diagnostics, data/MC validation, and systematic uncertainty propagation — is covered step by step in the course. Start with the free Module 1 and the Classifier Roadmap.

Educational material. It does not replace the internal review and approval procedures of your experiment's collaboration.