Five Mistakes Analysis Reviewers Catch in Boosted Decision Tree Analyses

Analysis review committees exist to catch problems before they reach a paper. If your analysis uses a boosted decision tree classifier — and most competitive analyses at the LHC do — there is a short list of mistakes that reviewers have seen enough times to check systematically.

Here are the five they look for first.

1. Overtraining

The problem. The classifier memorises the training sample instead of learning the true separation. Performance looks excellent on training data and degrades on independent data.

How reviewers check. They ask for the train-test overlay: histograms of the classifier output drawn separately on training and test samples. If the training histograms are more peaked, the model is overtrained. The KS test quantifies compatibility; TMVA (Hoecker et al., arXiv:physics/0703039) reports it by default.

How to avoid it. Reduce tree depth, lower the learning rate, add regularisation, or remove redundant input variables. Always hold out a test sample and show the overlay.

2. Data/MC disagreement in input variables

The problem. A boosted decision tree is only as good as its inputs. If the simulation does not describe the data in a variable that the classifier uses, the classifier will exploit that mis-modelling to separate "real signal" from "simulated background" — a separation that has nothing to do with physics.

How reviewers check. They ask for data/MC comparison plots of every input variable in a signal-depleted control region. Any variable with visible disagreement is suspect.

How to avoid it. Validate every input variable in control regions before training. Apply the corrective reweightings your experiment requires — track multiplicity, PID calibration (e.g. PIDCalib2 at LHCb), generator-level corrections. If a variable cannot be made to agree, do not use it.

3. Training on the sample you measure

The problem. You train the boosted decision tree on your full dataset, then apply it to the same dataset to extract a signal yield. The classifier has seen every event it will classify, so its output distribution is biased. This is distinct from overtraining — even a perfectly generalising model has a biased output distribution when applied to its own training data.

How reviewers check. They ask where the training sample came from and whether it overlaps with the measurement sample. If it does, they ask for a k-fold cross-validation procedure or a fully independent training sample.

How to avoid it. Train on simulation, apply to data. Or, if you must train on data (e.g. with sPlot weights), use k-fold cross-validation: train on k-1 folds, classify the held-out fold, rotate.

4. Optimising the cut on the classifier output using signal data

The problem. You scan the cut on the classifier output to maximise significance using the signal yield from your actual data. This biases your measurement toward the fluctuation at the optimal cut.

How reviewers check. They ask how the working point was chosen. If it involved scanning the measurement sample, the analysis has an unaccounted look-elsewhere problem.

How to avoid it. Use a figure of merit that does not require the true signal yield — Punzi significance is the standard choice. Or fix the working point on simulation before unblinding.

5. Not propagating systematic uncertainties through the classifier

The problem. The classifier is part of the measurement chain. If a systematic variation changes the input variables, it changes the classifier output, which changes the efficiency and yield. Ignoring this underestimates your systematic uncertainty.

How reviewers check. They ask whether systematic variations were propagated through the classifier — did you re-evaluate with each systematic shift and re-derive the efficiency?

How to avoid it. For each systematic source, generate varied MC (or reweight), re-evaluate the classifier, and recompute the efficiency. The difference from nominal is your systematic contribution from that source.

The common thread

All five mistakes share a root cause: treating the classifier as a black box that produces a cut value, rather than as a component of the measurement that must be validated end-to-end. A boosted decision tree is a powerful tool, but it does not replace the physicist's responsibility to verify that the inputs are well-modelled, the training is unbiased, and the uncertainties are honest.

The full validation sequence — from overtraining checks through systematic propagation to a production-ready classifier — is the subject of the later modules in the course. Start with the free Module 1 and the Classifier Roadmap.

Educational material. It does not replace the internal review and approval procedures of your experiment's collaboration.