Significance Calculation in an ML-Based Analysis

Getting the significance calculation right is one of the last steps in a particle physics analysis — and one of the easiest places to introduce a subtle error when a machine-learned classifier is involved. If you have already trained a classifier and are now asking "what is my discovery significance?", this article walks you through exactly what changes and what you need to watch.
Why a Classifier Changes the Significance Calculation
In a cut-based analysis the background estimate is usually straightforward: count events in a sideband, extrapolate with a transfer factor, or fit a smooth function. The logic is transparent and the systematics are well understood.
When you add a classifier, you are applying a nonlinear function of many observables to separate signal from background. That is powerful, but it couples several things that were previously independent:
- The shape of the background score distribution depends on the input features and the training procedure.
- Any systematic variation that shifts an input feature also shifts the classifier output.
- If the classifier was trained on simulation, any mismodelling in the training sample can bias the score distribution in data.
The core point is this: a significance calculation in an ML-based analysis requires a background estimate that accounts for all of these effects, not just the statistical uncertainty on the event count.
Step-by-Step: Calculating Discovery Significance with a Classifier
1. Define Your Signal Region in Score Space
After training, you apply the classifier to data and simulation and obtain a score (often a number between 0 and 1). Your signal region is a cut or a bin range on that score. The significance calculation machine learning practitioners care about — the expected number of signal events over the square root of background, or a full profile-likelihood ratio — is performed in this region.
Choose the cut point before looking at the observed count in the signal region. Use expected significance on simulation, or a blinded data sample, to set the threshold. This avoids sculpting your result.
2. Estimate the Background Independently of the Classifier
This is the step most likely to go wrong. You need a background estimate that does not depend on the classifier being perfectly calibrated on data. Good options include:
- ABCD method: Define four regions using the classifier score and a second, nearly independent variable. The background in the signal region is estimated from the other three.
- Sideband extrapolation: Train the classifier, then fit a smooth functional form to the score distribution in a background-rich control region and extrapolate into the signal region.
- Data-driven reweighting: Reweight simulation to match data in a control region before applying the score cut.
Each method has its own validity assumptions. The key is that your background estimate should be derived from data as much as possible, with simulation used to correct for acceptance and efficiency rather than to predict the absolute background yield.
3. Propagate Systematic Uncertainties Through the Classifier
This is the step that distinguishes a discovery significance ML analysis from a naive one. For each systematic source — jet energy scale, b-tagging efficiency, PDF uncertainty — you need to evaluate how the classifier score distribution changes when that systematic is varied.
Practically, you apply the varied simulation sample to the already-trained classifier (do not retrain) and observe the shift in the score distribution. That shift becomes a nuisance parameter in your likelihood. If the classifier is strongly sensitive to a particular systematic, your expected significance will degrade relative to the nominal estimate, which is the correct behavior.
Understanding how to build these likelihoods efficiently is covered in depth in the full HEP ML course.
4. Check for Score-Sculpting on the Background
A classifier trained to separate signal from background can inadvertently sculpt a bump or dip into the background score distribution, even in the absence of any signal. This is especially dangerous in bump-hunt analyses where you are looking for a localized excess.
Test this by running the classifier on a pure background sample (data sideband or simulation) and checking that the score distribution is smooth and matches your background model. Techniques such as decorrelation (DisCo, adversarial training, or mass-decorrelated classifiers) explicitly penalize the classifier for learning the invariant mass or another protected variable. You can read more about practical approaches to feature decorrelation on the blog.
5. Quote the Significance Correctly
Once you have a background estimate with uncertainties and a signal model, compute the profile-likelihood ratio test statistic. The asymptotic approximation (Wilks' theorem) is often adequate in the high-statistics limit but can fail when background counts are very small. In that regime, toys or exact counting statistics are safer.
Report both the observed and expected significance, and quote the dominant systematic contributions to the expected sensitivity. This lets reviewers understand whether the analysis is statistically or systematically limited.
A Note on Validation
Before unblinding, validate the entire chain — training, scoring, background estimation, and likelihood construction — on a signal-injected pseudo-dataset. If the injected signal is recovered at the correct significance, your pipeline is consistent. If you want a structured walkthrough of building and validating this kind of pipeline, the complete HEP ML course covers it end to end.
The one-line takeaway: a classifier improves sensitivity, but it transfers responsibility for the background estimate from a simple sideband count to a careful, data-driven procedure with fully propagated systematics — treat that step with the same rigor you would any other part of the analysis.
Want to go deeper?
Machine Learning for High Energy Physics: The Complete Course takes you from first principles to a defensible result in 6 structured modules. $97, 30-day guarantee.
See the course →Not ready yet? Grab Module 1 free →