The CLs Method for Limit Setting With ML Discriminants

Setting exclusion limits is one of the most consequential steps in any new-physics search, and it can also feel like one of the most opaque. If you have spent time building a classifier to separate signal from background, you might be wondering exactly how that discriminant score feeds into the CLs procedure — and whether the standard workflow still applies. It does, with a few important things to keep in mind.
What CLs Actually Does
The CLs method is a frequentist recipe for setting upper limits that avoids the uncomfortable situation where you exclude signal hypotheses your experiment was never sensitive to in the first place. At its core, you compute a test statistic from your data, evaluate how often background-only and signal-plus-background pseudo-experiments produce values at least as extreme as what you observed, and then form a ratio of those probabilities. If that ratio falls below a threshold you report an exclusion.
The key input is a likelihood, and the likelihood requires a model of what your data distribution looks like under each hypothesis. This is where your classifier enters.
From Classifier Score to Likelihood Input
A classifier trained to separate signal from background produces a continuous output — a score for each event. Treat that score as an observable, no different from a reconstructed invariant mass or a missing-energy value. In physics language, the classifier output is just another discriminating observable.
To build a likelihood you need to know the probability density of that score for signal events and for background events. In practice this means:
Step 1 — Produce score distributions
Run your trained classifier over the full simulated signal sample and over all background samples. Histogram the output score for each process. These histograms become your templates.
Step 2 — Choose a binned or unbinned approach
Most LHC analyses use a binned likelihood over the score histogram. Pick a binning that captures the shape variation without creating bins so narrow that statistical fluctuations in your simulation dominate. Coarser bins in the signal-poor region, finer bins near the high-score end, is a reasonable starting point.
Step 3 — Assign systematic uncertainties
Every shape uncertainty you would normally assign to a mass spectrum must also be assigned to the score distribution. Rate uncertainties (luminosity, cross-section) carry over unchanged. Shape uncertainties require re-evaluating the score distribution with each systematic variation — for example, varying jet energy scale or b-tagging efficiency — and storing the resulting shifted templates. This is identical to what you would do with any other observable, but it is worth doing carefully because classifier outputs can be more sensitive to modelling differences than simple kinematic variables.
Step 4 — Build the likelihood
Pass your signal template, background templates, and systematic nuisance parameters to a standard statistical tool. The profile-likelihood test statistic is then computed exactly as it would be for any other binned histogram input. Nothing in the CLs calculation itself changes.
What Actually Changes When You Use an ML Discriminant
The mechanics of exclusion limit setting with an ML discriminant are identical to those with a traditional cut-and-count or shape-fit analysis. What changes is sensitivity. A well-trained classifier exploits correlations among many observables simultaneously, compressing them into a single score. This tends to produce a score distribution that is more separated between signal and background than any individual kinematic variable, which in turn gives a tighter expected limit.
That improvement in sensitivity is only realised if the classifier score is well-modelled in simulation. If there is a mismatch between how the score is distributed in data versus MC, your background template is wrong, your likelihood is wrong, and your limit is unreliable. Validation in control regions — ideally regions where you can compare the score distribution in data against MC without sculpting the signal region — is not optional.
There is also a subtlety around reuse of data. If you optimised the classifier on the same events you fit, you can introduce a bias. Use a held-out test set for the final template, or use cross-validation, so that the score distributions are not accidentally tuned to statistical fluctuations in your training sample. For a deeper look at how to structure this workflow end-to-end, the full HEP ML course walks through classifier training, validation, and integration with downstream statistical inference.
Practical Checklist Before Running CLs
- Score distributions are produced on statistically independent samples from training
- Systematic variations are propagated through the classifier to produce shifted templates
- Control region comparisons confirm reasonable data-MC agreement in the score
- Bins with very low expected yields are merged to avoid numerical instability in the fit
- The observed limit and expected limit (with uncertainty bands) are both reported
If you are newer to the statistical side of limit setting and want to see how classifiers fit into the broader workflow of a physics analysis, our introductory module is a good starting point before diving into template fitting.
The one-line takeaway: a classifier output is just another observable — feed its histogram into your standard CLs machinery, propagate your systematics, and validate the modelling, and the rest of the procedure is unchanged.
Want to go deeper?
Machine Learning for High Energy Physics: The Complete Course takes you from first principles to a defensible result in 6 structured modules. $97, 30-day guarantee.
See the course →Not ready yet? Grab Module 1 free →