Nuisance Parameters and ML Classifiers: What Can Go Wrong

October 9, 2026 · 5 min read

If you have ever unblinded a fit and watched your signal significance shift uncomfortably when you pulled a jet energy scale nuisance parameter, you already understand the problem intuitively. A classifier trained on your nominal simulation lives in a world without systematic variations — and when your fit starts exploring that space, the classifier can behave in ways you did not anticipate. This article walks through exactly what can go wrong and how to catch it before it matters.

Why Classifiers Are Blind to Systematics by Default

A standard binary classifier — whether it is a boosted decision tree or a neural network — learns a decision boundary from whatever training sample you hand it. If that sample is your nominal Monte Carlo, the classifier has no knowledge of the alternative universes described by your nuisance parameters. It has never seen jets shifted up or down in energy, or parton shower weights reweighted, or pile-up conditions varied.

This is the core tension with nuisance parameters machine learning approaches: the score output by the classifier is a function of the input observables, but those observables shift when systematics move. When your downstream fit varies a nuisance parameter, the classifier input distribution changes, and the classifier output distribution changes with it — in ways that were never part of training.

What Can Actually Go Wrong

Shape Variations in the Score Distribution

The most common problem is that systematic variations introduce shape distortions in the classifier output that your fit did not model. You may have carefully derived a normalization uncertainty for your background, but the score shape under the up-variation looks different from the score shape under the down-variation in a non-trivial way. If your fit only interpolates linearly between those two shapes, and the true shape response is nonlinear, you are fitting with a mismodeled systematic template.

For systematic variations ML classifier problems, the shape effect can also be asymmetric. A jet energy scale shift might push events from one score bin into a neighboring bin, creating a migration that looks like signal. Your fit may partially absorb this into the signal strength, biasing your result.

Classifier Score Correlations with Nuisance Parameters

Even subtler: the classifier score can become correlated with a nuisance parameter even when the training observable is only weakly correlated with the underlying physics quantity the nuisance describes. Neural networks in particular are good at using combinations of features that you did not explicitly consider. If the classifier has found a way to use, say, a jet multiplicity observable that also happens to track your initial-state radiation modeling uncertainty, varying that nuisance will pull the score in a systematic way — and the pull will not be captured by your nominal templates.

Template Interpolation Breaks Down

Standard histogram-based fits interpolate between the up and down systematic templates. But the classifier score is a nonlinear function of observables, so systematic variations that were smooth in observable space can become jagged or discontinuous in score space. When you hand these templates to your fitting framework, the interpolation algorithm may struggle, and the resulting likelihood is not well behaved.

How to Diagnose the Problem

Before you freeze your analysis strategy, build a simple diagnostic. For each nuisance parameter of concern:

1. Generate the classifier score distribution under the up and down variations of that nuisance, keeping everything else at nominal.
2. Overlay those distributions on the nominal. If the shape change is qualitatively different from what you expect based on physics reasoning — for example, a normalization-only uncertainty producing a score shape change — that is a flag.
3. Check the correlation between the nuisance parameter weight and the classifier output score across events. A strong Pearson or rank correlation tells you the classifier has absorbed information about the systematic axis.

If you want to go deeper on the statistical mechanics of what a well-calibrated classifier should look like, the complete HEP ML course covers likelihood-ratio estimation and its connection to classifier outputs in a way that makes these diagnostics feel natural rather than ad hoc. You can also start with the free Module 1 to get a feel for the approach.

Practical Mitigations

The cleanest solution, where it is tractable, is to train the classifier on a mixture of nominal and systematically varied samples, so the decision boundary is averaged over the nuisance space. This is sometimes called adversarial training or decorrelation, depending on the implementation. The idea is to make the nuisance parameter classifier HEP relationship explicitly flat by construction.

A lighter-weight alternative is to derive your systematic templates from the varied samples directly — that is, run the up and down variations through the trained classifier and use the resulting score histograms as your templates — rather than trying to analytically propagate the variation. This does not make the classifier robust to systematics, but it does give your fit honest information about how the score responds.

Finally, consider whether the classifier output is the right summary statistic for a systematic-dominated region. Sometimes a cut-based selection with well-understood systematics outperforms a classifier whose systematic response you cannot validate.

The single most important habit is to never treat a trained classifier as a static function that lives outside your systematic model — the score is an observable like any other, and every observable responds to nuisance parameters.

References

Albertsson, K., et al. (2018). Machine Learning in High Energy Physics Community White Paper. Journal of Physics: Conference Series, 1085, 022008. arXiv:1807.02876.