Systematic Uncertainties and Machine Learning in HEP

July 26, 2026 · 5 min read

If you have ever replaced a cut-based selection with a BDT or neural network, you already know the uneasy feeling that follows: the model performs beautifully on nominal MC, but now you are not sure how to evaluate systematic uncertainties machine learning introduces — or inherits. This is one of the most common sticking points for physicists adopting ML, and it is worth working through carefully, because a classifier score you cannot propagate systematics through is not publishable.

Why Classifiers Do Not Escape Systematic Uncertainties

A common misconception is that a high-performing classifier somehow averages away systematics. It does not. The model learned a boundary in observable space, and any systematic that shifts that space — jet energy scale, b-tagging efficiency, lepton momentum resolution — can move events across the decision boundary. The effect can be larger or smaller than in a cut-based analysis depending on which observables the model weighted heavily.

Think of it this way: if you trained on a feature like missing transverse energy and that variable is sensitive to pile-up modeling, then uncertainty on the pile-up description becomes uncertainty on the classifier output distribution. The uncertainty BDT output propagation is not fundamentally different from propagating a systematic through a hand-tuned cut, but it is less transparent, because the boundary lives in a high-dimensional space you cannot easily inspect.

Step-by-Step: Propagating Systematics Through an ML Classifier

1. Identify Which Input Features Carry Systematic Uncertainties

Start by listing every input variable to your model and asking which experimental or theoretical systematics affect each one. Group them:

• Detector response systematics (energy scales, resolution smearing, efficiency corrections)
• Theory systematics (parton shower, PDF, renormalization/factorization scale)
• Modeling systematics (underlying event, pile-up, generator choice)

This is the same audit you would do before filling a histogram — you are just doing it for every feature simultaneously.

2. Apply Systematic Variations to Your MC and Re-evaluate the Score

For each systematic variation — up and down — rerun your MC through the frozen trained model and recompute the score distribution. Do not retrain; the model is fixed. You are asking: given a shifted detector or a shifted theory parameter, how does the score distribution change?

This is the most direct approach and is conceptually clean. The score is just another observable. You vary the inputs, you observe the output variation, and you treat that output variation as the systematic uncertainty on your score template. This maps naturally to the reweighting or shifted-sample approach you already use for cut-based selections.

3. Watch for Systematics That Affect Training, Not Just Inference

If you trained on a specific MC generator and your nominal analysis also uses that generator, then the model has implicitly learned the modeling choices baked into that sample. A mis-modeled feature could be driving discrimination you do not realize exists. This is the ML analog of fitting a chi-square to a biased template.

A useful diagnostic: retrain (or at least re-evaluate) on an alternative generator sample and compare score distributions. Large shape differences in the score distribution between generators — especially in signal-enriched regions — are a flag that generator modeling is a meaningful systematic uncertainty on your result.

4. Propagate Score Shape Uncertainties into the Fit

Once you have nominal and varied score distributions (templates), you pass them to your statistical model in the same way you would pass shifted histograms for a cut-and-count or shape analysis. Your fit then profiling over nuisance parameters constrained by control regions, just as before.

If you are using a profile likelihood fit, the score distribution becomes the discriminating variable, and each systematic is a nuisance parameter that morphs the templates. Tools like HistFactory and Combine handle this natively — you are not doing anything structurally new. ML systematics particle physics analyses are reviewed by the same standards as cut-based ones; the bookkeeping just requires extra care because the input feature list is often longer.

5. Consider Decorrelation Techniques When Needed

Sometimes a specific observable that carries a large systematic is also very powerful for discrimination. If you simply include it, you get good performance but large uncertainty. One option is to decorrelate the classifier from that observable — training with an adversarial term or mass-decorrelation technique — so the score shape becomes less sensitive to that systematic at the cost of some raw performance. This is an active area of development covered in depth in the full HEP ML course.

Practical Sanity Checks

• Compare the fractional change in signal efficiency between nominal and each systematic variation. If it is large relative to your statistical uncertainty, treat it as a leading systematic.
• Validate the score distribution in a data control region where you can compare data and MC directly. Discrepancies there are a warning sign of mis-modeling.
• Document which features drive the classifier and which systematics affect those features. Reviewers will ask.

For a broader foundation in building analysis-ready classifiers, the complete course covers training, validation, and uncertainty handling end to end.

A classifier score is an observable like any other — treat its systematic uncertainties with the same rigor you would apply to a reconstructed invariant mass, and your ML-based selection will be as defensible as any cut-based analysis you have ever published.

References

Albertsson, K., et al. (2018). Machine Learning in High Energy Physics Community White Paper. Journal of Physics: Conference Series, 1085, 022008. arXiv:1807.02876.