Template Fitting on a Classifier Output Distribution

October 12, 2026 · 5 min read

If you have ever trained a classifier to separate signal from background and then drawn a cut at some threshold on the output score, you have left information on the table. The score distribution itself — its shape across all events — carries more than a pass/fail answer. Template fitting on a classifier output lets you recover that information and extract a signal yield with the kind of statistical power that a hard cut simply cannot match.

Why Shape Fitting Beats a Hard Cut

When you cut on a classifier score you collapse the continuous distribution into a single number: how many events survive. Everything about how those events are distributed above the threshold is discarded.

A shape fit treats the classifier score as just another observable — like an invariant mass or a transverse momentum — and fits a mixture of signal and background templates to the full distribution. The discriminating power is additive across the entire score axis, not concentrated at a single operating point. Events that look "signal-like but not overwhelmingly so" still contribute to the measurement; they are not silently thrown away.

In physics language, a hard cut is like integrating your signal peak and using only the event count. A classifier score template fit is like fitting the mass lineshape: the shape itself constrains both the yield and the systematic uncertainties simultaneously.

Building Your Templates

Step 1 — Choose your score variable

Any well-calibrated classifier output works: a neural-network probability, a boosted-decision-tree response, or any other continuous discriminant. The requirement is that the score distribution differs in shape between signal and background — which, if training went well, it will. A shape fit on an ML discriminant lives or dies on how well the score is calibrated, so verify that your classifier output is actually a well-behaved probability or monotone discriminant before proceeding.

Step 2 — Build simulation templates

Use your Monte Carlo samples to produce two normalised histograms of the score: one for signal, one for background (or one per background component if you have multiple processes). These are your templates. Bin them with enough granularity to capture shape differences but coarsely enough to avoid template instability from limited MC statistics.

Validate the templates in control regions before touching the signal region. The background template in particular must describe data in a region where signal is negligible — the same care you would give any shape-based fit.

Step 3 — Set up the fit

The observed score distribution in data is modelled as a linear combination of the templates:

N_obs(s) = μ_sig · T_sig(s) + μ_bkg · T_bkg(s)

where μ_sig and μ_bkg are the free parameters (the yields), and T_sig, T_bkg are the normalised template histograms. Maximise a likelihood — Poisson bin-by-bin is standard — over those yields. Your analysis framework of choice (ROOT/RooFit, pyhf, cabianca, zfit) can handle this directly.

Step 4 — Add systematic uncertainties as nuisance parameters

Each source of systematic uncertainty that distorts the template shapes — jet energy scale, b-tagging efficiency, PDF variations — becomes a nuisance parameter. Let the templates morph continuously with the nuisance parameter (vertical or horizontal interpolation schemes both exist in standard toolkits). The fit then constrains nuisances using the shape information in the data, in exactly the same way a mass fit constrains them. This is where the real gain over a cut-and-count analysis becomes concrete: the shape provides additional handles on systematics that a single bin cannot.

Practical Pitfalls

Spurious correlations. If you train on simulation and fit in data, any mismodelling in the classifier input features propagates into the score distribution. Validate aggressively in sidebands.

Sculpting. Decorrelate your classifier from nuisance-sensitive variables (or use a mass-decorrelated classifier) if you later want to fit a second observable like invariant mass simultaneously. A sculpted score distribution produces biased yields.

Overtraining inflates shape differences. An overtrained classifier will show artificially sharp peaks in the score on training events; your templates will not describe independent data. Use proper train/validation/test splits, or better, cross-validation. Overtraining here is exactly the particle-physics analogue of fitting a chi-square to statistical fluctuations in your training sample. If you want a structured walkthrough of these training best practices, the complete HEP ML course covers model validation in depth.

Low-statistics templates. Smooth or parametrise templates when MC statistics are sparse, and propagate the template statistical uncertainty as an additional nuisance. Tools like the Barlow–Beeston method handle this automatically.

Connecting to the Broader Analysis Strategy

A classifier score template fit pairs naturally with simultaneous fits across multiple channels or with control-region constraints on background normalisations. Once you start treating the ML output as a proper observable, you can extend to multi-dimensional fits — score versus invariant mass, for example — unlocking further sensitivity. Explore how classifiers fit into a full analysis workflow in the HEP ML program.

Takeaway: Treat your classifier score as a physics observable and fit its shape — you will extract more signal yield, better constrain systematics, and leave nothing behind.