Profile Likelihood Fits With an ML Discriminant

September 21, 2026 · 5 min read

Getting a classifier score into a publication-quality profile likelihood fit trips up a surprising number of analysts — not because the idea is complicated, but because the pipeline has several places where things can go quietly wrong. This guide walks through each step so you can build a robust ML discriminant profile likelihood workflow for your own analysis.

Why Use a Classifier Score in a Template Fit?

A well-trained classifier compresses many correlated observables into a single discriminant that tracks signal-versus-background separation. Feeding that score into a profile likelihood machine learning framework lets the statistical machinery extract signal strength while simultaneously constraining nuisance parameters — the same way you would with any other powerful observable. The key insight is that the score itself becomes your observable; everything downstream is standard template fitting.

If you are new to how classifiers are trained for HEP in the first place, the full course covers the end-to-end pipeline from feature engineering through model selection before you ever touch the statistical layer.

Step 1: Choose Your Score Binning

The classifier outputs a continuous value, typically in [0, 1] for a binary classifier. You need to bin it into a histogram that becomes the template.

Practical guidance

• Use finer bins in the signal-rich region (high score) and coarser bins in the background-rich region. The statistical precision of each bin must be sufficient; a bin that is essentially empty in your signal Monte Carlo will destabilize the fit.
• Avoid bins whose MC statistical uncertainty dominates over the expected physics uncertainty you care about. If a bin is so narrow that a single MC event determines its yield, merge it with its neighbor.
• Fix the binning before you look at data in the signal region. Choosing bins after seeing data introduces a subtle but real bias.

Step 2: Build the Templates

For each process — signal, each background component — produce a histogram of the classifier score template in the relevant region using simulation (or a data-driven method for backgrounds).

Systematic variations

For every source of uncertainty, produce an "up" and "down" variation of each template by shifting the parameter by its prior width and re-histogramming. Common sources include:

• Jet energy scale and resolution — re-run the event selection with shifted four-vectors.
• b-tagging efficiency — reweight events by the per-event scale factor ratio.
• Generator and parton shower modeling — use an alternative MC sample.
• PDF uncertainties — use the PDF eigenvector or Hessian replicas.

The up/down histograms give the interpolation anchor points that the likelihood will use when it floats the nuisance parameters.

Step 3: Set Up the Profile Likelihood

The likelihood is a product over bins. In each bin i, the expected count is a function of the signal strength μ and all nuisance parameters θ:

ν_i(μ, θ) = μ · s_i(θ) + b_i(θ)

Each s_i and b_i is interpolated from your nominal and varied templates as a function of the corresponding θ. Standard frameworks (HistFactory-style tools are the norm in ATLAS and CMS, and similar structures exist in other experiments) handle this interpolation automatically once you provide the histogram inputs.

The profile likelihood ratio is then:

λ(μ) = L(μ, θ̂̂(μ)) / L(μ̂, θ̂)

where the denominator is the global maximum and the numerator is maximized over θ at fixed μ. The test statistic −2 ln λ is what you scan to extract confidence intervals.

Step 4: Propagate MC Statistical Uncertainty

Each template bin has a finite number of MC events. Ignoring this inflates your sensitivity artificially. The standard remedy is to add one nuisance parameter per bin per process (the "Barlow–Beeston lite" approach or its variants), constrained by a Poisson or gamma auxiliary term. Most fitting frameworks support this natively — enable it.

Step 5: Validate Before Unblinding

Run the fit on:

1. Asimov data (replace observed data with the expected yields) to verify the fit converges and pulls are near zero.
2. Signal-injection tests — inject a known signal strength and recover it within the expected uncertainty band.
3. Control regions — regions kinematically similar to your signal region but dominated by background. The post-fit agreement in control regions is your main check that systematics are behaving correctly.

You can also look at how much each systematic nuisance parameter is constrained or pulled. Large pulls on a dominant systematic are a warning sign that the template shapes are not correctly describing the data. Understanding which observables drive your classifier is helpful here — resources on interpretability and feature importance can guide that investigation.

For a deeper grounding in classifier training strategies that pair well with this fitting approach, the complete course walks through the full chain from raw data to statistical inference.

Step 6: Check for Sculpting

A poorly trained classifier can reshape the background score distribution in a way that mimics signal. Validate that your background template in the signal region is consistent with a data-driven estimate from a sideband, particularly if you trained on sidebands to avoid bias.

The single most important habit is to treat the binned classifier score exactly like any other observable: freeze your binning, propagate every systematic to a template variation, account for MC statistics, and validate in control regions before looking at signal — and the profile likelihood will do the rest correctly.