Common Review Committee Questions About ML Analyses

October 16, 2026 · 5 min read

Getting through a collaboration review of an ML-based analysis can feel like a gauntlet, especially if your reviewers are more familiar with cut-based methods. The good news is that review committee questions about ML analyses are remarkably consistent across experiments and institutions. Reviewers are not trying to block your result — they want to understand and trust it. If you anticipate their questions and build the answers into your documentation from the start, your collaboration review of a machine learning analysis moves from adversarial to constructive.

Why Reviewers Ask What They Ask

Reviewers have a responsibility to the collaboration's published results. When they see a neural network or boosted decision tree in the analysis chain, they are often asking one underlying question in several different forms: How do I know this model is doing what you think it is doing, and not something else?

Every practical suggestion below traces back to that question.

Question 1: "What Did You Actually Train On?"

Reviewers will want to see a clear separation between training data, validation data, and the data used to extract the final result. This is the ML equivalent of blinding your signal region before unblinding.

• State your splits explicitly. Describe the exact events used for training and make clear they never enter the final statistical fit.
• Show learning curves. Plot training loss and validation loss together as a function of epoch or tree depth. If the two curves diverge, you have overtraining — the ML version of fitting statistical fluctuations in a histogram rather than the underlying distribution. Reviewers recognize this analogy immediately.
• Report performance on a held-out test set. Never quote the number from your training sample.

Question 2: "Does the Model Describe the Data?"

This is where many ML analysis reviews stall. A model that discriminates signal from background well in simulation is only useful if the simulation actually describes the data in the relevant feature space.

• Show data/MC comparisons for every input feature, both before and after any reweighting. Reviewers will ask for these even if you think the variables are standard.
• Show data/MC comparison of the classifier output itself, not just the inputs. A well-calibrated set of inputs can still produce a poorly calibrated output if correlations are not modeled correctly.
• Identify control regions where you can validate the shape of the classifier output in data. A sideband that passes all your selection except the signal requirement is ideal.

If you want a structured walkthrough of validation strategies, the complete course covers this in the context of realistic HEP workflows.

Question 3: "What Are the Systematic Uncertainties on the Model?"

This is the question that trips up the most analyses. Reviewers performing an ML analysis review in physics want to see that you have thought carefully about how systematic variations propagate through the classifier.

• Vary your MC systematics through the full pipeline. If a jet energy scale shift changes your inputs, retrain or reweight and show the effect on the output distribution.
• Assess the impact of modeling uncertainties on the discriminant shape, not just on the event yield. Shape uncertainties in the classifier output can be large even when the yield effect is small.
• Consider retraining under each systematic. For some analyses this is practical; for others you can use a parameterized or reweighting-based approach. Either way, have a documented answer.

Question 4: "How Do I Know What the Model Learned?"

Reviewers are not necessarily asking you to explain every weight in your network. They want confidence that the model is using physical information and not latching onto an artifact.

• Provide feature importance or SHAP-style rankings. Show that the variables driving the score are the ones that make physical sense.
• Check that the model is not sensitive to variables it should not use. If you excluded beam spot position from training, verify the score is flat as a function of beam spot position in data.
• Scan the output score versus known non-signal variables (run number, luminosity block, pileup) to catch unexpected correlations. Reviewers will ask about this even if they cannot name it precisely.

For more on interpretability tools that fit naturally into HEP workflows, the HEP ML full course includes hands-on exercises you can apply directly to your own analysis.

Question 5: "Could a Simpler Method Do the Same Job?"

This question is not an attack. Reviewers ask it because complexity has a cost in maintenance, reproducibility, and trust. You do not need to justify using a neural network over a BDT in principle — you need to show what it buys you in practice.

• Run a baseline comparison. Show that your method outperforms a simple cut-based selection or a shallow BDT on a metric the reviewers recognize, such as signal efficiency at fixed background rejection.
• Frame the gain in physics language. "We recover the equivalent of X additional inverse femtobarns of data" is more persuasive than a raw AUC comparison.

Build the Answers In Early

The fastest path through a collaboration review of a machine learning analysis is to treat every question above as a checklist item during analysis development, not as a fire to fight at review time. Document your training procedure, validate in data, propagate systematics, and show your work. Reviewers who can see the reasoning will trust the result.

One-line takeaway: Anticipating review committee questions about ML before you submit your analysis note turns a slow, anxious review into a productive conversation.