Feature Importance and SHAP Values for Physics Models

July 26, 2026 · 5 min read

Getting a classifier to work is one challenge. Getting your collaboration to trust it is another. If you have trained a BDT or neural network that separates signal from background better than any cut-based approach you have tried, but you cannot explain why it makes its decisions, you will spend more time defending the model than using it. Feature importance and SHAP values are the tools that let you open the box, check what the classifier learned, and communicate that clearly to reviewers and collaborators.

Why Interpretability Matters in HEP

A collaboration review is not unlike a referee report: people want to know that your result is not an artifact. A classifier that implicitly learned a detector pathology, a kinematic boundary, or a badly modeled variable will produce a result that falls apart under scrutiny. Interpretability is not just a nicety — it is a cross-check.

The good news is that the same techniques used in industry to audit models translate directly to physics. You already think in terms of observables, correlations, and systematic uncertainties. Feature importance machine learning methods map naturally onto that vocabulary.

Start with Built-in Feature Importance for BDTs

If you are using a gradient-boosted decision tree (BDT) — via XGBoost, LightGBM, or scikit-learn — you get feature importance essentially for free.

What the score measures

A BDT splits nodes by choosing the feature and threshold that maximally reduce impurity (usually Gini or entropy, analogous to minimizing variance in a fit). The built-in importance score accumulates how much each feature contributed to those splits, weighted by the number of events passing through each node.

How to read it

Plot the importances as a bar chart, sorted from largest to smallest. Ask:

• Does the ranking make physical sense? If your leading discriminator is a variable you know has poor Monte Carlo modeling, that is a red flag worth investigating.
• Are any important features correlated with known systematics? High importance on a jet energy scale-sensitive variable means your classifier will be sensitive to that uncertainty.
• Is the importance surprisingly flat? That can indicate redundant features or that the model has not converged.

To interpret BDT feature importance correctly, keep in mind that this score is not a measure of how much removing a feature would hurt performance. It is a measure of how often and how usefully the feature was used during training. Correlated features can split importance between themselves and each artificially look less important.

Move to SHAP for a Deeper Understanding

Built-in importance gives you a global ranking. SHAP values physics gives you something more powerful: a per-event, signed explanation of why a specific prediction was made.

The core idea

SHAP (SHapley Additive exPlanations) borrows from cooperative game theory. For each event, every feature is assigned a value that represents its contribution to pushing the model output away from the average prediction. Positive SHAP value means the feature pushed toward signal; negative means it pushed toward background. The sum of all SHAP values for an event equals the model's output (in log-odds space), so the decomposition is exact.

This is the interpretability equivalent of looking at a pull plot: you see not just that something is off, but which variable is responsible.

Computing SHAP values in practice

Install the shap library and pass it your trained model and a test dataset. The library handles BDTs natively and efficiently. For neural networks it uses sampling-based approximations.

import shap
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X_test)
shap.summary_plot(shap_values, X_test)

The summary plot shows each feature on the y-axis, each event as a dot, color-coded by feature value, and the x-axis as SHAP value. This single plot tells you:

• Global importance (vertical spread)
• Direction of effect (positive or negative SHAP)
• Whether the relationship is monotonic or nonlinear (color gradient pattern)

Spotting problems

If a feature you expected to be unimportant appears with large SHAP values, investigate. If the SHAP dependence plot for a variable shows a discontinuity exactly at a detector boundary, your classifier may have learned a detector artifact. These are the conversations you want to have before unblinding.

Connecting Interpretability to Systematics

Once you know which features drive the classifier, you can prioritize your systematic uncertainty studies. Features with high SHAP importance deserve careful validation in control regions. This is where interpretability directly feeds back into analysis quality, not just collaboration politics.

If you want to build this workflow from the ground up alongside other HEP-specific ML techniques, the full HEP ML course covers classifiers, calibration, and uncertainty-aware training in a structured way. The free introductory module is a good place to check whether the depth is right for where you are.

For background on how classifiers are trained and evaluated before you reach the interpretability step, the course curriculum overview walks through the full pipeline from feature engineering to deployment.

A classifier you can explain is one your collaboration can trust — and SHAP values give you the language to have that conversation at the level of individual observables.