Gini Impurity vs Information Gain: Which Split Criterion Should You Use?

· The ML for HEP Team · 3 min read

Every decision tree — including every tree inside a boosted decision tree ensemble — must decide how to split each node. It considers every variable and every possible threshold, and picks the split that best separates signal from background. But "best" requires a scoring function. The two standard choices are Gini impurity and information gain (entropy-based).

If you have ever wondered which one to use, the honest answer is: it almost never matters. But it is worth understanding what they measure and why they converge.

Gini impurity

For a node with fraction p of signal and (1-p) of background, Gini impurity is:

Gini(p) = 2 * p * (1 - p)

This is zero when the node is pure (all signal or all background) and maximal at p = 0.5 (perfect mixture). A split is scored by the weighted average of the children's Gini values; lower is better.

Entropy and information gain

Entropy measures the same concept — node impurity — using the information-theoretic definition:

H(p) = -p * log2(p) - (1 - p) * log2(1 - p)

Information gain is the reduction in entropy from parent to children. Higher gain means a better split.

How they compare

Both functions are concave, symmetric around p = 0.5, and zero at the endpoints. Plot them side by side:

import numpy as np
import matplotlib.pyplot as plt

p = np.linspace(0.001, 0.999, 500)
gini = 2 * p * (1 - p)
entropy = -p * np.log2(p) - (1 - p) * np.log2(1 - p)

fig, ax = plt.subplots(figsize=(5, 3.5))
ax.plot(p, gini, label="Gini impurity", lw=2)
ax.plot(p, entropy, label="Entropy", lw=2, ls="--")
ax.set_xlabel("Signal fraction p")
ax.set_ylabel("Impurity")
ax.legend()
ax.set_title("Gini vs entropy")
plt.tight_layout()
plt.savefig("gini_vs_entropy.png", dpi=120)

The two curves have nearly the same shape. Gini peaks at 0.5; entropy peaks at 1.0 (in bits). But because the tree algorithm only cares about the relative ranking of candidate splits — which split produces the largest impurity reduction — the absolute scale does not matter. Two functions with the same shape rank the same splits in the same order.

When do they disagree?

The functions are not identical, so they can in principle rank two candidate splits differently. This happens when the splits produce children with very different sizes and the signal fractions land in the narrow region where the curvature of Gini and entropy diverge most (around p near 0 or 1).

In practice, this near-tie case is rare, and even when it occurs, the difference is a single split deep inside a tree that will be combined with hundreds of others in a boosted ensemble. Empirical comparisons across many domains consistently find that the final classifier performance is indistinguishable between the two criteria.

Scikit-learn uses Gini by default. XGBoost and TMVA use entropy-like objectives by default. Neither choice is wrong.

What actually matters for split quality

If the choice between Gini and entropy does not matter, what does? The parameters that have a measurable effect on boosted decision tree performance are:

  • Tree depth (max_depth). Deeper trees capture more complex interactions but overfit more easily. Typical values in HEP analyses: 2--5.

  • Number of estimators (n_estimators). More trees reduce bias but increase the risk of overtraining. Always validate with a train-test overlay.

  • Learning rate. A smaller learning rate requires more trees but regularises the ensemble. Values of 0.01--0.1 are common.

  • Minimum samples per leaf (min_samples_leaf). Prevents the tree from creating tiny leaves that fit noise.

These knobs move the ROC curve. The split criterion, in practice, does not.

The bottom line

Use whichever criterion your framework defaults to. Spend your time on tree depth, learning rate, overtraining diagnostics, and input variable selection. Those are the choices that determine whether your boosted decision tree classifier survives analysis review.

For the full treatment of decision tree mechanics, boosting, and the validation chain that takes a classifier from a notebook to a publication, start with the free Module 1 and the Classifier Roadmap.

Educational material. It does not replace the internal review and approval procedures of your experiment's collaboration.