Integrated Gradients

What Are Integrated Gradients?

Integrated Gradients (IG) is a method in the field of interpretable machine learning that aims to explain the predictions of complex neural networks by attributing importance scores to input features. Originally proposed for deep neural networks used in computer vision, it can also be applied to other data modalities (text, structured data, etc.). Integrated Gradients seek to measure how much each feature contributes to the predicted outcome, under the assumption that gradually “introducing” the feature from a baseline (neutral) input to the actual input reveals how crucial that feature is for the final prediction.

IG is considered a method for computing feature attributions that adhere to certain axioms of fairness and consistency. Unlike simple gradient-based saliency methods, which can be noisy or fail certain interpretability axioms, Integrated Gradients are designed to be more stable and theoretically grounded.

Conceptual Foundations

Gradient-Based Explanations:
A common approach to interpretability is to look at gradients of the model’s output w.r.t. the input features. The intuition: a high gradient magnitude means small changes in that feature value could significantly alter the prediction. However, raw gradients can be noisy and may not faithfully represent each feature’s overall importance. They capture instantaneous sensitivity rather than the contribution from a baseline to the actual input.
Baselines (Reference Inputs):
Integrated Gradients introduce the concept of a baseline input, a neutral or “absence” input that does not contain any discriminative features. For an image model, a common baseline could be a completely black image or a blurred image. For text models, it might be an empty sequence or a padded input. For tabular data, it might be a vector of median or zero values. The choice of baseline is crucial because it defines what “no contribution” means.
Path Integration:
Instead of looking at gradients at just the actual input point, IG considers a path (usually a straight line in feature space) from the baseline input $x_{baseline}$to the actual input $x$. Formally, consider a function $F: \mathbb{R}^n \to \mathbb{R}$ that represents the model’s output (e.g., the logit for a chosen class). Integrated Gradients define the attribution for the $i$-th feature as:
$$
IG_i(x) = (x_i – x_{baseline,i}) \times \int_{\alpha=0}^{1} \frac{\partial F(x_{baseline} + \alpha (x – x_{baseline}))}{\partial x_i} d\alpha
$$
In simpler terms, you start at the baseline and gradually morph it into the actual input, at each step taking the gradient. Then you integrate (accumulate) these gradients along the path. This integral reflects how the model’s output changes as you “add” the feature information from baseline to the original input. Multiplying by $(x_i – x_{baseline,i})$ scales the cumulative effect by the actual difference in feature value between the input and baseline.
Axiomatic Properties:
Integrated Gradients were developed with certain axioms in mind, including:
- Sensitivity: If a feature changes and that change affects the output, the attribution for that feature should be non-zero.
- Implementation Invariance: If two models differ only by functionally irrelevant transformations (like adding a layer that always passes input through unchanged), the attributions remain the same.
- Completeness: The sum of attributions for all features equals the difference between the model’s prediction at the input and at the baseline. This ensures a form of “conservation” of importance.

How to Compute Integrated Gradients in Practice

Select a Baseline:
Choose a baseline input that represents a “no signal” state. For images, black or zero images are common. For textual inputs, maybe a sequence of padding tokens. For tabular data, a zero vector or median values could be used. The baseline should be chosen thoughtfully—some baselines might produce more meaningful attributions than others.
Create a Path from Baseline to Input:
Consider a linear interpolation between baseline $x_{baseline}$ and input $x$:
$$
x(\alpha) = x_{baseline} + \alpha (x – x_{baseline})
$$
for $\alpha \in [0,1]$
Sample Points Along the Path:
In practice, the integral $\int_0^1 \frac{\partial F(x(\alpha))}{\partial x_i} d\alpha$ is approximated using numerical methods. One common approach is Riemann summation:
- Pick a number of steps mmm (like 50, 100, or more).
- For each step $k \in \{1,\ldots,m\}$:
  $$
  \alpha_k = \frac{k}{m} \quad \text{and} \quad x(\alpha_k) = x_{baseline} + \alpha_k (x – x_{baseline})
  $$
- Compute gradients $\frac{\partial F}{\partial x_i}(x(\alpha_k))$ at each step.
Integrate the Gradients:
$$
IG_i(x) \approx (x_i – x_{baseline,i}) \times \frac{1}{m} \sum_{k=1}^{m} \frac{\partial F(x(\alpha_k))}{\partial x_i}
$$
By averaging the gradients along these points and then scaling by the input difference, we get an approximation to the integrated gradients.
Attribution Visualization:
After computing IG for each feature dimension, you have a vector of attributions. For images, reshape this vector to match the spatial dimension and produce a heatmap. For text, align attributions with tokens. For tabular data, just interpret feature attributions as numerical scores.

Advantages of Integrated Gradients

Axiomatic Foundation:
IG was proposed to satisfy certain fairness axioms, making it a theoretically well-grounded method. This sets it apart from simpler gradient-based methods that may fail these axioms.
Model-Agnostic:
While IG is often used with differentiable models like deep neural networks, the approach is conceptually flexible. As long as you can compute or approximate gradients, you can apply IG.
Improved Stability Over Raw Gradients:
Raw gradients can be noisy and focus on local sensitivity. IG aggregates gradients along a path, providing a more global perspective that often yields smoother, more intuitively meaningful attributions.
Faithfulness to the Model’s Decision:
Because IG integrates from a baseline to the input, it captures how features cumulatively influence the output, potentially offering a more faithful representation of feature importance.

Limitations and Considerations

Baseline Choice:
Choosing a good baseline is critical. A baseline that is not meaningful (like all zeros that are far from your natural input domain) can produce attributions that may not make intuitive sense. Sometimes multiple baselines or domain knowledge is required to ensure a good reference point.
Computation Cost:
IG requires multiple forward and backward passes (one per step along the interpolation path). If your model and inputs are large and complex, this can be computationally expensive. Lowering the number of steps reduces cost but may degrade the quality of approximation.
Interpretability of the Final Attribution Map:
While IG provides a theoretical foundation and a sum of attributions that matches the output difference, the final attribution map still needs interpretation. If features are highly correlated or the model is complex, the attributions may reflect interactions that are not straightforward to read off directly.
Local vs. Global Explanation:
Like many attribution methods, IG explains one prediction at a time (a local explanation). It does not provide a global understanding of the model’s behavior across all inputs, only how features influenced the model’s output for a specific input-baseline pair.
Gradient Quality:
IG relies on gradients. If gradients are poorly conditioned or the model uses certain non-standard layers, this can affect the reliability of attributions.

Best Practices

Experiment with Different Baselines:
If possible, try multiple baselines. For image tasks, some recommend using a black image and a blurred image. Compare results to see which baseline leads to more plausible attributions. Domain expertise helps in picking a meaningful baseline.
Tune the Number of Steps:
Start with a moderate number of steps (like 50) and see if results are stable. If attributions vary too much with different random seeds, increase the number of steps for a finer approximation.
Combine with Other Methods:
Use IG alongside other interpretability techniques (such as SHAP, Grad-CAM for images, or LIME) to get a more holistic picture of model behavior. If multiple methods agree on which features matter, that increases confidence in their conclusions.
Use Normalization and Thresholding:
Attribution maps can sometimes be sharpened or normalized. For images, apply a colormap to highlight the strongest attributions. For text, maybe highlight tokens above a certain attribution threshold to focus on the most influential words.
Validate Results with Domain Knowledge:
Check if the attributions align with known domain insights. For example, if a medical imaging model highlights relevant anatomical structures, that’s a good sign. If it focuses on background or irrelevant text, further investigation is needed.

Example Scenario

Imagine you have a CNN classifying images of dogs vs. cats. You pick one image of a cat and choose a baseline of a completely black image. By applying Integrated Gradients:

You interpolate between the black image and the cat image, computing gradients at each step.
After integration, you visualize attributions as a heatmap over the cat image.
The resulting map might highlight the cat’s face and ears strongly, indicating these features are crucial for the model’s “cat” prediction.
If the model incorrectly highlighted the background or a label watermark, you might realize the model relies on irrelevant artifacts and consider retraining or addressing dataset biases.

Conclusion

Integrated Gradients provide a principled, gradient-based method to attribute a model’s output prediction to its input features. By integrating gradients along a baseline-to-input path, IG attempts to satisfy desirable axioms of interpretability, producing more stable and theoretically grounded attributions than simple gradient saliency maps.

Despite requiring careful baseline selection and computational overhead, Integrated Gradients remain a widely-used technique in model interpretability. They help stakeholders understand which input features are truly driving the model’s decisions, offering greater trust, debugging capacity, and opportunities for ethical oversight of AI systems.