Explainability Is Not a Feature You Bolt On Afterward

A recurring conversation in my lab goes like this: a collaborator has a trained model that works well, and they’d like an explanation method added on top so clinicians will trust it. The request is reasonable. The framing is where the trouble starts.

Post-hoc explanations answer a narrower question than people think

Methods like SHAP, LIME, and saliency maps answer: given this model’s actual decision boundary, which inputs influenced this particular output? They do not answer: is this model reasoning about the problem the way a domain expert would? Those are different questions, and post-hoc methods can make a model look more reasonable than it is, because the explanation is generated by approximating the model’s local behavior — not by inspecting whether the model’s underlying representations are meaningful.

In our GNN debugging work, we’ve repeatedly found models that pass every standard attribution sanity check while still relying on shortcut features that happen to correlate with the label in the training distribution. The explanation method wasn’t wrong about what influenced the output. It just couldn’t tell us the causal story it was interpreting was superficial.

What “explainable by design” looks like instead

The alternative isn’t necessarily switching to simpler models — that trade-off is real but often overstated. It’s building interpretability constraints into the architecture and training objective from the start:

• Structured intermediate representations that correspond to concepts a domain expert can independently verify, rather than opaque latent vectors.
• Attention or routing mechanisms that are trained to be meaningful, not just visualized after the fact and hoped to be meaningful.
• Evaluation protocols that test explanation faithfulness — does perturbing the features the explanation says matter actually change the prediction the way the explanation implies? — not just plausibility to a human reader.

Why this matters beyond research papers

The stakes of this distinction are highest exactly where explainability is most demanded: clinical decision support, credit decisions, criminal justice risk tools. In each of these domains, a plausible-looking but unfaithful explanation is arguably worse than no explanation, because it manufactures a false sense of scrutability. If we want AI systems that domain experts can genuinely interrogate, interpretability has to shape what the model learns to represent — not just how we describe it afterward.