By: Marco Virgolin (Unlayer AI) & Giovanni Cinà (Amsterdam UMC)
TLDR: Explainable AI (XAI) provides additional information about machine learning models. This information is inherently imperfect and typically reflects correlations, not causation. Naively interpreting XAI outputs as truthful or using them to infer new causal knowledge can be misleading or even harmful. Conversely, avoiding XAI altogether risks overlooking model failures and spurious patterns. We advise to use XAI, but carefully: by involving interdisciplinary experts and as a diagnostic tool to critically assess whether a model’s behavior aligns with pre-established domain knowledge.
Premise.
Here, by explainable AI (XAI) we mean the broad range of different techniques meant to give us a deeper understanding of how a trained black-box machine learning model (be it a random forest, a convolutional neural network, or an autoregressive transformer) comes to its predictions. In other words, feature attribution methods like LIME or SHAP, counterfactual explanation algorithms, and mechanistic interpretability approaches, all fit within our definition. Conversely, methods to produce models simple-enough to be immediately interpretable (such as optimal decision tree or equation discovery aka symbolic regression), are out of scope here – although some of the arguments we’ll make apply also to those methods.
Furthermore, while we use here a broad definition of XAI, we are certainly not implying that all XAI techniques are equally a good choice. Some techniques have been criticized and may not be fit for purpose, but this is a discussion for another post; here we focus on the quest of XAI as a research field and we assume an XAI technique can extract some reliable information about an AI model.
Limitations of XAI and counter-arguments.
XAI is an important field tasked with the goal of helping us understand how very complex models, typically called “black box models”, work. For example, Selvaraju et al. 2019 show how Grad-CAM, a technique to highlight pixels that contribute the most to vision network’s prediction, enables to spot gender-relevant features when a network mis-predicts a woman doctor for a nurse.
XAI methods have several important limitations. Recently, Cinà et al. published a position paper titled “Why we do need explainable AI for healthcare” (https://doi.org/10.1186/s41512-025-00209-4 (S'ouvre dans une nouvelle fenêtre), open access), in which they examine arguments against the use of XAI and provide corresponding counterarguments. Although the paper focuses on healthcare, its insights can be generalized to other industries and societal sectors. Across the various criticisms and responses discussed, the recurring theme that emerges is that XAI should be applied with care and supported by appropriate expertise.
A focus on causal overinterpretation.
Another argument against XAI, not listed in the paper by Cinà et al., is brought forward by van Rojen et al.’s “In humble defence of unexplainable black box prediction models in healthcare” (https://doi.org/10.1016/j.jclinepi.2025.112013 (S'ouvre dans une nouvelle fenêtre), open access). Again, this paper focuses on healthcare, but the argument can be generalized. May the authors forgive us for compressing it to what we believe to be the main message, which we quote from the paper itself:
“Prediction models and, by extension, the XAI techniques used to explain them, are not suited to answer such questions”
“such questions” refers to causal questions, such as “What if the patient had lower BMI? Would that lower cardiac risk?”. These questions are problematic because their answers could lead to incorrect interventions, e.g. “we must put the patient on a diet”. Prediction models use correlated factors to make predictions, ignoring whether these factors are truly causal.
Let us explain the problem with causal overinterpretation with an example. In the mid 90s, Cooper et al. trained several models to predict risk of death due to pneumonia. Neural nets achieved a remarkable accuracy. However, to get an intelligible representation of what these nets may be doing, a rule-based model was also trained – you can think of this as an early XAI approach: you train an interpretable model hoping that it will broadly reflect the patterns learned and used by the black box model. The rule-based model revealed that a controversial finding: the feature “the patient has asthma” was contributing to a lower risk of death by pneumonia. This was not a fluke: Caruana et al. reproduced this result with a generalized additive model. How could patients with asthma possibly be better off than those without asthma?
Turns out, the clinical practice was to send patients with asthma showing signs of pneumonia to the intensive care unit (ICU). These patients received much more medical attention and, indeed, ultimately died less than non-asthma patients. The predictive models were not wrong in correlating asthma to a lower risk of death! But we’d certainly be wrong in interpreting that asthma reduces this risk and concluding that patients with asthma require less attention than those without it – the opposite of what we should be doing!
Van Rojen et al. name the tendency of healthcare personnel to attribute a causal interpretation to predictive models -or, for black box ones, their XAI explanations- “causal overinterpretation”. They argue that it might be better to keep the black box shut, aka not use XAI, to prevent people from the risk of overinterpretation. Furthermore, they say that trying to uncover the underlying way in which a model works may in fact not be so important after all, because also for medicines and medical devices the mechanisms of actions are often opaque: ultimately, credibility and trust in medicine come from extensive validation (e.g. by means of carefully planned and executed randomized control trials).
The counter-argument.
While causal overinterpretation can certainly be a problem, purposedly looking the other way can also be problematic. When Caruana et al. spotted that also their model had asthma reducing pneumonia risk, they changed the parameters of their model to enforce that asthma increases risk. Had they not looked, their model would be problematic if deployed in hospitals with a different standard of care.
Therefore, in our view the crux of the matter falls back to the fact that:
You need interdisciplinary experts, who are knowledgeable about the technical limitations of the tools at hand (be it predictive models or XAI methods), and of the reality of the context around which the data is collected and prepared.
The goal of these experts using XAI techniques should not be to derive new knowledge about the world from the model, quite the opposite: to critically debug the model, checking that the model reflects the knowledge we already have about the world.
Choosing not to use XAI means to purposedly look the other way, deferring potential problems to a future stage. Furthermore, it is often much cheaper and faster to find and fix a problem early on – or simply, fail ahead of time. Indeed, a common saying in pharma, where only ~10% of new drugs make it to the market, is “fail fast, fail cheap”. Given the massive costs of the later phases of randomized control trials (easily hundreds of millions, see Sertkaya et al. 2024), pharma and medical device companies have great interest in trying to understand how their treatment works in as much detail as possible. This does not exclude that extensive validation is still important and necessary; our view is that one thing does not exclude the other.
References
Carriero et al. Explainable AI in healthcare: to explain, to predict, or to describe? Diagnostic and Prognostic Research. 2025
Caruana et al. Intelligible models for healthcare: Predicting pneumonia risk and hospital 30-day readmission. Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining. 2015
Cinà et al. Why we do need explainable ai for healthcare. Diagnostic and Prognostic Research. 2025
Cooper et al. An evaluation of machine-learning methods for predicting pneumonia mortality. Artificial Intelligence in Medicine. 1997
van Rojen et al. In humble defence of unexplainable black box prediction models in healthcare. Journal of Clinical Epidemiology. 2025
Selvaraju et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. IEEE International Conference on Computer Vision. 2019
Sertkaya et al. Costs of Drug Development and Research and Development Intensity in the US, 2000-2018. JAMA Network Open. 2024
Date
21/07/2026