Transparency inAgentic AI

A Survey of Interpretability, Explainability, and Governance

  • WhatCognitive objects
  • WhyAssurance objectives
  • HowMechanisms
  • WhenLifecycle stages
  • WhoStakeholders

Agentic systems plan, remember, and use tools. Transparency has to follow that lifecycle, not a single prediction.

Shaina Raza1 Ahmed Y. Radwan1 Sindhuja Chaduvula1 Mahshid Alinoori1 Christos Emmanouilidis2

1 Vector Institute 2 University of Groningen

Paper Code BibTeX

Abstract

Single-step explanations do not cover an agent.

Agentic AI systems—LLM-based agents with planning, memory, and tool use—introduce transparency challenges that are poorly served by explainability methods designed for single-step predictions. This article surveys and synthesizes interpretability and explainability techniques relevant to agentic behavior across the agent lifecycle.

We organize this survey using a five-axis taxonomy that categorizes prior work by (i) cognitive objects being inspected, (ii) assurance objectives being targeted, (iii) mechanisms employed, (iv) lifecycle stages, and (v) stakeholders served.

The transparency gap

Deployment is moving faster than the tooling that explains it.

Market projections show agentic deployment outpacing XAI tooling by about 6× by 2034. arXiv publication growth for agentic AI also outpaces XAI from 2022 to 2025. More than 90% of enterprises report interest on a three-year horizon, 23% are piloting, only 2% had deployed at scale by early 2025, and 15% report having XAI infrastructure. Work that combines agentic systems with transparency, safety, or governance appears in this survey’s references only from 2024. Banking, healthcare, and government face the strongest transparency requirements.

6× Projected lead of the agentic AI market over XAI tooling by 2034.
Projected agentic AI and XAI market size from 2024 to 2034, with a six-times gap in 2034.
(a) Market growth projection.
Compound annual growth for agentic AI and XAI markets and for arXiv publications.
(b) Growth rate asymmetry, markets and arXiv, 2022–2025.
Enterprise interest, piloting, and at-scale deployment of agentic AI compared with XAI infrastructure.
(c) Enterprise interest, piloting, and deployment.
Sector adoption of agentic AI compared with transparency requirements.
(d) Sector adoption versus transparency requirements.
References cited in this survey by year and source type from 2017 to 2026.
(e) Cited references by source type. 2026* is January–October.
References cited in this survey by theme, showing agentic transparency work beginning in 2024.
(f) Cited references by theme. 2026* is January–October.

Figure 2. The transparency gap across market size, research growth, enterprise adoption, sector requirements, and the references cited in this survey.

Literature positioning

Two literatures, and the space between them.

Prior surveys fall into two strands: XAI and interpretability surveys focused on static or single-step models, and Agentic AI surveys that cover architectures, planning, tools, and memory while treating transparency as secondary.

This survey differs in three ways: a five-axis taxonomy tied to lifecycle stages and stakeholder views, a shared trace those views derive from, and the Minimal Explanation Packet as a per-decision, outcome-time artifact.

This survey at the intersection of XAI and agentic AI research.
Figure 3. This survey occupies the intersection of XAI and Agentic AI research.
Evolution timeline showing the transparency gap from 2022 to the present.
Figure 4. Evolution showing the transparency gap period (2022–present).

Five-axis taxonomy

What, why, how, when, and who.

We organize transparency along five complementary dimensions.

What

Cognitive objects

What should be transparent?

  • Intent (Goals)
  • Beliefs (World Model)
  • Plans (Action Sequences)
  • Memory/State
  • Tool I/O
  • Policies
  • Outcomes

Why

Assurance objectives

Why is transparency required?

  • Faithfulness
  • Usefulness
  • Compliance
  • Robustness
  • Equity
  • Auditability

How

Mechanisms

How is transparency achieved?

  • Intrinsic
  • Post-hoc
  • Mechanistic
  • Operational
  • Social

When

Temporal stages

When is transparency required?

  • Design-time
  • Process-time
  • Outcome-time
  • Post-deployment

Who

Stakeholders

Who requires transparency?

  • End Users
  • Developers
  • Auditors
  • Regulators
  • Third Parties
Five-axis taxonomy organizing transparency across what, why, how, when, and who.
Figure 1. The Five-Axis Taxonomy organizing transparency across WHAT, WHY, HOW, WHEN, and WHO dimensions.

Minimal Explanation Packet

One record, several transparency objectives.

The MEP is a standardized record supporting multiple transparency objectives simultaneously.

MEP lifecycle from design-time specifications to outcome, with integrity gates.
Figure 6. MEP lifecycle from design-time specs to outcome with integrity gates.

Key findings

Coverage is uneven across the agent.

Interpretability coverage

Interpretability coverage across agent capabilities.
Figure 9. Significant gaps remain for tool use, memory, and multi-agent interpretability.

Explainability coverage

Explainability coverage across communication and attribution.
Figure 14. Major gaps in uncertainty communication and multi-agent attribution.

Evaluation landscape

Nine core evaluation areas for agentic AI systems.
Figure 13. Nine core evaluation areas for agentic AI systems.

Contributions

01

Comprehensive synthesis

Consolidating interpretability, XAI, and agentic systems monitoring across single-agent, multi-agent, and multimodal settings.

02

Five-axis taxonomy

Systematic organization along WHAT, WHY, HOW, WHEN, and WHO dimensions for comparable analysis.

03

Gap analysis

Mapping methods to governance frameworks and identifying critical research gaps.

Example

A tool-using agent, with the substrate made visible.

Tool-using agent execution flow with a transparency substrate.
Figure 5. Tool-using agent execution flow with transparency substrate.

Citation

Cite this survey.

Raza, Radwan, Chaduvula, Alinoori, and Emmanouilidis, 2026.

@article{raza2026transparency,
  title={Transparency in Agentic AI: A Survey of Interpretability, Explainability, and Governance},
  author={Raza, Shaina and Radwan, Ahmed Y and Chaduvula, Sindhuja and Alinoori, Mahshid and Emmanouilidis, Christos},
  year={2026}
}

Acknowledgements

Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring Vector Institute. This research was funded by the European Union’s Horizon Europe research and innovation programme under the AIXPERT project (Grant Agreement No. 101214389), which aims to develop an agentic, multi-layered, GenAI-powered framework for creating explainable, accountable, and transparent AI systems.

Vector Institute AIXPERT