Transparency inAgentic AI

A Survey of Interpretability, Explainability, and Governance

  • WhatCognitive objects
  • WhyAssurance objectives
  • HowMechanisms
  • WhenLifecycle stages
  • WhoStakeholders

Agentic systems plan, use tools, and keep memory over multiple steps. Transparency has to follow the trajectory, not a single output.

Shaina Raza1 Ahmed Y. Radwan1 Sindhuja Chaduvula1 Mahshid Alinoori1 Christos Emmanouilidis2

1 Vector Institute 2 University of Groningen

Paper GitHub BibTeX

Abstract

Single-step explanations do not cover an agent.

Agentic AI systems built on large language models plan, use tools, and maintain memory over multiple steps. Their risks and responsibilities depend on an execution trajectory rather than a single output. Despite progress, work on transparency for such systems remains scattered. Most explainability and interpretability research still targets static or single-step model outputs, while Agentic AI surveys emphasize planning, tools, and memory, giving limited attention to transparency and oversight.

Literature insufficiently addresses what should be subject to transparency and recorded during an agent’s lifecycle, and how to verify those records. This survey connects interpretability, explainability, and governance from design to deployment, synthesizing methods for plans, tool interactions, memory events, and coordination signals against assurance needs such as faithfulness, auditability, compliance, robustness, and equity.

It consolidates evaluation practices and highlights gaps in trajectory-level accountability, tool-mediated provenance, and multi-agent coordination. The Minimal Explanation Packet (MEP) is proposed as a standardized outcome-time artifact that bundles key lifecycle evidence into an audit-ready record.

The transparency gap

Deployment is moving faster than the tooling that explains it.

Market projections show agentic deployment outpacing XAI tooling by about 6× by 2034. arXiv publication growth for agentic AI also outpaces XAI from 2022 to 2025. More than 90% of enterprises report near-term interest, only 2% had deployed at scale by early 2025, and 15% report having XAI infrastructure. Work that combines agentic systems with transparency, safety, or governance appears in this survey’s references only from 2024. Banking, healthcare, and government face the strongest transparency requirements.

  • 6× Projected lead of the agentic AI market over XAI tooling by 2034.
  • 90%+ Enterprises reporting near-term interest in agentic AI.
  • 2% Had deployed agentic systems at scale by early 2025.
  • 15% Report having XAI infrastructure in place.
Projected agentic AI and XAI market size from 2024 to 2034, with a six-times gap in 2034.
(a) Market growth projection.
Compound annual growth for agentic AI and XAI markets and for arXiv publications.
(b) Growth rate asymmetry, markets and arXiv, 2022–2025.
Enterprise interest, piloting, and at-scale deployment of agentic AI compared with XAI infrastructure.
(c) Enterprise interest, piloting, and deployment.
Sector adoption of agentic AI compared with transparency requirements.
(d) Sector adoption versus transparency requirements.
References cited in this survey by year and source type from 2017 to 2026.
(e) Cited references by source type. 2026* is January–October.
References cited in this survey by theme, showing agentic transparency work beginning in 2024.
(f) Cited references by theme. 2026* is January–October.

Figure 1. The transparency gap across market size, research growth, enterprise adoption, sector requirements, and the references cited in this survey.

Literature positioning

Two literatures, and the space between them.

Prior surveys fall into two strands: XAI and interpretability surveys focused on static or single-step models, and Agentic AI surveys that cover architectures, planning, tools, and memory while treating transparency as secondary.

This survey differs in three ways: a five-axis taxonomy tied to lifecycle stages and stakeholder views, a shared trace those views derive from, and the Minimal Explanation Packet as a per-decision, outcome-time artifact.

This survey at the intersection of XAI and agentic AI research.
Figure 2. This survey occupies the intersection of XAI and Agentic AI research.

Five-axis taxonomy

What, why, how, when, and who.

We organize transparency along five complementary dimensions.

What

Cognitive objects

What should be transparent?

  • Intent (Goals)
  • Beliefs (World Model)
  • Plans (Action Sequences)
  • Memory/State
  • Tool I/O
  • Policies
  • Outcomes

Why

Assurance objectives

Why is transparency required?

  • Faithfulness
  • Usefulness
  • Compliance
  • Robustness
  • Equity
  • Auditability

How

Mechanisms

How is transparency achieved?

  • Intrinsic
  • Post-hoc
  • Mechanistic
  • Operational
  • Social

When

Temporal stages

When is transparency required?

  • Design-time
  • Process-time
  • Outcome-time
  • Post-deployment

Who

Stakeholders

Who requires transparency?

  • End Users
  • Developers
  • Auditors
  • Regulators
  • Third Parties
Five-axis taxonomy organizing transparency across what, why, how, when, and who.
Figure 3. The Five-Axis Taxonomy organizing transparency across WHAT, WHY, HOW, WHEN, and WHO dimensions.

Minimal Explanation Packet

One record, several transparency objectives.

The MEP is a standardized record supporting multiple transparency objectives simultaneously.

Minimal Explanation Packet in operation: lifecycle from design-time specifications to release, release-gate logic, and an example packet.
Figure 4. The Minimal Explanation Packet (MEP) in operation. (A) Lifecycle from design-time specifications to a release outcome. (B) Release gate: a decision is released only if its MEP is complete, signed, hash-consistent with the traces, and within the agreement and parity thresholds; any failure triggers flagging, human review, and rollback. (C) Example MEP.

Key findings

Coverage is uneven across the agent.

Interpretability timeline

Timeline of key milestones in AI interpretability, from classical neural networks to agentic systems.
Figure 5. Timeline of key milestones in AI interpretability, from classical neural networks to agentic systems.

Evaluation landscape

Sunburst of nine core evaluation areas for agentic AI and representative metrics in each area.
Figure 6. Agent performance evaluation landscape. Inner ring: nine core evaluation areas derived from the agentic AI literature. Outer ring: representative metrics for each area.

Contributions

01

Comprehensive synthesis

Consolidating interpretability, XAI, and agentic systems monitoring across single-agent, multi-agent, and multimodal settings.

02

Five-axis taxonomy

Systematic organization along WHAT, WHY, HOW, WHEN, and WHO dimensions for comparable analysis.

03

Gap analysis

Mapping methods to governance frameworks and identifying critical gaps in trajectory-level accountability, tool-mediated provenance, and multi-agent coordination.

Example

A tool-using agent, with the substrate made visible.

Tool-using agent flow from plan to tool use to synthesis, with dashed provenance into the Minimal Explanation Packet.
Figure 7. Tool-using agent. Solid arrows show the execution flow (plan → tool use → synthesis). Dashed arrows show the provenance that each stage contributes to the outcome-time Minimal Explanation Packet (MEP), which supports replay, verification, and stakeholder-specific views.

Citation

Cite this survey.

Raza, Radwan, Chaduvula, Alinoori, and Emmanouilidis, 2026.

@article{raza2026transparency,
  title={Transparency in Agentic AI: A Survey of Interpretability, Explainability, and Governance},
  author={Raza, Shaina and Radwan, Ahmed Y and Chaduvula, Sindhuja and Alinoori, Mahshid and Emmanouilidis, Christos},
  year={2026},
  url={https://papers.ssrn.com/sol3/papers.cfm?abstract_id=7030899}
}

Acknowledgements

Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring Vector Institute. This research was funded by the European Union’s Horizon Europe research and innovation programme under the AIXPERT project (Grant Agreement No. 101214389), which aims to develop an agentic, multi-layered, GenAI-powered framework for creating explainable, accountable, and transparent AI systems.

Vector Institute AIXPERT