Transparency in Agentic AI

A Survey of Interpretability, Explainability, and Governance

Shaina Raza1, Ahmed Y. Radwan1, Sindhuja Chaduvula1, Mahshid Alinoori1, Christos Emmanouilidis2

1Vector Institute    2University of Groningen

Code BibTeX

Abstract

Agentic AI systems—LLM-based agents with planning, memory, and tool use—introduce transparency challenges that are poorly served by explainability methods designed for single-step predictions. This article surveys and synthesizes interpretability and explainability techniques relevant to agentic behavior across the agent lifecycle.

We organize this survey using a five-axis taxonomy that categorizes prior work by (i) cognitive objects being inspected, (ii) assurance objectives being targeted, (iii) mechanisms employed, (iv) lifecycle stages, and (v) stakeholders served.

The Transparency Gap

Global Agentic AI market projections indicate that deployment is outpacing XAI tooling by approximately 6× by 2034. The sectors driving adoption—banking, healthcare, government—face the strictest transparency requirements.

Transparency gap analysis
Figure 2: The Transparency Gap showing market growth, research asymmetry, enterprise adoption, and sectoral requirements.

Literature Positioning

Prior surveys fall into two non-overlapping groups: XAI surveys focused on static models, and Agentic AI surveys with limited transparency coverage.

This survey bridges these strands by providing unified treatment of explainability and interpretability for LLM-based agentic systems.

Survey positioning
Figure 3: This survey occupies the intersection of XAI and Agentic AI research.
Evolution timeline
Figure 4: Evolution showing the "transparency gap" period (2022–present).

Five-Axis Taxonomy

We organize transparency along five complementary dimensions:

Five-axis taxonomy
Figure 1: The Five-Axis Taxonomy organizing transparency across WHAT, WHY, HOW, WHEN, and WHO dimensions.
WHAT

Cognitive Objects

What should be transparent?

  • Intent (Goals)
  • Beliefs (World Model)
  • Plans (Action Sequences)
  • Memory/State
  • Tool I/O
  • Policies
  • Outcomes
WHY

Assurance Objectives

Why is transparency required?

  • Faithfulness
  • Usefulness
  • Compliance
  • Robustness
  • Equity
  • Auditability
HOW

Mechanisms

How is transparency achieved?

  • Intrinsic
  • Post-hoc
  • Mechanistic
  • Operational
  • Social
WHEN

Temporal Stages

When is transparency required?

  • Design-time
  • Process-time
  • Outcome-time
WHO

Stakeholders

Who requires transparency?

  • End Users
  • Developers
  • Auditors
  • Regulators
  • Third Parties

Minimal Explanation Packet (MEP)

The MEP is a standardized record supporting multiple transparency objectives simultaneously.

MEP operationalization
Figure 6: MEP lifecycle from design-time specs to outcome with integrity gates.

Key Findings

Interpretability Coverage

Interpretability coverage
Figure 9: Significant gaps remain for tool use, memory, and multi-agent interpretability.

Explainability Coverage

Explainability coverage
Figure 14: Major gaps in uncertainty communication and multi-agent attribution.

Evaluation Landscape

Evaluation landscape
Figure 13: Nine core evaluation areas for agentic AI systems.

Contributions

1

Comprehensive Synthesis

Consolidating interpretability, XAI, and agentic systems monitoring across single-agent, multi-agent, and multimodal settings.

2

Five-Axis Taxonomy

Systematic organization along WHAT, WHY, HOW, WHEN, and WHO dimensions for comparable analysis.

3

Gap Analysis

Mapping methods to governance frameworks and identifying critical research gaps.

Example: Tool-Using Agent

Tool-using agent
Figure 5: Tool-using agent execution flow with transparency substrate.

Citation

Acknowledgments

Resources used in preparing this research were provided, in part, by the Province of Ontario, the Government of Canada through CIFAR, and companies sponsoring Vector Institute. This research was funded by the European Union’s Horizon Europe research and innovation programme under the AIXPERT project (Grant Agreement No. 101214389), which aims to develop an agentic, multi-layered, GenAI-powered framework for creating explainable, accountable, and transparent AI systems.