From Features to Actions: Explainability in Traditional and Agentic AI Systems

Over the last decade, explainable AI has primarily focused on interpreting individual model predictions, producing post-hoc explanations that relate inputs to outputs under a fixed decision structure. Recent advances in large language models (LLMs) have enabled agentic AI systems whose behaviour unfolds over multi-step trajectories, where success and failure are determined by sequences of decisions rather than a single output. While effective for static predictions, it remains unclear how existing explanation approaches translate to agentic settings where behaviour emerges over time. In this work, we bridge the gap between static and agentic explainability by comparing attribution-based explanations with trace-based diagnostics across both settings. We contrast attribution methods used in static classification tasks with trace-based diagnostics applied to agentic benchmarks (TAU-bench Airline and AssistantBench). Our results show that attribution methods produce stable feature rankings in static settings (Spearman ρ = 0.86) but fail to diagnose execution-level failures in agentic trajectories. By contrast, trace-based evaluation consistently localizes behaviour breakdowns and reveals that state-tracking inconsistency is 2.7× more prevalent in failed runs and reduces success probability by 49%. These findings motivate a shift toward trajectory-level explainability for evaluating and diagnosing autonomous agent behaviour.