Explaining how AI-driven decisions are made remains a challenge for many enterprises, even as AI technology has exceeded productivity expectations.
A recent FICO survey of 1,004 senior data, analytics, and AI leaders found that 85.1% say AI has met or exceeded ROI expectations. However, only 5.2% said they were very confident explaining AI-driven decisions to regulators and customers.
Rachael Hadaway, VP of AI product at FICO, said companies are currently getting ROI primarily from manual workload automation and other low-risk use cases. They haven’t yet tapped into higher-stakes decision-making: “I think a lot of companies are still getting ROI on just the manual workload stuff that they can easily do without getting in any sort of trouble.”
What Explaining an AI Decision Actually Requires
Scott Zoldi, chief analytics officer at FICO, said explaining how a model typically behaves is different from explaining why it reached one specific decision. He said companies need to trace what data drove decisions like credit eligibility or fraud-related transaction blocking, and where that data came from.
Knowing general model behavior doesn’t tell a regulator or customer why this person got this outcome. This traceability requirement points directly toward data lineage and governance, rather than just model architecture.
The Confidence Disconnect in the Data
The FICO survey found another mismatch between leaders’ confidence about regulatory readiness and the extent to which responsible AI standards were incorporated in practice. Of respondents, 82.8% describe their organization as in line with or ahead of anticipated regulatory requirements. Yet, responsible AI adherence ranked lowest of five operational standards measured — behind performance monitoring, security, compliance, and data quality monitoring.
Only 60.5% rated responsible AI adherence within their company a 4 or 5 out of 5. Zoldi argued that confidence may reflect how little is currently required to be explained, not actual readiness. There is a geographic caveat to the finding: 67% of respondents were based in North America, which likely shapes perceived regulatory urgency.
Fragmentation Hinders Governance at Scale
Zoldi identified technology fragmentation as a key part of the explainability problem. “I spoke to one financial organization that had 26 platforms. And those 26 platforms don’t talk to each other,” he said. Different business units often solve their own technical problems in isolation, making it harder to apply consistent standards to decisions affecting customers.
That same fragmentation extends beyond deployment platforms into the data itself. The unstructured data behind a decision — emails, case files, contracts, transaction records — often sits scattered across siloed repositories.
When that data lives in 26 disconnected systems, tracing a single decision back to its source becomes extremely difficult, and so does explaining that decision to a regulator. Solving this requires unifying the data under one governance layer that tracks lineage, sensitivity, and access consistently across every repository.
Hadaway asserted that privacy, legal, and trust teams need to be brought in during development, not after an AI system is already built. The same logic extends to the data those teams rely on. Privacy, legal, compliance, and records management all need to work from one shared, AI-ready data foundation that spans the entire data estate.
Survey respondents largely agree a single shared deployment platform could generate additional ROI (96.4%), yet only 8.2% say such a platform is fully in place.
Agentic AI Raises the Stakes
Nearly every organization surveyed has begun exploring agentic AI, but most are still in early stages: 38.2% are in early exploration, 33.3% are running pilots/POCs, and only 2.4% have reached broad deployment across multiple customer-impacting use cases.
The top barriers to agent ROI are security (40.1%), data integration and quality challenges (38.3%), and a lack of trust and predictability (5.6%). Zoldi used fraud operations as an example of how companies could delegate a workflow between different types of AI systems. A predictive model can determine whether activity appears fraudulent, while an agent handles surrounding tasks like gathering information and interacting with the customer.
Hadaway put it plainly: “It’s really easy to build an agent, and it’s really hard to manage and govern an agent.” As agents take on more of the work behind customer-facing outcomes, the explainability and traceability problem compounds across every system the agent touches.
Explainability Starts with the Data
Governing and tracing the data behind a decision is a prerequisite for explaining it. Fragmented data infrastructure and ungoverned data make confident, auditable AI decisioning harder to sustain as adoption scales. The companies deploying AI on a governed, traceable data foundation today will be the ones who are able to answer auditor questions when regulation tightens.
