Confabulated execution

Confabulated Execution

Definition

Confabulated execution is a Tier 3+ operational risk concept in the Agentic Tier framework where an AI agent progresses through a multi-step workflow using hallucinated intermediate states. Instead of basing each step on verified facts from the systems it interacts with, the agent invents internal facts mid-workflow and then treats them as ground truth. These fabricated states drive subsequent decisions, actions, and outputs.

A key characteristic is that the agent’s confidence presentation (e.g., assertive language, lack of uncertainty markers, or misleading confidence scores) conceals the underlying confabulation from human reviewers until late in the process—often only when the final or downstream outputs are examined.

Why It Matters

Confabulated execution is more dangerous than a single-step hallucination because:

• It compounds error across multiple workflow steps.
• It can trigger real-world actions (e.g., system changes, notifications, financial operations) based on fabricated internal states.
• It can evade detection at standard human-review checkpoints if those checkpoints rely on the agent’s own summaries or confidence framing.

Mitigation and Controls

Mitigation focuses on making each workflow step auditable and testable:

1. Per-step audit trail – Evidence Register (GOV-13)
   • Maintain a structured log of each step, including:
     • Inputs received from external systems.
     • Evidence or data sources referenced.
     • Intermediate conclusions or state transitions.
   • Enables post-hoc investigation to distinguish between:
     • Valid state transitions grounded in system data.
     • Confabulated states introduced by the model.
2. Confidence-score thresholding (where vendor supports it)
   • Use model- or vendor-provided confidence scores to:
     • Block or route to human review when confidence is below a defined threshold.
     • Require additional evidence or cross-checks for low-confidence intermediate states.
   • Apply thresholds not only to final outputs but also to intermediate state assertions.
3. AI Evaluation Harness (GOV-09)
   • Use the AI Evaluation Harness to test agents under multi-step workflow scenarios, including:
     • Synthetic and real-world workflows with branching logic.
     • Adversarial or ambiguous conditions that encourage state invention.
   • Measure:
     • Frequency and severity of confabulated intermediate states.
     • Whether confabulations are detectable via logs, confidence signals, or external reconciliation.
   • Feed results into model selection, prompt design, and guardrail tuning.

Framework Positioning

• Agentic Tier: Confabulated execution is a Tier 3+ risk, typically emerging when agents:
  • Maintain internal state across multiple steps.
  • Autonomously call tools, APIs, or sub-agents.
  • Are allowed to chain actions without continuous human supervision.
• Risk Governance: Governed primarily through GOV-09 (AI Evaluation Harness) and GOV-13 (Evidence Register), with additional alignment to broader operational risk controls (e.g., change management, incident response, and monitoring).

Usage Example (Conceptual)

• Step 1: Agent queries an inventory system but misinterprets a timeout as a valid response.
• Step 2: It confabulates that stock is available and logs an internal state of inventory_status = "in_stock".
• Step 3: Based on this fabricated state, it approves customer orders and triggers downstream fulfillment actions.
• Human reviewers see a confident summary (“Inventory confirmed; orders approved”) and only discover the issue when customers report non-delivery.

In this scenario, the risk is not just a mistaken answer but a fabricated internal state driving a chain of operational decisions, which is the hallmark of confabulated execution.