The Self-Confirming Error

An agent believes A causes B. She acts to maximize B. The outcome is consistent with A causing B. She continues believing A causes B. But A does not cause B — a confounding variable C causes both A and B, and her intervention on A happened to produce outcomes that looked confirmatory by coincidence of mechanism.

Halpern, Piermont, and Vierø (arXiv:2603.09387) formalize this trap. They embed decision-making into a structural-equations framework for causality, where the agent holds probabilistic beliefs over multiple possible causal models — she doesn’t know which causal graph is true. She acts optimally given her current beliefs, observes the consequences, and updates.

The key concept is the steady state: a fixed point where the feedback from the agent’s optimal actions, given her causal beliefs, is rationalized by those beliefs. At a steady state, the agent sees nothing anomalous. Her actions produce exactly the evidence her model predicts. She has no reason to revise.

The problem: wrong causal models can have steady states. An agent who believes she is pulling a lever that controls an outcome may instead be pulling a lever that does nothing, while the outcome is determined by something she never measures. If the correlation between her lever and the outcome persists — because both are driven by the same hidden cause — she will never encounter evidence that contradicts her model. The error is stable. The wrong belief is self-confirming.

This is distinct from simple confirmation bias. The agent is Bayesian — she updates correctly given her observations. The failure is not in the updating rule but in the causal model that structures what she observes. She cannot learn her way out of a wrong causal model when her actions produce evidence consistent with it.

The through-claim: rational agents can be permanently wrong about causation, not because they ignore evidence, but because their actions generate evidence that confirms their errors. The belief shapes the action, the action shapes the evidence, and the evidence reinforces the belief. The loop closes.