Anthropic Study Finds Experts Followed Claude Code's Coin-Flip Leans on 86% of Judgment Calls

Professional data analysts working with Claude Code chose whichever option their agent had been secretly told to favor on 86% of open analytical decisions, according to a randomized trial Anthropic's Alignment team published on October 27, 2026. The direction of each agent's preference was set by a coin flip, so the result isolates something the industry's "human in the loop" model takes for granted: whether the person who signs off on agent-built work is actually the one making the calls that matter.

The more consequential finding sits in the accompanying paper rather than the blog post. When the same model ran the task with no assigned preference at all, it produced an identical five-decision analysis plan in 112 of 123 runs, or 91%. Put the two numbers together and the risk is not one analyst being nudged. It is many independent analysts, all deferring to the same model, converging on the same defaults while each believes the conclusion is their own. The authors make a version of this point themselves, warning that apparent consensus could end up reflecting a model's priors rather than independent judgment.

The study, titled "Who's Aligned to Whom? How AI Agents Shape Experts' Research Judgment Calls," was led by Amaç Herdağdelen of the Anthropic Fellows Program and Anthropic researchers Hailey Joren and Alex Tamkin. Its pre-analysis plan was registered on the Open Science Framework, although the authors disclose that data collection began before registration, when 17 participants had already started the task and four of them had dropped out.

How the Coin-Flip Experiment Isolated Agent Influence

Anthropic recruited 61 analysts from a pre-screened professional pool. Each had passed a causal inference knowledge check, used Python or R regularly, and had hands-on experience with observational causal analysis, which means estimating cause and effect from real-world records rather than controlled experiments. Eleven chose not to use the agent, leaving 50 in the main analysis. Of those, 65% held a master's degree or doctorate or had six or more years of experience.

The task was a fictional consulting brief. A cooperative called Greenfields wanted to know whether a paid advisory program, which about 40% of members had joined, raised farm productivity enough to justify subsidizing a full rollout. The board would expand only if the gain cleared roughly 3% to 4%, but it never defined how productivity should be measured.

That ambiguity was deliberate. Analysts faced four forks, each with two defensible answers:

• Outcome metric: crop yield per hectare, or yield per unit of operating cost.

• Comparison method: a difference-in-differences design, which compares how participating farms changed before and after joining against how other farms changed over the same period, or a direct comparison of yield levels with statistical adjustment for differences between farms.

• Scope: the effect on farms that joined, or the effect across the whole membership.

• Weighting: every farm counted equally, or larger producers weighted more heavily.

A fifth decision, the final recommendation, asked whether to roll the program out or end it, with a one-year extension also allowed.

These forks are not cosmetic. In one session the paper reproduces, a difference-in-differences specification estimated a 4.6% gain, while a levels-based adjustment on the same data produced a 1.6% decline. One choice moved the estimate from above the board's threshold to below zero, and the analyst went on to recommend ending the program.

Hidden System Prompts Gave Each Agent an Honest but One-Sided Prior

The mechanism was a hidden system prompt, the instruction layer that shapes an agent's behavior before the user types anything. Each analyst's agent received one of 32 lean configurations, covering every combination of the two options across the five decisions, drawn from balanced randomized blocks. The prompt told the agent to treat each assigned lean as its own view, to raise it when the relevant decision came up, and to make the strongest case for it that the data allowed.

The prompt also set hard limits. The agent had to change its mind if the evidence turned decisively against it, and it could never misrepresent results, hide data, or make an argument the numbers did not support. In other words, this was not a test of deceptive AI. It tested what happens when an honest, fluent assistant argues one side well and the other side barely at all.

Participants worked in a browser-based version of the VS Code editor with Claude Code running Claude Opus 4.8 at High effort. Median completion time was 65 minutes.

Measuring what analysts actually chose was itself an AI-assisted pipeline, which says something about where research practice already stands. An LLM judge read each submitted report and, where needed, the analysis code. When the judge's reading disagreed with an analyst's own survey answer, the authors adjudicated; of 41 such disagreements, 37 were resolved in the judge's favor. For the transcript analysis, an author hand-coded 15 sessions and Claude Fable 5 applied the same scheme to every session, with the two agreeing on 93% of the 75 decisions both coded. Serious methodological errors were flagged by two other models, GPT-5.6-Sol and Gemini 3.1 Pro, and an author reviewed each flagged finding without knowing which group the analyst belonged to.

Analysts Matched the Agent on 86% of Decisions and Rarely Credited It

Agreement with the agent's lean ran 92% on the outcome metric, 82% on comparison method, 81% on scope and 90% on weighting, or 86% combined. With no influence, the expected figure would be about 50%.

The registered primary measure is stricter. It compares how often analysts chose a given option when the agent favored it against how often they chose it when the agent favored the alternative. That pooled "flip rate" was 0.72, with a 95% confidence interval of 0.60 to 0.83, across 195 decision trials. Put plainly, the coin flip shifted the probability of picking a given option by roughly 72 percentage points. On the outcome metric, not one analyst whose agent leaned toward yield per operating cost chose yield per hectare.

The final recommendation moved less. Analysts whose agent leaned toward rollout recommended it 54% of the time, compared with 36% when the agent leaned the other way. Under the registered primary estimator, that 0.17 shift was not statistically significant (one-sided p = 0.069). The authors argue the effect was weaker because the recommendation largely followed from whether the estimate cleared 3% to 4%. Yet the estimate was already shaped upstream. An exploratory model using only the four coin-flip leans correctly predicted 82% of recommendations among analysts who did not propose an extension, against a 61% baseline from always guessing the most common answer.

The persuasion extended to analysts who already had a view. In 38 cases an analyst stated a preference before the agent voiced its lean. When the agent then disagreed, analysts switched 60% of the time; when it agreed, 6% switched. The gap of about 54 points carries a wide confidence interval of 24 to 82 points, and the authors caution that people who volunteer opinions may differ systematically from those who do not.

The attribution data is the most uncomfortable. In the post-task survey, completed without access to the agent transcript, 77% of analysts repeated the agent's argument in at least one explanation, sometimes nearly word for word. Only 15% credited the agent anywhere. Some 52% said the final recommendation was more their own than the agent's, yet that group's choices shifted toward the agent's lean by about 65 points, within the margin of error of the 79-point shift among those who credited the agent. Analysts who followed the agent on all four decisions rated their confidence 71 out of 100, versus 73 for everyone else. Six of 48 respondents guessed the study concerned AI influence; none suspected the agent's preferences had been planted.

Expertise offered no protection. Of 10 tested traits, including years of experience, causal inference quiz scores, education, trust in AI and "need for cognition," none predicted deference after correcting for multiple comparisons. Age was nominally significant before correction, with older analysts following the lean more often.

This Is Conformity to the Model, Not Model Sycophancy

Most public concern about chatbots runs in the opposite direction: sycophancy, where a model tells users what they want to hear. This trial documents the inverse, closer to what human factors researchers call automation bias, a tendency to over-rely on automated decision aids that affects expert users as well as novices.

The paper's control experiments show the stock agent is not simply agreeable either. When a scripted analyst declared a choice that contradicted the unprompted model's baseline and asked only for code, stock Opus 4.8 raised the other option in 66% of 150 judged declarations and actively recommended or advocated it in 41%. That matters because it means a default model also carries opinions and voices them, just more weakly than the amplified agents in the main trial.

And those default opinions are strikingly consistent. Across 123 unprompted runs, the stock agent landed on the same profile 91% of the time: yield per operating cost, difference-in-differences, effect on participating farms, per-farm weighting, and a rollout recommendation. A real analyst population using an unmodified agent would therefore not face a random lean. Every one of them would face roughly the same lean.

That is the study's largest implication, and it reaches beyond individual autonomy. Science leans on independent replication: when separate teams reach the same answer, confidence rises because the judgments behind them are assumed to be separate. The well-known "many analysts" experiment, in which 29 research teams given one soccer dataset reached widely varying conclusions about whether referees give more red cards to darker-skinned players, showed how much legitimate analytical choices can spread results. If those teams all delegated to agents built on one model, their divergence could narrow, not because the question was resolved but because one model's priors were applied everywhere. Related research has found that generative AI can raise individual creative output while making the resulting work more similar across people.

Read more: Anthropic's automated AI researchers close safety gaps

Human Sign-Off in AI Safety Research Is the Pressure Point

The authors single out alignment research, and the context explains why. In August, Anthropic reported that Claude-based automated researchers found training fixes for 10 categories of alignment failure, including sycophancy and reward hacking, in smaller student models, and that Claude outscored 28 human safety researchers who had up to eight hours to devise methods. Anthropic cautioned at the time that the humans could not iterate on their submissions, so it treated the result less as a head-to-head contest and more as evidence for a workflow in which Claude proposes methods that humans refine. The case for that kind of automation rests partly on human reviewers acting as an independent check. This trial suggests that a reviewer's approval of an agent-run analysis may largely ratify the agent's own preferences on precisely the decisions where no answer key exists.

That also explains why the problem resists standard quality control. Error checks and comparison against a reference answer presume a correct answer exists. On judgment calls, either option produces work that looks sound, and asking experts afterward is unreliable because most did not notice the influence.

It fits a pattern in Anthropic's recent research on how AI assistance reshapes expert work. A January 2026 randomized trial with 52 mostly junior software engineers found those who learned an unfamiliar Python library with AI help scored 50% on a follow-up quiz, against 67% for those who coded by hand, with the weakest scores among participants who relied most heavily on AI to generate or debug code.

The Agents Did Improve Accuracy, but the Comparison Group Was Self-Selected

None of this amounts to evidence that analysts should work without agents. Among the 11 participants who declined the agent, six made methodological errors serious enough to invalidate their results; among the 50 who used it, three did. The no-agent group also took longer, with a median of 90 minutes against 65.

The authors decline to draw a causal conclusion, and the demographics show why. Only four of the 11 non-users held a master's degree or doctorate, versus 24 of 46 agent users, and six of 11 had two years of analysis experience or less, compared with 15 of 48. The people who opted out were simply less experienced.

The main trial carries its own limits, which Anthropic lists itself. The agents' leans were deliberately strengthened and voiced, so the 86% figure describes an agent that holds an opinion and states it, not average real-world behavior. The study covered one roughly hour-long task, one domain and one model. Participants were causal inference specialists but new to farming, and in real work they could interview stakeholders or question the board, which might narrow the options and weaken the agent's pull. The study also recorded which option analysts picked but not how strongly they held that preference, so some deference may reflect analysts who simply had no firm view.

Agent Design and Institutional Sign-Off Rules Are the Next Test

The authors outline three directions. The first is measuring deference in everyday work, where randomizing an agent's preferences is impractical; they point to a recent preprint on measuring AI reliance that estimates how a person would have done a task without AI as a starting point. The second is agent design: arguing the strongest case for each option, or asking for the expert's view before offering one. The tradeoff is real, because a single analysis can contain many unresolved choices and much of an agent's value comes from not forcing users through each of them. Deciding which forks deserve a human's attention is itself a judgment call. Tone matters too, since research on how language models express uncertainty links that expression to how much users rely on their answers.

The third is institutional. Organizations that treat an expert's signature as proof of independent review will need to decide which decisions require a person's own informed judgment, give people time to form it, and keep training the skills those decisions depend on.

For teams deploying coding and analysis agents now, the practical lesson is narrower than the headlines suggest but sharper. Code review, test suites and benchmark checks catch errors. They do not catch an unexamined default on a question with two reasonable answers. Until agent vendors publish what their models prefer on common analytical forks and how forcefully they argue for it, the most reliable safeguard is procedural: record the human's choice on each consequential fork before the agent weighs in, and log who decided. The next milestone to watch is whether any lab, Anthropic included, begins disclosing those default priors the way it now discloses benchmark scores.