Claude Opus 4.6 accuracy on BridgeBench hallucination test drops from 83% to 68%

TL;DR Highlight

Reports have emerged indicating a 15%p decrease in accuracy on the BridgeBench hallucination benchmark for the Claude Opus 4.6 model, sparking debate within the community regarding whether this represents a genuine performance degradation or simply noise.

Who Should Read

Backend/AI developers currently using the Anthropic Claude API in production and sensitive to changes in model quality, or developers interested in the reliability of LLM benchmarks.

Core Mechanics

BridgeBench is a benchmark for measuring the level of hallucination (the phenomenon where a model generates factually incorrect content as if it were true) in LLMs. Tests on Claude Opus 4.6 reported a decrease in accuracy from 83% to 68%, approximately a 15%p drop.
The results were published by the BridgeMind AI team (@bridgemindai) on X (formerly Twitter), but the original tweet is inaccessible without JavaScript, making it difficult to verify the details.
A 15%p difference is a relatively large margin and difficult to dismiss as mere noise, especially if the benchmark is designed to be tested over multiple iterations.
However, some methodological questions have been raised. The sample size and number of repetitions are not explicitly stated in the available information, raising the possibility that the results are based on a single run.
LLMs are inherently non-deterministic (they can produce different outputs even with the same input), so it is difficult to conclude that model performance has actually deteriorated based on a single run.

Evidence

One comment pointed out the lack of publicly available sample size and number of runs, stating, 'It seems like they only ran the entire test suite once.' The commenter argued that, due to the non-deterministic nature of the model, this is unlikely to be evidence of actual performance degradation.
A counter-argument stated, '15% is a huge gap.' The commenter claimed that if the benchmark is designed to be tested thoroughly over multiple iterations, this difference is significant, and also expressed frustration that Anthropic is restricting access to its top-tier models.
Some users expressed emotional dissatisfaction, stating, 'I want unrestricted access to the actual best model Anthropic uses, even if it costs more.' This reflects a long-standing distrust of model performance within the community.
Unrelated to the discussion, a spam comment was posted promoting the author's Substack article, claiming that 'Computational Semiotics has been empirically proven.'

How to Apply

If you are using the Claude API in production, it is good practice to build your own test set and run regular regression tests before and after model updates. Relying solely on external benchmark results can cause you to miss performance changes relevant to your actual service.
When interpreting benchmark results, be sure to check the methodological details such as sample size, number of repetitions, and temperature setting. If metadata is unclear, as in this case, it is difficult to judge the reliability of the results.
Considering the non-determinism of LLMs, important evaluations should be repeated at least dozens or hundreds of times and the average value should be used. Comparing performance between models or versions based on a single run can lead to incorrect conclusions.

Terminology

HallucinationThe phenomenon where an LLM confidently generates content that is not factual, as if it were true. For example, inventing a non-existent paper title.

BridgeBenchA benchmark test suite designed to measure the level of hallucination in LLMs, operated by BridgeMind AI.

비결정론적(Nondeterministic)A characteristic where different outputs can be produced even with the same input. LLMs are inherently non-deterministic due to parameters such as the temperature.

BenchmarkA test set used to measure and compare model performance in a standardized way. It serves as an exam paper.

Regression TestA test to verify that existing functionality that worked well before still works after software or model updates.