LLM analytics benchmark: the best model for your analytics agent

Background and Causes of Agent Errors

When an analytics agent provides an incorrect answer, the announcement states there are three possible causes. The necessary information might be missing or incorrect in the context. Alternatively, the information may be present, but the agent failed to find it. The third cause is that the model misread the information or wrote the wrong query after finding it.

The announcement explains that testing all model releases by hand takes too much time. To address this, the creators built a repeatable benchmark to measure answer quality, consistency, and cost. This allows them to verify agent behavior before releasing a new model to users.

Testing Methodology and Benchmarks

The test suite combines thirty cases based on a public Formula 1 database from Spider two point zero with sixty business cases inspired by real internal usage. The Formula 1 benchmark runs queries and compares results with the expected output. Meanwhile, the business benchmark uses an LLM judge to check whether the generated SQL is equivalent to the reference.

The harness used by the creators helps the agent find what it needs in large business contexts by navigating as a tree of business domains. It reads tables, metrics, and joins, then plans and writes the query. The authors also ran every model without the harness using plain text search to measure its added value.

Formula 1 Benchmark Results

On the Formula 1 benchmark, most models easily saturate the test, making it less effective for fine-grained rankings while serving as a regression check. With adaptive reasoning and medium effort, Sonnet 5, Sonnet 5.5, and Fable 5.1 matched all executed results across the evaluated cases. Other models achieved slightly lower match rates on these specific test cases.

The announcement notes that differences of a few points across twenty-five cases are not significant. When different model and setting combinations achieve identical scores, cost becomes the primary deciding factor.

Business Benchmark Results

The business benchmark separates the models with scores spreading from thirty-eight percent to seventy-five percent. Sonnet 5.5 achieved seventy-five percent correct at twelve cents per question, coming in first among the tested models. Fable 5.1 scored fifty-three percent at fifty-seven cents per question as the most expensive model tested.

Because of these results, the company is switching its default model to Sonnet 5.5. The announcement notes that Sonnet 5.5 scores highest and costs less than Sonnet 5, which was the only other model within noise range of it.

Impact of the Harness and Reasoning Effort

The custom harness makes a notable difference when the context is large. On the business benchmark, the harness adds ten points to Sonnet 5.5, eleven points to GPT-6 point one Sol, and fifteen points to GPT-6 Astra. Fable 5.1 was the exception, scoring fifty-eight percent without the harness and fifty-three percent with it.