Tuesday, November 10, 2026

@convequity Oct 29

Convequity on X

The Artificial Analysis (AA) Intelligence Index update is interesting, and the right reading is more nuanced than its one headline number. AA is an amateur score, aimed mostly at non-specialists. It is not the most authoritative benchmark, the most professional one, or the most accurate one, partly because of how AA sets up its own tests. On FrontierCode-AA, GPT-6 Luna (medium) is the language model that grades the answers of frontier models including GPT-6 Astra, Claude Fable and Claude Opus. AA uses that judge on FrontierCode 1.1 and 2.0. On FrontierCode 2.0, Luna scores about 25, while Claude Opus 5.5 scores about 46, GPT-6 Astra about 42 and Claude Fable 5.1 about 36. So the judge scores below each of the frontier models it grades. People still use AA because it turns "which model is better" into one number. That number is easy to quote, so labs put it in launch announcements and buyers use it to compare models quickly. Many labs intentionally overfit: they train a model on the answers and the tests from the benchmarks that made up AA's score, so it scores well on those tests without getting better at the work. About 60% or more of the benchmarks that constituted that score were public. On a public test, the questions and answers are published, so a lab can put them into training data and the model can memorize them. The published score then rises because the model has already seen that test, and that score is the number the market will quote. Labs under pressure to show a strong result overfit those public tests hard. The result is high scores on the tests and low scores in actual use. Real work brings new code, documents and problems the model has never seen, so memorized test items do not carry over. So far, $META's Muse, $GOOGL's Gemini and Moonshot's Kimi seem to be the most aggressive. Meta and Google face the most pressure to show they are back on top. They gamed the tests heavily, and their actual performance remains weak. A performance claim is now easy to check in actual use. Right after Opus 5.5 shipped, Anthropic's token use jumped and many people moved back to Claude Code, while Muse Code and Google Antigravity remain weak. Token spend and the coding product people actually open show what customers do with their own work, and a model trained on a public test cannot produce that shift. If Muse and Gemini had the lead their test scores implied, use would have moved toward Muse Code and Antigravity, and it did not. Benchmarking remains one of everyone's biggest problems. Weak benchmarks saturate. A weak test has a ceiling, and once the top models all reach the maximum score, the ranking stops separating them, even if one model is still better at the work. Strong benchmarks with public questions get gamed, because labs can train on the published questions. Closed benchmarks keep the questions hidden, which makes them harder to train on. The cost is weak transparency and reproducibility, and they are harder to treat as independent. A buyer cannot see the items, rerun the test from those questions, or check that the lab and the scorer stayed apart, so the result is less trustworthy. AA's new v5 index is 60% private or held-out, meaning the questions in that 60% are kept out of public view, and this is the first time a majority of the index is private. That private share is a different figure from the 60% or more of the earlier tests that were public. Even with that cost, I think AA made two right moves. It now uses more closed benchmarks, and it dropped its own poorly designed special benchmarks: GDPval-AA v2.1, SciCode and AA-LCR v1.1. Apart from the single headline number, the leaders on each test are still informative: Real code-repository development (FrontierCode): Claude Opus 5.5, 46% on FrontierCode 2.0 at max effort, the same result as above Terminal engineering tasks (Terminal-Bench): Kimi K3.1 Pro, 67% on Terminal-Bench 4.0 Terminal scientific research (Terminal-Bench-Science): GPT-6 Astra, 63% at max effort Complex physics research questions (CritPt): GPT-6 Astra, 72% on the private hard subset at high effort Enterprise knowledge work (AA-Briefcase): Claude Sonnet 5.5, 1,812 Elo Enterprise SaaS automation (AutomationBench): Gemini 4 Argon, 78% Complex PDF understanding (GDP.pdf): Muse Wonder Full, 33% all-pass HLE-Diamond, the refined hard-exam set: Claude Opus 5.5, 68% at max effort Knowledge reliability (AA-Omniscience): Claude Opus 5.5, 48 on AA-Omniscience v1.1