The problem
What needs solving
Make AI models say “I don’t know”, or how sure they are, instead of guessing, when they answer without looking anything up. Confident wrong answers should become rare without the models becoming useless.
Why it matters
AI states made-up answers in the same confident tone as true ones. On AA-Omniscience, a public test of hard factual questions, the best models get about two in three right yet answer 20% to 24% wrongly rather than declining. By mid-October 2026 a public database listed 2,199 court decisions dealing with AI-invented material, 871 of them involving lawyers. One reason is how models are scored: most tests give no credit for “I don’t know”, so guessing pays, as OpenAI and Georgia Tech researchers argued in 2025.
What counts as progress
(a) On AA-Omniscience, re-run by Artificial Analysis on fresh questions, or a successor, a general-purpose model gets at least 65% of questions right and at most 2% wrong, declining the rest. (b) On Humanity’s Last Exam, 2,500 expert questions where the AI also states its confidence, or its successor, a model that gets half or more right has confidence within 5 points of its real accuracy. (c) Solved when (a) and (b) hold for two model generations from two developers, and an outside audit of 1,000 citations in long answers written without lookups finds fewer than 1 in 1,000 invented.
Stater & version history
Stated by UNSOLVED editors
18 Oct 2026 — Stated by the UNSOLVED editors for the founding registry, from the research of October 2026.