Abstract
Shortly after its release, GPT‑6 Astra successfully solved five challenging Erdos‑related mathematical problems. These achievements are documented in the FrontierMath Erdos (FME) benchmark paper jointly published by Epoch AI and the University of Manchester. In this benchmark test consisting of 68 unsolved mathematical conjectures from human history, five top‑tier large‑model systems took part in the evaluation. GPT‑6 Astra became the only participant to achieve non‑zero scoring performance, reaching a 3 % success rate. The other four competitors — GPT‑5.6 Sol, GPT‑5.5, Claude Fable 5.1 and Claude Fable 5 — obtained zero verified valid submissions. This article unpacks the background and design philosophy of the FME benchmark, detailed test statistics, proof mechanisms, solved conjecture cases, existing limitations, and the broader implications for AI‑driven mathematical research. When deploying multiple reasoning‑focused LLM services for mathematical workloads in production, development teams may leverage an API gateway such as 4sapi to unify request routing, authentication and observability across heterogeneous model endpoints.
1. Background: Academic Criticism that Spurred the FME Benchmark
In recent years, AI‑assisted mathematical research has attracted widespread industry attention. In June 2026, OpenAI disclosed that AI had resolved the Unit Distance Conjecture. One month later, researchers using Claude Fable 5 reported progress on the 3‑dimensional Jacobian conjecture. OpenAI later published further results covering ten open‑standing problems across mathematics, theoretical physics and computer science, which included multiple Erdos‑style conjectures.
Nevertheless, Terence Tao raised pointed criticism during his presentation at the 2026 International Congress of Mathematicians (ICM). He argued many of these AI‑generated results were not obtained under controlled scientific conditions. Tao summarized five major concerns: selective reporting of successful cases while hiding failure distributions, undisclosed compute consumption, opaque levels of human guidance, possible contamination of training datasets containing answer hints, and lack of systematic cross‑model comparative evaluation.
FrontierMath Erdos (FME) is designed as a direct response to these criticisms. Its core design principles address each identified flaw. First, it uses genuinely unsolved conjectures to eliminate training‑set contamination risks. Since no complete human solution exists, models cannot retrieve pre‑written answers from training corpora. Second, all proofs must be formally constructed within the Lean proof‑assistant framework. Lean acts as impartial arbiter: formal compilation success counts as valid proof, while compilation failure means the argument is invalid. No subjective interpretation is allowed. Every model runs under identical experimental constraints. Each trial carries a fixed budget of 300 US‑dollars, with a maximum runtime of 72 hours per single attempt.
2. Comparison Between Prior Math Benchmarks and FME Improvements
Before FME, two major benchmarks targeted open mathematical problems: HorizonMath and FrontierMath: Open Problems (FM‑OP). Both operated under a “generate‑then‑verify” workflow. They could only handle statements that were already verifiable, for instance propositions about graphs or polynomials. Most open‑ended research‑grade mathematical questions could not fit within such constraints. Verification pipelines were also incomplete. HorizonMath authors acknowledged that some “provable” tasks might in fact be mathematically unsolvable. FM‑OP removed roughly 11 %‑40 % of test items for this exact reason.
FME makes key improvements through formalization. Any conjecture that can be encoded into Lean becomes eligible for benchmarking, regardless of whether the final outcome is proof or disproof. Even falsified hypotheses count as valid outputs. The testing environment is carefully sandboxed. Each test job runs inside two isolated Docker containers without mutual network access. The Agent container executes AI workflows with limited shell permissions. The separate Comparator container holds a clean Lean toolchain solely responsible for validation. AI‑submitted Lean source code is recompiled from scratch inside the Comparator. This architecture defends against multiple potential cheating vectors, including environment tampering, malicious code compilation, mis‑proving propositions, redefining dependencies and bypassing kernel checks.
3. Construction of the 68 Erdos Conjecture Test Set
Paul Erdos remains one of the most prolific mathematicians of the 20th century. His contributions span number theory and combinatorics, and he proposed hundreds of distinctive, far‑reaching open problems. The FME test suite draws its 68 conjectures from the erdosproblems.com website curated by Thomas F. Bloom, co‑author of the FME paper and researcher at the University of Manchester.
Only problems meeting strict selection criteria were included. A candidate problem must be sufficiently difficult and intellectually meaningful. Whether proven or disproven, a complete solution would be publishable in peer‑reviewed academic venues and spark follow‑up research within mathematics communities. Most selected items have attracted decades‑long attention from professional mathematicians.
Bloom previously cautioned against casually measuring AI progress using Erdos‑sourced problems, because difficulty and significance vary wildly across entries. Many earlier AI‑solved Erdos problems turned out to be relatively straightforward variations of known theorems. The FME set deliberately avoids such trivial cases. Problems like Unit Distance Conjecture (entry 90), Item 146 and Item 183 represent high‑stakes open challenges. Successfully handling these signals a new capability threshold for AI mathematical research.
Conjectures are selected to be mutually independent. Solving one problem should not logically unlock answers to others, so each solved case constitutes an independent scientific contribution. Erdos occasionally attached monetary reward bounties to his questions. FME includes both bounty‑bearing and non‑bounty problems, while filtering out high‑reward questions lacking sufficient intellectual appeal.
For technical implementation, 51 out of 68 conjectures were formalized in Lean4 directly referencing the Google DeepMind Conjectures repository. The remaining 17 items were formalized by paper authors. The full testbed ships with Lean4 toolchains, Mathlib, SageMath computer‑algebra libraries, Python mathematical utilities, plus an offline LaTeX corpus of 476 000 arXiv mathematics papers used as reference material.
4. Detailed Test Outcomes for GPT‑6 Astra
Within standard benchmark runs against the full 68‑conjecture pool, GPT‑6 Astra produced two formally verified results. Problem‑74 was disproved with total cost of 218 USD and runtime of 15 hours. Problem‑126 received a complete valid proof at the cost of 24 USD over 16 hours. All remaining 66 test items hit the 300‑dollar per‑attempt budget cap without yielding Lean‑verified outputs. The other four evaluated large‑model systems achieved zero valid formal proofs across all 68 tasks.
Beyond standardized benchmark rounds, research teams carried out additional multi‑round experimental campaigns on GPT‑6 Astra. Budget constraints and agent configuration parameters were adjusted. These supplementary trials are not counted as official benchmark scores, yet they still deliver valuable research insights. Across extended testing, GPT‑6 Astra produced three further verified solutions, bringing its cumulative total to five Erdos‑problem resolutions.
4.1 Problem 1: Disproving a dissociation‑set conjecture
First posed by Erdos in 1931 when he was 18 years old, this was described by Erdos as “my first serious problem”. The statement concerns dissociation sets within integer range {1,…,N}. GPT‑6 Astra constructed counterexamples demonstrating the conjecture fails for any constant ε ≤ 2⁻². Four separate attempts were executed; two succeeded, costing 405 USD and 1384 USD respectively. Bloom commented that the lattice‑oriented formal rewrite generated by the model offered a fresh perspective for human mathematicians.
4.2 Problem 74: Disproving Hajnal‑Szemerédi conjecture
This 1982 conjecture from Erdos, Hajnal and Szemerédi hypothesized that arbitrarily large chromatic number can persist even after removing O(n) edges from high‑order graphs. GPT‑6 Astra proved the opposite conclusion: bounded chromatic number emerges if edge‑removal quantity grows sufficiently fast at Θ(log log n / log log log n). Six total attempts were run; the cheapest success consumed merely 47 USD across 5 hours. Researchers observed three distinct argument strategies emerging across repeated trials.
4.3 Problem 126: Proof for Erdos‑Turán conjecture
Dating back to the first collaborative paper between Erdos and Turán in 1934, the original bound established S(A) ≫ log n. GPT‑6 Astra derived the stronger result S(A) ≫ n^(1/2). Four attempts all succeeded, yielding three separate proof variants with exponent parameters 1/8, 1/3 and 1/2 using varied methodological approaches.
4.4 Problem 548: Proof for Erdos‑Sós conjecture
This famous 1962 conjecture states that any graph with n vertices and more than ((k‑2)/2)·n edges must contain every k‑vertex tree as a subgraph. Partial results existed before, but complete formal proof remained missing. GPT‑6 Astra completed full formal verification combining ordering arguments, marking counting and inductive reasoning. One out of three attempts succeeded, costing 363 USD within a 20‑hour window.
4.5 Problem 571: Proof for Turán growth‑rate conjecture
Proposed jointly by Erdos and Simonovits, this conjecture addresses Turán exponent growth within bipartite graphs for rational α ∈ (1,2]. Only scattered partial cases were previously confirmed. GPT‑6 Astra delivered a complete formal proof for arbitrary rational α. One out of three attempts succeeded. This case represented the most resource‑intensive workload among all five solved conjectures. Bloom noted that human mathematicians still need further work to interpret the full meaning of this machine‑generated proof and its connections with existing mathematical literature.
Total experimental expenditure for all five successful proofs exceeded 220 000 USD, while official standardized benchmark testing only cost roughly 20 000 USD. Using FME’s nominal parameters of 300 USD per attempt and 3 % success probability, the expected cost for solving a single open‑hard conjecture approximates 1 million USD.
5. Limitations Documented in the FME Research Paper
The authors also point out important limitations of the benchmark. Formal‑proof translation overhead can artificially suppress measured scores. A model may discover logically correct reasoning chains but fail to translate arguments into compilable Lean code, especially when required mathematical results are not yet incorporated inside Mathlib.
FME measures only problem‑solving performance. In real‑world mathematical research, formulating meaningful new questions is often intellectually harder than proving existing statements. This capability dimension is not captured in the evaluation framework. Even with GPT‑6 Astra’s five verified solutions, 63 out of the original 68 Erdos conjectures within FME remain unresolved. Despite high‑profile progress, major barriers persist for AI mathematical research.
6. Industry Implications
The FME benchmark sets a higher bar for evaluating AI mathematical reasoning. By enforcing Lean formal verification, it eliminates subjective judging, dataset leakage risks and ambiguous “plausible‑looking but wrong” pseudo‑proof outputs. For AI practitioners, this signals that mathematical capability evaluation cannot rely merely on natural‑language demonstrations. Formal verification pipelines become essential infrastructure for validating advanced reasoning outputs.
Engineering teams building AI‑powered mathematical research platforms often orchestrate multiple model backends. Unified access control, request logging and load balancing become operational necessities. An API gateway can streamline such multi‑model deployments.
While GPT‑6 Astra’s five validated solutions mark tangible progress, the gap between narrow success on isolated conjectures and general‑purpose autonomous mathematical research remains substantial. Most historic hard open conjectures are still beyond current AI reach. Human mathematicians retain irreplaceable roles in interpreting machine outputs, identifying research directions and placing formal results within broader academic context.
International access: https://4sapi.com
Domestic access: https://4sapi.cn
