Back to Blog

AI Design Benchmark: Why 1931 Elo Isn't the Full Story

Daily News4475
AI Design Benchmark: Why 1931 Elo Isn't the Full Story

A stealth model called Gestalt topped a blind design leaderboard at Elo 1931, ahead of Claude Fable 5.2 and GPT-6.1 Sol — and ahead of the human reference works. The useful signal is not the ranking. It is the gap between the three metrics the same benchmark publishes.


1. The result, stated plainly

Contra Labs released the Frontier Design Benchmark on November 18. The leaderboard is built with BenchFlow and measured on real design briefs, not prompt-engineering puzzles.

The headline is an anonymous model. Codename Gestalt, still in stealth development, took first place with an Elo of 1931 and a 90% win rate against the field. Second place went to a reference set of human designers recruited from the Contra platform, at 1739 and 74%. Anthropic's Claude Fable 5.2 placed third at 1713, with GPT-6.1 Sol fifth at 1668.

RankModel / entryLabEloWin rate95% CIVotes
1Gestalt (stealth)undisclosed193190%86–92420
2Human designers (reference)Contra173974%71–78571
3Claude Fable 5.2Anthropic171373%69–76683
4Claude Opus 5.5Anthropic169173%69–76527
5GPT-6.1 SolOpenAI166870%66–74549
6Claude Sonnet 5.5Anthropic166769%65–73440
7Kimi K3.1 ProMoonshot AI166371%66–74519
8GPT-6 AstraOpenAI163866%61–70449
9Qwen3.8 MaxAlibaba143439%35–44428
10Kimi K3Moonshot AI141540%35–44425
11GLM-5.3 PrimeZ.ai140341%37–45464
12Gemini 4 CarbonGoogle136338%32–40546
13Muse Spark 1.3Meta133631%27–35503
14Muse WonderMeta124922%18–25486
15Grok 4.7xAI124723%19–26559
16Gemini 4 ArgonGoogle122128%—547
17DeepSeek V4 ProDeepSeek112112%9–15506

One flag on this table before any interpretation: the last three ranks invert the win-rate ordering. Gemini 4 Argon reports a higher win rate (28%) than Muse Wonder (22%) and Grok 4.7 (23%) yet carries a lower Elo. That divergence is not an error. It is what Elo is for.

Three numbers frame the whole table before any interpretation.

The first-place margin is enormous. 1931 versus 1739 is a 192-point gap. On an Elo scale, that is not a close race. It is a different tier.

The top eight are compressed into a 101-point band. From 1638 to 1739, seven models sit within a range that, at 420–683 votes each, is close to the resolution limit of a pairwise-vote Elo. Rank order inside that band is a claim the data does not strongly support.

There is a cliff after rank eight. 1638 down to 1434 is a 204-point drop. The field splits cleanly into a capable tier and a struggling tier. Everything below rank eight loses the majority of its comparisons, with win rates from 39% down to 12%.

That structure — one outlier, one dense band, one long tail — is more informative than any individual rank.


2. How the benchmark was built

The methodology decides what the numbers mean.

The pairwise stage is a blind head-to-head. Judges see two outputs, not two model names. Elo is computed from those comparisons, with judging covering color, type, layout, usability, and adherence to the brief.

That design has a specific virtue and a specific weakness. The virtue: blind pairwise comparison is the most reliable way to extract human preference, because it avoids absolute scoring, where raters drift. The weakness: preference is not quality. A designer can prefer the more persuasive artifact while still being able to name its defects. The benchmark's own rubric data proves this happens.


3. Head to head against humans: 20 of 28

The human comparison is the one worth isolating, because it is the only place where the benchmark measures a model against a non-model baseline.

In direct pairings between Gestalt and the human designer reference works, judges chose Gestalt in 20 of 28 comparisons. That is roughly 71%, consistent with the 74% win rate recorded for the human reference entry against the wider field.

Two readings are possible, and they are not equivalent.

The generous reading: a model now produces design outputs that professional designers, judging blind, prefer over work made by other professional designers on the same briefs.

The cautious reading: 28 comparisons is a small sample. A 20–8 split has a wide confidence interval. And the human reference set is a *reference*, not a curated team of top-tier designers — Contra describes it as designers recruited from the platform.

The benchmark supports the first reading directionally and cannot support it precisely. That distinction matters more than the headline.


4. The disagreement that makes this benchmark interesting

Here is the fact that separates this release from a routine leaderboard post.

On the Design Critique Rubric, Gestalt scored 83. Claude Fable 5.2 and Claude Opus 5.5 both scored 84. The human reference works scored 74.

The rubric ranks the models *above* Gestalt and ranks humans *last*. The pairwise preference ranks Gestalt first and humans second. The two metrics tell different stories about the same outputs, judged by the same population of designers.

Contra flags the tension explicitly. The company notes that the designers' rubric-level judgment and their actual execution quality do not fully align — the human reference works carry identifiable defects, yet still attract high preference in blind voting.

This is not a contradiction to explain away. It is the finding.

Three interpretations, all defensible.

Preference rewards the whole; the rubric scores the parts. A rubric decomposes design into color, type, layout, usability, brief adherence. Ten competent components can sum to a mediocre artifact, and one exceptional composition can absorb several weak components. If the rubric is additive and the vote is holistic, divergence is expected, not anomalous.

The rubric measures craftsmanship; the vote measures persuasion. Design output is consumed at a glance before it is inspected. The property that wins a blind comparison is immediate legibility and confidence of composition. The property a rubric rewards is correctness at the component level. A model optimized on human preference data will lead on the first and trail on the second.

The human baseline is a weak control. Human reference works were recruited, not curated for excellence. A 74 rubric score is a statement about that sample, not about human design capacity.

What the benchmark does *not* do is reconcile the three scores into one number. It publishes them side by side and lets the reader hold the tension. That is more honest than a composite index, and harder to market.


5. The Slop Detector and the problem of measuring human-likeness

The third metric is the most unusual, and the most exposed to criticism.

Contra runs a Slop Detector: a classifier that scores how strongly an output carries the fingerprints of AI-generated design. Lower is better. Gestalt's results show a notable AI fingerprint signal — described as approaching the level of human reference works.

Two structural issues follow.

A human-likeness classifier is a proxy, not a quality measure. If a model minimizes detectable AI signatures, it can improve this score without improving the design. The metric rewards indistinguishability, and indistinguishability is a target that can be optimized against directly.

It creates a second objective that competes with the first. A benchmark that simultaneously rewards (a) being preferred by designers, (b) scoring well on a critique rubric, and (c) not looking AI-generated is asking for three things that are not always aligned. A strong entry can trade them against each other. That is a design property of the benchmark, and it should be disclosed as one.

None of this makes the detector useless. Detecting AI-generated output is a real operational need — platforms screening submissions, agencies auditing deliverables, marketplaces filtering listings. But a detector score belongs in an authenticity column, not in a quality ranking, unless the benchmark states that human-likeness *is* the quality target.


6. The stealth variable

Gestalt is anonymous by design. Contra has not disclosed the development team, the model architecture, or the identity behind the codename. It is labeled as a stealth model.

That choice has consequences for how the result should be read.

Provenance is part of the result. An Elo score is a claim about a system, and a system includes its training data. Twenty-seven of the 33 tasks are adapted from real projects hosted on the Contra platform. If comparable projects are publicly visible, a model trained on public design work has a plausible advantage on exactly this distribution. That does not invalidate the score. It narrows the claim: this is performance on tasks drawn from one platform's project pool, judged by designers recruited from the same platform.

Stealth timing is a communication strategy as well as a security one. A codename, an undisclosed lab, and a first-place finish on a new benchmark produce a specific kind of attention. That does not make the result wrong. It does mean the burden of independent replication is higher than usual.

Verification requires a second run. The claim that survives is the one reproduced by an evaluator who did not build the benchmark, on tasks the model's developers did not see. Until that exists, 1931 is a well-measured number about an unidentified system.


7. Statistical caveats worth keeping


8. What this means if you are choosing a model for design work

Five operational conclusions, in order of practical weight.

Use the tier, not the rank. The defensible claim from this release is that one model sits clearly above the field, and that a group of seven frontier models is effectively tied for second. Procurement decisions built on "we picked the one ranked third" are reading precision into data that does not carry it.

Match the metric to the job. If the deliverable is judged by humans at a glance — ad creative, landing page hero, social asset — pairwise preference is the relevant signal. If the deliverable is handed to an engineering team for implementation — design systems, component libraries, spec documents — the rubric signal matters more, because it scores brief adherence and component-level consistency. The two metrics disagree in this benchmark, and that disagreement is exactly the reason to check which one your use case resembles.

Build a small pairwise harness on your own briefs. Thirty-three tasks from one platform's pool is a starting point, not a proxy for your distribution. A working eval needs your own briefs, your own judges, and enough votes to separate the models you are actually deciding between. Randomize presentation order to control position bias, and keep the judge pool fixed across rounds. Serve every candidate model through one endpoint, so that onboarding a vendor is a key change rather than an integration project — a single gateway, whether that is 4sapi or an internal proxy, keeps the comparison symmetric while it runs.

Treat human-likeness scores as an authenticity flag. They are useful for moderation and for detecting low-effort generated output. They are not a quality ranking, and optimizing against them produces a different artifact than optimizing for craft.

Route multi-model comparisons through one interface. A pairwise eval across seven or eight vendors means seven or eight SDKs, key sets, rate limits, and error formats — before any judging happens. Consolidating that traffic behind a single API gateway such as 4sapi keeps credentials in one place, standardizes request and response shapes, and makes per-model spend visible while the eval runs. That is infrastructure hygiene, not a scoring advantage: the numbers still come from your judges.


9. What the release does not settle

Each of these is answerable with one more run. None of them is answerable from the published table alone.


10. Bottom line

Tags:AI DesignGestalt AIContra LabsFrontier Design BenchmarkAI BenchmarksAI Model Evaluation

Recommended reading

Explore more frontier insights and industry know-how.