OpenVibeEval: An Open Frontend Benchmark Where You Vote Blind on AI Models

OpenVibeEval: A Living Frontend Benchmark

OpenVibeEval is officially live at openvibeeval.com. I built this platform because marketing claims and slide decks are the primary evidence people see for AI frontend tools. Those static artifacts hide structural failures. Developers need a transparent way to compare models on identical, interactive tasks without relying on closed leaderboards that obscure what actually broke during generation.[2]

The benchmark contains 233 committed results across 9 frontend prompts and 26 models. Ten harnesses appear in the dataset. The zcode harness leads with 110 runs. Command-code follows with 29 runs. Codex includes 23, opencode includes 18, and claude-code and antigravity-cli include 16 each. The collection also includes grok-cli, cline, kilo, and github-copilot.[1][3]

Every single run lives as an immutable record in a public repository. When you open a card on the site, you inspect the live sandboxed page. You examine the source code. You review raw accessibility scores. The goal is to move the conversation from vague opinions to inspectable artifacts.[2]

Why We Built the Blind Arena

Brand bias distorts how people evaluate generated code. When developers see a model name like Claude Opus or GPT-5 on a demo, they naturally overlook layout glitches or missing keyboard navigation. If the same output comes from an unfamiliar open model, reviewers scrutinize every flaw. The Arena feature removes that subjective distortion by keeping the model and harness identity hidden.[2]

When you enter the Arena, the interface presents two rendered frontend outputs side by side for a randomly chosen prompt. You test the interactive elements. You click through buttons. You assess visual cohesion without knowing who built either page. You vote on which result solves the prompt better.[1]

Only after you submit your vote does the interface reveal the model and the harness used. This mechanism forces every model to stand on its output quality alone. It eliminates marketing influence and lets actual execution dictate the score.[2]

The Mechanics of Blind Taste Testing

A blind comparison system requires safeguards against small sample noise. In OpenVibeEval, a matchup on the leaderboard stays locked until a result accumulates at least 15 independent community votes. This threshold prevents early anomalies or accidental misclicks from creating misleading rankings.[2]

We implemented automated task rotation. The Arena cycles between distinct prompt categories so visitors do not repeatedly judge the same matchup. One round might test a retro arcade game with canvas physics. The next round presents a multi-track web audio sequencer.[1]

Rotating tasks ensures that models are evaluated across diverse technical constraints. A model that excels at canvas particle math might completely fail at semantic form layout. Task rotation surfaces those specialized strengths and systemic weaknesses across the whole catalog.[2]

Verifying Output with Hard Accessibility Metrics

Visual aesthetics can hide severe structural defects. Many models generate visually striking user interfaces that are impossible for screen readers or keyboard-only users to navigate. To expose these problems, every committed result includes an automated accessibility audit based on the axe-core formula.[1]

Each result record stores DOM node count data. Depth metrics and total payload size are also captured in the result.json file alongside desktop and mobile screenshots. These numbers create an objective baseline that cannot be faked with flashy styling.[3]

The recent analysis of the stealth model 0x Alpha highlighted why this data matters. Social feeds praised the model for sweeping synthetic reasoning tests. When I ran it through OpenVibeEval, its accessibility scores cratered on specific frontend prompts. It scored zero on the smartwatch storytelling task. The empirical data contradicted the online narrative.[2]

The Harness Layer Dictates Code Output

Developers often assume the base model weights decide the final code quality. In practice, the agent harness is just as influential. The harness controls the system prompt, tool integration, file modification routines, and automated retry logic. Changing the harness often changes the rendered output more than upgrading the model version.[2]

OpenVibeEval labels every entry with both the harness and the model. For instance, running a model through command-code versus opencode produces different file structures and error handling patterns on the exact same prompt.[1]

Without harness attribution, benchmark scores become misleading. If a test fails because a CLI wrapper clipped the system prompt, blaming the underlying model is inaccurate. Documenting the wrapper gives teams the context needed to debug their actual developer toolchain.[2]

A Pure Git-Tree Architecture with Zero Servers

OpenVibeEval operates without a backend database and without an application server. The data store is a clean directory hierarchy inside a public git repository. Every evaluated run is filed under a predictable path: results/prompt-slug/model-slug/harness-slug/date/.[3]

A sync script aggregates these directory entries into a static JSON manifest. The browser downloads this manifest and renders the user interface through a vanilla TypeScript single-page application. Publishing a new run to the public site requires only a git commit and a push.[1]

This static approach guarantees transparency and longevity. Anyone can clone the repository, run the files locally, and inspect the raw HTML without relying on an external API service. The data remains accessible even if the host platform goes offline.[2]

Ingesting New Runs via CLI

To keep the benchmarking pipeline fast and repeatable, I created an ingestion command-line interface. The tool takes a generated index.html file, computes DOM metrics, runs the axe-core accessibility audit, and formats the output directory automatically.[3]

eval-cli ingest --prompt digital-garden --model gpt-5-6 --harness zcode --file ./output/index.html

The CLI captures desktop and mobile viewport screenshots, formats the result.json metadata file, and verifies that the output contains no broken relative assets. This standardized intake process keeps every entry in the 233-result dataset consistent across runs.[1]

Actionable Evaluation Checklist

Do not select an AI coding stack based on social media screenshots or aggregate synthetic leaderboards. Use this step-by-step workflow to establish an empirical evaluation pipeline inside your own development environment.[2]

Standardize your test prompts around a single self-contained index.html output with zero build steps. Explicitly record the harness configuration, system prompt version, and tool-calling parameters alongside the model identifier. Execute at least five diverse UI scenarios, covering canvas rendering, audio synthesis, and complex responsive dashboards. Audit every generated component with axe-core to record real accessibility violations. Vote on rendered outputs blindly before checking the provider attribution. Commit every result file and screenshot to a versioned repository to maintain an immutable audit trail.[3]