I Built an Open Benchmark for AI Frontend Generation

Why I stopped trusting AI leaderboards for frontend work

Aggregate leaderboards are vanity metrics. They collapse complex, multi-layered agentic behavior into single-digit scores that obscure the truth about real-world performance. When evaluating tools for frontend work, I stopped caring about these scores and started building OpenVibeEval. I wanted to move from abstract rankings to verifiable, rendered UI artifacts. You cannot build a production application based on a ranking found in a marketing deck. You need to see the code.

Most industry benchmarks focus on general reasoning capabilities or backend-heavy tasks. They fail to represent the specific, visual, and interaction-focused output that frontend engineers deal with daily. A screenshot in a slide deck is not a product. It is a marketing asset that no developer can actually verify. I wanted to see the model logic in motion. I wanted to confirm that a button actually triggered a state change, rather than simply looking correct in a static image file.

By shifting the focus to live, sandboxed rendering, my approach forces models to prove they can generate functional code. Every result on the platform is an interactive, browser-rendered artifact. This transparency ensures that I am comparing models on their ability to build usable applications. It is better than their ability to game a synthetic test suite. I built this tool to provide evidence for engineers who are tired of trusting corporate benchmarks that hide the messy reality of AI code generation.

The gap between an AI producing a correct answer on a coding benchmark and an AI producing a working frontend component is massive. A model might write syntactically correct logic that results in a broken layout. It might write a function that performs math perfectly but fails to bind to a DOM element. These nuances matter when you are trying to ship an actual interface. My evaluation site bridges this gap by rendering every output in a sandbox, providing immediate, visual proof of quality.

The harness variable

The industry obsession with frontier models ignores the harness layer entirely. This is the primary failure mode in modern evaluation. A model is only as good as the system prompt, tool wiring, and file-editing strategy that wraps it. You can swap the underlying model, but if your harness is brittle, your output will remain inconsistent. I learned this the hard way after watching the same model behave differently across two distinct development environments.

When my generated output disappointed me, I found that switching the harness often made a larger difference than upgrading to a newer model version. The harness determines the structure of the file edits, the scope of context, and the retry logic. Marketing materials from providers never surface this data. They leave engineers in a state of perpetual lock-in where we attribute success to the model provider rather than the engineering effort put into the integration tool.

OpenVibeEval explicitly labels every result with its corresponding harness. By isolating the harness as a primary variable, I can see how different integrations process the same prompt. This methodology prevents the mistake of attributing harness bias to model intelligence. If one harness consistently produces cleaner HTML and CSS while another fails, the evaluation makes that disparity visible for all to see.

The harness serves as the filter for everything the model produces. It handles the context window, it interprets the file system state, and it manages the multi-step reasoning required for a complex UI change. When you use a closed-source subscription service, you accept their black-box harness. By documenting which harness I use for every single entry, I keep the focus on the actual developer workflow instead of just the parameter count of the underlying neural network.

Standardizing inputs with a rigid code contract

To ensure my benchmarks remain reproducible and comparable, I impose a strict contract on every prompt. Each prompt requires a single, self-contained index.html file that relies only on vanilla JavaScript and CSS or CDN-hosted assets. I forbid build-step frameworks entirely. This is necessary because it removes the environmental variables that make testing so difficult in a modern web ecosystem.

This constraint trades generality for absolute clarity. Because I exclude complex build pipelines, I avoid the unpredictability that comes with environment configuration. Every result can render instantly in a plain sandboxed iframe. This portability is the only way to compare different models and harnesses side-by-side without architectural noise. If I allowed React or Next.js, the setup time and the potential for dependency errors would invalidate the purity of the generation evaluation.

I define the scope of every task as a frontend-centric objective. The prompts require the generation of functional UI components that can survive the transition from the model to the browser. By limiting the input space, I ensure that the output is always measurable. I am not testing the ability of a model to resolve a complex Node.js dependency conflict. I am testing its ability to translate an intent into a visual web interface.

When you standardize the input format, you create a baseline for fairness. The same prompt goes to every model. The same harness runs the test. The same container evaluates the output. This rigidity is the key to creating a data set that actually yields insights rather than just repeating company marketing claims about parameter sizes or token speed.

Building a benchmark as a git tree

Databases are unnecessary complexity for this scale. I chose a git-based filesystem architecture because it provides an immutable, version-controlled history. My data layer is just a folder tree. The structure is results/prompt-slug/model-slug/harness-slug/date/. This format allows me to store the entire state of the evaluation history in a format that is readable by any standard version control client.

This structure makes publishing trivial. Committing a directory to the repository automatically version-controls the evaluation state. There is no server to maintain and no database migration to run. For a solo project, this simple, static approach significantly outweighs the overhead of a managed backend. I do not have to worry about data integrity because Git manages that for me automatically.

By keeping the entire history in the repository, I can re-run old prompts against new models whenever I want. The repository effectively serves as both the data storage and the audit log. It ensures that my benchmark remains open and auditable by anyone. If a user wants to verify the result of a test run from months ago, they can check out the exact commit from that time.

This git-first architecture is the foundation of my performance. It keeps the ingestion pipeline predictable. Each directory represents a single unit of truth. When I want to add a result, I simply move the file into place and commit. This workflow is much faster than filling out forms in a web interface that might lose my progress or fail to save the metadata correctly.

The ingestion and rendering pipeline

The pipeline relies on a custom ingestion CLI that scaffolds the git tree. It accepts a single index.html and manual screenshot assets. It then generates a result.json file containing metadata like accessibility scores and harness details. This process ensures that I maintain high data hygiene without requiring an expensive cloud-based ingestion service or complex API authentication.

At runtime, a sync script copies this data into the public directory and generates a manifest. The frontend, a lightweight vanilla TypeScript SPA, fetches this manifest to render the grid of results. Because the SPA is dependency-free, it remains robust across builds. I do not need a framework that will go out of date in a year, and I do not need to worry about runtime performance costs.

The following code illustrates how the result manifest structure is kept simple to ensure the SPA can easily parse it for rendering.

{
  "id": "prompt-1",
  "models": [
    {
      "name": "gpt-5",
      "harness": "zcode",
      "path": "/results/p1/gpt5/zcode/2026-08-01/"
    }
  ]
}

The simplicity of this schema is intentional. By using a plain JSON object, I make it trivial for any future contributor to write their own ingestion script if they wish to add a new model. The manifest acts as the interface between the data and the viewer. Because the interface is decoupled, I can change the entire internal storage structure of the git tree without breaking the front end. I just update the sync script, and everything continues to work as expected.

Disclosure of sandboxing limitations

I intentionally choose restrictive sandbox policies for rendering results. Every output runs in an iframe with allow-scripts but without allow-same-origin. This breaks features like localStorage, sessionStorage, and relative fetch calls. Some might view these as technical failures, but I label them clearly as known limitations. Masking these constraints would provide a false sense of what the generated code is capable of in a real browser environment.

A benchmark that quietly works around sandboxing limitations misrepresents reality. I want the results to show how the browser handles the generated code exactly as the user will see it when they paste it into a file. If the code relies on origin-specific features, it should not work in the sandbox. This creates an honest representation of the utility of the AI-generated code. It forces me to be aware of the dependencies I am asking the model to implement.

Transparency is non-negotiable here. When a model fails to support standard accessibility practices, I record it as n/a rather than hallucinating a metric. This forces users of the benchmark to confront the actual trade-offs that models make in their output. It is better to have an honest result labeled as missing than to provide a false sense of security with a fabricated score that doesn't actually correspond to any real-world capability.

My sandbox policy is a direct reflection of my commitment to safety and clarity. I know exactly what permissions the iframes have. I am not leaking browser state or exposing the host environment. This allows users of OpenVibeEval to interact with the results without fear that the code might trigger malicious behavior. It creates a controlled environment where the focus remains entirely on the logic and the quality of the generated frontend.

Next steps for building your own evaluation harness

Move away from relying on vendor-provided scores today. Start by building a local evaluation artifact that you own. You can adopt this workflow by following these simple steps. Do not wait for a vendor to release an official benchmark that might favor their own architecture. Take control of your own data and build an evaluation suite that reflects your own production needs and constraints.

By following these steps, you will move from passive consumption of AI marketing to an active, engineering-led evaluation posture. You will develop a clearer understanding of which tools actually help your team ship and which ones only look good in a demo. Start with a single prompt and a simple folder. Build the habit of documenting your results. Over time, you will accumulate the data needed to make informed, evidence-based decisions about which AI models to use in your daily workflow.