key takeaways
- Arena AI, previously known as LMArena, is free to use on an individual basis and does not have a paid consumer plan as of September 2026.
- The service uses blind, crowdsourced model comparisons rather than reported industry benchmarks.
- Arena AI rebranded from its previous name LMArena in January 2026, after raising $150 million Series A at a $1.7B in funding.
- The rankings reflect aggregated anonymous human preferences rather than model performance on any particular set of proprietary prompts.
- For enterprise model evaluation needs, the company offers its services through a separate division called AI Evaluations, with custom pricing available upon request.
The short answer
A SaaS founder, on a Tuesday afternoon, tested out three language models for his support-ticket classifier. Five tabs in his browser, three API playgrounds open, and a spreadsheet tracking which model was better at the esoteric edge cases his classifier encountered. He hadn’t settled on his model by 4pm, delaying his launch by another week. This is the problem Arena AI solves.
Arena AI is good, but not great. It excels at a narrow task: it is free, effective, and largely eliminates the need for founders to perform due diligence in selecting a language model. Founded by a UC Berkeley team as an open-source project in 2023, the service sees tens of millions of model evaluations per month, giving it credibility in its niche.
However, good is a ceiling. Arena provides a model ranking based on general preference in an aggregated, anonymized leaderboard. It offers little guidance on which model will best suit your use case (accuracy, price, speed) and founders who treat it as an endpoint rather than a starting point will almost certainly regret it in a few months.
Who Arena AI is actually good for
Arena is a good choice when you know what work your model needs to do and want a quick unbiased first pass. If you are building a coding assistant, a support bot, or content feature and have three or four viable candidates for the job, running them through Arena’s blind battle mode before writing any integration code will save you time.
It also helps to stay up to date: new frontier models release every few weeks, and Arena’s leaderboard updates itself constantly as more votes come in, which saves you from having to re-test every release yourself.
Skip it (or at least don’t stop there) if you already have a specific use-case in mind like legal document review or medical triage copy, since crowd preference is not a good proxy for correctness in these specialized domains. The Arena site itself recommends the leaderboard as a starting point rather than an end-goal.
What using Arena AI actually looks like operationally
┌──────────────────────────────────────┐
│ 1. USE REAL PRODUCT PROMPTS │
│ Pull prompts from actual usage │
└──────────────────────────────────────┘
↓
┌──────────────────────────────────────┐
│ 2. RUN BLIND BATTLES │
│ Compare anonymous models in Arena │
└──────────────────────────────────────┘
↓
┌──────────────────────────────────────┐
│ 3. JUDGE THE OUTPUTS │
│ Vote for the response you'd ship │
└──────────────────────────────────────┘
↓
┌──────────────────────────────────────┐
│ 4. BUILD A SHORTLIST │
│ Keep the 2–3 strongest models │
└──────────────────────────────────────┘
↓
┌──────────────────────────────────────┐
│ 5. TEST ON YOUR OWN DATA │
│ Run shortlisted models in staging │
└──────────────────────────────────────┘
↓
┌──────────────────────────────────────┐
│ 6. MAKE THE FINAL DECISION │
│ Compare quality, cost & latency │
│ Then choose what goes to production │
└──────────────────────────────────────┘ ARENA → SHORTLIST
YOUR DATA → DECISIONArena AI pricing in 2026
Arena
Arena is an AI evaluation and comparison platform that lets users test multiple AI models against the same prompt. It’s designed for developers, researchers, and AI power users who want to compare quality, speed, and reasoning before choosing a model.
Arena’s product is free at its core. Access to the public arenas, leaderboards, and direct model testing is all free, with basic voting not requiring an account either.
The only way to get something of actual value here is to pay for AI Evaluations, Arena’s enterprise offering for AI labs and businesses that want structured, large-scale evaluations run through Arena’s community, as opposed to the public leaderboard. The pricing is custom and scoped on a per-engagement basis, set directly with Arena’s sales team, and not published anywhere on a pricing page.
In practice, this means that most of the founders reading this review, trying to find an article titled “Arena AI pricing” are likely to be looking for pricing that doesn’t exist for the individual or small-team level yet. Set your expectations to zero dollars for evaluation and not much else unless your team grows to the point of needing the enterprise evaluation service.
Wrong approach vs right approach
| Wrong approach | Right approach |
|---|---|
| Picking a model off a vendor’s launch-day demo | Running your own real prompts through blind comparison before committing |
| Treating Arena’s leaderboard rank as the final decision | Using the leaderboard to build a shortlist, then testing that shortlist on your own data |
| Testing with generic sample prompts | Testing with actual prompts pulled from real customer tickets or product usage |
| Choosing a model once and never re-checking it | Re-running the same test set whenever a new frontier model ships |
Three real limitations of Arena AI
Crowd preference is not the same thing as correctness. Blind voters reward longer answers, more certain answers, and more pleasant answers disproportionately often compared to factually correct answers. Works for a customer-facing chatbot that’s a reasonable proxy, but not a great signal for anything that actually requires precision.
The version you vote on may not be the version you deploy. Model providers have historically submitted specially tuned variants for Arena testing, and your production API can have different behaviors than the version that won the vote. Consider an Arena win a reason to poke at your production model, not a replacement for testing it.
There’s no cost, latency, or reliability data anywhere on the platform. A model can win the leaderboard but be demonstrably the wrong choice for your application if it costs 3x as much per token or has poor uptime in your region. Arena answers the question “which output do people prefer” and nothing else, and everything else in a real world deployment has to come from somewhere.
What I’ve seen working with customers is that they get the most value out of Arena when they treat it as a first step in a two-step process.
Pause and think: when was the last time you actually tested a model’s production API response to your own prompts, instead of trusting someones leaderboard ranking or a vendors demo script?
Self-audit: is your model selection process actually reliable?
- Have you run your top candidates through blind comparison using real prompts from your own product, not sample prompts from a demo?
- Does anyone on your team own re-testing the model choice when a new frontier release ships, or does the original decision just sit unquestioned?
- Have you checked cost and latency for your shortlisted models, separately from quality?
- Are you testing the exact production API endpoint you plan to integrate, not a variant?
- Would you be able to explain, in one sentence, why your team picked this specific model over the runner-up?
If more than one of these is a no, the gap probably isn’t the model. It’s that nobody owns the evaluation step after the initial decision gets made.
Alternatives if Arena isn’t a fit
Arena is free, public, and preference-based. Exactly correct for a first-pass model comparison but exactly wrong once your decision comes with material cost, compliance, or workflow weight. The two paths teams tend to pursue next fall into two categories: tools that help you adopt the model into your day-to-day workflow (after you’ve settled on one), and tools that are designed for a more rigorous evaluation than your public voting model can support.
For ongoing workflow, not one-time comparison
OpenRouter
OpenRouter routes the same prompt through dozens of models over one API, letting you compare cost & latency apples-to-apples. The gap Arena leaves open is exactly the kind of comparison you need to know which of your shortlisted models you can afford to use at your volume.
OpenRouter
OpenRouter is a unified AI interface and API that provides access to hundreds of AI models across multiple providers. It supports model routing, provider fallbacks, OpenAI-compatible APIs, model discovery, data-policy controls, and enterprise governance features.
Operator’s verdict: A good next step after Arena is exactly what you need once you’ve narrowed your models to 2-3 and need to know what they’ll actually cost you.
Vellum
Vellum allows teams to version prompts, evaluate models, and track performance over time, instead of throwing a prototype in a browser tab on a Wednesday and forgetting about it by Thursday.
Vellum
Vellum is an AI development platform with visual workflows, prompts, evaluations, deployments, SDKs, and agent-building capabilities. Its current product also provides an AI assistant with persistent memory, integrations, computer access, and the ability to take approved actions across connected systems.
Operator’s verdict: Worth it if more than one person is going to be responsible for making the model decision. Overkill if you’re just some founder testing out a hypothesis on a Tuesday.
For private or higher-stakes evaluation
Scale AI
Scale AI offers structured private human evaluation, controlled by you, with the ability to select who evaluates against which criteria, as opposed to a public crowd. This is useful when the potential use case has material implications like legal documents, medical intake forms, or other cases where the preference of an anonymous Arena voter may not directly apply to your domain. This is a quote-based product, meant for teams looking to evaluate at a scale where picking the wrong model has significant costs.
Scale AI
Scale AI provides data, evaluation, model-development, and enterprise AI infrastructure for organizations building advanced AI systems. Its products include Scale Data Engine, Scale GenAI Portfolio, evaluation capabilities, and specialized data services for areas such as generative AI, robotics, healthcare, and defense.
Operator’s verdict: Worth it for the sales call if a wrong model choice would cost more than the evaluation, not a priority for early-stage model selection conversations, that’s what Arena is for.
Surge AI
Both Surge AI and Scale AI offer private human evaluation with custom criteria, but Surge is more commonly used for fine-tuning or RLHF-style training data alongside straight comparisons for model selection. While both can be used for similar tasks, Surge is preferred when the evaluation is part of a larger data pipeline for training or alignment rather than a standalone comparison between two pre-trained models.
Surge AI
Surge AI provides data labeling, expert human feedback, evaluation, training data, and AI research services for advanced AI systems. Its platform is particularly focused on high-quality human data for language models, content moderation, coding, search evaluation, and other difficult AI tasks.
Operator’s verdict: A good indicator of whether Surge would be a good fit for a request is if the evaluation is one step in a larger dataset creation process, rather than an end in and of itself. It would be considered an unnecessary expense to use Surge for a binary choice between two pre-trained models, but a common use-case would be to use Surge to label a dataset alongside other processes for training or alignment.
HELM
HELM (Holistic evaluation of language models) is an automated benchmark designed to evaluate language models across multiple dimensions at once accuracy, robustness, fairness, and efficiency rather than asking humans to vote on their preferences. It is created by Stanford’s research team and produces a documented multidimensional score, rather than just a gut feeling.
HELM
HELM is an open-source Python framework from Stanford's Center for Research on Foundation Models for holistic, reproducible, and transparent evaluation of foundation models. It evaluates models across standardized scenarios and multiple dimensions including accuracy, efficiency, bias, toxicity, and other risks.
Operator’s verdict: The right choice when you need to justify your choice of models to a compliance or risk team not someone who is willing to take your word for it based on a couple of test runs. Too time-consuming for a quick decision.
None of the five tools reviewed above can replace Arena’s blind-testing format. They build on top of it: two of them are used after choosing a model and fine-tuning it to one’s needs, and three of them are intended for cases where the decision requires more responsibility than an arbitrary public vote can provide.
If Arena’s solution does not fit your organization’s needs for other reasons, you may want to take a closer look at your AI strategy before introducing another evaluation stage. It may be that the problem you are trying to solve is located elsewhere in your stack. For a wider look at the tools available to small organizations and teams, see the best AI tools for solo founders.
FAQs
Arena publishes all preference data and prompts for research, as a commitment to methodological transparency. If you are testing something that contains sensitive or proprietary information, we encourage you to operate this testing in your own private environment, and not in the public Arena.
Not as an individual account holder. AI labs use Arena to anonymously test pre-release models, but this is done through direct partnerships with the Arena team, and not through the individual free accounts that most founders will be using.
Not quite. Arena is designed around individual voting and a shared, public leaderboard. The private, structured and team-based evaluation is done through the paid AI Evaluations product.
Constantly, as votes are being added and new models are deployed. This is why periodic re-checks of the shortlisted models are more valuable than a one-time scan.
Not to vote or browse the public leaderboards. An account is necessary to save comparisons as a personal history, and to access the enterprise evaluation product.
Where this is heading
More founders are delivering product AI features every quarter, and picking a model is increasingly a maintenance task, not a one time decision, much like updates of dependencies or security patches. Tools like Arena reduce the cost of comparison, but do not encourage the iterative rechecking of alternatives.
The question is not which model wins in Arena for the current month, but whether one has an approach to consistently test the next option if the current leader will be replaceable in six weeks, and it will be.
Your next move
Take the last 3 real prompts your product actually generated this week (not the test prompts you write yourself) and feed them to Arena’s blind battle mode today. See which models win, and check if that matches your production spend.
