Leaderboard
Scored on public signals; components with none are listed as Unranked · Formula
Component
Domain
Vertical
| Rank | Score | Component | Description | Evidence | Last commit | Install |
|---|---|---|---|---|---|---|
| 1 | 99.536 | promptfooeval · ai-agents | CLI and library for testing, comparing, and red-teaming LLM prompts and agents with assertions and CI integration. | 24,737 stars · 2,255 forks | No one-command install · Source | |
| 2 | 99.512 | openai-evalseval · ai-agents | OpenAI's official framework for evaluating LLMs and LLM-powered systems, with a registry of community eval sets. | 19,509 stars · 3,093 forks | No one-command install · Source | |
| 3 | 99.505 | lm-evaluation-harnesseval · other | EleutherAI's unified framework for evaluating language models on hundreds of academic benchmarks. | 13,860 stars · 3,533 forks | No one-command install · Source | |
| 4 | 99.455 | deepevaleval · observability | Open-source LLM evaluation framework with 14+ metrics (hallucination, faithfulness, answer relevancy) and CI support. | 18,041 stars · 1,890 forks | No one-command install · Source | |
| 5 | 99.421 | ragaseval · search | Reference-free evaluation of retrieval-augmented generation pipelines; measures faithfulness, answer relevance, and context precision. | 15,853 stars · 1,727 forks · 1 mention · failed install test | No one-command install · Source | |
| 6 | 99.273 | phoenixeval · ai-agents | Arize Phoenix: open-source LLM observability with built-in evals, span tracing, and dataset curation for RAG and agents. | 11,286 stars · 1,086 forks · 2 mentions | No one-command install · Source | |
| 7 | 99.103 | swe-bencheval · ai-agents | SWE-bench: benchmark for evaluating LLMs on real-world GitHub issue resolution across 12 popular Python repositories. | 5,762 stars · 957 forks · 9 mentions | No one-command install · Source | |
| 8 | 99.052 | giskardeval · ai-agents | Open-source LLM testing framework for detecting vulnerabilities (prompt injection, hallucinations, bias) via automated scan. | 5,838 stars · 542 forks | No one-command install · Source | |
| 9 | 98.991 | agentaeval · ai-agents | Open-source LLM developer platform with prompt playground, evaluation pipelines, and A/B testing for iterating on LLM apps. | 4,670 stars · 661 forks | No one-command install · Source | |
| 10 | 98.956 | openai-simple-evalseval · ai-agents | OpenAI's lightweight benchmark suite (MMLU, HumanEval, MATH, GPQA, MGSM) for fast model capability comparisons. | 4,621 stars · 509 forks | No one-command install · Source | |
| 11 | 98.825 | big-bencheval · front-end | Google's Beyond the Imitation Game benchmark: 200+ diverse tasks designed to probe capabilities beyond standard NLP benchmarks. | 3,247 stars · 618 forks | Stale | No one-command install · Source |
| 12 | 98.783 | trulenseval · back-end | Evaluation and tracking for LLM and RAG applications with a feedback-function API and experiment dashboard. | 3,530 stars · 335 forks · 1 mention | No one-command install · Source | |
| 13 | 98.776 | xlang-ai-osworldeval · ai-agentsContributed by Sentinel | [NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments | 3,117 stars · 530 forks · 7 mentions | No one-command install · Source | |
| 14 | 98.743 | inspect-aieval · other | UK AISI's framework for safety evaluations of large language models, with task-based scaffolding and solver pipelines. | 2,683 stars · 685 forks | No one-command install · Source | |
| 15 | 98.706 | helmeval · other | Stanford CRFM Holistic Evaluation of Language Models: standardized benchmark suite covering accuracy, calibration, robustness, and fairness. | 2,898 stars · 412 forks | No one-command install · Source | |
| 16 | 98.688 | lightevaleval · ai-agents | Hugging Face lightweight evaluation library for LLMs across academic benchmarks, with fast local and remote inference support. | 2,533 stars · 553 forks | No one-command install · Source | |
| 17 | 98.324 | evalpluseval · other | Rigorous code generation evaluation framework built on top of HumanEval and MBPP with 80x more test cases. | 1,819 stars · 208 forks | No one-command install · Source | |
| 18 | 98.283 | webarenaeval · browser | WebArena: realistic web-based environment for evaluating autonomous agents on long-horizon browser interaction tasks. | 1,592 stars · 249 forks · 1 mention | No one-command install · Source | |
| 19 | 98.177 | tau-bencheval · ai-agents | Tau-bench: agent benchmark for tool-agent-user interactions in retail and airline domains with policy-grounded evaluation. | 1,416 stars · 215 forks · 1 mention | No one-command install · Source | |
| 20 | 98.141 | harbor-framework-terminal-bencheval · ai-agentsContributed by Sentinel | Measuring and evolving with the frontier of agent work | 588 stars · 434 forks · 13 mentions | No one-command install · Source | |
| 21 | 97.923 | wandb-weave-evalseval · ai-agents | Weights & Biases Weave evaluation framework for tracking LLM experiments, scoring model outputs, and comparing runs. | 1,130 stars · 170 forks · 3 mentions | No one-command install · Source | |
| 22 | 97.880 | langtraceeval · observability | Open-source observability tool for LLMs with OpenTelemetry-based tracing, automated evals, and annotation workflows. | 1,228 stars · 127 forks | No one-command install · Source | |
| 23 | 97.541 | xiaowu0162-longmemevaleval · commsContributed by Sentinel | Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025) | 1,049 stars · 81 forks · 10 mentions | No one-command install · Source | |
| 24 | 97.111 | princeton-nlp-webshopeval · ai-agentsContributed by Sentinel | [NeurIPS 2022] 🛒WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents | 589 stars · 107 forks · 3 mentions | Stale | No one-command install · Source |
| 25 | 96.786 | stonybrooknlp-appworldeval · ai-agentsContributed by Sentinel | 🌍 AppWorld: A Controllable World of Apps and People for Benchmarking Function Calling and Interactive Coding Agent, ACL'24 Best Resource Paper. | 500 stars · 78 forks · 4 mentions | No one-command install · Source | |
| 26 | 96.207 | continuous-evaleval · ai-agents | Relari's modular evaluation library for LLM pipelines with deterministic + LLM-based metrics for RAG and agent workflows. | 517 stars · 38 forks | No one-command install · Source | |
| 27 | 94.339 | metr-task-standardeval · ai-agents | METR's Task Standard: a specification and scaffold for creating agentic tasks used in autonomous agent capability evaluations. | 192 stars · 37 forks | No one-command install · Source | |
| 28 | 94.309 | harbor-framework-terminal-bench-2-1eval · otherContributed by Sentinel | Terminal-Bench 2.1 | 119 stars · 62 forks · 3 mentions | No one-command install · Source | |
| 29 | 90.554 | vellum-evalseval · observability | Vellum evaluation SDK for running LLM test suites with custom metrics, dataset pinning, and CI workflow integration. | 82 stars · 20 forks | No one-command install · Source | |
| 30 | 82.319 | braintrusteval · observability | Developer platform for logging, evaluating, and comparing LLM experiments with dataset versioning and scoring functions. | 27 stars · 12 forks | No one-command install · Source | |
| 31 | 80.342 | galileo-evaluateeval · observability | Galileo evaluation and observability SDK for detecting hallucinations, data errors, and model weaknesses in LLM pipelines. | 22 stars · 11 forks | No one-command install · Source | |
| 32 | 69.000 | humanloop-evalseval · ai-agents | Humanloop Python SDK with integrated evals, dataset versioning, and human + LLM judge scoring for production pipelines. | 12 stars · 3 forks | No one-command install · Source | |
| 33 | Unranked | agentbenchOurseval · observability | Use to put a number on harness quality — run an agent harness against a task set and get a score — so harness changes are validated by evidence, the… | No signals yet | No one-command install · Source | |
| 34 | Unranked | evals-cookbookseval · ai-agents | OpenAI Cookbook eval recipes: task-specific templates for summarization, QA, and classification evaluation using the Evals framework. | No signals yet | No commit datelisted | No one-command install · Source |
| 35 | Unranked | gaia-benchmarkeval · ai-agents | GAIA: benchmark of 466 real-world questions requiring multi-step reasoning, web browsing, and tool use for general AI assistants. | No signals yet | No commit datelisted | No one-command install · Source |
| 36 | Unranked | honeyhiveeval · auth | LLM evaluation and experimentation platform with session tracing, dataset management, and metric-based run comparison. | No signals yet | No commit datelisted | No one-command install · Source |
| 37 | Unranked | patronus-aieval · ai-agents | Automated LLM evaluation and hallucination detection platform with a Python SDK and judge-model scoring. | No signals yet | No commit datelisted | No one-command install · Source |
Score colour shows how many signals stand behind it, never how good it is: amber, three or more; dimmer amber, two; grey, one. The Evidence column names them.
Stars, forks and last commit are as GitHub reported them when Armory last read each repository: for most, or later. A repository may have changed since.