Leaderboard
Scored on public signals; components with none are listed as Unranked · Formula
Component
Domain
Vertical
| Rank | Score | Component | Description | Evidence | Last commit | Install |
|---|---|---|---|---|---|---|
| 1 | 99.536 | promptfooeval · ai-agents | CLI and library for testing, comparing, and red-teaming LLM prompts and agents with assertions and CI integration. | 24,737 stars · 2,255 forks | No one-command install · Source | |
| 2 | 99.512 | openai-evalseval · ai-agents | OpenAI's official framework for evaluating LLMs and LLM-powered systems, with a registry of community eval sets. | 19,509 stars · 3,093 forks | No one-command install · Source | |
| 3 | 99.273 | phoenixeval · ai-agents | Arize Phoenix: open-source LLM observability with built-in evals, span tracing, and dataset curation for RAG and agents. | 11,286 stars · 1,086 forks · 2 mentions | No one-command install · Source | |
| 4 | 99.103 | swe-bencheval · ai-agents | SWE-bench: benchmark for evaluating LLMs on real-world GitHub issue resolution across 12 popular Python repositories. | 5,762 stars · 957 forks · 9 mentions | No one-command install · Source | |
| 5 | 98.991 | agentaeval · ai-agents | Open-source LLM developer platform with prompt playground, evaluation pipelines, and A/B testing for iterating on LLM apps. | 4,670 stars · 661 forks | No one-command install · Source | |
| 6 | 98.688 | lightevaleval · ai-agents | Hugging Face lightweight evaluation library for LLMs across academic benchmarks, with fast local and remote inference support. | 2,533 stars · 553 forks | No one-command install · Source | |
| 7 | 97.923 | wandb-weave-evalseval · ai-agents | Weights & Biases Weave evaluation framework for tracking LLM experiments, scoring model outputs, and comparing runs. | 1,130 stars · 170 forks · 3 mentions | No one-command install · Source | |
| 8 | 96.207 | continuous-evaleval · ai-agents | Relari's modular evaluation library for LLM pipelines with deterministic + LLM-based metrics for RAG and agent workflows. | 517 stars · 38 forks | No one-command install · Source | |
| 9 | 69.000 | humanloop-evalseval · ai-agents | Humanloop Python SDK with integrated evals, dataset versioning, and human + LLM judge scoring for production pipelines. | 12 stars · 3 forks | No one-command install · Source | |
| 10 | Unranked | patronus-aieval · ai-agents | Automated LLM evaluation and hallucination detection platform with a Python SDK and judge-model scoring. | No signals yet | No commit datelisted | No one-command install · Source |
Score colour shows how many signals stand behind it, never how good it is: amber, three or more; dimmer amber, two; grey, one. The Evidence column names them.
Stars, forks and last commit are as GitHub reported them when Armory last read each repository: for most, or later. A repository may have changed since.