Leaderboard
Scored on public signals; components with none are listed as Unranked · Formula
Component
Domain
Vertical
| Rank | Score | Component | Description | Evidence | Last commit | Install |
|---|---|---|---|---|---|---|
| 1 | 69.000 | humanloop-evalseval · ai-agents | Humanloop Python SDK with integrated evals, dataset versioning, and human + LLM judge scoring for production pipelines. | 12 stars · 3 forks | No one-command install · Source | |
| 2 | 80.342 | galileo-evaluateeval · observability | Galileo evaluation and observability SDK for detecting hallucinations, data errors, and model weaknesses in LLM pipelines. | 22 stars · 11 forks | No one-command install · Source | |
| 3 | 82.319 | braintrusteval · observability | Developer platform for logging, evaluating, and comparing LLM experiments with dataset versioning and scoring functions. | 27 stars · 12 forks | No one-command install · Source | |
| 4 | 90.554 | vellum-evalseval · observability | Vellum evaluation SDK for running LLM test suites with custom metrics, dataset pinning, and CI workflow integration. | 82 stars · 20 forks | No one-command install · Source | |
| 5 | 96.207 | continuous-evaleval · ai-agents | Relari's modular evaluation library for LLM pipelines with deterministic + LLM-based metrics for RAG and agent workflows. | 517 stars · 38 forks | No one-command install · Source | |
| 6 | 97.880 | langtraceeval · observability | Open-source observability tool for LLMs with OpenTelemetry-based tracing, automated evals, and annotation workflows. | 1,228 stars · 127 forks | No one-command install · Source | |
| 7 | 97.923 | wandb-weave-evalseval · ai-agents | Weights & Biases Weave evaluation framework for tracking LLM experiments, scoring model outputs, and comparing runs. | 1,130 stars · 170 forks · 3 mentions | No one-command install · Source | |
| 8 | 98.688 | lightevaleval · ai-agents | Hugging Face lightweight evaluation library for LLMs across academic benchmarks, with fast local and remote inference support. | 2,533 stars · 553 forks | No one-command install · Source | |
| 9 | 98.783 | trulenseval · back-end | Evaluation and tracking for LLM and RAG applications with a feedback-function API and experiment dashboard. | 3,530 stars · 335 forks · 1 mention | No one-command install · Source | |
| 10 | 98.991 | agentaeval · ai-agents | Open-source LLM developer platform with prompt playground, evaluation pipelines, and A/B testing for iterating on LLM apps. | 4,670 stars · 661 forks | No one-command install · Source | |
| 11 | 99.103 | swe-bencheval · ai-agents | SWE-bench: benchmark for evaluating LLMs on real-world GitHub issue resolution across 12 popular Python repositories. | 5,762 stars · 957 forks · 9 mentions | No one-command install · Source | |
| 12 | 99.273 | phoenixeval · ai-agents | Arize Phoenix: open-source LLM observability with built-in evals, span tracing, and dataset curation for RAG and agents. | 11,286 stars · 1,086 forks · 2 mentions | No one-command install · Source | |
| 13 | 99.455 | deepevaleval · observability | Open-source LLM evaluation framework with 14+ metrics (hallucination, faithfulness, answer relevancy) and CI support. | 18,041 stars · 1,890 forks | No one-command install · Source | |
| 14 | 99.512 | openai-evalseval · ai-agents | OpenAI's official framework for evaluating LLMs and LLM-powered systems, with a registry of community eval sets. | 19,509 stars · 3,093 forks | No one-command install · Source | |
| 15 | 99.536 | promptfooeval · ai-agents | CLI and library for testing, comparing, and red-teaming LLM prompts and agents with assertions and CI integration. | 24,737 stars · 2,255 forks | No one-command install · Source | |
| 16 | Unranked | honeyhiveeval · observability | LLM evaluation and experimentation platform with session tracing, dataset management, and metric-based run comparison. | No signals yet | No commit datelisted | No one-command install · Source |
| 17 | Unranked | patronus-aieval · ai-agents | Automated LLM evaluation and hallucination detection platform with a Python SDK and judge-model scoring. | No signals yet | No commit datelisted | No one-command install · Source |
Score colour shows how many signals stand behind it, never how good it is: amber, three or more; dimmer amber, two; grey, one. The Evidence column names them.
Stars, forks and last commit are as GitHub reported them when Armory last read each repository: for most, or later. A repository may have changed since.