Browse

plannotator

Interactive plan review UI that intercepts ExitPlanMode via hooks, letting users visually annotate plans with comments, deletions, and replacements before approving or denying with detailed feedback.

swe-bench

SWE-bench: benchmark for evaluating LLMs on real-world GitHub issue resolution across 12 popular Python repositories.

giskard

Open-source LLM testing framework for detecting vulnerabilities (prompt injection, hallucinations, bias) via automated scan.

getzep-zep

Examples, framework integrations and tools for Zep Cloud, Zep's hosted agent memory service; the repository says it is not the product itself. The open-source knowledge-graph engine behind Zep is Graphiti (getzep-graphiti).

Contributed by Sentinel

agenta

Open-source LLM developer platform with prompt playground, evaluation pipelines, and A/B testing for iterating on LLM apps.

openai-simple-evals

OpenAI's lightweight benchmark suite (MMLU, HumanEval, MATH, GPQA, MGSM) for fast model capability comparisons.

caviraoss-openmemory

Local persistent memory store for LLM applications including claude desktop, github copilot, codex, antigravity, etc.

Contributed by Sentinel

big-bench

Google's Beyond the Imitation Game benchmark: 200+ diverse tasks designed to probe capabilities beyond standard NLP benchmarks.

flowelement-m-flow

Use when recall should follow associations between memories rather than nearest-neighbour similarity alone.

trulens

Evaluation and tracking for LLM and RAG applications with a feedback-function API and experiment dashboard.

xlang-ai-osworld

[NeurIPS 2024] OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments

Contributed by Sentinel

inspect-ai

UK AISI's framework for safety evaluations of large language models, with task-based scaffolding and solver pipelines.

mirix-ai-mirix

Use when what the agent should remember is what actually happened on screen, consolidated into structured memories.

helm

Stanford CRFM Holistic Evaluation of Language Models: standardized benchmark suite covering accuracy, calibration, robustness, and fairness.

lighteval

Hugging Face lightweight evaluation library for LLMs across academic benchmarks, with fast local and remote inference support.

kayba-ai-agentic-context-engine

Use when an agent should carry forward what it learned from its own successes and failures into later runs.

memodb-io-memobase

User Profile-Based Long-Term Memory for AI Chatbot Applications.

Contributed by Sentinel

zilliztech-memsearch

Use when several coding agents should share one memory store instead of each keeping its own notes.

kingjulio8238-memary

Use when you want a well-known reference implementation of agent memory over a knowledge graph to read or fork.

tdd-guard

A hooks-driven system that monitors file operations in real-time and blocks changes that violate TDD principles.

redplanethq-core

Use when one memory graph should serve Claude Code, Codex and your other assistants at once.

evalplus

Rigorous code generation evaluation framework built on top of HumanEval and MBPP with 80x more test cases.

webarena

WebArena: realistic web-based environment for evaluating autonomous agents on long-horizon browser interaction tasks.

langchain-ai-langmem

Long-term memory for agents: tools that extract what matters from conversations, refine prompts from feedback and keep memory across sessions, with LangGraph's store built in.

Contributed by Sentinel