Browse

xiaowu0162-longmemeval

Benchmarking Chat Assistants on Long-Term Interactive Memory (ICLR 2025)

Contributed by Sentinel

princeton-nlp-webshop

[NeurIPS 2022] 🛒WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents

Contributed by Sentinel

modal-labs-modal-client

Modal — serverless GPU/CPU containers for running agents, sandboxes and model inference from Python (this is the client SDK).

Contributed by Sentinel

stonybrooknlp-appworld

🌍 AppWorld: A Controllable World of Apps and People for Benchmarking Function Calling and Interactive Coding Agent, ACL'24 Best Resource Paper.

Contributed by Sentinel

continuous-eval

Relari's modular evaluation library for LLM pipelines with deterministic + LLM-based metrics for RAG and agent workflows.

metr-task-standard

METR's Task Standard: a specification and scaffold for creating agentic tasks used in autonomous agent capability evaluations.

harbor-framework-terminal-bench-2-1

Terminal-Bench 2.1

Contributed by Sentinel

vellum-evals

Vellum evaluation SDK for running LLM test suites with custom metrics, dataset pinning, and CI workflow integration.

braintrust

Developer platform for logging, evaluating, and comparing LLM experiments with dataset versioning and scoring functions.

galileo-evaluate

Galileo evaluation and observability SDK for detecting hallucinations, data errors, and model weaknesses in LLM pipelines.

e2b

Open-source secure cloud sandboxes (Firecracker microVMs) for running AI-generated code. ~150ms cold start.

modal

Serverless cloud platform for running Python functions, containers, and AI workloads with zero infra management.

railway

Zero-config cloud platform for deploying agent backends, databases, and services from a Git push.

vercel

Frontend cloud platform with serverless functions and AI SDK integrations for deploying agent-facing UIs.

humanloop-evals

Humanloop Python SDK with integrated evals, dataset versioning, and human + LLM judge scoring for production pipelines.

agentbench

Use to put a number on harness quality: run an agent harness against a task set and get a score, so harness changes are validated by evidence. It is the eval backbone of a self-improving loop.

aws-lambda

Serverless function-as-a-service platform for event-driven agent compute without provisioning servers.

beta9

Open-source serverless GPU container runtime for running AI workloads with fast cold-starts on bare-metal.

blaxel

Cloud runtime and control plane for deploying, scaling, and observing production AI agent workloads.

cloudflare-sandboxes

Isolated V8 sandbox environments for multi-tenant agent workloads on top of Cloudflare Workers.

cloudflare-workers

Serverless edge-compute platform for deploying agent functions and MCP servers at the network edge.

coder

Self-hosted remote development environment platform for provisioning agent dev workspaces on any cloud.

deno-deploy

Edge serverless runtime for deploying TypeScript agent workers globally with zero config and V8 isolation.

evals-cookbooks

OpenAI Cookbook eval recipes: task-specific templates for summarization, QA, and classification evaluation using the Evals framework.