The hardest part of working with LLMs is getting consistent results. You get a prompt working across a set of tasks, add another one, tweak the prompt slightly, and suddenly one of the other tasks stops working. A tool call gets missed. The structured output comes back wrong.
Hopefully you noticed. But without evals, you’re flying blind.
This isn’t actually a new problem. The non-deterministic part is, sure, but testing inputs against expected outputs goes back to the early days of computing. What’s different with AI is:
The inputs are unstructured, and you’ll never test every permutation.
The output is also unstructured, and you need a way to quantify “success.”
And that’s just for the green/red testing. What about how many tokens you burned getting there? Benchmarking used to be about latency. Now it’s about cost.
The industry is starting to figure out the best practices, and the open source community is stepping up. I want to focus the start of this Commit Log on those projects and teams.
Let’s dive in.
Test in production
As agent workloads move to production, “does this work?” gets a lot harder to answer. Teams are quietly building the CI layer for agents: frameworks that run suites of tasks, grade outputs deterministically, and tell you when something went wrong.
ragas (13.6k stars, +25 this week). One of the most widely-deployed LLM evaluation library on this list. It started out focused on RAG quality, then grew into a full eval platform covering agents, pipelines, and custom metrics. ragas.io
langfuse (25.3k stars, +62 this week). The observability layer that includes evals. Langfuse sits between your agent and your datastore, logging every trace, then letting you run eval datasets over them. langfuse.com
UKGovernmentBEIS/inspect_ai (1.9k stars, +5 this week). The UK AI Safety Institute’s evaluation framework for LLMs. Task-based, model-agnostic, and built on the idea that evals should be reproducible artifacts rather than ad-hoc notebooks. inspect.aisi.org.uk
opik (19k stars, +31 this week). Debug, evaluate, and monitor LLM applications, RAG systems, and agentic workflows.
vllm-project/guidellm (1k stars, +8 this week). Evaluation focused on the deployment side: latency, throughput, cost per token across different serving configurations. Most eval frameworks ask “did the model get it right?” GuideLLM asks “can you actually afford to run it, and how fast?”
skillsbench (1k stars, +8 this week). Evaluates how well agent skills work and how effectively agents use them. Most evals test the model in isolation; SkillsBench tests the model-plus-tool system, which is what actually fails in production. Written in PDDL, which is either a red flag or the right move depending on how seriously you take formal task specification. skillsbench.ai
SanityHarness (216 stars, +3 this week). Lightweight, universal harness for evaluating coding agents across a broad set of tasks and languages. sanityboard.lr7.dev
ClawBench (70 stars, +3 this week). Benchmark for browser AI agents: 153 everyday tasks across 144 live websites. Uses a 5-layer recording stack, DOM-match scoring, and an LLM judge. claw-bench.com
terminal-bench (2k stars, +7 this week). Benchmark for LLMs on complicated terminal tasks. The terminal is where coding agents spend most of their time, and surprisingly few benchmarks focus there. tbench.ai
ParseBench (281 stars, +25 this week). Document parsing benchmark for AI agents, from the LlamaIndex team. Backed by an arxiv paper.
The token wars
It feels like there are now more repos dedicated to not spending tokens than there are dedicated to spending them well. The compression niche has exploded, and the results range from clever to absurd.
So essentially, AI coding tools have gotten good enough that the bottleneck has shifted. It’s no longer “can the model do this?” It’s “can I afford to keep asking?”
claude-mem (63k stars, +2,008 this week). The week’s star. A Claude Code plugin that captures everything Claude does during a session, compresses it with AI, and injects relevant context back into future sessions. The thesis: your AI shouldn’t have to re-learn your codebase every Monday morning. 63k stars suggests a lot of people agree. claude-mem.ai
caveman (39k stars, +2,133 this week). “Why use many token when few token do trick.” A Claude Code skill that cuts 65% of tokens by making Claude respond in caveman-speak. Technically this is satire. Technically it also works. getcaveman.dev
rtk (30k stars, +810 this week). CLI proxy that sits between your terminal and your LLM, reducing token consumption by 60-90% on common dev commands. Single Rust binary, zero dependencies. rtk-ai.app
context-mode (8k stars, +184 this week). Context window optimization for AI coding agents. Sandboxes tool output. Claims 98% reduction. context-mode.com
zilliztech/claude-context (6k stars). Code search MCP for Claude Code. Make your entire codebase the context for any coding agent. From Zilliz, who make Milvus. This one’s going the other direction: not compressing, but making the context actually relevant instead of just big.
token-savior (468 stars, +40 this week). MCP server claiming 97% token savings on code navigation, plus persistent memory across sessions. 69 tools, zero external deps.
sqz (98 stars, +21 this week). Just: compress LLM context to save tokens. Written in Rust.
engram (92 stars, +14 this week). The context spine for AI coding agents. 8 providers, 88% claimed token savings, 5 IDE integrations, hook-based Read interception, tree-sitter AST, local SQLite. Technically the most ambitious of the small-but-interesting entries.
EverMind-AI/MSA (3.2k stars). Memory Sparse Attention. A scalable, trainable latent-memory framework for 100M-token contexts. This is the research answer to the same problem: instead of compressing input, build models that don’t need you to.
The plumbing nobody talks about
Agents need a floor under them. Not just models and prompts, but actual infrastructure too.
manifest (5.5k stars, +233 this week). Smart model routing for personal AI agents, claiming up to 70% cost reduction. manifest.build
bifrost (4.1k stars, +65 this week). Enterprise AI gateway in Go. Claims 50x faster than LiteLLM, sub-100µs overhead at 5k RPS, adaptive load balancing, cluster mode, guardrails, 1000+ models. getmaxim.ai
TencentCloud/CubeSandbox (1.3k stars, +72 this week). Instant, concurrent, secure, lightweight sandbox for AI agents, written in Rust. cubesandbox.ai
libkrun (2k stars, +46 this week). A dynamic library providing virtualization-based process isolation, also in Rust.
lucebox-hub (563 stars, +45 this week). Hand-tuned LLM inference built for specific consumer hardware, written in C++. lucebox.com
Who’s watching the agents?
codeburn (3.3k stars, +102 this week). Interactive TUI dashboard showing exactly where your AI coding tokens go across Claude Code, Codex, and Cursor. Most developers have no idea how their token costs are distributed until they see a chart.
Anthropic-Cybersecurity-Skills (5.5k stars, +160 this week). 754 structured cybersecurity skills for AI agents, mapped to MITRE ATT&CK, NIST CSF 2.0, MITRE ATLAS, D3FEND, and NIST AI RMF. This is the offense-and-defense reference library for anyone building agents that touch anything security-adjacent.
agentshield (525 stars, +31 this week). AI agent security scanner: detect vulnerabilities in agent configurations, MCP servers, and tool permissions. Available as a CLI, GitHub Action, ECC plugin, and GitHub App. MCP server configs are the new .env file people paste into places they shouldn’t.
ai-attestation (75 stars, +10 this week). Track AI-generated code in your repository. Open standard, CC0 spec, detects 19 AI coding tools. Trying to answer the increasingly futile question of - “what percentage of this codebase did a human actually write?”
Agents doing actual work
swarms (6.4k stars, +25 this week). “Enterprise-grade” multi-agent orchestration. swarms.ai
gepa (3.9k stars, +50 this week). Optimize prompts, code, and more with AI-powered “Reflective Text Evolution”. The prompt tuning loop that everyone does manually and nobody wants to admit takes as long as it does.
ouroboros (2.6k stars, +31 this week). “Stop prompting. Start specifying.” Move from prompt-as-instruction to spec-as-source. One to watch
AnyCrawl (3k stars, +21 this week). Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google, Bing, Baidu, and others. anycrawl.dev
ElatoAI (1.6k stars, +23 this week). Realtime voice AI with 100+ models running on Arduino ESP32 over secure websockets and edge functions. elatoai.com
council-of-high-intelligence (432 stars, +45 this week). 18 AI personas (Aristotle, Feynman, Kahneman, Torvalds, and 14 others) deliberate your hardest decisions across multiple LLM providers in structured multi-round discussion. Whether it produces better decisions is unverified. Whether it produces more entertaining ones seems assured.
Not everything is agents
Finally, just other great stuff to keep an eye on.
perry (1.3k stars, +194 this week). A native TypeScript compiler written in Rust that compiles TypeScript directly to executables using SWC and LLVM. perryts.com
mini-tokyo-3d (4k stars, +52 this week). A real-time 3D digital map of Tokyo’s public transit system in JavaScript. minitokyo3d.com
TablePro (2.2k stars, +17 this week). Native macOS database client for MySQL, PostgreSQL, SQLite, MongoDB, Redis, and 15+ more. Free and open-source. tablepro.app
ascii-studio (934 stars, +25 this week). Turn videos into ASCII frames: real character-based frames that run in sequence. asciistudio.space