HELM

The Art & Science of Benchmarking Agents — Vincent Chen, Snorkel AI

Jun 4, 2026 · 23:25

Vincent Chen, a research fellow at Snorkel AI, argues that the ability to measure AI has fallen behind the ability to build it, and benchmarks must shape future capabilities rather than just measure past progress. Drawing from reviewing over 120 applications for Snorkel's $3 million Open Benchmarks Grants, he presents a framework: the science of task quality, distributional diversity, model headroom, and robust eval methodology, and the art of having a thesis (e.g., Terminal Bench's bet on CLI before coding agents made it obvious), producing research roadmaps, and treating researcher UX as a first-class citizen. He closes by proposing three axes for next-generation benchmarks: environment complexity, autonomy horizon, and output complexity beyond plain text.

Scaling Agents on Kubernetes with acpx and ACP — Onur Solmaz, OpenClaw

May 21, 2026 · 19:00

Onur Solmaz, a founding engineer at textcortex and OpenClaw maintainer, built acpx, a headless CLI for the Agent Client Protocol (ACP), to automate processing of 300–500 daily AI-generated pull requests on OpenClaw. He replaces manual PTY scraping with structured agent-to-client communication, driving Codex sessions through a node-based workflow: reproduce the bug, judge the implementation, check for conflicts, run a review loop, and emit structured JSON. Solmaz runs parallel Codex sessions from five Discord channels while traveling, one channel per task. He also presents an open-source Kubernetes operator called Spritz (from textcortex) that provisions disposable full-compute pods per task, wires them into Slack, and tears them down when work is done, advocating for on-demand disposable agents over persistent chat apps.

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

May 5, 2026 · 18:30

Filip Makraduli of Superlinked introduces SAI, an open-source inference engine for small models that addresses gaps in embedding infrastructure by enabling dynamic model loading, hot-swapping, and memory-aware eviction on a single GPU. He argues that provisioning separate GPUs for each small model wastes idle capacity, and that the real challenge lies in supporting diverse model architectures (e.g., BERT, Qwen, Colbert) with different attention mechanisms and positional embeddings. The engine re-implements forward passes with variable-length FlashAttention and handles model swapping via a least recently used eviction policy. Makraduli also explains that context management for agents requires small models to pre-process data, referencing Andrej Karpathy’s graph-based knowledge bases and Chroma’s own model. The talk details the infrastructure layer including routing, auto-scaling with Prometheus, and GPU provisioning using spot instances, all open-sourced as SAI (Superlinked Inference Engine) with Helm charts and Docker images.