Terraform

The Small Model Infrastructure Nobody Built (So We Did) — Filip Makraduli, Superlinked

May 5, 2026 · 18:30

Filip Makraduli of Superlinked introduces SAI, an open-source inference engine for small models that addresses gaps in embedding infrastructure by enabling dynamic model loading, hot-swapping, and memory-aware eviction on a single GPU. He argues that provisioning separate GPUs for each small model wastes idle capacity, and that the real challenge lies in supporting diverse model architectures (e.g., BERT, Qwen, Colbert) with different attention mechanisms and positional embeddings. The engine re-implements forward passes with variable-length FlashAttention and handles model swapping via a least recently used eviction policy. Makraduli also explains that context management for agents requires small models to pre-process data, referencing Andrej Karpathy’s graph-based knowledge bases and Chroma’s own model. The talk details the infrastructure layer including routing, auto-scaling with Prometheus, and GPU provisioning using spot instances, all open-sourced as SAI (Superlinked Inference Engine) with Helm charts and Docker images.

Identity for AI Agents - Patrick Riley & Carlos Galan, Auth0

Jan 14, 2026 · 1:22:12

Patrick Riley and Carlos Galan from Auth0 (Okta) present new identity and access management features for AI agents, including Async Auth, Token Vault, and MCP integration. Token Vault securely stores and refreshes upstream resource tokens, enabling agents to access APIs on behalf of users. Async Auth allows agents to request user approval for risky operations, such as placing an order, via out-of-band notifications like push to the Guardian app. They demonstrate a Next.js chatbot that uses Token Vault to read a user's stock portfolio and Async Auth to require explicit consent before executing trades. The MCP server is secured with OAuth 2.0, supporting Dynamic Client Registration (DCR) and fine-grained scopes. The solution bridges user identity, agent identity, and upstream API authorization, ensuring agents can act autonomously but with user-defined boundaries.

Software Development Agents: What Works and What Doesn't - Robert Brennan, OpenHands

Jul 25, 2025 · 16:46

Robert Brennan, creator of the open-source coding agent OpenHands (formerly OpenDevin), argues that while coding is going away, software engineering remains about critical thinking, not typing. He explains that agents like OpenHands operate via a loop between an LLM and tools like a diff-based editor, terminal, and web browser, all sandboxed in Docker. Brennan advises starting small with rote tasks like merge conflicts (99% success) or PR feedback, being specific about frameworks and files, and always reviewing AI-generated code to avoid tech debt. He shares that 90% of his code now goes through the agent, with 10% requiring manual intervention, and warns against auto-merging without human accountability—OpenHands now assigns PRs to the human who triggered them. Key use cases include fixing failing tests, infrastructure changes, database migrations, and greenfield internal apps where vibecoding is acceptable.

Case Study + Deep Dive: Telemedicine Support Agents with LangGraph/MCP - Dan Mason

Jun 22, 2025 · 1:56:13

Dan Mason of Stride presents a case study on building autonomous agents for telemedicine support, replacing a human-driven button-pushing workflow with LangGraph, Claude, MCP, and a Node.js/React/MongoDB stack, achieving roughly 10X capacity increase. The LLM-driven virtual operations associate ("Ava") assesses patient messages, updates state with anchors and scheduled messages, and passes proposals to an evaluator agent that scores confidence and complexity, escalating to humans below 75% confidence. Mason explains how treatment "blueprints" in Google Docs are read directly by the LLM instead of using RAG, enabling new treatments to be added without writing code. He details the eval system using LLM-as-a-judge via PromptFu and retry logic for tool-calling errors, and discusses trade-offs around confidence scoring, prompt caching, and model selection (Claude for steerability). The team includes two software engineers, one designer, and Mason, who wrote most LangGraph code with AI assistance (Kline).