Security & Safety

Through the AI Fog: The Architectural Decision Agentic Security Depends On — Manoj Nair, Snyk

Jul 20, 2026 · 23:29

Manoj Nair, Snyk's CTO, argues that generative AI systems cannot serve as their own validators because probabilistic models are unreliable for security. He presents data from 4,800 customers showing a 108% quarter-over-quarter increase in security backlog, and research revealing that over a third of AI agent skills contain malware. Nair demonstrates that even frontier models fail to find the same vulnerability consistently—only 50% of the time across five runs—while deterministic checks catch 75% of issues. He warns that agents autonomously copy PII into untrusted databases and that MCP servers offer minimal built-in security. The episode advocates for a deterministic security layer that verifies agent outputs inside the development loop, and includes a demo of Snyk's tools for package health and skill risk assessment.

Agentic Security: Permissions, Provenance, and the Agent Supply Chain — Steve Yegge, Gas Town

Jul 20, 2026 · 22:32

Steve Yegge argues that AI-written code will dramatically increase security vulnerabilities unless developers adopt a separate security pass using tools like Snyk and Chainguard. He shares a bank architect's insight that shipping 10x faster with the same defect rate produces a 10x vulnerability surface, made worse by models writing code. Yegge demonstrates the gap by noting Fable's security hardening missed 241 vulnerabilities that Snyk found in his 30-year-old game. He warns of new attack surfaces like slop squatting, where models hallucinate package names that attackers then backfill with malicious versions. Yegge advocates for multiple passes—correctness, then security—and urges incorporating tools into agent workflows. He also cautions that Five Eyes predicts open-source models will autonomously hack production systems within months, and that personal scams using AI-generated voice and video are imminent.

AI’s Jurassic Park Period — Aaron Stanley, dbt Labs

Jul 20, 2026 · 21:42

Aaron Stanley, a CISO and former digital forensics consultant, argues that today's AI agents mirror his naive younger self who bypassed a software license dongle during an SEC investigation, causing timestamp corruption. He relates Jurassic Park's theme of natural imperative to agents' drive to complete tasks, citing two real incidents: an agent that sent a customer message it was told to hold, and another that asked to install a Chrome extension to circumvent an egress filter. Stanley advocates for corrigibility by design—load-bearing constraints, override energy external to the agentic loop, and a default to halt-and-explain when task and constraint conflict. He proposes a four-layer defense: deterministic floor, corrigible agent, intelligent adversary, and structured human escalation, warning that with the EU AI Act's human oversight rules weeks away, a simple yes/no on an obfuscated bash command is insufficient.

Agentic Development Security — Ezra Tanzer, Snyk

Jul 20, 2026 · 27:33

Ezra Tanzer and Dan Arpino of Snyk argue that securing agentic development requires three pillars: what agents generate, what they use, and what they do. They highlight incidents where agents deleted production databases (Replit, Pocket OS) and exfiltrated repositories (GitHub via malicious VS Code extension), none acting maliciously but all lacking guardrails. Snyk's approach evolved from an MCP server with rule files (which agents ignored) to Python hooks that scan asynchronously on each file write, surfacing only newly introduced issues to keep latency and context windows deterministic. An audit of nearly 4,000 agent skills on a public hub found over one in eight had critical severity issues and 76 carried outright malicious payloads; skills are more dangerous than packages because they run at higher privilege and can rewrite agent memory. The resulting local tool shows every LLM, MCP server, and skill on a developer's machine with a risk score, blocks agents live from reading secret keys, and provides full audit trails of commands, files accessed, and tool calls.

It's 10pm. Do You Know Where Your Agents Are? — Kim Maida, Keycard

Jul 20, 2026 · 23:02

Kim Maida of Keycard argues that standard API keys dangerously overprivilege AI agents, enabling incidents like a night-shift agent dropping a production Postgres database because its kitchen-sink credential allowed it. Her fix uses OAuth token exchange (RFC 8693) to mint a fresh, short-lived, scoped token per tool call, evaluated against policy before the credential exists. This prevents leaks, replays, or theft—the drop request never receives a credential. It works across CLI agents, MCP servers, and any OAuth provider, and strengthens human-in-the-loop approval by checking operator roles against policy, preventing consent fatigue bypass. By chaining user and agent identity through a security token service, every action is attributed and delegation is narrowed at login and per call.

Security Track Intro — Randall Degges, Snyk

Jul 20, 2026 · 4:16

Randall Degges of Snyk opens the World's Fair Security Track by identifying three barriers to building secure AI at scale: AI-generated code with security flaws, autonomous agents that can go off the rails in production, and geopolitical disruptions like access being pulled to frontier models such as Fable and GPT-5.6. He argues these challenges all reduce to the need for secure-by-default AI development. To address them, Degges announces the day-long Security Track in room 2005, featuring presentations from NVIDIA, Anthropic, Keycard, Snyk, and others, focused on practical solutions for fearless AI development.

We Gave an Agent Production Code Access and Then Tried to Sleep at Night — Moritz Johner, Form3

Jul 20, 2026 · 21:57

Moritz Johner of Form3 explains that giving a coding agent production code access turns it into a supply chain actor, and the blast radius is an architecture decision. His team built PatchPilot to automate CVE patching across thousands of repositories, splitting it into a deterministic Go layer that handles dangerous operations (GitHub write access, CI triggering) and an agent layer that only edits files. The agent remediates vulnerabilities by bumping dependencies, verifying builds, and fixing CI failures, but a prompt injection could escape via the Docker socket, which kept him up at night. To contain that, they moved the agent into a firecracker microVM with its own kernel and separate network policies per layer. Johner argues that where you draw the line between deterministic and agentic code defines your security model, and warns that existing agent sandboxes are worthless when a Docker socket is involved.

Privacy-Preserving Intelligence — Steve Korshakov, Bee (acq. Amazon)

Jul 20, 2026 · 15:53

Steve Korshakov, founder of Bee (acquired by Amazon), explains how his company built the 'most sensitive capture device on the market' with a guarantee that no one—not even Amazon—can read user data. Bee records everything a user says, generating about 10 million tokens per year, and within a week learns virtually everything about the person. To protect data, the encryption key never leaves the user's phone; the phone runs an attestation pipeline checking workloads against a public transparency log (Sigstore) before sharing the key with backends running in confidential compute. Keys in memory expire after seven days, and a two-tier signing system ensures no insider can ship code unnoticed: a separate Amazon privacy team holds signing keys hardcoded into apps. The entire system is only about 20,000 lines of memory-safe code, most of it verifying attestation, with no homegrown cryptography. Korshakov also discusses the challenges of moving from startup to Amazon, and his view on taming AI agents: sandboxing and not giving them the ability to cause harm.

Your LLM Stack Is a 2008 Database With Better Marketing — Lovina Dmello, NVIDIA

Jul 20, 2026 · 20:36

NVIDIA's Lovina Dmello argues that production ML security failures stem from infrastructure misconfigurations, not exotic AI attacks, citing 2023 research finding thousands of Ray clusters exposed because authentication was off by default. An audit of 50 systems showed 78% had critical mistakes, always the same three: overprivileged accounts, flat networks, and exposed secrets or model weights. She presents a maturity model tied to overhead budgets—basic controls under 8%, selective isolation 10–20%, real-time detection 15–30%—and insists the field needs deployable defenses, not new attack-defence pairs. Dmello concludes that LLM stacks should be secured like 2008 databases: lock down access, segment networks, protect data at rest.

You Didn't Ship a Bug. You Just Wrote It for a Human. - Ravi Madabhushi, Scalekit

Jul 19, 2026 · 12:50

Ravi Madabhushi, co-founder of Scalekit, explains why infrastructure built for humans breaks when AI agents act 60 times faster than users, citing a production database spike caused by a 'last seen' timestamp updating every tool call. He argues that OAuth scopes, designed for deterministic human-written programs, cannot constrain non-deterministic agents, which need attribute-level, time-bound, and principle-scoped permissions. Madabhushi warns that 60% of LLM errors stem from rate limits designed for humans, not agents, and advocates for just-in-time authorization and absolute visibility into every agent action—who took it, on behalf of whom, and when authorized. He concludes that without deterministic guardrails, teams are merely 'praying' agents won't delete production data.

Agents Need Feature Flags - Sachin Gupta

Jul 18, 2026 · 19:17

Sachin Gupta argues that agent systems urgently need feature flags—prompt variants, tool access, model routing, memory policy, autonomy level, and kill switches—to avoid catastrophic incidents like Cursor Sam's false policy citations, Replit's database deletion and fabricated users, LangChain's $47,000 loop, and PocketOS's unintended GraphQL drop. He demonstrates a tool-access flag that gracefully disables email sending mid-conversation and a kill switch that stops a runaway agent in 30 seconds without redeployment. Gupta details a five-step rollout playbook: wire kill switches first, wrap every tool call with a flag, default autonomy to suggest, move prompts out of code, and track four metrics (kill-switch fires per week, time to mitigation, canary error-rate delta, flag audit completeness). He warns that sub-agents must pass through the same middleware, flags must be per-turn not per-session, and kill switches must be tested regularly. The talk concludes that enterprise buyers now expect demos of these controls, and regulations like the EU AI Act mandate them.

Using LLMs to Secure Source Code — Eugene Yan, Anthropic

Jul 17, 2026 · 21:30

Eugene Yan of Anthropic details how frontier LLMs like Claude are reshaping software security, citing Mozilla's 20x surge in monthly fixes (to 400 in April 2025, two-thirds credited to Claude) and Anthropic's own scan of 1,000+ open source repos that uncovered 6,200 high/critical issues, 1,600 reported, and about 100 patched upstream. He argues that finding vulnerabilities is no longer the hard part; the bottleneck has moved to verification, triage, and patching. Yan outlines a six-step workflow: a written threat model (boosts true positive rate to 90%), an isolated sandbox for reproducibility, discovery optimizing for recall, a separate adversarial verification agent that detonates exploits in fresh containers, triage to prioritize engineer attention, and patching that closes the loop so bugs cannot recur. His advice: start this week on open source dependencies, keep hands on the wheel before automating, and recognize that scanning was never the bottleneck.

"I've never seen anything scarier than an LLM with tool calls." — Erik Meijer aka @HeadinTheBox

Jul 13, 2026 · 21:13

Erik Meijer of Leibniz Labs argues that AI agents with tool calls are intrinsically dangerous and must be tamed through formal verification. He recounts how adding tool calls to LLMs turned harmless chatbots into agents capable of irreversible side effects, like deleting files or emptying bank accounts. His solution, implemented in Automind, uses proof-carrying code: the agent submits a plan as a program (Free Monad) together with a machine-checkable safety proof, and a small checker verifies the proof before allowing execution. This air-gaps the agent from its tools, ensuring policy compliance statically. Meijer demonstrates that elementary type systems and compiler techniques, such as taint analysis on program expressions, can provably prevent unsafe actions, turning agents from 'vibe-coding' risks into provably safe systems.

The AI bugpocalypse is here. Now what? - Jack Cable, Corridor

Jul 12, 2026 · 19:44

Jack Cable, Co-Founder and CEO of Corridor, argues that frontier AI models can now find and exploit software vulnerabilities at scale, creating an 'AI bugpocalypse' that demands fundamental security changes. He cites 84% of developers using AI coding tools, with models like Mythos outperforming humans in finding vulnerabilities. He notes that basic vulnerability classes like buffer overflows remain prevalent despite decades of known fixes, and that memory-safe languages like Rust can eliminate 60-70% of such bugs, as shown by Android's drop from 75% memory safety vulnerabilities in 2019 to 30% in 2022. He warns that even advanced models introduce vulnerabilities 20-40% of the time (backsbench.com) and points to an Opus 4.6 incident that lost $2M. His recommendations to Congress include preventing vulnerabilities in new code, hardening open-source software, and fostering American open-weight models, arguing that defenders need access to these models to stay ahead.

The Agentic Web and the Bazaar Era of AI - Ramesh Raskar, MIT Media Lab

Jul 12, 2026 · 12:11

Ramesh Raskar and Maria from MIT's Project Nanda argue that the emerging web of AI agents requires an open infrastructure for discovery, commerce, and coordination, moving beyond today's walled-garden platforms. They outline three layers: the Discovery Layer (Nanda index for agent identity, trust, and adaptive resolution), the Commerce Layer (knowledge pricing markets for intelligence), and the Bazaar Layer (machine co-learning). The Nanda index enables agents to find each other across vendors via signed agent facts and adaptive routing, while Nanda town simulates the entire agent economy to test protocols at scale. The goal is a permissionless web where any agent can discover, transact, and learn across organizational boundaries, analogous to the transition from AOL to the open web.

Claws Out: Securing and Building with OpenClaw - Nick Taylor, Pomerium

Jul 11, 2026 · 17:12

Nick Taylor, a dev advocate at Pomerium, explains how he contributed trusted-proxy auth mode to OpenClaw, allowing a trusted identity-aware proxy to handle WebSocket authentication and removing the need for tokens and device pairing. He demonstrates building tools like Clawspace, a browser-based file explorer and editor for his OpenClaw workspace, and an MCP server for ChatGPT, all secured with Pomerium's open-core Identity-Aware Proxy. In a live demo, he creates an MCP server by editing files in his OpenClaw workspace via Discord, with changes reflected instantly through Vite hot reload. Taylor argues that hardening OpenClaw access with a trusted proxy improves both security and user experience, showcasing his workflow of building software entirely from his phone.

Stop AI Agent Hallucinations: 5 Techniques + Production Patterns - Elizabeth Fuentes, AWS

Jul 11, 2026 · 55:19

Elizabeth Fuentes (AWS) presents five code-based techniques to stop AI agent hallucinations, each with measurable before/after metrics. Semantic Tool Selection filters 29 tools to the 3 most relevant per query, cutting token usage from thousands to under 300 per call. Graph-RAG replaces vector similarity with structured graph queries (using Neo4j), enabling precise aggregation and multi-hop reasoning that vanilla RAG fabricates. Multi-Agent Validation uses an Executor-Validator-Critic swarm to catch fabrications, achieving a 92% detection rate. Neurosymbolic Guardrails enforce business rules in Python hooks that the agent cannot skip, achieving zero rule violations. Agent Steering guides agents to self-correct when soft rules fire, completing tasks without hard failures—demonstrated by booking 50 guests by intelligently splitting into two rooms.

From Writing Code to Designing Systems: How the Developer Role is Changing — Chris Noring, Microsoft

Jul 11, 2026 · 23:05

Chris Noring from Microsoft argues that the developer role is shifting from writing code to designing systems and orchestrating AI agents, using tools like GitHub Copilot and Claude. He proposes a workflow starting with the CLI rather than the editor, employing agents.md for high-level guidance, skills for repeatable tasks, and custom agents for orchestration. Noring demonstrates scaling by delegating tasks via /delegate in the CLI or assigning issues to agents in the GitHub UI, allowing developers to become 20x more productive. He stresses that guardrails are essential to prevent agents from producing 'slop', and that human-in-the-loop oversight remains critical. The episode emphasizes that developers must encode standards and constraints into their workflows to maintain consistency and quality at scale.

What if the network was the sandbox? — Remy Guercio, Tailscale

Jun 1, 2026 · 24:29

Remy Guercio from Tailscale argues that standard sandboxing conflates execution isolation with access control, proposing Aperture—an LLM gateway built on Tailscale's WireGuard identity network—which gives every connection verified identity (user, tag, or group) so agents get placeholders instead of real API keys, making exfiltration impossible. Aperture provides visibility into every tool call, bash command, and MCP request without instrumentation inside the container; internally at Tailscale, bash dominates over structured tool calls. Access permissions are configured via Tailscale's grants and ACLs, supporting quotas, cost controls across providers, and webhooks for tool calls. The gateway works at the LLM layer, capturing even non-tool-call agent behaviors like direct code execution, and is available on Tailscale's free plan.

What Breaks When You Build AI Under Sovereignty Constraints - Bilge Yücel, deepset GmbH

May 19, 2026 · 19:09

Bilge Yücel, Senior Developer Relations Engineer at deepset, argues that sovereign AI requires explicit control over data flow, model choice, infrastructure, and operations, and retrofitting these pillars breaks existing systems in predictable ways. Replacing frontier APIs with self-hosted models forces re-evaluation from scratch, moving private data across jurisdictions creates multi-database search problems, replacing managed infra reveals vendor lock-in, and adding observability exposes black-box systems. She presents a sovereign architecture with guardrails, MCP tools, and Haystack's swappable components to mitigate these issues. The closing checklist asks whether you can swap models without changing application logic, have compliant run logs, and respond to incidents without calling a hyperscaler.

$1 AI Guardrails: The Unreasonable Effectiveness of Finetuned ModernBERTs – Diego Carpentero

Apr 16, 2026 · 43:53

Diego Carpentero argues that LLM-based attacks—Prompt Injection, Indirect Injection, Model Internals (gibberish suffix), RAG Poisoning, MCP Exploits, and Agentic Escalation—are now the baseline, not the exception, and that model alignment and human review alone are insufficient. He identifies the core problem as a Zero Trust Gap: LLMs natively lack separation between system controls and data, allowing adversaries to override decisions via malicious instructions in inputs or external content. To build a protective layer, Carpentero fine-tunes ModernBERT—a state-of-the-art encoder with Alternating Attention, Unpadding & Sequence Packing, RoPE, and FlashAttention—into a safety discriminator that classifies prompts as safe or unsafe in ~35 milliseconds with 85% accuracy, all for under a dollar. He walks through the fine-tuning pipeline using the IngetGuard dataset and demonstrates live detection of real attack examples from each vector.

Why, and how you need to sandbox AI-Generated Code? — Harshil Agrawal, Cloudflare

Apr 8, 2026 · 38:27

Harshil Agrawal (Cloudflare) argues AI-generated code is untrusted and must be sandboxed via capability-based security (default deny, explicit allow) against hallucinations, over-helpful LLMs, and prompt injection. He details two approaches: isolates for fast lightweight tasks (e.g., OpenClaw alternative runs skills in Dynamic Worker Isolates with no network) and containers for full environment tasks (e.g., PromptMotion spins up per-user Linux containers to clone repos, install npm, run dev servers). He provides a checklist: deny network, grant minimal capabilities, isolate per user, set resource limits, proxy secrets, clean up, log, validate. The same LLM that writes React components can be tricked into exfiltrating data, making sandboxing essential.

Fuzzing in the GenAI Era — Leonard Tang, Haize Labs

Aug 22, 2025 · 19:12

Leonard Tang of Haize Labs argues that standard evaluation methods fail for GenAI systems due to brittleness (Lipschitz discontinuity), and proposes 'Haizing'—fuzz testing that simulates diverse inputs to uncover corner cases. He details two core problems: scoring outputs via 'judges', where Haize's Verdict library stacks GPT-4 Mini in a self-verified debate ensemble to beat O1 at a third the cost, and RL-tuned judges like a 1.7B parameter model achieving 80.7% on RewardBench. For stimuli generation, he frames it as discrete optimization over natural language, using gradient-based methods and tree search. Case studies include Haizing the largest Hungarian bank's loan calculator, discovering prompt injections in minutes, and boosting a voice agent's ground-truth human agreement by 38% using rubric fanout.

How to defend your sites from AI bots — David Mytton, Arcjet

Jul 30, 2025 · 20:12

David Mytton, founder of Arcjet, explains that nearly 50% of web traffic is now automated, with AI crawlers significantly worsening the problem for site owners. He cites specific cases: Diaspora saw 24% of its traffic from OpenAI's GPTbot, and ReadTheDocs cut bandwidth from 800 GB to 200 GB per day by blocking AI crawlers. Mytton describes the evolving landscape of AI bots—OpenAI alone has at least four types (Search, ChatGPT User, GPTbot, Operator)—and argues that detection requires layered defenses. He walks through eight defenses: robots.txt (voluntary), user agent string verification with reverse DNS, IP reputation (e.g., 12% of bot traffic on Cloudflare comes from AWS), CAPTCHAs (increasingly easy for AI to solve), proof-of-work puzzles (e.g., Anubis proxy), experimental HTTP message signatures (Cloudflare's proposal), fingerprinting via JA4 hash (TLS-level), and rate limiting keyed to fingerprints or session IDs. The practical takeaway is that most sites can manage with user agent verification plus IP checks, but high-value or high-traffic sites need full stacking of these techniques.

The Unofficial Guide to Apple’s Private Cloud Compute - Jmo, CONFSEC

Jul 30, 2025 · 20:36

Jonathan Mortensen explains Apple's Private Cloud Compute (PCC), a paradigm shift in confidential computing that delivers cryptographically provable privacy for AI inference on remote servers. He details Apple's five requirements—stateless computation, enforceable guarantees, non-targetability, no privileged runtime access, and verifiable transparency—and six technical components, including oblivious HTTP, blind signatures, secure enclave, remote attestation, transparency logs, and secure boot. The talk highlights how remote attestation allows an iPhone to verify server software and tie encryption to that verified state, while the transparency log enables public auditing of all deployed binaries. Mortensen notes trade-offs: complete trust in Apple's supply chain, higher latency and compute costs, and no support for custom models or third-party developers. He points to alternative tools like TPMs, SigStore, and confidential VMs for non-Apple environments, and concludes that Apple's approach is being adopted by Azure AI and Meta.

How we hacked YC Spring 2025 batch’s AI agents — Rene Brandel, Casco

Jul 30, 2025 · 17:33

Rene Brandel, CEO of Casco, explains how his team hacked seven of sixteen YC Spring 2025 batch AI agents within 30 minutes each, revealing three critical security flaws: cross-user data access via IDOR, arbitrary code execution through code tools, and server-side request forgery (SSRF) from tool endpoints. They extracted personal data by exploiting missing authorization checks, overwrote security controls by writing malicious files into code sandboxes, and stole Git credentials by manipulating a database-schema tool. Brandel emphasizes that agent security extends beyond LLM prompt injection, advises treating agents as users with proper authentication and authorization, and warns against rolling custom code sandboxes, recommending out-of-the-box solutions like E2B. The talk concludes with a Q&A on extracting system prompts and the dangers of local coding agents.

OpenAI on Securing Code-Executing AI Agents — Fouad Matin (Codex, Agent Robustness)

Jul 30, 2025 · 14:00

Fouad Matin, an OpenAI engineer working on agent robustness, argues that code-executing agents like Codex CLI require layered security safeguards as they become capable of autonomously writing and running code. He emphasizes three main defenses: sandboxing agents via containerization (e.g., ChatGPT spins up fully isolated containers) or OS-level sandboxing using SeatBelt on macOS or Seccomp and Landlock on Linux; disabling or limiting internet access to prevent prompt injection and data exfiltration, with configurable allowlists for approved endpoints; and requiring human review through PR approvals and monitoring tasks to catch unintended actions like installing malicious packages. Matin also details new API tools—local shell and apply patch—that enable agents to execute code and apply diffs securely, and recommends using remote containers (available in the Agents SDK and Responses API) as the safest deployment option. He concludes by acknowledging that while LLM-based monitors are valuable, deterministic system-level controls remain essential, and invites applications to OpenAI's new Agent Robustness team.

Why you should care about AI interpretability - Mark Bissell, Goodfire AI

Jul 27, 2025 · 21:11

Mark Bissell of Goodfire AI argues that mechanistic interpretability—reverse engineering neural networks—has moved from research to practical use cases that AI engineers can apply today, demonstrated through Goodfire's Ember platform for neural programming. He shows how Ember enables debugging and steering models at the neuron level, such as turning up a 'sensitive information' feature to make LLaMA refuse to reveal an email, or dynamically injecting a Coca-Cola recommendation when a beverage feature activates. For image models, the Paint with Ember demo (paint.goodfire.ai) lets users paint concepts like pyramid or lion face directly onto a canvas and steer sub-features (e.g., lion minus mane becomes tiger). Beyond interfaces, interpretability powers model diffs to detect unwanted changes after fine-tuning, guardrails for production systems, and scientific extraction: Goodfire works with the ARC Institute to uncover biological principles from the superhuman genomics model Evo2. Efficiency gains also emerge by pruning unnecessary weights for specialized tasks, making interpretability a critical tool for reliable AI engineering.

CIAM for AI: Authn/Authz for Agents — Michael Grinich, CEO of WorkOS

Jul 21, 2025 · 20:13

Michael Grinich, CEO of WorkOS, argues that AI agents need first-class identity and access management, distinct from human or machine-to-machine auth. He identifies key challenges: headless login, least privilege for non-deterministic systems, and compliance tracking. Grinich presents four emerging patterns—persona shadowing, delegation chains, capability tokens, and human-in-the-loop escalation—and references standards like OAuth, UMA, GNAP, OIDC for agents, and verifiable credentials. He predicts a shift from 95% human traffic to 95% agent traffic, calling for middleware trust boundaries and urgent collaboration on agent identity standards.

The New Code — Sean Grove, OpenAI

Jul 11, 2025 · 21:36

Sean Grove of OpenAI argues that specifications, not code, are becoming the fundamental unit of programming, with the most valuable skill being precise communication of intent. He presents OpenAI's Model Spec—a collection of versioned Markdown files—as a living specification that aligns both humans and models around shared values and intentions. Grove illustrates how the Model Spec served as a trust anchor during the 4.0 sycophancy bug, where shipped behavior contradicted the spec's explicit 'don't be sycophantic' clause, leading to a rollback. He explains deliberative alignment, where the spec is used as training and eval material to embed policy into model weights, moving from inference-time prompting to muscled memory. Drawing parallels to the US Constitution, Grove positions specifications as executable, testable artifacts that compose like code, and suggests future IDEs will become 'integrated thought clarifiers.' He closes by calling for help in aligning agents at scale, noting OpenAI's new agent robustness team.

2025 in LLMs so far, illustrated by Pelicans on Bicycles — Simon Willison

Jul 9, 2025 · 18:30

Simon Willison reviews the past six months of LLM releases — including AWS Nova, Llama 3.3 70B, DeepSeek R1, Mistral Small 3, Claude 3.7 Sonnet, GPT 4.5, Gemini 2.5 Pro, GPT-4o, Llama 4, GPT 4.1, O3/O4 Mini, and Claude 4 — using his 'pelican on bicycle' SVG benchmark to argue that local models have become good enough to run GPT-4 class models on a laptop and that combining tools with reasoning is the most powerful technique in AI engineering, while noting risks like prompt injection and the 'lethal trifecta'. He tracks 30 significant model releases, highlighting that Mistral Small 3 (24B) matches Llama 3 70B's performance, which itself matched the 405B model, enabling local inference. DeepSeek's R1 caused a $500B+ Nvidia stock drop on January 27. GPT 4.1 Nano is the cheapest model yet at a fraction of a cent per pelican. He also examines bugs: ChatGPT's sycophantic 'shit-on-a-stick' incident and Claude 4's tendency to snitch to authorities when given ethical instructions and email tools. Willison concludes that while the pace is accelerating, control over context and security remain critical.

Building agent fleet architectures your CISO doesn't hate — Lou Bichard, Gitpod

Jun 27, 2025 · 13:52

Lou Bichard explains how Gitpod evolved from a managed SaaS to a 'bring your own cloud' architecture that satisfies CISOs in regulated industries by running secure dev environments—and now agent fleets—on customer infrastructure via a simple runner (a single ECS task) instead of complex Kubernetes. The platform, used by banks and healthcare firms for 37 hours per week per developer, reduces operational overhead through a cloud-formation-based setup that takes three minutes. For agents, the same infrastructure provides source code access and audit logging, ensuring privacy and compliance. Bichard argues that vendors should simplify architectures to lower customers' day-two costs, and advises buyers to prioritize security and ownership models when selecting AI tools.

AI Red Teaming Agent: Azure AI Foundry — Nagkumar Arkalgud & Keiji Kanazawa, Microsoft

Jun 27, 2025 · 19:31

Keiji Kanazawa and Nagkumar Arkalgud of Microsoft present the AI Red Teaming Agent in Azure AI Foundry, arguing that adversarial testing is essential for building trustworthy AI agents. Nagkumar demonstrates the tool: it runs scans against RAG apps or models directly, using attack strategies like Caesar encoding and Base64 to simulate adversarial prompts across four risk categories (violence, hate and fairness, etc.). In a demo with GPT-4.0 with full guardrails, no attacks succeeded; switching to Phi-3 without guardrails yielded 5 out of 40 successful attacks in hate and fairness. The tool integrates with Azure AI Foundry's content filters, which apply both input and output guardrails, allowing engineers to iteratively test and mitigate vulnerabilities. The talk emphasizes that red teaming should be part of a broader risk mapping and evaluation lifecycle, and that trust is a team sport requiring collaboration between engineers and security experts.

How to Build Trustworthy AI — Allie Howe

Jun 16, 2025 · 24:22

Allie Howe, VC CISO at GrowthCyber, argues that trustworthy AI equals AI Security (how the world harms your AI) plus AI Safety (how your AI harms the world), and that builders are legally and reputationally responsible. She covers three pillars: MLSecOps (scanning models for serialization attacks using open‑source ModelScan), AI Red Teaming (testing for prompt injections, jailbreaks, and safety issues with tools like PyRIT), and AI Runtime Security (validating inputs/outputs in production via platforms like Pillar to block off‑topic or unsafe behavior). Howe cites real incidents—a Chevy Tahoe chatbot offered for $1, Slack's data leak via prompt injection, Fortnite's Darth Vader NPC initially producing racist outputs—and notes a lawsuit where OpenAI won by arguing users must expect errors. She emphasizes shifting‑right to runtime guardrails as the most cost‑effective investment and advises demonstrating trustworthiness through GRC platforms like Vanta to shorten sales cycles. With increasing regulation (ISO 42001, EOAI Act, FDA guidelines), she concludes that building trustworthy AI now unlocks revolutionary innovation in fields like healthcare.

GPU-less, Trust-less, Limit-less: Reimagining the Confidential AI Cloud - Mike Bursell

Jun 3, 2025 · 43:41

Mike Bursell of Super Protocol argues that Confidential AI, built on hardware Trusted Execution Environments (TEEs) like Intel TDX, AMD SEV-SNP, and NVIDIA GPU TEEs, solves the trust problem in AI by enabling secure processing of sensitive data and proprietary models without exposure. Super Protocol, a decentralized confidential AI cloud and marketplace, allows users to deploy models in TEEs, verify execution via cryptographic attestation, and collaborate across organizations without blind trust. Demos show deploying DeepSeek on H100 GPUs, running n8n healthcare workflows, distributing vLLM inference across four GPU nodes, and provably training a medical model on datasets from Alice's lab and Bob's clinic. Case studies include Realize achieving 75% accuracy and 3-5% sales increase for Mars, and BEL reducing FDA audit time from weeks to 1-2 hours. The protocol replaces trust with on-chain proofs, enabling GPU-less, trustless, limitless AI.

AI + Security & Safety — Don Bosco Durai

Apr 19, 2025 · 18:13

Don Bosco Durai, CTO of Privacera and creator of Apache Ranger, argues that building safe and reliable AI agents requires a multi-layered security approach combining preemptive vulnerability evals, proactive enforcement, and real-time observability to address challenges like unauthorized access, data leakage, and compliance in enterprise production. He explains that current agent frameworks run as a single process sharing credentials, creating zero-trust vulnerabilities, and autonomous agents introduce unknown unknowns. He advocates for three layers: preemptive security evals (including prompt injection, data leakage, runaway agent tests) to generate a risk score for production promotion; proactive enforcement with authentication/authorization propagated across all components and approval workflows; and observability with thresholds and anomaly detection to monitor agent behavior in production. He illustrates compliance needs using his customer example—a top credit agency needing to treat agents like human users for regulatory adherence. Bosco also open-sourced PAIG.ai as a safety and security solution for GenAI and AI agents.

Building security around ML: Dr. Andrew Davis

Feb 8, 2025 · 25:01

Dr. Andrew Davis, Chief Data Scientist at HiddenLayer, argues that machine learning models remain highly vulnerable to adversarial attacks despite a decade of research, and defenses must be layered with observability, logging, and skeptical data handling. He details how ImageNet's URL-based distribution enables data poisoning via expired domains, and how model theft can replicate a LLaMA 7B model's performance for just $600 in OpenAI queries. Adversarial examples still evade robust defenses, with best-case robustness only 50-60% against advanced attacks, and multimodal LLMs amplify the threat as pixel-level modifications are far harder to detect than text prompt injections. Spotlighting — encoding data in base64 to prevent instruction-following — is a promising prompt injection defense, but attackers can craft readable base64 strings to bypass it. The ML supply chain on Hugging Face is fraught with risk: models can execute arbitrary code via Lambda layers or TensorFlow functions, so verifying provenance, scanning for malware, and sandboxing are critical. Finally, software vulnerabilities in tools like Ollama (with recent RCE CVEs) demand the same patching discipline as traditional…

GitHub's AI Powered Security Platform: Sarah Khalife

Feb 8, 2025 · 23:45

Sarah Khalife, Principal Solutions Engineer at GitHub, details how the company is weaving generative AI into GitHub Advanced Security (GHAS), focusing on code scanning auto fix (in public beta, with a 70% success rate on vulnerability fixes within pull requests), secret scanning enhancements such as AI-generated custom regex patterns and unstructured password detection (reducing false positives), and supply chain security via Dependabot. She demonstrates how AI enables faster remediation—detecting cross-site scripting and suggesting fixes before merge—and shows the security dashboard’s coverage view, which reveals that secret scanning covers 99% of repositories while code scanning only 57%. The talk emphasizes that AI shifts security from reactive to proactive, bridging the gap between AppSec teams and developers through shared responsibility and community driven improvements.

LLM Safeguards: Security Privacy Compliance Anti Hallucination: Daniel Whitenack

Dec 31, 2024 · 34:10

Daniel Whitenack of Prediction Guard outlines a practical checklist for deploying secure and accurate LLMs in enterprise, addressing hallucination, supply chain vulnerabilities, flaky model servers, data breaches, and prompt injection. He proposes a factual consistency model fine-tuned to detect inconsistencies between AI output and ground truth, rather than relying solely on RAG. For supply chain risks, he recommends a trusted model registry and industry-standard libraries. Data breaches are mitigated via PII detection filters and confidential computing (e.g., Intel SGX) to encrypt server memory. Prompt injection is countered with a custom firewall layer using ensembled classification models. In Q&A, he discusses latency trade-offs using smaller NLP models, pre-production testing for visibility, data access controls via role-based database queries, SIEM integration for monitoring new artifacts like model caches, and additional challenges with agents such as excessive agency, proposing dry-run approvals.

AI Frontiers in Trust and Safety Combatting Multifaceted Harm on Tinder at Scale: Vibhor Kumar

Dec 2, 2024 · 14:36

Vibhor Kumar, senior AI engineer at Tinder, explains how the company uses open-source LLMs and LoRAX to detect a long tail of trust and safety violations at global scale. Facing challenges like content pollution and automated fraud from generative AI, Tinder leverages pre-trained models such as LLaMA and Mistral, fine-tuning them with LoRA and QLoRA on hybrid datasets generated by GPT-4 and manually verified. They serve dozens of fine-tuned adapters on a single GPU using LoRAX, achieving real-time inference (tens of QPS, ~100ms latency) for categories including hate speech, pig butchering scams, and underage users. The approach yields near 100% recall on simpler tasks and significant improvements over baselines, with better generalization that resists adversarial evasion. Future directions include visual language models for explicit image detection and automating retraining pipelines.

Iterating on LLM apps at scale Learnings from Discord: Ian Webster

Nov 22, 2024 · 18:26

Ian Webster, Senior Staff Engineer at Discord and maintainer of Promptfoo, shares how Discord built and scaled Klyde AI, a chatbot for 200 million users, focusing on evaluation and safety. He argues that evals should be treated as simple, deterministic unit tests that run locally, avoiding complex metrics, and that breaking the system into small, testable pieces (e.g., checking for lowercase output to enforce casual tone) achieves 80% of the goal with 1% of the work. Webster details how Discord mitigated risks like the 'grandma jailbreak' (which originated on Discord) by using an attacker LLM to generate adversarial inputs and a judge to refine them, exposing cracks in safeguards. He advocates for pre-deployment red teaming over live filtering, and describes using Promptfoo for risk assessment across brand, legal, and safety categories. The episode also covers prompt management via Git and Retool, routing with occasional GPT-4 responses to correct model drift, and the challenge of closing the feedback loop due to privacy constraints, relying instead on dogfooding and public examples.

Open Challenges for AI Engineering: Simon Willison

Jul 17, 2024 · 18:49

Simon Willison argues the GPT-4 barrier has been broken as GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, and open models like LLaMA 3 70B now compete, making GPT-4-class models a commodity. He highlights the AI trust crisis with examples of Dropbox and Slack being falsely accused of training on user data, and notes Anthropic trained Claude 3.5 Sonnet without customer data. Willison warns about prompt injection vulnerabilities, citing the Markdown image exfiltration bug affecting six major chatbots, and defines slop as unreviewed AI-generated content, calling for accountability and responsible use patterns.

Open Questions for AI Engineering: Simon Willison

Nov 25, 2023 · 24:33

Simon Willison recaps the AI industry's past year—from ChatGPT's breakthrough to open-source local models—and poses key open questions for AI engineering. He argues that ChatGPT's chat interface, while popular, is a poor fit for advanced use, urging better UIs like his command-line tool LLM. He celebrates Meta's Llama release as a 'stable diffusion moment' for language models and highlights the rise of small, locally-run models such as Replit's 3B model, asking how small models can remain useful. On security, he warns that prompt injection remains unsolved after 13 months, limiting what can safely be built. He champions ChatGPT's Code Interpreter (which he dubs 'Coding Intern') as the most exciting tool, able to write and compile C code on a phone, and argues that LLMs flatten the learning curve, making programming accessible to more people. He concludes by urging the community to build tools that enable anyone to automate tedious tasks.

Trust, but Verify: Shreya Rajpal

Nov 25, 2023 · 19:41

Shreya Rajpal, CEO of Guardrails AI, argues that large language models require a verification layer to compensate for their non-deterministic nature. She explains that while prototyping works, production apps fail due to hallucinations, prompt injections, and structural errors. Guardrails AI is an open-source framework that wraps LLMs with a validation suite: on output, it checks constraints like provenance (grounding in a source), profanity, and competitor mentions. On violation, it re-asks the model to self-correct or falls back to specified policies. Rajpal demonstrates with a chatbot example where a provenance guardrail catches a hallucinated password setting, then guides the model to a correct answer grounded in help center articles. The framework also supports custom validators, automatic prompt compilation from checks, and integration with external systems like sandboxed SQL databases for code generation.