HiddenLayer

How we hacked YC Spring 2025 batch’s AI agents — Rene Brandel, Casco

Jul 30, 2025 · 17:33

Rene Brandel, CEO of Casco, explains how his team hacked seven of sixteen YC Spring 2025 batch AI agents within 30 minutes each, revealing three critical security flaws: cross-user data access via IDOR, arbitrary code execution through code tools, and server-side request forgery (SSRF) from tool endpoints. They extracted personal data by exploiting missing authorization checks, overwrote security controls by writing malicious files into code sandboxes, and stole Git credentials by manipulating a database-schema tool. Brandel emphasizes that agent security extends beyond LLM prompt injection, advises treating agents as users with proper authentication and authorization, and warns against rolling custom code sandboxes, recommending out-of-the-box solutions like E2B. The talk concludes with a Q&A on extracting system prompts and the dangers of local coding agents.

Building security around ML: Dr. Andrew Davis

Feb 8, 2025 · 25:01

Dr. Andrew Davis, Chief Data Scientist at HiddenLayer, argues that machine learning models remain highly vulnerable to adversarial attacks despite a decade of research, and defenses must be layered with observability, logging, and skeptical data handling. He details how ImageNet's URL-based distribution enables data poisoning via expired domains, and how model theft can replicate a LLaMA 7B model's performance for just $600 in OpenAI queries. Adversarial examples still evade robust defenses, with best-case robustness only 50-60% against advanced attacks, and multimodal LLMs amplify the threat as pixel-level modifications are far harder to detect than text prompt injections. Spotlighting — encoding data in base64 to prevent instruction-following — is a promising prompt injection defense, but attackers can craft readable base64 strings to bypass it. The ML supply chain on Hugging Face is fraught with risk: models can execute arbitrary code via Lambda layers or TensorFlow functions, so verifying provenance, scanning for malware, and sandboxing are critical. Finally, software vulnerabilities in tools like Ollama (with recent RCE CVEs) demand the same patching discipline as traditional…