How to defend your sites from AI bots — David Mytton, Arcjet
Jul 30, 2025 · 20:12
David Mytton, founder of Arcjet, explains that nearly 50% of web traffic is now automated, with AI crawlers significantly worsening the problem for site owners. He cites specific cases: Diaspora saw 24% of its traffic from OpenAI's GPTbot, and ReadTheDocs cut bandwidth from 800 GB to 200 GB per day by blocking AI crawlers. Mytton describes the evolving landscape of AI bots—OpenAI alone has at least four types (Search, ChatGPT User, GPTbot, Operator)—and argues that detection requires layered defenses. He walks through eight defenses: robots.txt (voluntary), user agent string verification with reverse DNS, IP reputation (e.g., 12% of bot traffic on Cloudflare comes from AWS), CAPTCHAs (increasingly easy for AI to solve), proof-of-work puzzles (e.g., Anubis proxy), experimental HTTP message signatures (Cloudflare's proposal), fingerprinting via JA4 hash (TLS-level), and rate limiting keyed to fingerprints or session IDs. The practical takeaway is that most sites can manage with user agent verification plus IP checks, but high-value or high-traffic sites need full stacking of these techniques.
