# Why (Senior) Engineers Struggle to Build AI Agents — Philipp Schmid, Google DeepMind AI Engineer · 2026-05-30 Philipp Schmid from Google DeepMind argues that senior engineers struggle to build AI agents because they carry years of implicit context that agents don't, designing tools that assume it. He highlights five key shifts: text replaces structured state, so agents can handle semantic meaning like approving a research plan with additional input; errors are inputs, not restart triggers, since long-running agents (5-15 minutes) can't afford to start over; evals replace unit tests because agents are non-deterministic and success is measured by how often it works, not fixed outputs; and agents evolve while APIs don't, so tools like a `deleteItem` endpoint need agent-friendly docstrings because the agent only sees the schema. He advises to hand over control to the LLM, design for recovery, and build to delete since models improve rapidly. ## Questions this episode answers ### Why do senior engineers struggle when designing APIs for AI agents, according to Philipp Schmid? Philipp Schmid explains that senior engineers carry years of implicit context; for them, an endpoint like `deleteItem` is self-explanatory. But an agent only sees the function schema and docstring, not the code or background. Therefore, we need to build agent-ready tools with semantic, self-documenting interfaces, avoiding assumptions of long-year developer expertise. [7:52](https://aie.addtry.com/22bbb2ca-4f15-4aec-ad2d-72ba93b2d9e2?t=472000) ### How should errors be handled differently in AI agent workflows compared to traditional software? Schmid says that in agents, errors must be treated as normal inputs, like user inputs, rather than restarting the whole process. Since agents can run for 5 to 15 minutes, a full restart wastes compute and loses context. We need to provide errors back to the model and design for recovery so the agent can continue forward. [5:04](https://aie.addtry.com/22bbb2ca-4f15-4aec-ad2d-72ba93b2d9e2?t=304000) ### Why can't unit tests guarantee correctness for AI agents? Schmid notes that agents are non-deterministic; the same input does not always yield the same steps or result. Thus, we must move from unit tests to evals that measure how often something works. For example, a customer agent that succeeds only 1 out of 10 times is too flaky. We should use LLM-as-judge or human experts to assess outcomes. [6:14](https://aie.addtry.com/22bbb2ca-4f15-4aec-ad2d-72ba93b2d9e2?t=374000) ## Key moments - **[0:00] Intro** - **[0:45] The Shift** - **[2:10] Text is State** - [2:10] Text is our new state: Agents operate on context and semantic meaning, not rigid data structures, allowing dynamic adjustments. - **[3:58] Hand Over Control** - **[5:04] Errors as Inputs** - [5:04] Errors are just inputs. - **[6:14] Evals over Tests** - [6:14] Move from unit tests to evals: Agents are non-deterministic; Philipp Schmid says we need to test how often something works, not if a fixed input yields a fixed output. - **[7:52] Agent-Ready APIs** - [7:52] A `deleteItem` endpoint is obvious to the developer who built it. An agent only sees the function schema and docstring. - **[9:05] Takeaways** - [9:54] Build to delete. - **[10:21] Outro** ## Speakers - **Philipp Schmid** (guest) ## Topics Agent Engineering, Agent Skill Design, Agent Evaluation ## Mentioned DeepMind (company), Google (company), Gemini (product) ## Transcript ### Intro **Philipp Schmid** [0:15] Okay, cool. Awesome. Hi, hi everyone. Uh, my name is Philipp, I work at DeepMind, everything related to agents on Gemini or Gemini APIs. So if you have some questions afterwards, some concerns, some bugs, some issue, please let me know. We're going to talk today, 10 minutes, about why engineers struggle to build agents. And I see this every day internally at Google, but also externally at Google. And I brought 5 examples on, like, what's really different to how we built traditional software a few years ago and to now how we build agents. And if we, like, on a high level, compare them,right? When we wrote software, we created a spec, a PRD, wrote code, sometimes created tests to make sure our code works. We deployed it, and then our user used it. ### The Shift **Philipp Schmid** [0:59] And when building agents, things are a little bit different. We define instructions on what we want our agent to do. We run it, we observe what it does, we maybe adjust our prompts, maybe we adjust our tools, we run it again. And we have, like, this iterative loop of how can we improve and make our agent way more reliable, which is very different to how we build software. And, like, something I like to compare it to is, like, traditional software is more like we acted as a traffic controller,right? We had control over the streetlights, over how fast you can go, which road you can use, basically how the car drives. And now with agents, we are more of a dispatcher. We tell the agent, "Hey, I want to go to London, and I'm from, like, Germany. I could use the train, I could fly, I could use my car and go, like, under the water." And it's more about, okay, we define the goal on what we want the agent to do, but we don't define the exact step the agent needs to take to achieve that goal. And I mean, every one of you has probably seen in their coding agent that sometimes it does something very weird, but at the end it achieves the outcome. And that's what we want to do. So starting with the first example, text is our new state. I mean, traditionally we had data structures and everything was kind of mapped to Boolean or to, like, flags we could check. So initially when we created, for example, a deep research agent, a deep research agent returns a plan to you, "Okay, I'm going to research this and that." ### Text is State **Philipp Schmid** [2:27] In traditional software we might have had an accept plan or deny plan, but we couldn't catch semantic meaning. And now what we have with LLMs is they can understand the semantic meaning. So for example, if I have a deep research, um, request on, like, doing some market research, I can approve the initial plan, but I can also, at the same time, provide additional information. So maybe I want to focus on, like, the US market and ignore California. Maybe I want to provide something additional and not have, like, these multiple steps,right? Traditionally I would probably set decline, and then it has a follow-up. I might need to provide more input, create a new plan, and continue. And another good example is everything related to memory and personalization we do cannot really be mapped to data structures,right? The example I hear I have is, like, um, I'm from Europe, so I mostly use Celsius. But what if I would like to use Fahrenheit for cooking,right? Previously we might have some flags on, like, the user profile, is Celsius or is Europe, or use Fahrenheit. But I couldn't, like, dynamically adjust based on the user preference, based on what I provide. So really it's all about text and context. I mean, it could be images, video, or audio as well, but we no longer are really operating in those clear structured data concepts. The other thing is we should start handing over control. And the, the trap or the example which we might have from, like, previous customer support is, like, when a user reached out, "Hey, I want to cancel my subscription." I might have had a classification model which kind of classified the intent, okay, the user wants to churn. ### Hand Over Control **Philipp Schmid** [4:14] And then I had a predefined workflow of, okay, do you try to sell it, do you cancel the subscription. But there was no, like, dynamic kind of option to, to react to it dynamically. And maybe instead of, like, um, going through the subscription cancel flow, what if your agent, like, kind of tries to understand the meaning and, like, offers something, uh, in except to, like, the, the subscription and the user changes their mind and now you have, like, a whole different intent. And it's very hard to model all of those differences and uniqueness and to, to, like, all of those stateful workflows we had before. So we need to, like, trust into the LLM or, like, hand over control that we are no longer working in those purely deterministic, um, environments. ### Errors as Inputs **Philipp Schmid** [5:04] The third one is errors are just inputs. So if something in your agent flow fails, we need to treat it as a normal input, as very similar to a user input. In Go, we already do this,right? A function call can be an error or can be a value, and we treat them kind of equally. And we have to do this for agents very similarly. In the past, HTTP requests were very cheap. When some search, some product search failed, you just rerun your request, you redid all of the work, which was okay. But now if you have, like, an agent which takes 5 minutes, 15 minutes, and something in the flow breaks and you would start all over, you would need to spend, you need to spend a lot of compute again to, like, do all of the previous steps. And you also might lose the existing context. So we cannot, like, just start over the whole process. We need to kind of understand and treat errors differently, provide them back to the model, maybe have some other workarounds, some additional checks that we basically keep going forward in the flow and not, like, starting over from the beginning. The fourth, uh, example or step is, like, we need to move from unit tests to evals. So when building software before,right, we wrote integration tests, unit tests, smoke tests, and all kinds of different tests. And we assume that when we provide input A for our code B, we will always get C as an output. ### Evals over Tests **Philipp Schmid** [6:32] And that's no longer the case with agents. Agents are non-deterministic. We cannot always guarantee that the same input will lead to the same steps and the same result. So we need to move from unit tests to eval. We need to test how often something works because agents are only successful if they are really reliable,right? If you have a customer agent and the same prompt only works 1 out of 10 times, it's nothing really you want to put in production and it becomes very flaky. So we need to test on evals, on how many times it passes, and compare to traditional software. Results are very subjective,right? An outcome can be very different if you ask it to create a research report, if you ask it to create a customer feedback kind of scenario. And we need more, like, qualifying, uh, feedback. Um, LLM as a judge or human expert, for example, is a good way. And we always need to trace what the agent is doing, but we need to create on the output. We maybe the agent decides for, like, one user it needs to do, like, 4 more steps to do more research than for the other user. It consumes maybe a few more tokens, but at the end the outcome is really what we need to measure and want to measure in terms of, uh, success. And then the last part is, uh, agents evolve and APIs don't. And if you have worked on the backend and if you have built an API, you might have seen a lot of methods, API endpoints, which feel very self-explaining to you. Like, deleteItem feels very self-explaining if you are working on, like, the product API. ### Agent-Ready APIs **Philipp Schmid** [8:10] But an agent doesn't see the code. An agent doesn't have the context and the background from all those years from you working on the API. So we need to build APIs or tools which are really agent-ready, which are self-documenting with semantic interfaces. I would assume if you have, like, a product microservice and you have an deleteItem endpoint with an ID, you don't need to, like, define a docstring what the ID is or what happens if something fails. But our agents only see, like, the function schemas and the docstrings and the tool definitions. So on the first look, they don't really see what the deleteItem method does. That's why we need to really adjust to, hey, we need methods, tools which are written for agents to be used and not assume long-year developer expertise and people who have built the API. So to, to summarize everything, we need to give trust, but we also verify. We should stop fighting the model. You should not, like, try to force the model into this one specific workflow with step one, do this, step two, do the other thing. ### Takeaways **Philipp Schmid** [9:19] We need to preserve meaning. Um, everything is a context now. We no longer have those very well-defined data structures for all of our applications. We need to design for recovery. Models are not perfect. Agents are not perfect. Especially if we have longer running agents, there will be some very weird things happening. So you need to design for recovery. We need to evaluate and don't only assert. Agents are not 100% reliable. We need to find theright balance between how many times our run need to be successful to provide it to the user. And last but not least, build to delete. Um, the bitter lesson is what every one of us is learning is, like, software is disposable. We are going to rebuild many, many times the same things with better models, better agents. Things will change. And, um, yes, um, it's also available on my blog. So if you want to, like, look a bit deeper with some code examples. And if not, if you have any questions, feel free to, to reach out to me. And perfect on time. Thanks. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.