Enterprise Agents Have a Structure Problem - Ishita Daga, Tesla

Hello everyone, and welcome to my talk: "Enterprise Agents Have a Structural Problem." A quick introduction about myself: I'm Ishita Daga, I work as a machine-learning engineer in Tesla, and I'm building enterprise agents for our organization.

Before we get into the framework and the solution space, I actually want to talk about why these agents fail in the first place. So when an agent gives a bad answer, the first reflex that we have is that we need a bigger model, we need the latest model, we want a model with a lot of context so that it can hold a lot of information, or we need to add a lot of knowledge base in terms of .md files, documents, or MCP servers, plugins, etc.

But while all of these are fair solutions, they're not the answer to actually improving the data agent itself.

Model

the agent actually has these three main problems. One is ambiguity. So it doesn't know what table isright, what column isright, which data source or which knowledge base to access when, which one holds the source of truth or like the cleanest information, which one is just a bit more flexible.

So we have a lot of these knowledge bases, but it's not defined which one is the best, or which one to use when. So ambiguity, I feel, is one big problem. The second is staleness. We all know context updates so quickly.

A lot of decisions change, a lot of definition KPIs, they change so often, processes they get updated so often. But to keep our data agents accurate, we need to update this context, this context lifecycle, and that's a second part of the problem.

The last one is preference, which is more so capturing an individual or a team preference on what metric to use to answer a specific question, how to use a query, or what filters to use in a query. Although we have like a canonical query, let's say, each team has a different preference on what filters or what definitions to use.

So this, I feel, is a very open-ended problem, still needs a lot of research, a lot of industries or the frontier labs are actually working on solving this one. So let's start with the first one: ambiguity. The agent needs to understand what source of truth to use, or what knowledge base to use.

It can't weight all the knowledge bases equally; there has to be a bit more structure to the one it chooses to answer a particular question. In my opinion, the source of truth actually is a hierarchy, which goes from the cleanest, least flexible source of truth to the messiest but most flexible, most dynamic source of truth.

And the way to answer any question is to start from the cleanest to the messiest, or the most dynamic one. We have divided the sources of truth in three different buckets for this framework. The first one being the semantic layer, which is the best source of truth: a very curated list of all the different queries, KPI definitions, metric, or how to calculate the metric, the business definitions, and everything combined into just one semantic layer, which the model can use to answer a query.

So the agent just looks into the semantic layer, finds the closest KPI, and references all of these different data points from that KPI to actually give you an answer. The second part is canonical tables. These are more so the parametric tables, or queries, sorry, where you give the agent a list of different parametric queries and it sort of tries to understand which ones can it use to actually answer your question.

It gives it a bit more flexibility in choosing or writing its own query, or its own filters, etc., and can answer a bit more wider set of questions. The last one is the database graph, which I feel is the most trickiest because it takes a lot of effort, but it gives you a lot of flexibility in the kind of questions that can be answered.

So the idea here is you connect each table to all the different columns, each table to the metric that can be answered through this table, and vice versa, and sort of create this enormous database graph that can be served to the agent and it can answer or query on top of it.

While it is tricky, it is also very difficult to maintain or update. So I feel if an enterprise is approaching or adding sources of truth, they should start adding the first and the second layer first. These are easy to set up, solve, I think, 80% of the problems, and then the 20% can be solved by the database graph.

Moving on to our second problem, which is staleness. I think the word speaks for itself, but the context gets rotten, or it gets deprecated, or processes change so often that it's hard to maintain .md files or keep on updating your skills with the most latest context.

It's just not easy, and we need a better solution for that.

My answer to this problem is creating a context lifecycle, which basically has two different components. The first one being embedding live data sources or knowledge sources. By live I mean something or the data sources that are going to be always updated, would be reviewed and well curated, and always provide the newest or the most up-to-date data.

This can include your GitHub, your CRM tools, or your Tableau or dbt semantic layers. Whatever you want to include can be there, but it has to be a mandatory data source, like a data source that is updated most often.

The second part is a feedback loop, which is something that a lot of enterprise or data agents actually miss. The idea here is you should be able to log each and every event, and by event I mean whenever someone says that this database is incorrect, or this definition is incorrect or updated, or there's a new way to calculate a metric, or there is a new filter that you need to use.

All of these events need to be captured, logged, and used to update your data agent context. Once you have captured these events, the second part is to actually evaluate your data agent, and you can do that by either curating an evaluation suite with human-annotated questions or human evaluations, or you can create an automated evaluation where you see all the questions that have been asked in the last couple of days and the actual answers, and how close the new answers are to the actual answers.

So I think there are a lot of ways you can do that, but a lot of teams do not focus on evaluation that much, and which is why the agents actually fail so often, because you don't know how the agent is progressing or how is the performance.

You don't keep a track of that. So I feel creating this feedback loop, or a context loop, where you log, evaluate, update the context every so often is very important.

The last problem is preference, which is a very subjective problem, because preferences they change very often in terms of what kind of metric should be used, what's theright calculation. Different teams will calculate the same metric differently, or use the same query but different filters.

So there is a lot of subjectivity which needs to be captured, but it's very hard to do that on individual or team level. To give an example,

so let's say there is Team A and Team B, and they both want to calculate the average time taken for a particular milestone. Team A calculates it by looking at the previous milestones, or completion of the previous milestone to completion of the current milestone, while Team B actually looks at the start of the current milestone to the start of the next milestone.

So while both of these are correct metrics, or the correct way to calculate a metric, they both will give you very different answers, and it's just about preference. We just need to understand what an individual means when they say, "What's the average milestone time?"

So while this is a great question, I feel the industry still does not have a correct answer or a correct way to solve the problem itself. I feel like there are sort of two ways to solve it. One is a semantic layer, which is something that I'm trying, wherein you store all the different ways to calculate a metric as a semantic layer, and then the individual sort of can prompt the model which one do they want to use.

So you can solve it, but it still is not storing the preference of the user. And the second thing is you again bump into the first challenge, which is ambiguity. The agent doesn't know which metric is theright one.

You still have to prompt it. The second part is agent memory, which is your mem0, or like storing a memory.md file, but it still is not the best way. It stores your preference, but it can't understand the distinction between two different metrics, or which one to use when.

So I feel there are ways to solve it, but none of the ways are actually solving the preference problem itself. What we actually want is a way to route the agent to theright metric based on

what team or which individual is using that agent. So still an open-ended problem, but yeah, I think something we should work on or research on a bit more. Which brings us to the end of this presentation. So just to conclude, I feel there are these three big problems: ambiguity, staleness, and preference.

And what we really need to solve for is a better structure, a better way to manage the context and evaluate the context. And then preference, which is such an open-ended problem, it requires not just understanding how the agent works, but also embedding an individual preference, like creating a hive mind for your data agent.

That's about it. Thank you so much.