Grounded Reasoning Systems for Cloud Architecture - Iman Makaremi

Hello everyone, my name is Iman Makaremi, I'm the co-founding head of AI at Catayaw. Today I'm talking about Grounded Reasoning systems for cloud architecture, and how we use multi-agent orchestration to build an AI copilot at Catayaw. But why now?

Cloud architecture needs reasoning, not just automation. We see that cloud systems are growing in complexity based on people, both users and also developers, and in terms of tools and constraints and expectations are always on the rise. Statistic tools don't scale with the diversity of decisions that need to be made around cloud architecture.

And we need systems that understand, debate, and justify, and also plan in order to solve such problems. And this is not automation; this is reasoning. The architecture stack isn't just technical; it's also very cognitive. Architects don't just assemble infrastructure; they are constantly negotiating trade-offs based on how defined the requirements are, or how much time they have, or what resources are available for building the application that they are supposed to deliver.

And they rely on this structure and implicit context in order to make this decision. And capturing this for AI requires understanding how architects think. So, where AI meets architecture, there are three challenges that we capture at Catayaw at a higher level in order to

solve architecture design problems. One is about requirement understanding. So where is the requirement coming from, in what format, and what are the important pieces, how global or specific they are. These are all the things that need to be understood as valid requirements.

On the other side, it's about architecture identification. There are many, many different components that could exist in an architecture, but based on the components and based on where they are, they could have very different functions. In order to understand how an architecture works, knowing different functionalities and the functions that different components are playing is very important too.

And putting all these things together is about architecture recommendation. It's that so we know the requirements, we understand what the architecture state is today, what are the recommendations that need to be provided in order to either match the requirements or improve the architecture

to be matching best practices.

To turn these problems into more specific AI-related challenges, what we're dealing with is a mix of semantic and graphics context. On one side we have requirements, which are mostly textual documents, and on the other side architecture, which is inherently graph data.

How we can make these two different sources of data to come together and make theright connections in order to do higher-level reasoning is very important. And through that comes complex reasoning scenarios. The kind of questions that come in could be very complex, very vague, very broad, and that needs to be broken down into different parts and be planned properly in order to get theright answer to the question.

And above all of these things, how we evaluate and provide feedback to such a large AI system that has many different moving parts. So we want to go through

these challenges and share some of our learnings, what we have experienced, and what we have done, and what we are doing in order to improve to the next level. So the first one is about grounding agents in a specific context.

So LLMs need proper context about architecture to reason effectively. Translating natural language into meaningful architecture to reasonable tasks isn't straightforward, and also providing it fast makes it even harder. So we tried multiple things here, both on architecture retrieval and on the requirement retrieval part.

One was that it was about semantic enrichment of architecture data. For each component, we collected enough relevant semantic information to enrich that component in order to make it more searchable and findable in a vector search. Another part was about graph-enhanced component search.

When you're searching for a specific component or a type of component within an architecture, through graph search, we also used all sorts of graph algorithms in order to make sure that we retrieve theright pieces of information from an architecture.

For requirements specifically, we did some early score enrichment of requirement documents for fast retrieval, because requirements could be about many different topics. And

if we want to understand a large corpus of text properly, we probably want to understand different concepts that exist in that context. We did something a little simpler in our first situation, which was about figuring out what are the important concepts and requirements for us within a large organization.

And based on that, we scored different documents, and through that, we made retrieval tasks faster. What we learned was that semantic grounding improves reasoning but doesn't always work. We saw some really good results for architecture retrieval; however, we also saw where it falls short and cannot be scaled or gets very detailed responses.

Prompt design is critical in soft grounding. We cannot always provide

the exact structure, and telling the agent exactly what to focus on and what to retrieve is very important. And the other part is that graph memory supports continuity and not just accuracy. It is very important that we, in our graph searches, we are able to find different nodes within a graph, but also be able to connect these different nodes and figure out why we have these different nodes pop up in our searches and add more context that could help us to do proper reasoning in the next steps.

So what we see here, for example, this is one of our early designs on architecture retrieval. On the left side we have a storing step, on theright side we have the query step. And on the left step, we're basically breaking the architecture down by components and turning the JSON data into natural language, and then enriching it with connection data and embedding it and putting it into a vector DB.

And on the other side, when the search time comes, a query comes, we embed that search in the vector DB and find the component. We saw some good results with this, but one thing we found was that semantic search doesn't work very well for

this type of graph data, and it has its own limitation. And that's why, from here, we went more towards graph-based searches. First of all, enrich this approach with some graph searches, and from here we are moving towards making this more based on the graph, a knowledge graph.

This is another design that we did initially for understanding requirements, and part of it is very familiar for most of us, which is about taking input documents, split pre-processing it, splitting it, embedding it. What we did extra here was coming up with, first of all, having requirement templates that had a specific structure on what requirements are.

In this way, we were able to basically structure what we are extracting from the documents in order to, as we call it, pods, to make sure that the extracted information is relevant to what we're going to use downstream.

This helped us with fast retrieval. It has its own advantage in terms of finding and structuring the business requirements and finding theright pieces for agents to do the further recommendation tasks. However, we also learned that this also has some limitations.

As we increase the number of documents, we see that we sometimes lose the context in the larger searches. So, and here is another part that finding theright context within larger corpus becomes very important, and that's where we see some graph analysis could also help on this side.

Another part is about complex reasoning scenarios. Good design involves conflicting goals, trade-offs, and debates. That's part of the architecture design world. It never comes easy. And we also need agents that can collaborate, argue, and converge on justified recommendations.

So what we tried was basically building a mass orchestration with role-specific agents. This multi-agent system is able to basically make multiple agents work together at the same time and give them multiple properties. We'll cover more of that in the next slides.

Another thing we tried was having structured message formats. Initially, we had to start with generating XMLs and parsing XMLs, which wasn't fun, and I'm glad those days are gone. But having the structured message definitely helped with building better workflows and having multiple agents to work with each other, and having more agents also work with each other in a longer chain.

Customer memory management was an important piece because the way that we wanted to have the agents collaborate with each other, because there were multiple levels of interaction between agents, we wanted to make sure that the conversations are isolated to, first of all, make sure that we don't waste tokens.

And plus, we had done some experimentations that increasing the memory showed an increase of hallucination in the results. So we decided to keep the memory isolated between agents. Cloning of agents for parallel processing for certain tasks was an important piece that we did and helped us to

speed up some of our processes. And that required also some memory management in order to, like, what do we duplicate for agents from a certain point that we decided to do the cloning, and how we manage the memory past that cloning point.

And what we learned was that structured outputs improve clarity and control. It's super, super important to do structured outputs for better programming. We hear a lot also about how structured outputs could reduce

the reasoning abilities of the model. It is a real trade-off, but however,

we are seeing very good results with the level of structured output that we have.

Mass let agents resolve trade-offs dynamically and not just execute static plans. We give a lot of, we are working also on different ways of orchestration for agents and what are they exactly doing in each step that they're taking as part of their interaction with other agents.

But we are learning that the more dynamic we make it, the higher creativity they show in planning and reaching the results. And that's what we are trying to optimize. Another thing is that successful mass orchestration does require control flows.

It's not possible to

have multiple agents to just work with each other and

hope for the best that the result is going to come at the end. So

the major part of our stack that is in production and is working is recommendation that is based on multiple agent systems. As we see, here's a screenshot of some of the recommendations that we have on some mock-up data.

But everything else that you see here in terms of the categories that we have, for example, architecture, messaging, and queuing, API integration, and so on. And for each of these, the multi-agent system comes up with multiple recommendations, and each recommendation is broken down into different parts, such as description, target state, gap analysis that shows, for example, what the requirement was, what the agents couldn't find in the architecture based on their requirements, and what are the recommended actions in terms of filling out

the discovered gaps.

The system that we have that does this has multiple agents in it. One is a chief architect that basically oversees and coordinates higher-level tasks. And then there are 10 staff architects, each specialized in a certain domain, for example, infrastructure, API, IAM, and so forth.

And then

there are two retrievers, requirement retriever, that

basically has access to requirements data, and the architecture retriever that understands what's the current architecture state and can answer questions about what components are there and so forth. The mass workflow that we have is like this, that there are basically three main tasks that happen in a sequence in order to

generate recommendations at the end of the flow. One is list generation, where at the end of it, we have a list of possible recommendations. And then there is conflict resolution, which is about

having the chief architect looking at a generated list of recommendations in order

to figure out if they are in conflict or if there are redundancies and so on. And basically, prune the list of recommendations in order to go further down the workflow. And the last step is design proposal, which is basically writing a full-on design proposal based on each recommendation topic in terms of, again, what are the gap analyses that have been done, what are the proposals in terms of improvement of the architecture, and so on.

And the way it works is that there is a chief architect, and the chief architect requests a list of recommendations from a staff architect. And the staff architect, in order to do that, reaches out to, or sends a request to the architecture state agent and requirements agent.

And these calls happen in parallel. They happen a number of times based on the budgets that exist. And at some point, it all comes back to the staff, and the staff gets back to the chief with a list of possible recommendations.

And this also happens among multiple staff architects. So this could be tens of calls that happen at the same time. And then after resolving the conflicts, we go again back to the staff architect in order to generate those design proposals.

And again, this flow will happen. And this time, there is a cloning happening, which is an important part of this whole flow, is that each staff architect is going to be cloned by the number of recommendations that's supposed to generate.

And each clone has access to the past history and generates its own current history, which is separated from the other clones. And this way, the agent has access to all the existing knowledge in order to make sure that it can write up the best design proposal at the end of the workflow.

And

the last important thing here is evaluation and feedback. And

the challenge is, how do we know if a recommendation is good? We build this multi-agent system that has many, many agents, and there are rounds and rounds of conversations that need to be

monitored. What's going to work the best here? And we had to close the loop with human scoring and structured feedback and revision cycles. What we learned, especially at the stage that we are, the best

evaluation is human evaluation. And we noticed that LLM evaluations are good, but they do not give us what we want in order to make the improvements that we needed to. For that purpose, we built our internal human eval tool.

We called it Eagle Eye. And it basically helps us to look at every specific case, the architecture, the requirements that are extracted, the conversations between the agents, and also the generated recommendations. And in this way, we are able to look at them and basically do relevance, feasibility, and clarity studies and give them scores.

And based on that, make some decisions in terms of what we want to focus next. And what we learned is that LLM confidence is not correctness. It can help, but it is not

what can be trusted all the time. And human feedback is essential early on when we are building such systems from scratch. And eval must be baked into system design, not added later. This is a design thought that we came down to that is very helpful, is that as soon as we start thinking about

designing a new AI system, we also start thinking about, okay, what are the evaluations that are going to be around it? And we're able to evaluate whatever we have as we go forward. It could be a human eval tool.

It could be a monitoring dashboard. Or it could be LLM-based evaluation feedback loops.

This is a screenshot of our

evaluation tool. We basically can see various things about the interactions in the multi-agent system and go through the conversations and read them and make sure that they make sense or not. And it actually helps to do that. For example, this is a case of hallucination early on.

This is a staff architect, network security, that sent a message to requirements retriever and

asked for scheduling a workshop and provided some dates on when that workshop could be in order to basically solve whatever they were trying to solve. Handling these kinds of cases is very important

in the prompts and also as part of the structured output that each LLM gives. And also, it's fun to come across some of these hallucinations mid-building such systems.

So to wrap it up about reasoning systems, is that it's not assistance, and there is a slight difference there. Building a copilot is not generating answers. It's about designing a system that can reason. We want to build a system that can have a good view of a large amount of data, especially such as architecture data, where we're talking about thousands or millions of components and a large number of documents that need to be in scope in order to answer various kinds of questions with various types of stakeholders, from developers to CTOs that could be interacting with

such a copilot.

And that takes roles, that takes workflows, that takes memories and structure in order to build such a reasoning system. And

putting all these things together requires a lot of experimentation in terms of what works based on the data that exists. We are learning that based on the data that we have, there are certain patterns that work better than the others.

For example, graphs are becoming more and more important in our designs. And the part of it is what kind of interactions agents can have with each other and what level of autonomy we want to give to each agent.

We are learning that we have experimented with various

frameworks for building multi-agents, and we are coming down to using

LangGraph for building some of the agent workflows. And again, having a manager on top of that in order to manage these various LangGraph workflows. And then also using Flyte to basically handle some of these higher-level management and

using graphs to capture as much of the memory as possible in order to make sure that the AI always has theright context as per task. And yes, and we are just getting started, and we believe this is how AI will design software.

And if there are any questions, please feel free to reach me at iman@katayo.tech. Thank you for your attention.