What Breaks When You Build AI Under Sovereignty Constraints - Bilge Yücel, deepset GmbH

Hello everyone, can you all hear me well? Allright, amazing. Uh, thank you for joining the session. In this one we're going to talk about sovereignty. My name is Bilge Yücel, I work as a Senior Developer Relations Engineer at Deepset, but I want to start with a question: so how many of you are familiar with the term of sovereignty?

Wow, okay, amazing. Maybe I don't need to do a lot of introduction for this one. And, um, just a little bit of context here about Deepset: Deepset is the company behind the open-source orchestration framework called Haystack. We also have our enterprise platform, and with those products we are solving custom AI challenges for big organizations like Airbus, Bosch, Siemens, but also public sector organizations like European Commission, Federal Ministry of Research and Technology and Space in Germany, and other ministries as well.

So you can imagine sovereignty is a very big and important topic for us.

So here is a, like, a policy definition you can say: what sovereign, sovereign AI is. Sovereign AI is the ability of an organization to design, deploy, and operate AI systems on its own terms. But we are all engineers here, we are not policymakers, we are not lawyers.

So if you, if you turn that into a technical, uh, definition, it's basically having explicit control over data flow, model choice, infrastructure, observability, and operations.

And, um, for us to understand it better, I want to split that into four pillars. So the first one is data sovereignty. So it's about where you store your data, where does it get processed. The second one is model sovereignty: who controls the running models, the origin of the training data.

The third one is infrastructure sovereignty, so where does compute happen. And the last one is operational sovereignty, so is your, uh, application, is your system traceable, who can update it, who owns the incident response. So let's go into the depths of this for a minute.

Um, data sovereignty. Data is the most important asse- assets that we have in, in an enterprise. And for us to have data sovereignty, data should be processed and stored within trusted jurisdictions to meet compliance requirements. What does that mean?

So basically GDPR says that your, the European citizen data should stay within Europe. But if you send that data to an embedding model, to an embedding API hosted in Virginia, in the US, then you are already losing the control of your data.

So that's against data sovereignty. And the other aspect is access permissions. So this is not exactly about how the data is stored and processed, but if there are users in your organization, within your organization, that are, that has access to data that are not, that they are not supposed to see, that is also a breach of data sovereignty.

The other one is infrastructure sovereignty. So where does compute happen? It means, like, we, we have the AI application layer, so we have our RAG pipeline ingestion, we have agents, a, these tools, the tools for the agents, and they all run somewhere.

And where this application layer runs defines the infrastructure sovereignty of your system. Um, and this, and there's this whole spectrum from max control to max convenience. Maybe you run everything in an air-gapped environment, it gives you the EU AI Act safe.

Maybe you run on a, uh, private VPC, this gives you GDPR safety. There is also sovereign cloud, this kind of depends on the provider that you are going with. And there's also, at the very end, there's SAS. So with SAS you kind of face the Cloud Act risk.

Meaning that if you are, for example, using a US headquartered company, although you store, you run all your data, all your application within Europe, uh, they have ability to get access to your running data. So that's against the sovereignty that you have over your system.

The other pillar is model sovereignty. So who controls the model and the origin of the training data. Um, so the, you should have the freedom to choose and switch models. But what it means that if your system can only work with one specific model, you are tightly coupled with that model provider.

If they are, if the API is down, then you lose access. If they increase the price, then you also have a cost issue. Um, this is completely against the model sovereignty idea. And the other one is swappability without inf architectural changes.

So maybe you're not coupled, tightly coupled with that model provider, maybe you can use other models as well. But if your system, if your, uh, code doesn't allow that immediately, then you are, although technically you are not tied to that provider, but you cannot just change the whole code base within one day.

And the last one is training data origin. So this is a bit controversial because we don't have a way to know, like, where the model was trained, which data was used to train this model. Uh, but, uh, if a model provider is a European company, then they have a better advantage than other companies based in the US.

The, the last pillar is operational sovereignty. So, like, building the whole AI system is one aspect, but monitoring, so basing, basically maintaining this whole system by monitoring, evaluating, managing is also another. Uh, operational sovereignty is about monitoring how these systems behave in production, including model inputs and outputs.

And in high-stake environments like HR or finance, it requires human-in-the-loop. And managing versioning, updates to models, and the application layer, uh, in a controlled, auditable way also goes under this operational sovereignty.

But the good news is sovereignty is a spectrum. So not everyone needs, like, needs to be sovereign in all of these pillars. Of course, like, uh, if you are in finance, in healthcare, in hi-high-stake environment, in high-stake domains, you might need an fully air-gapped solution.

But if you are an enterprise or startup working in a different domain, then maybe you don't need everything at all, you're, you don't need to comply with every sovereignty pillar that I showed you. The important thing here is you need to know the level of control, so the, the level of vendor lock-in you have with your system.

So you are here in the conference this week, but next week on Monday you're going to go back to work and maybe your manager or CIO comes and says, like, okay, now we need to make, like, we have this working system, but now we need to make it sovereign.

Uh, and in this slide I want to show you what you do first usually and what you break in this existing system. So probably intuitively the first thing that you do is the model. And you replace the frontier API that you have with a self-hosted model.

And what happens? Then you need to translate this whole API logic to this new model architecture. Maybe you need to update the prompts. Basically you need to evaluate the performance of this system from scratch and you need to write a lot of code.

The other one is you move private data into the required jurisdiction because you notice that, okay, we have some information hosted in, in the US, now we need to take them somewhere in Europe. And you do that, but you find yourself managing multiple databases and instances, then search becomes a problem.

So how do you handle search? Do you do classification, like query classification first, or do you send the request to both of those databases and get information from those? That's just one challenge that comes with this change. And the other one is you replace managed infra with on-prem and you immediately notice how much you had vendor lock-in in this area.

You, because now you also start thinking about, okay, Kubernetes cluster management, how do I, like, deal with all these, um, all these things that all these model pro, all these cloud providers were handling for you. And there are also hardware limitations because now you need to think about, okay, I have this application layer running in GPU, CPU, but now I have my model run- running GPU, so I need to connect them in a way.

And this connection management, network management also becomes a problem. And the last one is you incorporate observability and tracing. I mean, if you, if you didn't have observability until this point in your system, that's an important issue already.

But if you, uh, now you, when you need to incorporate it, you notice that you have this fully black box thing. So you don't know actually what's happening in this AI application layer. So you need to understand now because you need to log it somewhere so that your system is auditable.

And there is also version control aspect. How do you do version control for your application layer for this whole system that you are managing? And I want to hear shamelessly plug Haystack because a good orchestration framework, I think, solves some of these problems.

I mean, good orchestration cannot solve your GPU limitation, but can help you with so many different stuff. Uh, the, the first thing is, uh, Haystack is, has this consistent interface. So basically when you want to, uh, transition your system from cloud to a self-hosted, uh, application, you can easily do so.

Just changing a couple lines of code and just focus on the hardware that you have and this hardware connection. The second one is explicit data flow. So every input and output in a Haystack application is typed and declared.

So you can read the pipeline definition or, or this whole application, um, in a, in a, in a regular way and know exactly what data was where. And even in less deterministic architectures like agents and, uh, like agents, the whole data, these tools, tool calls and tool outputs are traceable.

The third one is YAML. So the Haystack applications are serializable to YAML, making them very easy to version. So you can just, after creating your application with Haystack, you can turn that into YAML, put it into your version control, and then when you need to go back to the history, you just need to go back to the, the commit and you, you can see the hash.

And the last part is it's truly open source. So there is no black box, no hidden assumptions. And when you need to customize some code or extend one of the components, you can easily do so because you actually understand what's happening under the hood.

And here is one of, here's a sovereign architecture that I want to show you, um, today, just as, like, a high-level overview. Uh, imagine you are building this agent, but it needs to be some sort of sovereign. So first thing that you do, probably you add some guardrails before.

And then because this guardrail needs to check if there's, like, a prompt injection coming from the user import, coming through the user input, and it needs to also check some specific regulatory checks maybe because this is, like, a very specific agent that needs to be used in a certain way and you want to check if user has this intention.

And if it's unsafe, it just leaves the application layer immediately. Uh, and if it's a safe, uh, request, safe input, then it goes to the agent. And the agent here is basically an LLM with a system prompt and lots of different tools.

And these tools can be API calls. Maybe they are connected to your knowledge base, so you are, you are doing some sort of search. Maybe you are using other agents if you are dealing with a, um, multi-agentic systems.

And there are also MCP servers, of course, that are connected to your agent. And agent creates an input, so does the, uh, does the work for you. And there is the last guardrail doing compliance checks because you also don't want to leak sensitive information to your user.

And then there becomes an output. And, uh, but of course, like, how you design a system as a sovereign system is a challenge. And the, the duty of Haystack here is to make sure that everything is swappable, traceable, and without vendor lock-in.

Um, so you know you can work with theright level of sovereignty, uh, that you need. And here are some of the, uh, the tools that I picked from, uh, that I picked that you can work with Haystack. Um, so maybe you, uh, you decide you need, like, different models based on the task.

Maybe you decide that for public data you can steal some proprietary models from those providers, so you don't worry about those. But for some of the tasks, for guardrails, maybe for your knowledge base, for, uh, for your LLM, you think that you need local models running on, on, like, self-hosted local models from other different providers like, uh, Mistral, Google, Nvidia, uh, and Jina.

And then of course, uh, there's traceability aspect. And, uh, as I said, in Haystack, everything, like, every input coming to a component and every output, but as well as the traces in the agent is easily visible. So you can just connect those spans to your LLM observability tool.

And you can, since there is open telemetry integration, you can also implement your own observability. And as the last part, there's the storage. I wanted to pick some of the providers who can give you cloud and also the open source version of them.

So you can easily host these ones, uh, locally, uh, on your prem.

And here is the, uh, here is the guardrail, um, code. So, uh, basically how you connect a guardrail or how you define a guardrail is quite simple with Haystack. You just start with a model provider. For example, in this one it's an Nvidia chat generator.

You give the name of the model that you want to use. And by connecting it to your LLM message router, it helps you do the classification. So it checks the input. If it's safe, it goes to the safe route.

And if it's unsafe, it goes to the unsafe route.

And then you add, you start adding tools. So you connect your MCP server that you host locally with, uh, with MCP tool set, and you give the names of the tools that you want to pick from that MCP server.

Because probably you have lots of tools running on that, on that MCP server. You don't want to get all of them. Uh, you don't want to have access to all of them with just one agent. And you pick, for example, knowledge base search and generating PDF report one.

And, but of course, like, you can add different tools. You can add just, like, you can start defining tools from a Python function or you can connect other components in Haystack, like agent component, uh, into a tool. Or maybe there is some functionality that you define, like data ingestion or rack pipeline, and you can also convert those workflows into tools as well.

And by after defining all these tools, you put them in a searchable tool set. Uh, so this gives you a dynamic tool search with BM25 because you probably now have more than hundreds of tools for your agent, but you don't want to fill in the whole context with just tool definitions.

Then there's this agent component. So bas-basically you define a system prompt. You say, like, you're a sovereign agent with access to multiple tools and you define the intention of this agent. And then you put a brain to this agent, basically a chat generator.

If you are, uh, running this model locally on-prem, maybe you create your own custom component saying that this is my on-prem chat generator connected to this internal company, uh, URL and getting the model inference from that. But of course, if you have an OpenAI API compatible endpoint, you can already use an existing component in Haystack.

And then you give the tools and then you define confirmation strategies. So basically you incorporate human in the loop saying that, so if the user wants to submit a request, the agent should always ask for human approval. But if the agent wants to use the less payment request tool, maybe ask for permission first from the human and then you can use this tool as much as possible, uh, as much as you want, uh, in this whole cycle.

Then you bring it all together. So you start defining your pipeline. You first add the component as a tracer that connects to your, uh, LLM observability. Then you put the input guardrail. You connect your agent, the output guardrail, and you just, you can just run this agent saying that pull the outstanding payment request for Q3 and generate a PDF for me.

And by using the tools that this agent has, it creates the PDF for you and saves it under this, uh, directory under this name.

Um, so with that, um, I want to give you just, like, when we are coming to the end of the presentation, I want to show you a checklist. So if you want to check if your, uh, system is sovereign, you can, you can take a look at these questions.

So you can think about, can you swap models without changing the application logic? Do you have reproducible run logs stored in a compliant way? And can your team respond to an incident without calling a vendor, uh, like one of those hyperscalers?

Um, thank you for joining the session. Uh, and you can get the presentations and ask the questions to me by filling in the, uh, the form. And you can also find me on social media if you want to connect there.

I'm happy to talk about you about, um, agents, Haystack, and especially in a sovereign setting. Thank you.