Bypassing the Multimodal Tax: Hybrid RAG, SQL RRF & UI Telemetry - Abed Matini, Ogilvy

Hello everyone, I'm Abed Matini, I'm Senior Backend Developer at Ogilvy. Thanks to AI Engineer World Fair 2026 for giving me a time slot for the online track, and I'm going to walk you through bypassing the multimodal text today, how we can have a framework-free Hybrid RAG, with raw SQL RRF, and live telemetry.

As you can see, I have got my live demo on theright side of the screen, so any slide that I pass, and if there is anything related

to my presentation, I can show you. I have a few tabs also ready here for you to show and walk you through

the slides and the demos.

Okay, let's begin.

So,

we have 2 problem for every document, uh, we have 2 problems that we try to solve today. One is when we start to chat with any LLM, and if we have to upload a document, we're usually going to just drag and drop a PDF, or a Word doc, or an image, and ask the chatbot to get ready for the questions that's going to come up after we upload those documents.

The issue with that is, as soon as we upload these documents, we're going to

basically lose some of our tokens that we're supposed to have only for

processing these documents, without even asking a question. So we already spend some tokens there without even asking any questions. That would be the first issue. And then, if we're building a chatbot and it's, uh, we want to have vector database there, we want to have keyword search and semantic search there, and we want to just add more tools, this is going to be too many tools

combined altogether. So in the production, in the production basically, chatbot, we're going to have

too many tools, and it's going to be complicated for us to manage

our production chatbot properly. So today I'm going to walk you through in 3 main sections. One is how we can upload our documents and get our documents and make it ready for our chatbot before users going to chat.

So on theright side, I do have a FAQ assistant chatbot that's running locally now for me, and

I'm going to show you the dashboard, the dashboard that I've created for it, and how we can chat and how we get the results. So this is going to be a supposedly FAQ

assistant for a sample employee handbook. So imagine that you have an HR company, and this HR has uploaded the handbook

for the employees to ask their question, any question they have, about leave, about,

like, different kinds of leaves, like sickness or parent, like, parental leave, or any other questions related to, or possibly an HR would have as a common question that they get every day. So that would be one sample that we're going to use to upload.

So our documents can get in the different formats. We can upload in different formats of PDF, like PowerPoint documents, and Word documents, or even image. So we're going to talk about that. We're going to talk about how we have to chunk these, because we're talking about RAG here.

We need to know how we can chunk this information properly based on our documents. So we're going to talk about

different types of chunking here as well. We can show you in the admin how we can choose different,

different strategies to chunk the documents. Then we're going to see how the search is going to work, how the embedding is going to work, how the database is going to be, how the search is going to be based on the keyword search or the semantic search.

And we're going to see about how RRF in the Python is going to work. And then we're going to talk about the observability, how we're going to check the safety, how we're going to observe all the process with LangFuse, and how we can prevent

prompt injection pattern or

any risky question that might come to LLM. I'm planning this talk for about under an hour, and this, the whole thing, the plan is the entire codebase can be easily be running on a GitHub codespace, which you can just pull from the repo, and one command, run, and it's going to be ready to test on GitHub codespaces.

So the stack we're using here are Python, FastAPI for the backend, React for the frontend, our PostgreSQL for our database, Docker for just easily being able to have the containers, and we can just reproduce it anywhere, rerun it anywhere, for example in codespaces or codespaces or any other environment we want.

Then we have Ollama, local language models, some local language models, and embedding models that's running only locally. That can be run on any server as well. It doesn't need a GPU. The CPU is going to be good enough for it.

So any staging server, any codespace,

going to be fine for it. And LangFuse for tracking what's going on and how we can see the chat and the latency

so we can improve our chats better.

This is the end-to-end blueprint of what we're going to talk about. So we're going to have a document, a few documents, as a source of truth for our chatbot. So we're going to upload some documents. So what we can do with Python is we're using Docklink to convert those raw documents to Markdown files, and then we're going to chunk those Markdown files into our Postgres.

And then we can also do some scan as well here, so that would happen as well. Then we would have some queries from the user. So the employee is asking questions, HR-related questions, and it's going to be retrieved from the database.

The result, relevant result, when the top rank would be passed through the LLM, and LLM would return the, in our chatbot that we have here, in a format of chat. So we can see, we can get our questions answered here, and we can observe in the LangFuse what's going on in our system.

So,

optimizing data ingest. So we have 2 different ways of optimizing. So if we first, I'm going to just throw it into the, we have a document, and if we're just uploading it on our

cloud chatbot anywhere, the issue we just, having, the one issue we have is we're spending tokens before we're doing anything. Another issue is you don't know how that document looks like there, and that chunking, how it's being chunked, and how basically that LLM is seeing it.

So that would be a risk quality. If there is a table that we don't know how it's been read there, so we kind of don't see. We're trying to have more clarity about how the document's being stored in our database and how it's being retrieved.

So another way would be a structure first. So we use the local CPU. If we're running it here, we can upload our documents. Our Docklink would convert it to a Markdown file, and we would know the better chunk, how the chunking is going to work.

I'm going to show you a few examples. And of course it's going to be a cheaper prompt if after that you want to talk to

an online cloud. But if you want to, if you want to have it locally, then it would be actually no cost for you. So the Docklink pipeline would be,

we would be running it, it's like a local, locally running it on our server. The Python is going to get that document, it's going to export it to the Markdown and save that Markdown file. Then it's going to chunk it based on different strategy.

And we're using embedding models to embed it to our database and insert that to our vector database. So if we say, why local? Because imagine if 1 or 2 pages, the cost is next to nothing, but if you have 200 pages, then the cost is going to be too much.

And if you have a big company with lots of employees, and they want to just ask, keep asking the same questions, or if they want to upload documents, then it's going to be, going to be a big cost.

So

there are 4 ways of chunking a document. Let me first, before I show you the 4 ways, let me show you the admin page that what I'm talking about. So imagine here you have, we are, we're having our FAQ admin page.

So

imagine you have, let me delete a few of the documents that I have chunked before here. So

you want to upload a new FAQ file for your HR, for your HR policy. I want to show, give your chatbot data updated with our RAG system. So imagine you have,

you have some question answers here. Before I do that, let me show you here what I mean. So this is the heading FAQ, yeah. So

one way that we can feed our chatbot with information is if we actually create a question answer file. So that would be easier to digest for our

system to understand what's going on. So part of that is, imagine we're having this entire FAQ file. So the entire, basically handbook. If we want to chunk this handbook and just upload it, imagine we're uploading the entire PDF.

This is going to be the way our PDF is going to look like after chunking. So it started by acknowledgement, then you see a few chunks that doesn't have any meaning, and then something starts here. There are some chunks that doesn't have any meaning.

For example here, signature date, that's basically kind of useless. So this is going to slow down and reduce the accuracy of our chat. It's going to increase hallucination. So

instead of uploading straight 28 pages, we can, if we can divide our documents to different chunks and just up to different files, relevant files, and clean up that data a bit, as much as we can, and then upload it, then we will have a better result.

So one way of

uploading in the FAQ case is to actually have your, you're turning your data to question and answers.

So talking about working hours, talking about different situations here. So you can have it based on questions and answers. Then this way you know when you reference your documents, you're having the correct reference and won't be any confusing.

And there are different ways to upload your documents. One way is you have the question and answer, and then you would like to upload it. This is my FAQ PDF file, same PDF file, and I would like to upload it based on

heading-based chunking strategy. That means, well, while I'm explaining the parsing, Docklink is happening in the background. So that means each

chunking would be based on the heading that Docklink would find and the answer that comes based on that. So each heading and answer is going to, heading as a question and answer is going to become a chunk. So we know our chunking is clean and it's easy to reference.

So when this chunking finishes in a second, we would see, okay, this is the first FAQ. It could chunk successfully to, based on the headings. So it recognized the heading question

and then the relevant answer. So each chunk has its own question and answer here. So if we say,

if we say, for example,

as our questions here,

do we, do we have, or something like this. So we should expect a kind of answer with the reference that's coming to that chunk. So we're talking about the heading chunk. It's going to understand the documents based on headings that you have.

That's how it's going to chunk each document, heading by heading. So as you can see here, our question was about how much, this was the question we prepared, and this was the answer we prepared. Sambo Kokar, 90% of individual employee premium and 75% dependent premium.

What we see here as a reference, we're getting that answer. And not only that, you're getting the correct answer. You're also getting the, you see that it's been, this answer has been generated based on this document, based on this chunk.

So it's easy to reference. It's easy, you know, where the issue is. If there is anything not working, you know why, then you can go track it, why thisright chunk hasn't been retrieved. So in this way you have more clear path towards how you can debug it.

Because I was, in our system, I was asking for

return top 2 chunks. It would suggesting me, I'll talk about all these models here. It would bring top 2 response. So then it was also checking that sample employee handbook, the big file here, sample employee handbook. As you can see here, that's a big chunk.

And if actually, it's a good example, because if you compare these 2, because we just drag, uploaded the entire chunk here, you can see although it's returned some information, but it's not that accurate. And you don't know, of course it went through that file and found something, but

you don't know how accurate that information could be. And you can't follow because the chunk is just

untrackable that easily. So one way of chunking is chunking based on the headings. Second way of chunking is based on the paragraph. So technically when you upload your documents, you would say, just chunk each paragraph you catch, have it as a separate chunk.

Doesn't matter there is a heading or not heading or anything. So if I want to go back to the, to our documents and say, let me first choose, I want to do a paragraph chunking. I want to say, okay, for paragraph,

let me find the paragraph. General policy for paragraph, I want to upload that.

So for general policy, it's uploading here as you would see, the whole file is just being paragraphed by paragraph being chunked. So there is no heading, there is no separate information. And if you look at the, if you look at that information as well, if I can find it here quickly.

General,

general paragraph, general policy paragraph. As you can see, the information is just paragraph by paragraph. So you kind of cleaned up your data again, but dividing it based on the different system. So instead of heading, you're talking about paragraphs.

That's another way of

basically feeding your RAG for your information. Another way would be a fixed 512 character. You can say, okay, one of the best practices for RAG is you just divide based on 512 character and you have 64% overlap between the chunks.

So it won't lose the track. So if your data is just too, is not organized, if there's just random data that you have and you can't clean that up, that's another way you can upload your document. So that would be, instead of this, you can choose another file and

you can say, basically, there's another paragraph for global travel

file here. So that's going to, when it's going to overlap here, when it's going to see here, the, about this chunking strategy, you would see, okay, it started for chunking and then it ended in the 512. And then it's again using that, I think it was 64, it was 64 overlap, yeah.

So it's using the 64 overlap here again. So

you would see it's basically

chopped it from here. And so from here till here is 512, and from here to here is 32, then another 32 from here to

here. So this is the 32, 32, 64 in total. So that's how it's relating each chunk together. It's happening same for here. As you can see, it's happening same for here. It's the way that, but you see the issue with that is, it's coming here to assist, an assistant.

So it's breaking. It's not the best, but in some cases it works for you. Another way that we can do is sentence base. So it can be based on, it's counting based on this number of sentences and it chop it there.

There are different use cases for each one. So another interesting one is, imagine that you, there's some maintenance coming, you're receiving some email and you just want to quickly upload

the document in your chat without,

without, because you don't have the, it's just a screenshot and you can't do anything. Imagine you just receive an email and says, okay, there is a maintenance plan for this day from this hour and you just want to upload this in your chat for the user.

So if me as an employee ask, okay, what is, is there any maintenance plan for this weekend, then I would know immediately. So another way that you can easily upload that screenshot as well.

Let me find, easily upload that.

Oh, sorry, that was in my screenshots. So the screenshot you can upload, then we're using another

model that just turns that image to text and text to .md. And then we're using this,

basically, what's the name, the sentence model for the emails that can convert

that image to a text and then it can feed. So we can see we're using the chunking strategy, a sentence group. It's turned that image to text and it's already good for the, for our chatbot. So we can, we can index it into our knowledge base now.

So then it's going to be a next one that's been uploaded here, screenshot. We can see how it's been chunked. Because this is just an email, it doesn't have heading, it doesn't necessarily have all the paragraph chunked properly.

So, and if it's a shortage of time, it's a small message, temporary change. You don't need to sit and clean up the data for here. You just want to drag and drop, update it for a weekend and everybody knows, okay, this weekend we're having a maintenance plan.

So when they will come here and ask, is there any maintenance plan for this weekend, then technically we should get the result of, okay, this is the maintenance plan that's been happening and the reference documents. So the maintenance, this is the exact maintenance window that we got.

And actually the reference, you can see that's a screenshot that we uploaded just now. So if we do, if we had I done the screen,

the file name better, there would have been even a more relevant. And somebody says, okay, according to this information that I got from here, we have a maintenance plan tomorrow from 6:00 PM onwards till midnight. So, and I can easily archive, delete any of these and then it won't be visible as well.

So these are basically different ways that you can chunk. That's one of the, that's basically the, one of the most important part of the whole system. Because most of the LLMs would just work. But the way you want to feed that data and you want to clean that data and not having just drag and drop the entire handbook here and have a messy data, that would help you a lot to have a more successful, in this case, FAQ chatbot.

Yeah, so these different strategies would help you as well. And yeah, let's move on to the next slide.

So we're using, we're using Postgres and

we are also having a few agents that they are just Python codes. They are not really any, we're not calling another LLM as an agent to do something for us. The one reason for that is the speed. Because I'm running this locally, I'm having running this on Ollama.

I don't have, I don't need to, three, four agents to loop three times, four times. Because then I had to, I have to wait 20, 30 seconds for it and the user would lose interest. You just want one natural language to run.

The rest, if something can be done by a Python function, let the Python function then run it for you. You have also the full control over the function. It won't be any hallucination because you need to, you're going to write your test suite to just cover most of the situations.

So that would help you. The agents, I just want to show you something here in the settings. What can it be? So I can, I can

off and have it, agents off and on as well. So I can say, if somebody wants a current date, then this function easily can be called and retrieved. You don't need the LLM to bring you back the current date.

You don't need to LLM to calculate for you something. Because also another plan for expanding this, expanding this chatbot is having it for customer service for, that's going to talk about the products and the prices and calculations. So that's another thing, product info or even having another deeper search on the agent mode.

But the point with that is then, because it's going to be another search, then you would lose that referencing on that. That would be important. So while I'm here, I'll show you what do I have

as an LLM actually I'm using here. I'm using the smallest Qwen 2.5. So it's 0.5 billion is parameters, instruct model. And it's, I think it's 400 megabytes only. So it's that small. And I'm using this model also for, for the agent mode if I want to run.

So I would run in two, I would run another one and I have to run the agent mode. It's going to make it slightly slower. And then, yeah, let's go back to the slides here.

So, yeah, this way of working, you have a full visibility on your Python function. That's what's going on. So you only pass the needed information, clear information to LLM and receive a clear information. So it would reduce the chance of hallucination dramatically.

Also showing you here about

vectors and text in the database. So, so when we're going to chunk, we're going to chunk the information. This is based on the ID and the section and what kind of vector embedding are we doing. And then we have our chunk.

We have, we can see

basically how it's going to find the nearest neighbor here. And

also the keyword match at the same time. Because if you're having a customer service product chatbot, you don't need the nearest information. You need the exact information. If it comes to the number as well, if it comes to the product as well, you need the exact, if it's a medical chatbot, you need the exact medication.

You don't need something similar or close to it. So we need to have both, both basically

keyword search and the semantic search together. So we'll have a hybrid search that's giving you a better result. And we're talking about the number of retrievals that we're going to have. I have it in the system limited to two because in this case you don't need more than that.

But if you want to retrieve some information, if it's a product, you might have, you don't want to just bring top two products that's being there for the user because you might have the same brand, like 20 product or 50 product of the same brand.

If you don't let the window, your window to pull those information, those products never going to be sold and never going to be shown in the chatbot. So it's important how many you can fetch.

And then the, the more about the retrieval that when we're using the BM25 to get the exact keywords. So the previous slide was about to get the most closest with the cosine distance that we have. Because when we upload information with the, in our vector, in our vector database, the information with the similar meanings are going to sit next to each other.

And then when we user search for something and ask for something, then

our information, our, and our, our search is going to convert that question, that query in that database and go find the similar closest information to that question and will return it. So that's the reason we can't just return one.

We need to return more than those. So we have multiple answers and more closer answers. But here in the sparse retrieval, we need the exact answer. So we would need to, we would need to filter it based on the language, for example.

We need to based on the ID or exact product name, SKUs or

brand name or something. So that would be the, the keyword retrieval

for us. And then how we do the ranking based on, which is very important because when we have the both vector and the BM25, we want to mix these and still bring the top,

top two or top five relevant answers that we need for us. So more accurate if we bring too many, if we bring top 20 here and

bring, shows top 20 answer is going to be quite confusing for the user. So you just want to have, make sure you bring in theright amount based on your

use case, based on your use case. So if you're having products, you would bring more, you retrieve more. If it's something medical, you'll retrieve less. You don't want to, information to be all over the place. You want to accurate, more accurate information because you would have more liability, obviously, on that.

Another point here is,

so this is, this is more about the

reranker. So when we have our reranker here, that would also

bring the information, bring that, and so after that score,

how many of those information that shows by default. So I'm only showing two there. So I've briefly talked about the agent mode via direct RAG. The direct RAG is, is just following the, it's a fixed pipeline. It's not going to go calling anything.

You know, the path is quite clear. You have embedded, embedded query, you have a hybrid retriever and you just bring up the answer. It's good if compliance is quite important for you. But agent mode, you have a few extra tools that's going to be called.

So a search is a compare product. These can bring more information back. It's less in your control. So

that's quite important and it takes longer because you have another agent that's going to work. So basically it's getting this information and then you have agent mode based on your request is going to call. So that would, that would take slightly longer, slightly longer.

About the, about the telemetry and application guardrails. So

can use LangFuse for, we integrated LangFuse in our functions. So we can see what kind of answers we have. It's quite easy to, everything is local. It's no outside calling or anything. So it's easy to track the information that I was chatting with the widget and I wasn't logged in.

It's a FAQ, FAQ chat. That's FAQ agent in his name of the, the whole projects that I'm running there. So each conversation has an ID.

You can track if there was any maintenance for this weekend and you can track the answer for it. You can see what model I've been using. You can see I've returned the top two answers and the, because it was top two is referenced twice on that image.

Agent mode was off. It was a chat. Two chunk of information was returned and it took this long milliseconds. And yeah, that's basically the information we have. If you have external models, you can even estimate the cost that you're spending.

That's another thing you can, you can use that for. And if you have different sessions, different users, you can track the logins. So that's give you a good insight of how users are treating

with your chats. So,

yeah, so as I said about the user sessions and IDs

and then you would have the guardrails as well. So something, because of using functions for it, we can have a better control over it. So if I ask this chatbot, how do I

treat flu? I have created a section to say without going to chat, but the point of lots of these guardrails is that the issue should stop before we

the prompt injection or anything that we have or a question that is we're not supposed to respond. We shouldn't even send it to the LLM to generate anything. So before even in the code, I can have, so we have created a section here to say if there is a medical escalation, bring this medical message back.

And you can see how the system prompt is quite small because I'm basically prompting everything in the code. So although the system prompt looks quite small and just few sentences, I'm actually prompting way more in the code instead of prompt.

We use, I'm using the code to say what do's and don'ts. And another example is the medical escalation and you can add more based on your

concerns you would have for your product. So yeah, so this is a example for that. Can be, can needs to be expanded with different keywords, with different kind of sentences. It should be cured and see how many times somebody could pass this or not.

But technically that would be example of how we should block, how you have our guardrails should block any query that's not related. Can do the same with the injection and so on.

What else do we have here? We talked about the LangFuse and we just need to create a, generate a key and you have it in your system and it would, it would work. It's just a little bit about observability that I've talked about.

And then,

then the injection as well. So the tools for prompt injection, again, it shouldn't go to the LLM and being stopped there. It should be stopped before, before it's being sent there. And yeah, we're using intent regex and term dictionaries and LLM classifier to stop any of these issues.

And of course we can expand it more. But the good thing about that, because it's just a code, it doesn't, it doesn't need the long prompt. It doesn't need anything. Outcome would be obvious. You know what you are blocking and what you're not blocking.

So it's not like a LLM that sometimes likes to listen to you as your, as the instruction and sometimes just escape and does its own thing. So part of that integrating all of these in your Python before sending it is that you can have rigid test and to say, okay, this is what I'm blocking and this is why.

And it passes all the tests. So you can be more sure about that.

So yeah, just a small slide about the, we can just create your, the, and I'm having it as a, I'm having it here as a, as a, as a widget. It could be also done here instead of in a bigger chat as well.

So another thing is you have to, in your chat as a part of the rules, you should say also if somebody's understand the terms that they're using. So if not, the person within LLM will start the chat, you know.

And

also I gave the, the widget capability if I want. So user, you can reset the consent as well. It's just for my demo purposes. So, and then the, the widget would show here. If I say no, this wouldn't work.

If I reset the consent, if I say yes, then it will start.

So yeah, so that's also about the widgets and is because I'm using React, it can be used anything else. Any other framework as well. I just use it for simplicity. I'm here saying about, okay, the models I used.

So I've used two models.

I pulled the Qwen 2.5. Actually first I've had the 7 billion, but it took longer. So for a smaller server, I've changed it to, let me show you the admin here. I've changed it to

0.5. Otherwise I have different options. So I was trying with different models. If you have a bigger model, it's just going to take longer and it might be too wordy. Interesting finding from my side was you don't need the biggest model.

You, if you can vet your data before sending it to your LLM, a smaller, smallest model can sort you out and get a good

answer for you. And it would give you less hallucination. It would, it's safer because it's not going to make up information. If it doesn't have information, it simply is going to say, no, I don't have that information. So that's another way of guardrailing to not, to not have that issue of your LLM randomly generating something.

I have BGM3 embedding models. So when I upload anything here that, upload anything here, that embedding models basically, that embedding models do the embedding for me. So technically you need two models only. The smallest chat model and one of the smallest,

one of the smallest model for embedding. And then if you have it on Docker, one click everything's going to be running. You would need your Python requirements to be installed. And then you have your React working on the front end.

And then that's all you need. One more thing that I should be adding here. I haven't, the LangFuse also

should be added here. That's going to be running in a different port. That's missing from here. So the key takeaway for

this is the structure Markdown first. So we use Docklink for do the heavy job. And then we use the Postgres vector database to just store everything. Didn't use any specific framework. We just used

Ollama for our chat and no other paid framework anywhere else. We use LangFuse for telemetry. That was also free to use. And this can be escalated to different products as well. It can be document, catalog, or from any database.

Thank you very much for your time. I would like to get connect with you and if you have any questions, if you have any concern or something that you figure that would be better to integrate, please share your idea with me either in the comments or message me on LinkedIn.

Yeah, thank you so much. Thank you, AI engineer, for the opportunity and we'll see you next time. Cheers.