Information Retrieval from the Ground Up - Philipp Krenn, Elastic

Let's get going. Audio is okay for everybody? I have some slight feedback, but I'll try to manage. I hope it's okay for you. Um, hi, I'm Philippe. Let's talk a bit about retrieval. I'll show you some retrieval from the ground up.

We'll keep it pretty hands-on. You will have a chance to follow along and do everything that I show you as well. I have, like, a demo instance that you can use, or you can just watch me. If you have any questions, ask at any moment.

If anything is too small to read, shout, and we'll try to make it larger. We'll try to adjust as we go along. So,

I guess we're not over RAG yet, but RAG is a thing, and we'll focus on the R in RAG. The Retrieval Augmented Generation, we'll just focus on the retrieval. Um, just, let's see where we are with retrieval. Quick show of hands.

Who has done RAG before? Okay, that's about half or so. Who has done anything with vector search and RAG? Do I need vector search for RAG, or can I do anything else?

Can you do anything with Probstash then?

Yeah. Yeah, so you can do anything. Retrieval is actually a very old thing. Depending on how you define it, it might be 50, 70, whatever years old. Retrieval is just getting theright context to the generation. I, like, ignore all the generation for today.

We'll keep it very simple. We'll just focus on the retrieval part of getting theright information in. Partially from the old stuff, like the classics, but we'll get to some new things as well as we go along. Who has done keyword search before?

Just, that is fewer than vector search, I feel like. Which is, um, which almost reminds me of, like, 15 years ago or so when NoSQL came up. Like, more people had done MongoDB, Redis, whatever else, rather than SQL.

That has changed again. I think it will be kind of similar for retrieval. The way I would always put it is that vector search is a feature of retrieval. It's only one of multiple features, or many features, that you want in retrieval.

And we'll see a bit why and how, and we'll dive into those details. So I work for Elastic, the company behind Elasticsearch. We're the most downloaded, deployed, whatever else, search engine. We do vector search, we do keyword search, we do hybrid search.

We'll dive into various examples. Everything that I will show you works, well, the query language is Elasticsearch, but if you use anything built on Apache Lucene, everything behaves very similarly. If you use something that is a clone or close to Lucene, like anything built on Tentivii or anything like that, it will be very similar.

The foundation of keyword search and vector search will apply broadly everywhere. So let's get going. We'll keep this pretty hands-on. Who remembers in Star Wars when he's making that hand gesture? What is the quote?

I'm not sure what's your quote.

These are not the droids you're looking for. We'll keep this relatively Star Wars-based. Feel free to come in and filter on the side, so whatever. I'm afraid we have, I think, one chair over there otherwise, and one down there otherwise.

It's getting a bit

full. Okay, let's look at this of what "These are not the droids you're looking for" does for search. And I will start kind of like with the classic approach. Keyword search, or lexical search, is like you search for the words that you have stored, and we want to find what is relevant in our examples.

If you want to follow along, there is a gist which has all the code that I'm showing you. It's at elast/ai.engineer.

There is one important thing. It's, I have one shared instance basically for everybody, so you can all just use this without signing up for any accounts or anything. So this is just a cloud instance that you can use.

There is, my handle is in the index name. If you don't want to fight and overwrite each other's data, replace that with your unique handle or something that is specific to you, because otherwise you will all work on the same index and kind of like overwrite each other's data.

You can also just watch me. If you don't have a computer handy, that's fine. But if you want to follow along, elast/ai.engineer, there will be a gist. It will have the connection string, like there is a URL, and then the credentials are workshop, workshop.

If you go in to log in, it will say log in with Elasticsearch. That's where you use workshop, workshop. Then you'll be able to log in, and you can just run all the queries that I'm showing you. You can try out stuff.

If you have any questions, shout. I have a couple of colleagues dispersed in the room, so if we have too many questions, we'll somehow divide and conquer. So let's get going and see what we have here. And I'll show you most of the stuff live.

I think this is large enough in the back row. If it's not large enough for anybody, shout, and we'll see how much larger I can make this.

And let me turn off the Wi-Fi and hope that my wired connection is good enough.

Let's refresh to see. Ooh, maybe we'll use my phone after all.

Okay.

Let's try this again.

Okay, this is no good.

Out you go.

Okay, hardest problem of the day solved. We have network. Okay, so we have the sentence, "These are not the droids you're looking for." And we'll start with the classic keyword or lexical search, like what happens behind the scenes.

So what you generally want to do is you basically want to extract the individual words and then make them searchable. So here I'm not storing anything. I'm just looking at, like, how would that look like if I stored something.

I'm using this underscore analyze endpoint to see what I will actually store in the background to make then searchable. So I have, "These are not the droids you're looking for." And you see, "These are m not m the droids you are looking for."

In Western languages, the first step that hap or, every time or everywhere, the first step that happens is the tokenization. In Western languages, it's pretty simple. It's normally any white spaces and punctuation marks where you just break out the individual tokens.

Especially Asian languages are a bit more complicated around that, but we'll gloss over that for today. And we have a couple of interesting pieces of information here. So we have the token. So these is the first token. We have the start offset and the end offset.

Why would I need a start and an end offset? Why would I extract and then store that potentially? Any guesses? Yeah?

So you have a reference back to the original text?

Yes. Especially if you have a longer text, you would want to have that highlighting feature that you want to say, "This is where my hit actually was." So if I'm searching for these, which is maybe not a great word, but you would very easily be able to highlight where you had actually the match.

And the trick that you're doing in search, generally what differentiates it from a database is a database just stores what you give it and then does basically almost everything at query or search time. Versus a search engine does a lot of the work at ingestion or when you store the data.

So we break out the individual tokens. We calculate these offsets and store them, so whenever we have a match afterwards, we never need to reanalyze the actual text, which could potentially be multiple pages long. But we could just highlight where we have that match because we have extracted those positions.

We have a position. Why would I want to store the position with the text that I have?

Yeah?

For annotations?

Yeah, annotations. So the main use case that you have is if you have these positions, and later on we'll briefly look at if you want to look for a phrase, if you want to look for this word followed by that word.

So you could then just look for all the texts that contain these words, but then you could also just compare the positions and basically look for n, n plus 1, et cetera. And you never need to look at the string again, but you can just look at the positions to figure out, like, this was one continuous phrase, even if you have broken it out into the individual tokens.

Most of the things that we see here is alpha num for alphanumeric. An alternative would be synonyms. We'll skip over synonym definition because it's not fun to define tons of synonyms. But this is all the things that we are storing here in the background.

You can also customize this analysis, and that is one of the features, again, of full-text search and lexical searches, that you pre-process a lot of the information to make that search afterwards faster. So here you can see I'm stripping out the HTML because nobody's going to search for this emphasis tag.

I use a standard tokenizer that breaks up, for example, on dashes. You will see that. Alternatives would be whitespace, that you only break up on whitespaces. I lowercase everything, which is most of the times what you want because nobody searches in Google with proper casing, or at least maybe my parents, but nobody else searches with proper casing in Google.

We remove stopwords. We'll get to stopwords in a moment. And we do stemming with the snowball stem. What stemming is, it basically reduces a word down to the root. So you don't care about singular, plural, or, like, the flexion of a verb anymore, but you really care more about the concept.

So if I run through that analysis, does anybody want to guess what will remain of this phrase, or which tokens will be extracted and in what form?

Not a lot will remain.

Two?

Droid and look?

Yeah, close. So we'll actually have three. So we have droid, you, and look. And you can see all the others were stopwords, which were removed. The stemming reduced looking down to look because we don't care if it looks, looking, look.

We just reduce it to the word stem. So we do this when we store the data. And by the way, when you search afterwards, your text would run through the same analysis, that you would have exact matches. So you don't need to do anything like a like search anymore in the future.

So this will be much more performant than anything that you would do in a relational database because you have direct matches. And we'll look at the data structure behind it in a moment. But what we get is droid, you, look with theright positions.

So for example, if we searched for droid, you, we could easily retrieve that because we have the positions, even though that is a weird phrase. Do we start indexing at 0 or 1?

0 but 1.

0, yes. It's the onlyright way.

There's no discussion here. So we are the positions are based starting at 0, and these are the tokens that are remaining. If you do this for a different language, like you might hear I'm a native German speaker. This is the text in German, and you would, if you use a German analyzer, it would know the rules for German and then would analyze the text in theright way.

So then you would have remaining droid, den, such.

Anybody wants to guess what happens if I have the wrong language for a text?

It will be tokenized?

It will go very poorly. Because the so how this works is, basically you have rules for every single language. It's like, what is the stopword? How does stemming work? If you apply the wrong rules, you basically just get wrong stuff out.

So it will not do what you want. So what you get here is, like, this is an article, but, well, in English, the rule is an s at the end just gets stemmed away, even though this doesn't make any sense.

So you apply the wrong rules, and you just produce pretty much garbage. So don't do that. Just to give you another example,

French. This is the same phrase in French. And then you see droid, la, and rechercher are the words that are remaining in these examples. Otherwise it works the same, but you need to have theright analysis for what you're doing.

Otherwise you'll just produce garbage.

A couple of things as we are going along. The stopword list by default, which you could overwrite, is relatively short. This is linguists have spent many years figuring out what are theright lists of stopwords, and you don't want to have too many or too few.

In English, it's I always forget. I think it's 33 or so. This is where you can find it in the source code. It's, I don't want to say well hidden, but it's not easy to find either. So every language has, like, a list of stopwords that are defined that will be automatically removed for, "These are not the droids you are looking for."

By accident, more or less, we had a lot of stopwords that why not a lot remained here in the phrase. And then for all other languages, you will have a similar list of stopwords.

Should you always remove stopwords?

Yes, no.

Like, remove context when you have things like not?

Yes. That is, by the way, another not is a very good so I'm not sure if everybody heard that. The comment was about not. One important thing here, we're talking about lexical or keyword search, which is dumb but scalable.

It doesn't know understand if there is a droid or there is no droid. It's just defined as a stopword. It does just keyword matching. That is, in vector search or anything with a machine learning model behind it will be a bit of a different story afterwards, where these things might make a difference.

But this is very simple because it just matches on similar strings, basically. It doesn't understand the context. It doesn't know what's going on. That's why the linguists decided not is a good stopword. You could overwrite that if for your specific use case, this is not a good idea.

Always removing stopwords? Yes, no, maybe. So our favorite phrase is, "It depends." And then you have to explain, like, what it depends on. So what it depends on is there are scenarios where removing all stopwords does not give you the desired result.

And maybe you want to have, like, a text with and without stopwords. Like, sometimes stopwords are just, like, a lot of noise that blow up the index size and don't really add a lot of value. That's why we have defined them and tried to remove them by default.

But if you had, for example, "to be or not to be," these are all stopwords. This would all be gone when you run it through analysis. So it is tricky to figure out, like, what is theright balance for stopwords or what works for your use case.

But you might have unexpected surprises in all of this. Okay, we've seen the German examples.

Let's do some more queries. Or let's actually store something. So far we only pretended or we only looked at what would happen if we would store something. Now I'm actually creating an index. Again, if you're running this yourself, please use a different name than me.

Just replace all my handle instances with your handle or whatever you want. Since this is a shared instance, if we have too many collisions, I might jump to another instance that I have as a backup in the background.

But what I'm doing here is I'm creating my this analysis pipeline that I have looked at before. Like, I'm throwing out the HTML. I use a standard tokenizer, lowercasing, stopword removal, and stemming. And then I call this my analyzer, and then I'm basically applying this my analyzer on a field called quote.

This we call this a mapping. It's kind of like the equivalent to the schema in a relational database, but this defines how different fields behave.

Okay, and somebody did not replace

the query.

By the way, you need to keep user underscore. Let me quickly do this

myself. Oops.

I should have seen this coming.

We want to replace and will use

oops.

Please don't copy that.

And I want to

okay. Looks like it worked.

Let's try it again.

So we're creating our own index, and now just to double-check, I'll just, again, run this underscore analyze against this field that I've set up to just double-check that I've set it up correctly. And now I'm actually starting to store documents.

Bless you. So we'll store these are not the droids you're looking for. I have two others that I'll index just so we have a bit more to search. No, I am your father. Any guesses what will remain here?

Father.

Father, yeah.

Okay, let's try this out. Let me copy my

this one actually has way fewer stopwords than you would expect. Let's quickly do this.

Since I didn't do the HTML removal, let's take these out manually. So what you get is, no, I am your father. And this was stupid because this was not what I wanted. We need to run this against theright analysis.

This happens when you copy-paste. Okay.

Ah, sorry. And we'll do text.

No, I think I've patched this back together. Okay, I am your father. So no is the only stopword in this list, actually. No was on the stopword list. All the others are not.

Okay, let's try another one. Obi-Wan never told you what happened to your father.

How many tokens will Obi-Wan be? Two, one.

Three.

No, Obi-Wan will be two. Like, Obi-Wan because we used a default tokenizer or standard tokenizer, that one breaks up at dashes. If you had used another tokenizer, like whitespace, that would keep it together because that breaks up at whitespaces.

So there are various reasons why you want or would not want to do that. I don't want to go into all the details, but there are a lot of things to doright or wrong when you ingest the data, which will then allow you to query the data in specific ways.

So for example, if you would have an email address, that one is also weirdly broken up. Like, you might use, like, there is a dedicated tokenizer for URL email addresses. So depending on what type of data you have, you will need to process the data theright way because pretty much all the smart pieces are kind of, like, at ingestion here to make the search afterwards easier.

So you can easily do that. Let's see. Let's index all my three documents so that we can actually search for them. Now, if I start searching for droid, it should match. These are not the droids you're looking for, yes or no?

Because this one is singular and uppercase, and the droid that we stored was plural and lowercase. Will that match, yes or no?

Yes.

Why?

Because you got the stemming.

Yes, we had the stemming. We had the lowercasing. And when we search, so we store the text, it runs through this pipeline or the analysis. And for the search, it does the same thing. So it will lowercase the droid.

It has stemmed down the droids in the text to droid, and then we have an exact match. So what the data structure behind the scene actually looks like. The magic is kind of like in this so-called inverted index.

What the inverted index is, is these are all the tokens that remained that I have extracted. I have alphabetically sorted them, and they basically have a pointer and say, in this document, like, with the IDs 1, 2, 3 that I've stored, we have how many occurrences?

Like, 0, 1. Yeah, nothing had 2. And then we also know at which position they appeared. So search for droid now. This is what I have stored. I will lowercase the droid to droid. I have an exact match here.

Then I go through that list and see, retrieve this document, skip this one, skip this one. And at position 4, you have that hit. And then you could easily highlight that. So you have almost done all the hard work at ingestion, and this retrieval afterwards will be very fast and efficient.

That's the classic data structure for search, the inverted index where you have this alphabetic list of all the tokens that you have extracted to do that. And this will just be built in the background for you, and that's how you can retrieve all of this.

Let's look at a few other

queries and how they behave.

If I search for robot, will I find anything? No, because there was no robot. There was a droid. We could now define a synonym and say, like, all droids are robots, for example. Who likes creating synonym lists?

Nobody anymore. Okay. Normally I would have said that's the Stockholm syndrome because there is sometimes somebody who likes creating synonym lists because they've done that for so many years. But it got easier nowadays. Now you can use LLMs to generate your synonyms, so it can get a bit easier to create them.

But they're still limited because you have always this mapping. So with synonyms, you can expand theright way, where it gets trickier if you have homonyms. If a word has multiple meanings, like a bat could be the animal, or it could be the thing you hit a ball with.

There it just gets trickier because there is no meaning behind the words or no context. So you just match strings, and that is inherently limited. But like I said, it's dumb, but it scales very well. And that's why it has been around for a long time.

And it does surprisingly well for many things because there's not a lot of things that are unexpected or that can go totally wrong. Now, other things that you can do. You could do a phrase search where you say, I am your father.

Will this find anything?

Yes.

Yes, because we had no, I am your father.

What happens if I say, for example, I am

let's say, I am not your father? Yes, no?

No.

No. Why? So you'reright.

Looking for an exact match based on the position. So since it's not, it won't.

But not as a stopword.

But you'reright because the positions still don't match. So the stopword not would be filtered out, but it still doesn't match because the positions are off.

That is one of the things that sometimes can be confusing. So even if something is a stopword and will be filtered out, it doesn't work like that. One thing that you can do is, though, that the factor is called slop, where you basically say, if there is something missing, it would still work.

So I am your father, and I am father with slop 0, that's kind of like the implicit one, will not find anything. But if I say 1, then I basically say, like, there can be a 1 off in there.

Like, one word can be missing.

However, I am his father. Here, his would not match, so this still will not work. The slop is really just to skip a word. Yeah.

What about I am your father?

I am your father.

I assume that no, I apostrophe am your father. I assume that won't work, but.

That will not work.

How would you get that to work?

There you might need to do something like a synonym where you say slash am gets replaced by am. Or we will need to have some more machine learning capabilities behind the scenes to do stuff like that.

Are there any libraries that would predefine contractions like that, or?

So what is built in is generally a very simple set of rules. What you will need to do for things like this is normally you need a dictionary. The problem around these is they're normally not available for free or open source.

Funnily enough, they're often coming out of university, the dictionaries, because they have a lot of free labor, the students. That's why the universities have been creating a lot of dictionaries, but they often come out under the weirdest licenses.

That's why they're not very widely available. But yes, there is a smarter or more powerful approach if you have a dictionary and you can do these things. For example, one thing to show is, like, or maybe that's a good thing to also mention.

You don't always get words out of the stemming. It's not a dictionary. It doesn't really get what you're doing. It just applies some rules. So for example,

blackberry. Blackberry. Ah, sorry. Blackberries. I think that this will be stemmed down differently. Ah, sorry. I need English. Without English, this will not work.

So this will stem down to this weird word, blackberry. And it will also stem down the singular to blackberry. So there is a rule that applies this, but it's just a rule. It's not dictionary-based. It's not very smart.

And it only has some rules built in that work for this, but you will definitely hit limits. The other thing, by the way, and why I picked blackberry as an example, you have some annoying languages like German, Korean, and others that compound nouns, like blackberry, where you have basically two words.

Black would never find blackberry in the simplest form because it's not a complete string. There are various ways to work around that that all come with their own downsides. And either you have a dictionary, or you extract the so-called n-grams.

It's like a group of words, and then you match on group of words. But all of those are one of the many tools how we try to make this a bit better or smarter, but it all has limitations.

I hope that answers the question and makes sense. So there are dictionaries, but they're generally not free or not under an easy license available. For some languages, by the way, even the stemmers are not freely available. I think there is a stemmer or analyzer for Hebrew.

I think that has also, like, some commercial license, or at least you can't use it for free in commercial products.

Though licensing with machine learning models is also its own dark secret. Yeah.

I guess this maybe not exactly fair because you don't know how this is going to rank. But, like, why not just have, like, much smaller tokens, like subword tokens? And then you'd be able to, like, match a lot of things.

You're going to have, like, more false positives, but presumably things which are true positives will have more matches.

Yes. That is what an n-gram is doing. Let me see if I can.

An n-gram is normally a word group, normally a tri-gram. This is way too small.

Somehow I have weirdly overwritten my command plus, so I can't use that. Let me make this slightly larger.

OK. Here, we basically use one or two letters as word groups, which is way too small. But just to show the example, and this is very hard to read. Let me copy that over to my console that there you can.

There you can. Oops. There you can see this. But this is a great question. So we'll use n-gram for quickfox. And then you can see the tokens that I extract here are the first letter, the first two, the second, the second and third, et cetera.

You end up with a ton of tokens. The answer is, A, you have to do more work when you store this. B, it creates a lot of storage on disk because you extract so many different tokens. And then your search will also be pretty expensive because normally you would at least do three, like, tri-grams.

But even that creates tons of tokens and tons of matches, and then you need to find the ones with the most matches. And it works. But A, it is pretty expensive in disk, but also query time. And it might also create undecided results or results that are a bit unexpected for the end user.

It is, I would call it a it's again, it's a very dumb tool that works reasonably well for some scenarios, but it's only one of many potential factors. What you could potentially do is, and I don't have a full example for that, but we could build it quickly.

What you would do in reality probably, you might store a text more than one way. So you might store it, like, with stopwords and without stopwords and maybe with n-grams. And then you give a lower weight to the n-grams and say, like, if I have an exact match, then I want this first.

But if I don't have anything in the exact matches, then I want to look into my n-gram list. And then I want to kind of, like, take whatever is coming up next. So even keyword-based search will be more complex if you combine different methods.

N-grams are interesting, but again, they're a dumb but pretty heavy hammer. Use them with theright at theright scenario.

Sorry, quick question about this n-grams. Is it by default one or two?

Yes. But you could redefine that. So we can let me go back to the docs. The n-gram, you can say min-gram and max-gram. If you set both to three, you would have tri-grams, where it's always groups of three, like 1, 2, 3, 2, 3, 4, et cetera.

You could also have something called edge n-gram, where you expect that somebody types the first few lettersright. And then you only start from the beginning, but not in the middle of the word, which sometimes avoids unexpected results and, of course, reduces the number of tokens quite a bit.

So somewhere in here,

edge n-gram.

Let's just copy that over so I won't type.

And so here we have edge n-gram with quick, and you can see it only does the first and the first two letters, but nothing else. And in reality, you would probably define this like 2, 2, 5, or more, whatever else you want.

But here we only do from the start and nothing else, which reduces the tokens tremendously. But of course, if you have blackberry and you want to match the berry, you're out of luck.

Make sense?

Anybody else? Anything else?

Sorry, another question.

Sure.

So if you're working with a language like English and then Hebrew, where it'sright to left, how do you deal with the indexes and stuff like that?

Yeah. So if you have multiple languages, do not mix them up. That will just create chaos. Because we'll get to that in a moment, but how keyword search works is basically word frequencies. And if you mix languages, it screws up all frequencies and statistics.

So what you would do is either you would have

you could do two things to do. If you have English and Hebrew, you could either do field English, and then you would have

field whatever the abbreviation for Hebrew is. Hebrew. Oops. And then you would have that. And then you would need to define theright analyzer for that specific field. But so you break it out either into different fields, or you could even do different indices.

And ideally, we even have that built in. We have a language analyzer. Even if you just provide a couple of words, it will guess or not guess. It will infer the language with a very high degree of certainty.

Especially Hebrew will be very easy to identify. If you have your own diacritics, it's easy. But even if you just throw random languages at it, it will have a very good chance, like, just with a few words, to know this is this language, and then you can treat it theright way.

Good. Let's continue.

So we have done all of these searches. We have done slop. One more thing before we get into the relevance. One other very heavy hammer that people often overuse is fuzziness. So bless you. If you have a misspelling, so I misspelled Obi-Wan Kenobi.

We already know that this is broken out into two different words or tokens. It will still match your Obi-Wan because we have this fuzziness, which allows edits. It's like a Levenstein distance. So you can have one. By default here, you could either give it an absolute value, like you can have one edit, which could be one character too much, too little, or one character different.

You could set it to two or three. You can't do more because otherwise you match almost anything. And auto will auto is kind of smart because auto, depending on how long the token that you're searching for is, will set a specific value.

If you have zero to two characters, auto fuzziness I think is one from two to no, zero to two characters is zero. Three to five characters is one. And after that, it's two.

So you can match these.

Will this one match?

Yes, no, and why?

No, because you've got the V and the W.

Yes. So we have both of those are misspelled.

It still matches. Why?

They get tokenized separately, and each one has a single Levenstein distance.

Yes. That is a bit of a gotcha. So yes. You need to know the tokenizer. So we tokenize with standard. So it's two tokens. And then the fuzziness applies per token, which is another slightly surprising thing. But yes, that's how you end up here.

OK. Now, we could look at, like, how the Levenstein distance works behind the scenes. But it's basically a Levenstein automaton, which looks something like this. If you search for food and you have two edits, this is how the automaton would work in the background to figure out, like, what are all the possible permutations.

It's a fancy algorithm that was, I think, pretty hard to implement, but it's in Lucene nowadays. OK. Now, let's talk about scoring. One thing that you have seen that you don't have anywhere or in a non-search engine or just in a database is, like, we have these scores.

Like, how well does this match?

How does a score work here? Let's look at the details of that one.

So the basic algorithm, which is also most of us, or pretty much all of us here, term frequency, inverse document frequency, or TFIDF. It has been slightly tweaked. Like, the new implementation is called BM25, which stands for Best Match.

And it's the 25th iteration of the Best Match algorithm. So what they look like is you have the term frequency. If I search for droid, how many times does droid appear in the text that I'm looking for? And it's basically the square root of that.

So the assumption is if a text contains droid once, this is the relevancy. If I have a text that contains droid 10 times, this is the relevancy. The tweak between TFIDF, that one just keeps growing. BM25 says, like, once you hit, like, five droids in a text, it doesn't really get much more relevant anymore.

So it kind of, like, flattens out the curve. That is the idea of term frequency. The next thing is the inverse document frequency, which is almost the inverse curve. The assumption here is over my entire text body, this is how often the term droid appears.

So if a term is rare, it is much more relevant than if a term is very common. Then it's kind of, like, less relevant. Basically, the assumption is rare is relevant and interesting. Very common is not very interesting anymore.

And then it's kind of, like, just works its curve out like that. And the final thing is the field length norm is, like, the shorter a field is and you have a match, the more relevant it is, which assumes, like, if you have a short title and your keyword appears there, it's much more relevant than if there is a very long text body and your keyword and you have a match there.

And these are the three main components of TFIDF. So let's take a look at how this looks like.

You can make this a bit more complicated.

And this will show you why something matches. Don't be confused by the or let me take that out for the first try. So I'm looking for father. And I am no, I am your father. And Obi-Wan never told you what happened to your father.

One is more relevant than the other. Why is the first one more relevant than the second one?

Because they showed the first one.

Yeah. Term frequency is the same. Both contain father ones. The inverse document frequency is also the same because we're looking for the same term. The only difference is that the second one is longer than the first one. And that's why it's more relevant here.

So this is very simple. And you can then, if you're unsure why something is calculated in a specific way, you can add this explain true. And then it will tell you all the details of, like, OK, we have father.

And it then calculates basically all the different pieces of the formula for you and shows you how it did the calculation. So you can debug that if you need to. But it's probably a bit too much output for the everyday use case.

And then you can customize the score if you want to. Here I'm doing a random score. So my two fathers this is a bit hard to show. They will just be in random order because their score is here randomly assigned.

But you could do this more intelligently, that you combine, like, the score and, like, if you have, I don't know, the margin on the product that you sell or the rating, that you include that in the rating somehow.

And you can build a custom score for things like that. So you can influence that any way you want.

One thing that I see every now and then that is a very bad idea and we'll skip this one because it's probably a bit too much. This, by the way, is the total formula that you can do. Or maybe I'll show you this part that I skipped.

What happens if you search for two terms? And they're not the same. They don't have the same relevancy. So what the calculation behind the scenes basically looks like is let's say we search for father. Father is very rare.

That's why it's much more relevant than your. Your is pretty common. And then we have a document that contains your father. It's kind of like this axis. This is, like, the this will be the best match. But will a document that only contains father be more relevant or only your?

Intuitively, the one with just father will be more relevant. But how does it calculate that? It basically calculates, like, this is the relevancy of father. This is the ideal document. And this is your. And then it looks like which one has the shorter angle.

And this is the one that is more relevant. So if you have a multi-term search, you can figure out which term is more relevant and how they are combined. And then you can also have the coordination factor, which basically rewards documents containing more of the terms that you're searching for.

So if I'm searching for three terms, like,

you, I am father, whatever, if a document contains all three, this will be the formula that combines the scores of all three and multiplies it by 3 divided by 3. If it only contains two of them, it would only have the relevancy of 2/3 and with 1, 1/3.

And then you put it all together. And this is the formula that happens behind the scenes. And you don't have to do that in your head, luckily. Cool. We have seen these. One thing that I or we see every now and then is that people try to translate the score into percentages.

Like, you say, this is a 100% score. And this is only, like, a 50% match. Who wants to do that?

Hopefully nobody because the Lucene documentation is pretty explicit about that. You should not think about the problem in that way because it doesn't work. And I'll show you why it doesn't work or how this breaks. Let's take another example.

Let's say we take this short text. These are my father's machines. I think of a good Star Wars quote to use here, but bear with me. So what remains if I run this through my analyzer? My father machine.

These are the three tokens that remain. Now, I will store that. You remember the three tokens that we have stored. And if I search for my father machine, you might be inclined to say this is the perfect score.

This is, like, 100%. Agreed? Because all the three tokens that I have stored in these are my father's machines are there. So this must be, like, my perfect match. So it's 3.2 that would be 100%. The problem now is every time you add or remove a document, the statistics will change.

And your score will change. So if I delete that document and I search the same thing again, I don't know what percentage this is now. Is this now the new 100%, the best document? Or is this a 0 point or, I don't know, 20%?

How does this compare?

And then you can play funny tricks where these droids are my father's father's machines. And you can see I have a term frequency of two for father here. So if I store that one then and then search it,

is this now 100%? Is this now 110%? So don't try to translate scores into percentages. They're only relevant within one query. They're also not comparable across queries. They're really just sorting within one query to do that. OK. Let me get rid of this one again.

Now, we've seen the limitations of keyword search. We don't want to define our synonyms. We might want to extract a bit more meaning. So we'll do some simple examples to extend. I will add from OpenAI text embedding small.

I'm basically connecting that inference API for text embeddings here in my instance. I have removed the API key. You will need to use your own API key if you want to use that. But it is already configured. So let me pull up the inference services that we have here.

I have done or I have added two different models. One sparse, one dense. Let's go to these. By the way, if you try to do this with the 100% score, don't do this

because it will just not work. OK. Not everybody has worked with dense vectors,right? So I have a couple of graphics coming back to our Star Wars theme just to look at how that works. So what you do with dense vectors is we'll keep this very simple.

This one just has a single dimension. And it has, like, the axis is pretty much, like, realistic Star Wars characters and cartoonish Star Wars characters. And this one falls on the realistic side. And that other one is just cartoonish.

And you have a model behind the scenes that can rate those images and will figure out, like, where they fall. Now, in reality, you will have more dimensions than one. And you will also have floating point precision. So it's not just, like, minus 1, 0, or 1.

But

you will have more dimensions. So for example, here in human and machine and in a realistic model, you don't have, like, the dimensions are not labeled as nicely and clearly understandable. The machine has learned what they represent. But they are not representing an actual thing that you can extract like that.

But in our simple example here now, we can say this layer character is realistic and a human versus, I don't know, the Darth Vader is cartoonish and, I don't know, somewhere between human and machine. So this is their representation in a vector space.

And then you could have, like I said, you could have floating point values. And then you can have different characters and similar characters. Like, both of those are human. Without the hand, he's only, like, not quite as human anymore.

So he's a bit lower down here.

So he's a bit closer to the machines. So you can have all of your entities in this vector space. And then if you search for something, you could figure out, like, which characters are the closest to this one.

And again, in reality, you will have hundreds of dimensions. It will be much harder to say, like, these are the explicit things. And this is why it works like that. It will depend on how good your model is in interpreting your data and extracting theright meaning from it.

But that is the general idea of dense vector representation. You have your documents or sometimes it's, like, chunks of documents that are represented in this vector space. And then you try to find something that is close to it for that.

Does that make sense for everybody or any specific questions?

So it's a bit more opaque. I want to say it's not quite as easy because you say, like, these five characters match these other five characters here. But you need to trust or evaluate that you have theright model to figure out how these things connect.

So let's see how that looks like.

I have one dense vector model down here. We have OpenAI embedding. This one is a very small model. It only has 128 dimensions.

The results will not be great. But it's actually for demonstrating it actually helpful. So we'll see that. The other model that we have and let me show you the output of that. So if I take my text, these are not the droids you're looking for.

This is the representation. It's basically an array of floating point values that will be stored. And then you just look for similar floating point values. And then you have these are not the droids you're looking for. Here on the previous one,

dense text embedding. This one here does sparse embedding. Sparse is the main model used for that is called Splade. Our implementation of Splade is we call it Elser. It's kind of like a slightly improved Splade. But the concept is still the same.

What you get is you take your you take your words. And this is not just TFIDF. This is a learned representation where I take all of my tokens and then expand them and say, like, for this text, these are all the tokens that I think are relevant.

And this number here basically tells me how relevant they are. Again, not all of these make sense intuitively. And you might get some funky results, for example, with foreign languages. This currently only supports English. But these are all the terms that we have extracted.

Normally, yeah, you get, like, 100 something or so. So the idea is that this text is represented by all of these tokens. And the higher the score here, the more important it is. And what you will do is you store that behind the scenes.

When you search for something, you will generate a similar list. And then you look for the ones that have an overlap. And you basically multiply the scores together. And the ones with the highest values will then find the most relevant document.

This is insofar interesting or nice because it's a bit easier to interpret. It's not just, like, a long array of floating point values. Sometimes these don't make sense. The main downside of this, though, is that it gets pretty expensive at query time because you store, like, a ton of different tokens here for this.

When you retrieve it, the search query will generate a similar long list of terms. And if you have a large enough text body, a query might hit a very large percentage of your entire stored documents with these OR matches because basically these are just a lot of ORs that you combine, calculate the score, and then return the most or the highest ranking results.

So it's an interesting approach. It didn't gain as much traction as dense vector models. But it can be as a first step or an easy and interpretable step. It can be a good starting point to dive into the details here.

So this is your entire vocabulary?

So these are not the droids you're looking for. It's basically represented by this embedding here. So it's, like, this entire list of terms with these

with this relevancy, basically. This is the representation of this string. And then when I search for something, I will generate a similar list. And then I basically try to match the two together, like, for what has the most or the highest matches here.

Make sense?

Can you run a query? Can you search for Robot?

Yes. We'll do that in a second.

It's a bit because with the OR,

in fact, it doesn't tell you exactly this is what matched. But yes. So I will need to create a new index. This one keeps the configuration from before. But I'm adding this semantic text for the sparse model and the dense model.

So I've created this one. And now I'll just put three documents. I have my other index. As you can see here, it says three documents were moved over. So we can then start searching here. And if I look at that, the first document is still these are not the droids you're looking for.

You don't see, like, for the for keyword search, you don't see the extracted tokens. Here, we also don't show you the dense vector representation or the sparse vector representation. Those are just stored behind the scenes for querying. But there's no real point in retrieving them because you're not going to do anything with that huge array of dense vectors.

It will just slow down your searches. You can look at the mapping. And you can see I'm basically copying my existing quote field to these other two that I can also search those. OK. So if I look for machine on my original quote, will it find anything?

No?

Yes.

No because it only had these are not the droids you're looking for. And this is still the keyword search.

So this is just a query from before just to show you that, yes, we are not matching anything because it only had these are not the droids you're looking for. But here we're looking for machine. It was still a keyword matching.

Doesn't work. Shouldn't work. That's exactly the result that we want out of this here. Now, if I say Elser and I say machine, then it will match here. These are not the droids you're looking for. And you can see this one matches pretty well, I don't know, at 0.9.

But it also has some overlap with no I am your father. I mean, it is much lower in terms of relevance. But something had an overlap here. And only the third document, Obi-Wan never told you what happened to your father.

Only that one is not in our result list at all. But there was something here. I don't know the expansion. We would need to basically run

where was it? We would need to run this one here for all the strings and look then for the expansion of the query. And then there would be some overlap. And that's how we retrieve that one.

So is there an example of threshold that you have?

You could define a threshold. It will, though, depend

let's see. These are not the droids you're looking for. Let's say if

I say I'm not sure if this will change anything.

I mean, the relevance here is still it's still 10x or so. But yeah, this one still

will just have a very low match. It's not the terms you look for. The score just totally jumps around. So it's a bit hard to define that threshold because here you can see in my previous query, we might have said 0.2 is the cutoff point.

But now it's actually 0.4 even though it's not super relevant. So it might be a bit tricky. Or you might need to have a more dynamic threshold depending on how many terms you're looking for and what is a relevant result.

In the bigger picture, the assumption would be if you have hundreds of thousands or even millions of documents, you will probably not have that problem that anything that is so remotely connected will actually be in the top 10 or 20 or whatever list that you want to retrieve.

So for larger proper data sets, this should be less of an issue. With my Hello World example of three documents, it can be a bit misleading. But yes, you can have a cutoff point. If you figure out what for your data set and your queries is a good cutoff point, you could define a cutoff point.

No, sorry. You have three documents. How come it's only showing two? Is it because there's a.

So the query gets expanded into, I don't know, those 100 tokens or whatever. And then for those two, there is some overlap. But the third one just didn't have any overlap.

Oh, wow.

But I so we what? OK. We can do that. It's just a bit tricky to figure out the term that has the overlook the overlap. So we will need to take this one. Machine, no I am your father.

Let's take this one. What you need to do is to figure that one out. Oh, no. Actually, we should be able let me see.

Let's see.

This

is a pretty long output. Somewhere I was actually hoping that it would show me the term that has matched here. OK. I see something.

OK. There is something pop it that seems to be the overlap. How much sense that term expansion for the stored text and the query text makes is a bit of a different discussion. But in here with that explain true, you can actually see how it matched and what happened behind the scenes if you have any really hard or weird queries or something that is hard to explain to debug that.

But the third one didn't match. Now, if I take the dense vector model with OpenAI and I search for machine, how many results do you expect to get back from this one?

0, 1, 2, 3. Yes, 3. Why 3?

There will always be some result.

Yes, because there's always some match. That is the other or let me run the query first.

Here, these are not the droids you're looking for. This one is the first one. I don't think that this model is generally great because here the results are super close. It is, I mean, the droids with the machines.

That is the first one. But the score is super close to the second one, which is no I am your father, which feels pretty unrelated. And Obi-Wan never told you what happened to your father. Even that one is still with a reasonably close score.

But why do we have those? Because if we say, what is the relevance? I mean, it's further away. But it's always like there is always kind of like some angle to it even if the kind of like the angle here or depending on the similarity calculation that you do.

But it's still always related. There is no easy way to say something is totally unrelated. That is, by the way, one good thing about keyword search where it was relatively easy to have a cutoff point of things that are totally not relevant where you're not going to confuse your users versus here, if you don't have great matches, you might get almost it's not random.

But it's potentially it looks very unrelated to your end users what you might return just because it's very hard to show. Yes?

Is it fair to say then that the, like, the OpenAI embedding search is worse for this kind of toy example because the magnitude of difference is so much?

I'm careful with worse because it's really a Hello World example. So I don't take this as a quality measurement in any way.

I'm, yeah, I mean, the OpenAI model with 128 dimensions is very few dimensions. I think it will probably be cheap but not give you great results necessarily. But don't use this as a benchmark. I think it's just a good way to see that this is now much harder because now you need to pick theright machine learning model to actually figure out what is a good match.

With keyword-based search, it was a bit of a different story. There you need to pay more attention to, like, how do I tokenize? And do I have theright language? And do I do stemming or not stemming? But most of that work is relatively, I want to say, almost algorithmic.

And then you can figure that out. And you configure it. And then it's very predictable at query time. Versus with the dense vector representation, you really need to evaluate for the queries that you run and the data that you have.

Like, is that relevant? And is there an improvement or not? It's very easy to get going and just throw a dense vector model together. And you will match you will always match something that might be an advantage over the lexical search where you don't have any matches, which sometimes is there the problem that nothing comes back and you would want to have at least some results.

Here, it might just be unrelated. So that can be tricky. The you want to have some results is, by the way, a funny story that a European e-commerce store once told me. They said they accidentally deleted, I think, 2/3 of their data that they had for the products that you could buy.

And then I asked them, like, OK. So how much revenue did you lose because of that? And they said, basically nothing because as long as you showed some somewhat relevant results quickly enough, people would still buy that. So only if you have no results, that's probably the worst.

So for an e-commerce store, you might want to show stuff a bit further out because people might still buy it. But it really depends on the I'm coming to you in a moment. It really depends on your use case.

E-commerce is kind of like one extreme where you want to show always something for people to buy. If you have a database of legal cases or something like that, you probably don't want that approach because that will go horribly wrong.

So it is very domain specific. That's, I think, also the good thing about search because it keeps a lot of people employed because it's not an easy problem. It's almost job security because it depends much on the this is the data that you have.

And this is the query that people run. And this is the expectation of what will happen. And this is for this domain theright behavior. So there is no easyright or wrong with a checkbox. And the other thing is you might make if you tune it, you might get for one thing worse for two things.

That's why a robust abbreviation set is normally very important, though very rare. A lot of people YOLO it. And you will see that in the results. And for the e-commerce store, it probably works well enough. Sorry. You had a question.

Can I limit the semantic enrichment to a subset of my index based off of properties of the document? So if I have a very large shared index with a lot of customers and I want to enable AI for a subset of the index, can I say, hey, only do the semantic enrichment if the document has this property where maybe it's like an AI customer?

Yeah. So the way we would do it in our product is that you would probably have two different indices with different mappings.

Yeah. But then it's not so fun. Like, the customer upgrades and I have to migrate them to the new index.

Dave, please.

Yeah. So if you, for example, have an index in Elasticsearch, you can think of it almost like a sparse table,right? So there's no penalty for having a field that is not populated. So either in your application or in an ingest processor, you could have an if statement, say.

Yeah. That's how we do it now.

I was wondering if it's.

Only move it over.

With this automatic way where you kind of turn it off.

No. The problem is the data structure. Like, if the field is there so the data structure that we build in the background is called HN. And either we build the data structure or we don't build it.

Yeah. So if you had 10 billion entries in your vector index, your index that is set up for vectors,right? And you just don't populate the thing that is either putting in a dense vector or is triggering the inference to create the dense vector to put into there, then it's just going to be a bunch of the index is just a bunch of pointers.

And none of them head towards the HNSW. And it won't show up in search results. The penalty to you is nothing,right? But you're going to have to manage what does or does not create the vector. You could do that in an ingest processor by just saying, hey, we're going to use the copy command to have two copies of the text, one that's meant for non-vector indexing, one that's meant for actual vector indexing.

But you'd have to manage that with some tricky complex AI technology called if-then-else,right, somewhere inside of your ingesting pipeline. And it would work just fine.

Yeah. One more question. When we did HNSW with Elastic, we found that it was extremely slow at write time. And the community suggested that we freeze our index if we were going to use HNSW. Force merge or.

Yeah. I think just freeze writes. They said build the index and freeze it. Otherwise, you'll put a ton of load on the CPU.

I mean, yes.

What we found is that some of the default kind of like have been around in Elasticsearch for 10 years. Settings for the merge scheduler are really optimized for keyword search. And for high update workloads on HNSW, we've got some suggestions.

They take a little bit of parameter sweep tuning to go find somethingright for your IOPS and for your actual update workload. But sometimes it's about the merge scheduler and not doing kind of an inefficient HNSW build when it's not important for your use case.

The other thing we'd say is that sometimes friends don't let friends run on Elasticsearch 8.11 upgrade, upgrade, upgrade. They put a lot of optimization work into here because it should be simple.

So the reason.

It used to be that. You were correct.

The reason why that is, it's like merging so because you have the immutable segment structure in Elasticsearch. And HNSW, you cannot easily merge. You basically need to rebuild them. The one trick I forgot which version it was. I'm not sure, Dave, if you remember.

I think it was even before 8.11. But basically, if we do a merge, we would take the largest segment with not deleted documents and basically plop the new documents on top of them rather than starting from scratch from two HNSW data structures.

There's another optimization somewhere now in 9.0 that will make that a lot faster. So it really depends on the version that you have. And there are a couple of tricks that you can play. But yeah, that is one of the downsides of, like, the way immutable segments work and HNSW is built that you can't easily just merge it as easily together as other data structures because you really need to rebuild the HNSW data structure or, like, take the largest one and then plop the other one in.

OK. Some of the things where we just, like, we fixed it in the next version of the scene. So I want you to.

We found, like, KNN was too slow. And then ANN broke our CPU. And then we moved to Pinecone. And then now we're curious.

Stay curious.

Yeah. Might have been a while ago. Yeah.

So for, like, traditional document search, you know, I'm saying, like, hey, please find me a document that contains my search query,right? For R in the context of RAG, it might be something more like, hey, come up with a fun plan for my weekend,right?

And then the documents that we want to find don't necessarily look like the search query,right?

Yeah.

So, like, one approach to that is you just give it an agent. And you give it a search tool. And it searches,right? But, like, I'm just curious what you think about that in general.

Yeah. I feel like RAG has been very heavily abused. It's or, like, the mental model, I think, started off as, like, you do retrieval. And then you do the generation. But you could do the generation earlier on as well that you do the query rewriting and expanded query.

So my favorite example for that is you're looking for a recipe. You don't need to have the LLM regenerate the recipe. You just want to find the recipe. But maybe you have a scenario where you forgot what the thing is called that you want to cook.

And then you could use the LLM, for example, to tell you what you're looking for. Like, you say, like, oh, I'm looking for this Italian dish that has, like, these layers of pasta and then some meat in between.

And then the LLM says, like, oh, you're looking for lasagna. And then you basically do the generation first or a query rewriting and then search and then get the results. As a very let's see explicit example here, your example will look very different and probably smarter than my example.

But query rewriting is one thing. There's also this concept of hide where your documents and your queries often look very different and that you use an LLM to generate something from the query that looks more close to the documents that you have.

And then you match the documents together because they are more similar in structure. So there are all kinds of interesting things that you can do. Like I said earlier, the "it depends" is becoming a bigger and bigger factor.

But yeah, your use cases probably might be, yeah, maybe a multi-retrieval where you figure out, like, oh, you look I don't know. I know the example from an e-commerce store where it's like, I'm going to a theme party from the 1920s.

Give me some suggestions. And then the LLM will need to figure out, like, what am I searching for? And then it can retrieve theright items and rewrite the query and then actually give you proper suggestions. But it's not just running a query anymore.

Yeah?

We use this instruction for tearing into many models. And we have to, like, you're gifted this. You can still do it in several passes. And then, like, along with your query, you can say, like, this is the kind of thing I am doing.

And you can similarly invent your documents in this way. So you can, like,

you can have document query embeddings. We've tried to embed queries relevant to the documents we're going to find versus just, like, the text. But you can have instruction fine-tune embedding models that you instead of saying, like, I'm matching queries and documents, you can match appropriate things to appropriate things.

And that doesn't require you to do a separate two-pass process, which can be slow.

Yeah.

Yeah.

How should we be thinking about the number of dimensions in the embedding model? Is like a 512-dimensional model necessarily better than a 128?

Definitely not necessarily. It's an interesting question. That feels almost like a blast from the past. I remember, like, two or three years ago, there was this big debate of, like, how many dimensions does each data store support? And, like, how many dimensions should you have?

And at first, it looked like, oh, more dimensions is always better. But then it turned out more dimensions are very expensive. So it really depends on the model and what you're trying to solve. Like, if you can get away with fewer dimensions, it's potentially much cheaper and faster.

But I don't think there is a hard rule. Like, maybe the model with more dimensions can express more because it just has more data. And then it will come in handy. But maybe it's not necessary for your specific use case.

And then you're just wasting a lot of resources. I don't think there is an easy answer to say, like, yes, for this use case, you need at least 4,000 dimensions.

It will depend. But it depends on the model how many dimensions it will output. And then maybe you have some quantization in the background to reduce that again or reduce either the number of dimensions or the fidelity per dimension.

So there are a lot of different trade-offs in that performance consideration. But it will mostly rely on, like, how good does the model work for the use case that you're trying to do.

How should we go about thinking

or building evaluation framework for vector search, especially the search similarity score that just comes up? And again, it's not equivalent to a percentage,right?

Yeah. So that is one area. So I want to say historically what you would do is you would have a golden data set. And then you would know what people are searching for. And then you would have human experts who rate your queries.

And then you run different queries against it. And then you see, like, is it getting better or is it getting worse? Now LLMs open a new opportunity where you might have with human experts in the loop to help them out a bit.

But they might be actually good at evaluating the results. So almost nobody has, like, the golden data set and tests against that. But you can either use it, look at the behavior of your end users and try to infer something from that.

Or you have an LLM that evaluates, like, what you have. Or you have a human together with an LLM evaluate the results. So you have various tools. But again, it's and it depends. And it's really not an easy question of saying, like, this is theright thing.

Maybe you can get away with something simple. So the classic approach I want to say is, like, you looked at the click stream of how your users behaved. And then you saw, like, they clicked on the first or up to the third result.

The result was potentially good. And they didn't just go back and then click on something else. But they stuck on the page. If they don't click on anything and just leave, it might be very bad. If they go to the second or third page, it might also not be great.

So there are some quality signals that you can infer from that. Or you really look into the quality aspect and try to evaluate, like, what people were doing and how it behaves. But you can make this from relatively simple to pretty complicated.

What else? What else?

OK. Obviously, if I search for dad with square extension, it will find my father example. And this one here will still, again, match my droids, pretty much like the OpenAI example. One thing that I wanted to show you what is also happening behind the scenes here.

This is a very long segment. Like, it's a lot of information with different speakers. What I have created here, though, is we have created multiple chunks behind the scenes. And if I search for that, I think looking for murder in the Skywalk saga works pretty well here.

It finds the document that I've retrieved. But it can also highlight. So here I say, show me the fragment that actually matched best here. And if I search here for murder, it didn't find anything. But I think the term that it found was in this highlighted segment here, it found kill.

And it was that one that was expanded here. So here I've broken up my long text field into multiple chunks. And there are multiple strategies. You do that by page, by paragraph, by sentence. You could do it overlapping or not overlapping.

Many strategies will depend on how you want to retrieve what works best for your use case. But you want to kind of, like, reduce the context per element that you're matching because there's only so much context that a dense vector representation can hold.

So you want to chunk that up, especially if you have, like, a full book. You want to break up those individual at least pages and then find the relevant part where the match is. And then you can actually link back to that.

The point in this query here is also to show you I didn't define any chunks. I didn't say, like, OK, send this representation of a dense vector there. And then when it comes back, interpret again. This is all happening behind the scenes just to make this easier.

So the entire behavior here is still very similar to the keyword matching, even though there's a lot more magic happening behind the scenes just to keep that very simple.

OK.

How does everybody feel about long-adjacent queries?

We'll see about alternatives. And maybe we can make this a bit simpler again. But let me show you one more way of looking at this. We call them retrievers. They're a more powerful mechanism to actually combine different types of searches.

Combining different types of searches let me get from my slides, actually, when we talk about combining searches and how this all plays together. This is kind of my little interactive map of what you do when you do retrieval or what your searches do.

So we started here in the lexical keyword search. And then we run a match query. And we are matching these strings.

This often combined with some rank features are often what we call full-text search. The rank features could be either you extract a specific signal or it could also be something however you influence that ranking. It could be, like, the margin on a product, like how many people bought something, what the rating is.

There are many different signals that you could include, like, not just with the match of the text, but any other signals that you want to combine for retrieving that. And then you have, like, full-text search as a whole.

On top of that, I kept that kind of, like, to the side here. You might have a Boolean filter where you have, like, a hard include or exclude of certain attributes. This does not contribute to the score. This is just, like, black and white.

This is included or excluded. Versus this here calculates the score for you, how you match. And then this was kind of, like, the algorithmic side. And then we have this machine learning, the learned side or the semantic search where you have a model behind the scenes split into the dense vector embeddings and the sparse vector embeddings for vector search or learned sparse retrieval.

I think those are the two common terms.

And the interesting thing is all of these, including the sparse one, these are the sparse vector representation in the background. And only this one here is the dense vector representation. And then when you combine any grouping down here to combine for one search, this is then what we would call hybrid search, even though there can be big discussions of, like, what is exactly hybrid search or not.

I will definitely stick to the definition that as soon as you combine more than one type of search, it could be sparse and dense. Or it could be dense and keyword. Or maybe if you combine two dense vector searches, then it's hybrid search because you have multiple approaches.

And then you can either boost them together. You could do reranking, which is becoming more and more popular. One thing that we lean heavily into is RRF, which is reciprocal rank fusion that doesn't rely on the score. But it relies on the position of each search mechanism.

So it basically says, like, the lexical search had this document at position four. And the dense vector search had it at position two. And then it kind of, like, evens out the position and gives you an overall position by blending them together rather than looking at the individual scores because they might be totally different.

So this is kind of, like, the information retrieval map overall. And we have OK, we didn't do a lot of filters. But I think filters are intuitively relatively clear that you just say, like, I'm only interested in users with this ID or whatever other criteria.

It could be a geo-based filter, like only things within 10 kilometers or only products that came out in the last year, like a hard yes or no. All the others will give you

a value for the relevance. And then you can blend that potentially together to give you the overall results. That is kind of, like, the total map of search.

Could you give an example of the signal one, please?

Yeah. For signal, so we have our own data structure for these rank features. It could be, for example, I don't know, the rating of a book. And then you combine

the keyword match for, I don't know, you search for murder mysteries. But then another feature would be how well they are ranked. And then you would see that. Or it could be your margin on the product or the stock you have available.

And you would want to show the products where you have more in stock. There can be or it might even be as simple as a click stream, like what have people clicked before. There are a lot of different signals that you could include in all of this searching then.

Any other questions? Or everybody good for now? Yeah?

When you do this, like, scoring, you said you do, like, a blended approach.

RRF does a blended approach, yes.

Those scores, are they kind of standardized? Or are they different based on the type of search that you're doing, like the relevancy scores that you're kind of combining?

Yeah. So there are you would have to normalize them. There will be so depending on, like, if you have depending on the comparison that you do for dense vectors, it might be between 0 and 1. But you saw that for the keyword search, also depending on how many words I was searching for, it might be a much higher value.

There is no really real ceiling for that. Or you could add a boost and say, like, this field is 20 times more important than this other field. There is no real max value that you would have here. You could normalize the score and then basically say, like, I'll take, like, the highest value in this sub-query as 100% and then reduce everything down by that factor.

And then I combine them. Maybe that works well. RRF is a very simple paper. I think it's, like, two pages. And it really just takes the different positions. I think it's 1 divided by 60, which is, like, a factor that figured out made sense plus the position.

And then you add the scores or, like, the positions for each document together. And then that value gives you the overall position. It's really just it doesn't look at the score anymore. But it blends the different positions together and, like, how they are interleaving and what should be first or second then.

Yeah?

If I just want vector search, why should I use Elastic or PgVector or something like Quadrant? I'm sure there are some trade-offs. Because if my data is already in a database, every change probably via CDC, change data capture has to go to Elastic.

So there's one extra hop during ingestion versus, like, PgVector, like, it'sright there outside the database. So I was just curious what sort of systems maybe people who already had keyword search requirements, like, what have you seen in production?

What systems choose PgVector or Quadrant versus?

I mean, PgVector will always be there because, like, if you are already using Postgres, it's very easy to add. I think then the question is, like, does it have all the features that you need? For example, Postgres doesn't even do BM25.

It has some matching. But it's not the full BM25 algorithm because I don't think it keeps all the statistics. It will be a question of, like, scaling out Postgres can be a problem and then just, like, the breadth of all the search features.

If you only need vector search, I think my or our default question back to that is, like, do you really only need vector search? Maybe for your use case. But for many use cases, you probably need hybrid search.

One area, for example, where vector search will not do great is, like, if somebody searches for, like, a brand because there is no easy representation in most models for the specific brand. And it will be very hard to beat keyword search.

So there will be very and also, your users get very angry when they know you have this word somewhere in your documents or in your data set. But you don't give me the result back. So there are many scenarios where you probably want hybrid search.

Or I feel like that's the we started two years ago. We started with just vector search. But I feel like the overall trend is coming more to hybrid search because you probably want some sort of keyword search. And then you want to have that combined probably with some model for the added benefit and extra context.

But you often want the combination. It might also depend a bit on, like, the types of queries that your users run. So if your users run single-word queries, like I've done in my examples, that's often not really ideal for vector search because you live off, like, any machine learning model because you live off extra context.

So depending on that, I've seen some people build searches where it's, like, if you search for one or two words, they do keyword search. But if you search for more, they might fall over to vector search. So it depends a bit on the context what works.

If you really only need vector search and PgVector is small enough to do all of that and Postgres is your primary data store, then that's probably where you will do well. But there are plenty of scenarios where that will or not all of those necessary boxes will be ticked.

One last question. Specifically for code, let's say you have a file. And it's a you have a Git repository with, like, thousands of commits,right? The file doesn't change across many commits. But you change one byte. And then technically, like, the embedding of the chunk has changed.

And so you have two options. Either you embed at a file level or you embed at a chunk level,right? But I don't want to pay the penalty across thousands of unchanged like, the file isn't changed for these thousand commits.

But just for this one, it has changed,right? So I kind of need shadow copies of the same thing. And have you seen, like, what are some, like, tips and tricks that people use to not have exploding storage costs?

And you're like, this might not be just an Elastic problem but a general vector DB and retrieval problem. Like, how do I just pay the penalty once of storing the same embedding and only when it changes, like, I re-embed and, like, ingest it?

But so you would create so it's one data set basically with thousands of files that all are chunked together. And so one change would invalidate all of them?

No, no. So imagine I have a Python repository,right? It has, like, 5,000 commits. And there's one file, 4.5, let's say. It didn't change for 4,000, 9, 9, 9 commits. But on the 5,000th commit, it did change,right? And so if it hasn't changed, I want to only pay the penalty once of the embedding,right?

But only when the file contents have changed, I want to re-ingest, re-embed, whatever,right? But for those 4, 9, 9, 9 times, I don't want to store that, like, this Git hash has the same embedding, same embedding.

Ah.

I'm not sure if this is a problem you have seen with some of your customers.

Maybe so that.

This company just throws a problem.

I think the way we might solve it is that if you create the hash of the file and use that as the ID and you only use the operation create and it would reject any duplicate writes, you would at least not ingest and then create the vector representation again.

You will still send it over again. And it would need to get rejected.

The doc ID will have to be the hash of the file.

Yes. If you have that doc ID and then you need to set the operation to just create and not update or upsert, then it would just be rejected. And you would only write it once.

I'm not sure if that is a great use case or if you might want to keep, like, I don't know, an outside cache of, like, all the hashes that you've already had and deduplicate it there. But that would be the Elasticsearch solution of, like, using the hash as the ID and then just writing to that.

I'll tell you what we did. We just broke it down into chunks and did that only. I was curious if.

OK. And also with create on the yeah. That is, I think, the intuitive or most native approach that we could offer for that, yeah. I think there was some other question somewhere. Yeah?

Yeah. I just wanted to add on to the Postgres question a minute ago. Postgres does have a native full-stack search engine in it. That's pretty good. It's the TS Vector extension I've been in for a while. There has also been recently a lot of work.

And so there is a Rust-based BM25 search engine implementation in Postgres now. It's called, like, TensorFlow or something like that.

But from what I remember, the default Postgres full-stack search does not do full BM25. But it only does it doesn't have all the statistics, I think, from what I remember.

Correct.

Yeah.

Any other questions?

I do.

Joe, please. Go ahead.

Do you plan to cover more on the retriever thing, like the retriever concept and.

To show now, yeah. I mean, how much JSON do you want to see?

Well, it depends.

I'm particularly interested in, I mean, your modeling. So you have multiple I love the retriever concept. It's a great concept. And then you have multiple different candidate retrievers. What kind of flexibility do you have on top of that?

Because I'm not interested in using this. I'm interested in re-scoring them, not necessarily looking at anything. But I'm modeling you're just retrieving a bunch of stuff, candidates. And then you're actually evaluating your real scoring function.

Yes.

I'm sure there's something that was.

Maybe for before we dive into that, for everybody else, like, re-scoring is, like, let's say we have a million documents. And then we have one cheaper way of retrieving them. And we retrieve the top, I don't know, 1,000 candidates.

And then we have a more expensive way but higher quality way of actually re-scoring them. Then we will run this more expensive re-scoring on just the top 1,000 to get our ultimate result list of results. But the re-scoring algorithm would be too expensive to run it across, like, the million documents.

And why you don't want to do that, that's why you want a two-step process. And that's why you might want to have the re-scoring. So yes, we have a you can in Elasticsearch now, you can do re-scoring because it becomes more and more popular.

I don't have a full example there. But we do have, like, we do have a re-scoring model built in by default now. Let me pull that up.

Not this one. So we have currently, it's the version one re-ranking. But we have a built-in re-ranking model now as well. So for one of the tasks that we can do, you can see here, we have the other tasks, like, for example, the dense text embedding.

Now we have a re-ranking task that you can also call. Your question?

How do you express that in the whole retriever concept? Like, if you have.

It might be too detailed.

No, re-ranking is good. Let me

somehow my keyboard binding is broken. This is very annoying.

OK. We re-rank results. Let me see. Somewhere here, there should be so there's learn to rank.

But it should not be the only one. This is what we want. OK. We have our re-ranking model.

Unless, Dave, you know from the top of your head where we have theright docs for this.

Organization of our docs can help you with. But retrievers.

Yeah. Retrievers could find them.

Starting in 8.16 or 8.17, the Retrievers API added a specific, like, parent level retriever that you it's like you nest it around outside the retrievers that go inside of it. And it's specifically called the text re-ranking retriever. And so if you have a cross-encoder, say, from Hugging Face or you're using the Elastic re-ranker or using one of these things, then it complies with that kind of inference task of taking a bunch of things and re-ranking them against a query,right, taking the full token stream,right, and the full token stream with the context and doing that.

It can target a parent level text field of the documents that are being retrieved. So it works really well for the one document per chunk kind of re-ranking use case. I've also seen people just do it outside in a second API call, say, if you wanted to do it on a highlighted thing or if you wanted to do re-ranking of sub-document chunks.

That works pretty well just hitting the inferences API kind of directly as a secondary call. But yeah, there is a text

re-ranker retriever clause that specifically got added in, like, 8.16 re-ranking.

Yeah. So I think this is a simple example. Like, we have a standard match. Like, this will be very cheap. And then we have the text simulated rarity re-ranker, which uses our Elastic re-ranker that falls back to that model behind the scenes.

Yeah. So you can think about it I was a functional programmer, so don't mind the parentheses. But you would have, like, the text re-ranker retriever. Inside of that, you would have the RRF. Inside of that, you would have lexical and KNN as peers.

And it works from the inside out. Hey, do each of those retrieval methodologies. Do, like, the Venn diagram. Find the best results. And then take the full text of those results and run them through the re-ranker. It's almost like a little mini LLM saying, which one of these actually answers the question?

And then what outcomes? It's pretty good. The cool thing about the re-ranker is you can run it on structured lexical retrievals,right? You don't have to run it on a vector search. You can run it on anything you want.

So if you don't want to pay to vector search everything or maybe the text is too small to vector search, without needing there to actually have the model lock on the stuff, the re-ranker, when you run it on just kind of actual customer data sets and say, how did it do?

They're like, yeah, our evaluation score is bumped by 10 points. Basically for free, it feels like cheating,right? So when you run against, like, a Gemini API and it's like, wow, why is this 10 points better than the Amazon one?

It's because they threw on their retriever and didn't tell you,right? So there's a lot of black box stuff out there that we're exposing. So don't be scared that we're showing you how it works inside. But this is what the leading retrieval technologies are doing under the hood and then reselling it to you as if they're, you know, it's all AI,right?

So it's all just code. Train model, et cetera.

Yeah.

Does that answer the question?

Yeah.

Yeah. So my wish list is I want to be able to do, in the Retrievers API, re-ranking on sub-document hits. A lot of my things are about sub-document knowledge retrieval. Andright now, I got to do it outside of the Retrievers API.

But I'm bending the gear to the developer to get it done now.

Yeah. So

just to give you the example of, like, I don't think I have a re-ranking example here. But this one

uses a classic keyword match for retriever. And then we have we normalize here the score. I think somebody else asked about the normalize or we had a discussion about the normalizing. We do a min-max normalizing. We weight this with two.

And then I use the OpenAI embeddings with a, again, normalized with a weight of 1.5. And then they will get blended together. And you get the results. It won't surprise you that. These are not the droids you're looking for.

If you search for droid and robot, will be by far the highest ranking document. You had a question somewhere. Yes?

How much control does Elastic if you're doing the re-rankers thing?

Currently, we use something similar. But we, like, do the different steps of the re-rankers at kind of, like, different levels of the distribution hierarchy. So, like, you have, like, you have, like, sharded query processing nodes. And you do some ranking there.

And then, like, once you, like, rejoin before you return the result from the query engine, that's when you run the final re-ranker, just for, like, cost reasons, like, performance reasons. How much control does the Retrievers API give over that sort of thing?

Or is that, like, internally optimized? What parts of re-ranking happen where?

I mean, so in the first so we would retrieve, like, X candidates. And you could define the number of candidates. And then we would run the re-ranking on top of those. So that it will be a trade-off for you.

Like, the larger the window is, the slower it will be. But the potentially higher quality your overall results will be because you will just have everything in your data data set that you can then re-rank at the end of the day.

Is that what you meant? Or you wanted something per node or.

Yeah. I mean, like,right now, we, like, are actually controlling, like, specifically, like, where kind of, like, where in the compute that happens so that you can, like, fan out more aggressively for the, like, cheap re-ranking and then do the expensive re-ranking at once.

But maybe that doesn't apply in the case of Elastic. Like, I think it's about how the queries mechanically run on the machines.

Yeah.

That's going to be normal.

I don't think that's how we do it. So what you can control here is, like, this is the window of, like, what you might retrieve. And then we have the minimum score, like a cutoff point, to throw out what might not be relevant anyway to keep that a bit cheaper.

That's what we have here.

Those are the retrievers. And then you could do the RRF that I've explained where you blend results together. All of that is easy. One final note.

If you got tired of all the JSON,

we have a new way of defining those queries as well. Where here, we have a match operator, like the one we've used all the time, that you can use either on a keyword field. But it could also be either a dense or a sparse vector embedding.

And then you can just run a query on that and then just get the scores from that. So it is a piped query language. It's a bit more like, I don't know, like a shell. But if you don't want to type all the JSON anymore, this is how you can do that.

And here, my screen size is a bit off. But yeah, you get the quote that we retrieved, the speaker, and the score. And maybe I'll take out the speaker to make this slightly more readable.

Now it broke.

Oh.

This is you could write queries with a fraction of the JSON. This will also support funny things like joins. It doesn't have every single search feature yet. But it's getting pretty close. So this is more like a closing out at the end.

If you're tired of all the JSON queries, you don't have to write JSON queries anymore.

This is nice both for, like, observability use cases where you have, like, just, like, aggregations and things like that. But it's also very helpful for full-text search now if you want to write different queries. I think the main downside is that the language support in the different languages, like Java, et cetera, is not very strong yet.

You basically give it strings. And then it gives you a result back that you need to parse out again. So it is not as strongly typed on the client side yet as the other languages.

Any final questions? Yes?

You just talked about hybrid search. And I was just curious, like, what is kind of the recommended best practices? Like, we also do hybrid search today. What we do is we trigger two Elastic queries. One to do, like, a basic keyword search.

The other one to do, like, a vector search. And then we run through some, like, a coherent re-ranking API to get the final result. But, like, from the Retrievers, you could just show it feels like it's better, like, to combine those two queries in one Retriever.

And, like, would that make the results might potentially be better? Like, sort of it sounds like each of the query will have its own stats and normalizer. And it will mix things in, like, more, I don't know, just, like, in general, sounds better.

I mean, we can make your life easier.

Yeah.

It's just all behind one single query endpoint. So you could use the two different methods to retrieve. And then you could still re-rank but all from one single query. So you don't have to do it yourself. I mean, it's not like we want to stop you.

But you don't have to. And we can make your life a bit easier.

But, like, I'm just curious. Instead of just, like, simplify the process, would that impact the performance in any way as force or.

I mean, it's only one single query that you need to run and, like, one single round trip to the server that you need to do.

Yeah. But, like, I was just curious comparing to the two queries methods what we did today. Would that make any difference?

I mean, if you still need to do the retrieval, like, you do the retrieve, like, all the individual pieces are still there. If you have two parts of the query, you will still retrieve those if that is the main cost.

And then you have the re-ranking. So you're not getting out of those completely. But you can just do it in one single request that you send. We take care of all of that for you and then send you one result set back rather than sending more back to your application.

So

it will potentially be a little less work on the Elasticsearch side. But it will mostly be less work on your application side.

OK.

Yeah. You're doubling the utilization of your thread pools. So if your thread pools don't have any problems, you may not notice. But you're running two map reduces,right, when you could be running one. And you're denying the optimizer the opportunity to do any short circuits to say, oh, there's no more results that are better.

I can just stop. Yeah. So you're potentially going to be a little bit more performance and resource efficient by hybrid query time. But it's easier to do separately

profiling the query separately. Which is what people do before they turn it into a query. So if it's not hurting you, by all means, keep going,right? But at some point, you're going to start vertically scaling your hardware when you don't need to.

You could get further if it was just a hybrid search.

Perfect. Thank you so much. I hope everybody learned something. I will let the instance running for today or so so you can still play around with the queries if you feel like it. Thanks a lot for joining. If you want stickers, we have stickers up there.

We also have a booth the next few days. Come join and get some proper swag from us there. Thank you. See you around.