Evaluating Domain Specific LLMs for Real World Finance — Waseem Alshikh, Writer

Hello everyone, my name is Waseem, I'm one of the co-founders and CTOs at Writer.

Today I'm going to just tell you a quick story about actually why we're building at Writer, what we're doing. But before we dive in, I would love just to give you a quick history of Writer. So Writer's, we started the company in 2020.

We love to say the story of Writer is the story of the Transformer. We started building those decoder-encoder model in the early days, and we started, we kept building those models, and we built a lot of them. Today we have a family of models, I believe around 16 we published, we have another 20 coming in the way, and we keep building those models.

And you're going to see from this list, those models come in two categories: general model like Palmyra X, Palmyra 3, 4, if you have a Palmyra 5 is coming soon. And we have a lot of what's called domain-specific model: creative, financial services, Palmyra Medical.

Now,

early 2024, basically last year almost, we start seeing this trend with all the LLMs basically get very high accuracy in general with any publish mark. We're seeing the accuracy moving and just growing, and I believe everyone noticing this accuracy.

Today average accuracy for a good general model is between 80, maybe close to 90.

So that basically make a bring a question inside the company saying, is it worth it for us to start building and keep building domain-specific models if the accuracy today with general model achieving around 90% and we have domain-specific models, should we just keep building general models, fine-tune it, maybe go direction with what you call reasoning or thinking models, and that will be more than enough actually.

And we don't need those financial or what you call domain-specific model. Now, to answer this questions, we need data. So whatever we're going to present next actually could be a clickable to financial services domain-specific model, sorry, to medical-specific model, customer support domain-specific model, and all different domain-specific model.

Today I'm going to talk specifically about the financial, what you call the financial punch mark for domain-specific model. We have something similar for medical, but we believe we are, but we start seeing similar result. Now, let me dive in.

Just to remind you, we're trying to answer these questions: general model, domain-specific model, should we keep build them, where we actually going from here. We start actually saying, great, we don't know the answer, let's actually do the evaluation, let's create the data.

And we created something called FailSafe. The idea behind it, let's create real-world scenario to evaluate those model and let's see actually of those new model can really give you the accuracy that we promise or the accuracy that we see today from the punch marking on domain-specific.

We created two type of categories in this evaluation, something called query failure. In query failure, basically we introduce three type of subcategories, something called misspelling queries, you know, when you go ask the LLM questions but you do some spelling error, segment error, you do some comment typo issues.

We introduce that to the eval set. We introduce something like what you called incomplete queries, you're missing some keyword, some stuff not clear. We introduce what you call out-of-domain queries. If you are not expert in the field or you decide to copy-paste some general answer, try to answer about something very specific.

And also we introduce a second category is what we call the context failure. In the context failure, basically, and this could be very interesting, we introduce three subcategories, what you called basically missing context. We basically ask the LLM question about context not exist in the request itself in the prompt.

We introduce what you call OCR error. Today when we do any kind of OCR or convert physical document to text, we introduce a lot of errors like, you know, character issues, distance between them, the word between when you do the OCR could be merged together.

So we introduce that type of errors. And also we did what you call unrelevant context. Let's say you want to ask question about specific document and you end up basically uploading completely wrong document. Does the LLM going to still answer?

Is the LLM just actually figure out you have a completely unrelevant context? Now, when you put all this data together in domain and financial specific, financial service specific, you need some kind of diversity. This is a quick screenshot just tell you what amount of data, how much token.

Something worth mentioning, the white paper, the data, the evaluation set, the leaderboard, all actually open source today available in GitHub and Hugging Face. So anyone please check it out. And we introduce very simple, what you call it, evaluation key metrics.

Basically we need to look to two things. If the model give the correct answer, can the model actually give good follow to the grounding or context grounding or basically what you call it here the context. This is quick or high level way of how we do the calculation.

So to evaluate, we select the group of models. Today we can see a lot of chat model and also thinking models. This is basically the two list we have here. I'm sure you're familiar with this list. And then we run the evaluation and we start seeing very interesting result.

I'm going to dive in directly to the result. And basically we start getting something fancy with all this color. Let me switch to the what basically see. What's start getting very interesting, we're seeing really good behavior in all thinking models actually they don't refuse to answer.

This sound good most of the time, but in reality when you give something those LLMs wrong context, when you give them wrong data, when you have a completely different grounding, those model actually fail to follow this part and they still give you an answer and that basically get you way higher hallucination.

If you start focusing on the answer itself, can the model give me answer or not, you can see basically almost every model from the domain-specific to general model, they give you some kind of answer or them close to each other.

Actually reasoning or thinking model they get to even higher score a little bit from there. But we get to the grounding and concept grounding, this is when stuff get more interesting. You can see specifically in task like text generation or question answering, it's just not performing well.

Now all this chart look great, what I prefer is the numbers.

This is the same data we use to generate the chart. We can go through this really quick. And if you look at this number here, especially for example like the 01 or 03 or Palmyra Fen, you can start noticing the stuff those model doing amazingly.

And basically when you ask was it misspelled, when you got stuff and complete out of domain, the numbers look amazing. The model can take a query with misspelling, wrong grammars, or even out of domain and still can give you the answer.

But

when you start going to grounding, this is when stuff get very interesting. I'm going to hold this slide for a second here. Did you notice something different?

Smaller is better.

Yep. And also those bigger more thinking give you the worst result. You're getting almost 70, 50% to 60% worse in the grounding. Meaning the model is just not following. You attaching context, you ask the questions and the answer is exist outside the context completely.

Same thing coming to stuff around it, unrelated context. So you can look at the data and you'll see smaller model actually performing better than all this model overthinking at that side. And this is basically it get us about is this thinking or just a chain of thought.

You know, this could be a lot of argument. At least from the data we have, in domain-specific task, those model not thinking at that stage. Meaning hallucination is really high, causing a lot of issues, especially in this

bunch mark we run here in financial use cases. Also we can see there's a huge gap between what you call robustness and the hallucination and getting the answer correct. So definitely we still have a lot work to do to build those model and better performance.

But also that get me to, you know, to the main idea. If you go back really quick here, even with the best model between all the slide, we still not getting between robustness and context grounding more than 81%.

Sounds a great number. If you think in reality, you're saying every 100 request, 20 of them it just completely wrong. So that basically what we start seeing, believe at least today with the technology we have, with the current model we have, until we have something completely different, we're seeing you need full stack, you need the RAG system, you need the

grounding, you need everything from guardrails and the build around the system itself to actually have something reliable utilized today. In the same time, I would love to go back and answer the first questions. And our first question here, do you still need to build the models?

At least today from the data we have, from running those punch mark, the answer simply yes. We still need to build and continue domain-specific model at least with the today implementation. Even accuracy is keeping growing, but the grounding, the context, following all the context correctly, it's still way, way, way behind from everything we see today in the market.

Thank you so much.