Trust, but Verify: Knowledge Agents for Finance Workflows - Mike Conover

I'm Mike Conover. I am founder and CEO of Brightwave. Uh, we build a research agent that digests very large corpuses of content in the financial domain. So you can think of due diligence in a competitive deal process: you are pre-term sheet, you step into a data room with thousands of pages of content.

You need to get to conviction quickly ahead of other teams, you need to spot critical risk factors that would diminish asset performance. It's a fairly non-trivial task. You think about mutual fund analysts: it's earnings season, you've got a universe of coverage of 80, 120 names.

There are calls, transcripts, filings. It's a fairly non-trivial problem to understand at a sector level, but also at the individual ticker level, what's happening in the market. Or, goodness, you get into confirmatory diligence and you've got 80, 800 vendor contracts, and you need to spot early termination clauses.

You need to understand thematically, how is my entire portfolio negotiating their vendor contracts? It's, frankly, not a human-level intelligence task. And the reality, as we've stepped into this space, is that these professionals just get put in a meat grinder.

Junior analysts are tasked to do the impossible on extremely tight deadlines. I come from a technical background. Prior to Brightwave, I was at Databricks and created a language model called Dolly that was one of the earlier models to demonstrate the power of instruction tuning for eliciting instruction-following behavior from open-source technologies.

And as I have met with these professionals, I have developed a deep sense of empathy for the stakes and the human cost of doing this work manually. When it comes to the role of the individual in finance workflows and financial research, we think of the parallels to early spreadsheets.

You go to an accountant or finance professional in 1978, before the advent of computational spreadsheets. You say, "What's your job?" "Well, I run the numbers." It's cognitively demanding. These people write this stuff out by hand on literally wide pieces of paper called spreadsheets.

It's cognitively demanding, it's important to the business, and it's time-intensive. It feels like real work. And now nobody wants that job. And it's not because there aren't finance professionals, it's not because nobody's doing analysis. It's the sophistication of the thought that you can bring to bear on the problem has increased so substantially because there are tools that allow us to think more effectively, more efficiently.

What we're seeing, what we're hearing from our customers is that a system like Brightwave that is able to digest—and not just Brightwave, this class of knowledge agents—is able to digest volumes of content and perform meaningful work that accelerates by orders of magnitude the efficiency and also time to value in these markets.

And so the purpose of this talk is to relate

sort of the intelligence that we've developed in the course of building this high-fidelity research agent, and just things that we're seeing both technically but also in terms of product affordances. I mean, the design problem that you have to solve is: how do you reveal the thought process of something that's considered 10,000 pages of content to a human in a way that's useful and legible?

That is not a UI/UX problem, it's not a product architecture problem that existed three years ago. And the final form factor has not been determined. Chat, everybody's very target-fixated on chat. That's probably not enough. So the first thing that I'll observe is that non-reasoning models are performing agreed to local search.

So the Bloomberg talk highlighted that sort of fidelity issue. Like a really concrete example: you put a Reuters article in 40 and you ask it to extract all the organizations. Goodness, if it's not going to give you products.

And if you have a 10, 5 percent error rate and you chain calls like that, you're going to introduce, sort of in an exponential way, the likelihood of error being in these models. And so the winning systems will perform end-to-end RL over tool-use calls, where the results of the API call are in fact part of the RL sequence of decisions so that you can make locally suboptimal decisions in order to get globally optimal outputs.

The reality is that that's still an open research problem. You know, how do I avail myself of a knowledge graph? Or, "I did not do that."

How do you avail yourself of these tools in an intelligent way so that you get globally optimal outputs? It does seem like that that is not a solved question. So the reality—and I think it's like heartening to see this as a theme, and I think everybody in this room can be sort of comforted by this—you've got to build a product today.

And like, there's going to be this talk of the bitter lesson: that more data, more compute, better models dominate all other approaches. Like, nobody wants an expert system. Nobody wants to use SpaCy to do named entity recognition. The—sorry, that was not in the speaker notes.

You can think of being more circumspect about what is the scope of behaviors that the system, the agent, is going to engage in, sort of like a regularization parameter, which constrains the complexity of the model. And that limits the likelihood, reduces the likelihood, that it will go truly off the rails and begin to produce degenerate output.

You can think of it sort of like a—like multi-turn. The most interesting interactions I've had with language models are deep into a conversational tree, where you can think of selecting, at each branch, each response, there are a set of reactions that I can have to the model output.

And I'm steering, I'm choosing. This is what knowing how to use language models—that's a skill. And many people who have real full-time jobs may not invest in developing that skill. This is not dissimilar to what these RL systems are doing.

And if you can think of a multi-turn conversation as not just establishing a human-orchestrated chain of thought, but really that set of tokens defines the activations of the model. And if you think of the activations of the model as defining a program, what you are doing when you respond to the model and say, "No, not quite like that, more like this," is if you think of the activation weights or the activations as a point in a vector space, you are nudging the activations to a place where they can finally solve the problem at hand.

And I think that's what the chain of thought process, or the sort of reasoning monologue, is performing. It's getting the activations to a position where it can actually solve the problem. So it's actually not—I don't—it's cute that you can interpret it, but I would prefer if it just got to theright set of activations automatically.

And so from a product affordance standpoint, people are not going to want to really become prompting experts in a deep way. And frankly, it takes easily a thousand hours. And so the scaffolding that products put in place in order to orchestrate these workflows and shape the behavior of these systems, I think, you know, these verticalized product workflows are probably going to be enduring because they specify intent.

They take that weight off the user. So some of the things that we see with respect to archetypal design patterns in this space: consider a basic autonomous agent. You really want to mimic the human decision-making process and decompose what is it that a person would do.

Well, if I need to understand how this polypropylene resin manufacturer is managing costs, I might look for public market comparables. And that would, you know, maybe entail going to the SEC filings or earnings call transcripts. And I would assess content potentially from a knowledge graph constructed from previous deals that, you know, I as a private equity investor have done.

News corpuses, assess which document sets are relevant to me, distill down from those documents findings that substantiate premises or hypotheses that I might have about this question or this investment thesis, and then enrich and error-correct those findings. And so a couple of points on this.

One is that it is actually—so I forget who it was, but they were talking about—it was the deep research team talking about, on that next step, what are my intermediary notes? What is it that I believed on the basis of what I found?

That's actually an extremely useful think-out-loud about what do we believe, given the facts as they have materialized on that first pass through the dataset. Enriching individual findings that are distilled down from documents is an extremely powerful design pattern.

Likewise, it's kind of—you can ask these models, you know, "Is this accurate?" For that Reuters example, you can say, "Is this factually entailed by this document?" Or, "Is this actually an organization?" And the model can frequently self-correct. And what we've noticed is that it is—you can do that in the JSON as sort of like a chain-of-thought behavior, but it's also—it's actually more powerful to do it as a secondary call because the model is kind of primed to be credulous.

It says, "Well, you know, I told you it was, and so I'm probablyright." So it's interesting how you can tease apart some of these steps into multiple different calls. And then through this process of synthesis, you're able to weave together fact patterns across many, many, many documents into a coherent narrative.

And that control loop, we think that obviously human oversight is extremely important. The ability to nudge the model with directives or sort of selecting, "This is an interesting thread, I want you to pull that," is extremely important. And that's because the human analyst always is going to have access to information that has not been digitized.

That's a conversation with management. That's your portfolio manager thinks this class of biotech is just hairbrained. That taste-making, I think, is going to be where you see the most powerful products lean. I firmly believe, with respect to the nodes in that knowledge graph, and we probably—many people in this room probably reached some of this conclusion as well, but you still see this, "Oh, we've got a portfolio manager agent, and this is the fact-checker."

And that sort of needless, like, anthropomorphizing of these systems, it constrains your flexibility if the design needs of your compute graph change. And this is—this is 1970, I think it was 1978, and, you know, the Unix philosophy, it's like, you think about piping and teeing on the Bash command line.

I guess I date myself. I still use Bash, not C-shell. Just simple tools that do one thing and that work together well. And text is the universal interface. It's 40 years ago. 50 years ago, geez. So our friends at Latent Space put together this plot.

With respect to the structure of these graphs, obviously that Pareto frontier, which is the sort of efficiency frontier—it's two batten a thousand a day—the efficiency frontier for compute and performance tradeoff, or price-performance tradeoff, that frontier is going to continue to move out, but there will, I believe there will, for at least an enduring time, be a frontier.

And what's notable about that is that you have to select then which tool, which system, which model am I going to use for each node in the compute graph. And the reason that this is important is

what I call the latency trap. If you think about the plot of time to value and realized value for agentic systems, and I think this is extremely important, it's very easy to think, "Oh, man, it's going to do all of these things.

It's going to, you know, I'm going to check it and error-correct, and then, you know, in 25 minutes it's going to be banger." And I think even with high-quality products like OpenAI's Deep Research, it's, you're not always sure that what you're going to get out is high quality.

So there's kind of like a question of like, which side of the diagonal? It's probably not a straight line, but is that product on? But also from a rep's standpoint, the impulse response for the user, how well refined is—you can think of like my expectation for what the report is going to look like and what the report actually looks like is the loss.

And the user's mental model is developing a sense for how do my prompts elicit behaviors from these models? If it's an eight-minute feedback loop, it's a twenty-minute feedback loop, goodness, you're not going to do many of those in the course of a day.

And your faculty with the system and the product is going to be low. So synthesis is really where a lot of the magic happens in these systems. And

a couple of observations. So notice that it—I don't know, has anybody in this room ever had a 50,000-token response from any model? No? They say that, you know, O1 is 100,000 context output context length. I'm not so sure.

And it's because the instruction tuning demonstrations, these human-generated, synthetic or human-generated outputs that are used to post-train the models, have a characteristic output length. It's hard to write 50,000 coherent novel words. And so the likelihood that the models are able to produce, I mean, even O1 still is about 2,000, 3,000 tokens.

It's better than 4.0. And so what happens, it's kind of like a—there's a compression problem. If I have a very, very large context window for input, I'm compressing that information into a set of tokens. And so it's like the difference between writing a book report and a synopsis of each chapter.

You can be more focused and specific about what is it that I want those 10,000 tokens to be focused on. Here we have, you know, I said, "Write an analysis of the global financial crisis." Goodness, if I don't think the rise of the shadow banking system warrants more than three sentences.

And so if you can be more granular and more specific, you can get higher quality, higher fidelity, more information-dense outputs out of these systems by decomposing your research instructions into multiple sub-themes.

Additionally, the last point I'll make on this problem is that of the presence of recombinative reasoning demonstrations in the instruction tuning and post-training corpuses is low. So it is easy to say, "Here, you know, given the text of the Great Gatsby, this is the epilogue, and write a new epilogue for the Great Gatsby," because the cost of internalizing that corpus is fixed.

Effectively, you read the book and then you write five epilogues, and it's like, "Goodness, I got it." Synthesis really is about weaving together disparate fact patterns from multiple documents. Think about the applications to biomedical literature synthesis. I need to read all of these papers and then have something useful to say that actually brings together the facts from these documents.

Now there's like a cute trick you could try, which is to say, "Given the bibliography of any given paper, write the abstract as a post-training exercise." But it's just really hard to get high-quality, intelligent, thoughtful analysis of many, many, many different documents.

And so there are limitations in practice for even state-of-the-art models in terms of how they are able to manage complex real-world situations, factors like temporality, perplexity at a—well, so temporality is hard. And being able to understand, you know, something like a merger and an acquisition, you know, these pro forma financial statements are different from those that came before the event.

If there are addendums to contracts, it's important to propagate with evidentiary passages, metadata that contextualizes, "Why do I care about this? What do we think about this document? How should I consider this in relation to the other pieces of evidence in the context window?"

So I'll now shift a little bit with some examples from the product that we've built, which is, how do you reveal the thought process of something that's considered 10,000 pages of text? And I think that it is more like a surface and one where you're able to—it's kind of like this.

Like people you may know, the Facebook and LinkedIn recommendation algorithm for

connections feels uncanny good in part, not because, I mean, the algorithms are okay, not great, have gotten a lot better over time, but in your visual cortex, there is a bundle of nerves that are exclusively dedicated to face recognition.

And the ability to say in a, you know, six-by-six grid of faces, "Goodness, I know that person." And so you attend to the things that matter, even if it's actually a low-precision product experience. And so the ability to give the person details on demand is extremely important.

We'll see. So here we have a Brightwave report. You know, I think the ability to click on a citation and then get additional context about this, not just what document is it from, but how should I be thinking about this?

What was the model thinking in the course of this? As well as structured interactive outputs that give you the ability to pull the thread and say, "Well, tell me more about that rising CapEx spend." In Brightwave, you can highlight any passage of text.

So it's not just the citations, but you can highlight any passage of text and say, "Tell me more. What are the implications of this?" I think OpenAI gestures towards this with respect to Canvas and the ability to increase the reading level of a passage.

Having a continuous surface that's not just these citations, but in fact, any finding should be interrogable.

Likewise, you can think of—I'm actually going to pause. It's not going to pause. I'm going to go back and do this again.

You can think of the set of things that the model has discovered. It reads all of these documents. It develops a view. It weaves the facts together. As a high-dimensional data structure and the report is one view on that data structure, it's kind of a low-effort point of entry into the space of ideas.

You want to be able to turn over that cube and see, especially in finance, the receipts. What's the audit trail for this system that's read all of these materials? And so being able to, in this example, click into the documents is one level, but having all of the findings laid out for you, whether it's a fundraising timeline, ongoing litigation, I'm able to, if something catches my attention, click on it.

This is where that investor—hello?—investor analyst taste comes into play. I'm able to say, "Tell me more about that." It's like a magnifying glass for text. Something catches my eye. This patent litigation, goodness, that seems important. You had a factory fire in Mexico that wiped out, you know, a critical single-source supplier.

What are you going to do about that? That ability to drill in and get additional details on demand is extremely important in these systems. And I think, candidly, we do not yet have the final version, the final form factor of this class of products, but it's an extremely interesting design problem.

And I will say, we are hiring. So these QR codes, not only is it a great place to work, we've got people from Goldman Sachs and UBS and Meta and Instagram and Annaplan, and we just hired a senior staff software engineer from Brave.

Goodness, we've got a stacked team. We also have a $10,000 referral bonus. So I'm going to see a lot more phones come out now. $10,000 referral bonus for all of these roles, primarily the product designer and the front-end engineer.

We're hiring staff and senior staff-level professionals. We have a small team of extremely experienced individuals. And this is structured like the DARPA Red Balloon Challenge, if you're familiar. So if you refer the person that refers the person that we hire, you get 1,000 bucks.

And so on and so on and so on all along that exponentially exploding referral tree. So we're Brightwave. We build knowledge agents for finance workflows. I appreciate your time today.