Analyzing 10,000 Sales Calls With AI In 2 Weeks — Charlie Guo

I want to start with a question: how many sales calls can you listen to and take notes on, in a single day? If you assume each call is 30 minutes and you work an 8-hour day with no lunch break, that gives you 16 calls.

If you have absolutely zero work-life balance and only stop to sleep for 8 hours a day, you might get to 32 calls. And if you do that every single day, you'll get to 224 per week doing nothing but listening to sales calls over and over again.

Last year, our CEO wanted to analyze 10,000 sales calls in order to do a wide-ranging analysis of our ideal customer profile, and I had 2 weeks to do it. Two years ago, this would have been impossible, or it would have at least required a dedicated team working for several weeks.

Today, it's something a single AI engineer can accomplish in about a fortnight, and I want to show you exactly how I did it. At Pully, our ideal customer profile, or ICP, was venture-backed startups. It's a useful north star from a product perspective, but much less helpful when it comes to marketing.

We knew we needed to get more specific: think CTO of an early-stage, venture-backed crypto startup, instead of merely "founder." And as every founder knows, when you're trying to understand your customers, there's really no substitute for talking to them.

In my case, I had the next best thing: thousands of hours of our sales reps talking to customers directly. The only problem was, it was too much data. I want to paint you a picture of what a manual analysis of this sales call database would look like.

It would require downloading each transcript, reading the conversation, deciding if that conversation matches the target persona you wanted to analyze, scanning hundreds or thousands of lines of conversation for key insights, trying to remember everything while writing reports, and compiling notes as well as citations for a future reference in a research report.

And then, of course, doing this 10,000 times in a row. Even if you wanted to do this manually, we're talking about 625 days of continuous work, nearly 2 years. The human brain simply isn't wired to process that much information.

It's like trying to read an entire library and then write a single book report about it. Before LLMs, traditional approaches to doing this kind of analysis generally fell into two categories: a manual analysis that was high-quality but completely unscalable, and a keyword analysis that was fast and cheap but would often miss context and nuance.

This is where modern large language models come in. This intersection of unstructured data and pattern recognition, it's a sweet spot for AI projects. But here's what I learned: what looks simple in hindsight—just use AI to analyze the sales calls—actually required solving several interconnected technical challenges.

Our first major decision was choosing theright model. At the time, GPT-4.0 and Claude 3.5 Sonnet were the most intelligent options that we had access to, but they were also the most expensive and the slowest. The temptation to use smaller and cheaper models was pretty strong, but our experiments showed their limitations rather quickly.

They produced an alarming number of false positives. They would classify transcripts as being related to crypto companies because a sales rep mentioned blockchain features on the platform, or they might decide that a prospect was actually the founder of a company despite zero supporting evidence in the transcript.

In our case, the worst outcome was a bad analysis. If we couldn't trust the model's data, the entire project would be pointless. So we made the choice to shell out for the more expensive models, since they produced a hallucination rate that was acceptable for our needs.

In the end, we ended up going with Claude 3.5 Sonnet. I mention the hallucination rate here because it also wasn't as straightforward as feeding the transcript in and asking for answers out. We ended up developing a multi-layered approach to reducing hallucinations in this system.

We started with the raw transcript data, which we then enriched via retrieval-augmented generation, both from third-party sources as well as other internal sources that we had access to. We employed some prompt engineering techniques, like chain-of-thought prompting, in order to get the model to produce more reliable results.

And finally, we made sure to get structured JSON outputs where possible in order to generate citations. This, taken together, created a system that could reliably extract both accurate company details and meaningful insights. And more importantly, it had a verifiable trail back to the original transcripts where it was pulling information from, meaning we could be confident about the final results.

The main problem was, doing all of this analysis and making sure we had low error rates drove up our costs pretty significantly. We often hit the 4,000-token output limit for Claude 3.5 Sonnet, which would require multiple requests per transcript analysis.

Luckily, we were able to leverage two features that were experimental at the time to dramatically lower our costs. The first was prompt caching. A lot of the analysis that we did involved reusing the same transcript repeatedly, both to extract metadata and to extract insights.

By caching the transcript content, we were able to reduce costs by up to 90% and latency by up to 85%. The other feature we leveraged was extended outputs. At the time, Claude had an experimental feature flag that could give access to double the original output context.

Using this flag let us generate complete summaries in single passes rather than having to break the analysis into multiple turns and burning multiple rounds of credits. As a result, we turned a $5,000 analysis into a $500 one, and we got results in days instead of weeks.

But what ultimately surprised me the most about this project wasn't the technical capabilities that we discovered or the challenges that we overcame. It was the wide-ranging impact of our analysis. What started off as a project for the executive team, to produce insights that would normally take tens, if not hundreds, of hours, ended up being more useful across the organization.

The marketing team, for example, were able to pull customer quotes for branding and positioning exercises that they needed. And the sales team was able to automate transcript downloads on the back of this system, saving them dozens of hours every week.

On top of all this, teams started asking questions that wouldn't have been considered before because doing manual analysis like this would have been too daunting. In the end, we transformed our mountains of unstructured data from a liability into an asset.

So what did we learn from all this? I want to share three key takeaways. First, models matter. Despite the push for open source and smaller, cheaper models, Claude 3.5 and GPT-4.0 could handle tasks that other models simply couldn't.

We chose Claude for its prompt caching capabilities and its accuracy, but in the end, theright tool isn't always the most powerful one. It's the one that best fits your specific needs. Two, I want to point out that good engineering still matters.

Despite AI's capabilities, we gained significant wins from good old-fashioned software engineering, leveraging JSON structured output, good database schemas, and proper architecture for this system as a whole. AI engineering is knowing how to build effective systems around large language models.

That means that the AI can't just be bolted on or used as an afterthought. It needs to be thoughtfully integrated into existing systems and architectures. And we learned that we needed to consider additional use cases for our tech.

We didn't just stop at a single report or white paper. We built out an entire UX around our AI analysis, with features like search, filters, and exports. And by building a simple yet flexible tool, what could have been a one-off project actually became a company-wide resource.

At the end of the day, this project showed how AI can transform seemingly impossible tasks into routine operations. It's not about replacing human analysis. It's about augmenting it and removing human bottlenecks. To me, that's the real promise of tools like Claude, ChatGPT, Gemini, and all the rest.

Not just doing things faster, but unlocking entirely new possibilities. So here's my challenge to you: what customer data are you sitting onright now? Sales calls, support tickets, product reviews, user feedback, and social media interactions. These are all valuable sources of insight that go untouched in most companies, but are now very much accessible via large language models.

The tools exist today. The techniques work. The only question is, when will you stop ignoring your data and start turning it into gold? If you liked this, you can read much more about it and other projects like it at my blog, Artificial Ignorance.

Thanks for watching.