The Coherence Trap: Why LLMs Feel Smart (But Aren’t Thinking) - Travis Frisinger

Hi, my name is Travis Frisinger. I'm the technical director of AI, AIthlight, and I'm here to talk to you about the Coherence Trap, or why prompting feels like magic but isn't. This is a top-down talk focused on how I understand LLMs.

It's about explaining why LLMs work so well when we know they're not intelligent and they lack intent or desire. It's very much a story about chasing a feeling across experiments, analysis, and finally formulating a theory.

Let's step back to November 2022. GPT-3.5 was released, a lot of hype in the news about it, and I promptly went to go check it out. I'm sure like a lot of us did. I was very quickly disappointed.

While it had shown some advancements in certain areas, like it could help me improve an email and do other things that I wasn't used to doing with a chatbot, my experience with chatbots had been kind of very customer support type of focused activities.

It was very brittle in its understanding. Its surface-level fluency often collapsed at edge cases. It had prompt sensitivity, so small wording changes, obviously context limits, and overall felt very hyped up,right, overhyped.

And then GPT-4 came out not much later, I think it was about January of 2023, and you know, it was a very uncanny moment. My brain started to tingle as the words seemed to align effortlessly on the screen.

So what happened here? You know, I wasn't alone. Other folks, like Microsoft Research, published a paper, "Sparks of Artificial General Intelligence: Early Experiments with GPT-4." Ethan Mollick, a pretty well-known academic in this space, started conducting some more research, I think doing a little writing as well around this.

And overall, you know, there was something we were all noticing,right? Like it surpassed anything we'd known before. Just that experience, that interaction, that perceived level of utility was something that we hadn't encountered, or at least I hadn't encountered, in my experiences with AI.

Something had fundamentally shifted. Specifically, you know, when output felt like understanding. It's the moment when AI transcends mere text generation and displays something eerily similar to comprehension. Hence the Microsoft paper on AGI. I didn't feel that that was adequately capturing what I was feeling.

I felt there was a space in between these really, like, very dumb chatbots to AGI, and I wanted to explore that a little bit more. So what did I do? Well, I do what I do: I started doing experiments.

I'm a bit of an engineer and a scientist, you know, just as comfortable writing some research papers as I am, like, producing production code. So I started digging to understand. Where did I start? Not where I would have expected.

I started with a live stream, mainly because it was a low-friction way. My colleague I was talking to about the idea. I'd recommended it. She'd known others that, you know, gotten traction with this, not specifically around AI, but just thought leadership.

So I fired up the YouTube stream and started working with ChatGPT in a live scenario. I did paired programming. I called it chat-assisted programming. Others called it chat-oriented programming. And finally, we know it as vibe coding today. It was a really interesting experience because it took a lot of effort to get just a few hundred lines of producible, usable code out of the system,right?

It wasn't a great experience. I was building in public, you know, a lot of nerves, but ultimately it did balloon into something bigger. And these weren't just hacks. These were prototypes for what AI pair of programming would become.

And I still use a lot of that same technique today in tools like Cursor or Windsurf. At the end of the day, I developed one really useful utility called Webcat. It was a Python Azure function that would scrape a web page and bring back the contents for me.

Why was this super useful? Well, and then paired with a custom GPT. Why was this super useful? Because the early ChatGPT-4 didn't have internet access. I was often copying web pages in, trying to chat with them, explore ideas.

While, you know, it had this vast corpus of knowledge, it just wasn't very useful for the things I was trying to solve. So I made a utility to try and address that.

Which then led on to, you know, deeper collaborative experiments. I ended up building my blog, aibuddy.software. Go check it out if you're interested. It's been running for a little over a year, probably about a year and four months now.

And the idea was, can I work with AI to build this? Not that AI is super smart and it's going to give me all the answers, but, you know, kind of lean into this collaborative essence that I was feeling along the way here.

So I let it pick the platform using Ghost. I had a server to put it on already. You know, I didn't—I had some weird constraints. I didn't want to use, like, Medium or other, you know, public services. I wanted to keep this low cost.

So it gave me the instructions. I followed them. We together, the AI and myself, built a template, you know, that fit the need. And we started live streaming. Again, I say we. I started live streaming while working with ChatGPT.

I was using Webcat to pull in, you know, article snippets to talk about those in relation to ideas and have it help me build a blog around various concepts. And it's worked successfully. You know, I have a very, I think, you know, pretty well-trafficked thought leadership blog.

So I would say that's been awesome and successful. Which led me on to, you know, thinking about how can I make a bigger project. It's good. It's kind of good for code at this point. It's definitely good at content generation.

What else can I do? Can I make music? Now, I do have a background in music. I'm a trained pianist, so I understand music theory and other things. But I wanted to produce a whole concept album. You know, I figured, hey, my cat, I like my cat, I like metal.

Can I make a feline metal album? Let's call it "Mr. Fluffle's Reign of Tiny Terror." And the goal was just to produce something fun and silly, use multiple tools, use ChatGPT for the lyrics and the music composition, image editing as well.

Could I keep images relatively consistent across interactions? This was the first time that you could actually work with the ChatGPT image generation and ask it to refine. Prior to that, it had been a lot like a slot machine.

So, you know, there was new creative partnering prospects here, so I wanted to lean into those and understand them. And I'd done—I thought I'd done everything to handicap this process,right? Like, it's Cat Metal. Like, who's going to be in the Cat Metal?

It's AI-generated. Like, am I going to be able to produce anything of value, or is this going to just be AI slop? Well, much to my surprise, I just uploaded the videos to YouTube. Didn't let anyone know they were out there.

And within a month, I had over 3,000 views, which, you know, for feline Cat Metal AI-generated stuff seems pretty impressive. And what's even more impressive is I had a lot of likes. I had a lot of positive comments.

I clearly was able to create something outside of just my own capabilities in assistance with AI that was of value, and people recognized the value. So this got me thinking. Huh, okay. So it's working across a variety of things.

I can create a single coherent concept. Maybe there's something here. Maybe I need to start looking for

other behaviors. So I decided, hey, you know, decision intelligence is a thing. It exists within the traditional or tracked AI space. It's very useful for understanding how to work with AI systems where, you know, you need to make decisions.

And I'm like, hey, I'm an engineer. I've been doing pair programming since the early 2000s. Like, you know, maybe these two things are kind of the secret sauce. So let me go looking there. I didn't know what else to look for either.

So I figured, like, a good jumping-off point. Can I—could I go look for elements of decision intelligence, and could I go look for elements of pairing behavior? Well, sure. I broke up the vibe coding skills and built an analysis tool.

I downloaded my ChatGPT history, and I started building a tool to pull out each chat history, run it through with a prompt to look for qualitative and quantitative metrics around decision intelligence, pairing behaviors. You know, was there this thing happening?

You know, was there a concept in there that I could name, or was it just kind of me going, "Oh, yes, of course, you know, I'm doing these things. I know about them." Now, I did end up writing a research paper on this.

I wrote a 21-page case study that outlines the technique, that outlines the prompts, that outlines, you know, the things I was looking for. It's all there on my AIbuddy site/papers if you're interested. But it was a really neat experience.

It really allowed me to kind of bridge the kind of, you know, business world with the academic world again. So what did I pull out? I pulled out something I'm calling the AI Decision Loop. I feel like, you know, at the initial stage it was very academic.

I was trying to make sure I checked all the boxes and had all these delineated phases. I would probably roll it into a four-step process, which we'll talk about on the next slide. But the idea is that you frame the problem,right?

You come with the problem in the context. Think of this as prompt engineering. It's the one place where, I guess, the hype makes sense. Then you're going to generate some outputs, maybe one, maybe many, depends on what your prompt is.

And here's where most people, I think, fall off. They don't really engage with the output. They kind of take it and carry on. But you really need to judge. You need to evaluate the quality and the fit. You know, if you're at extra level rigor, you can think about validating it.

Maybe there's some external requirements or you need to go through and fine-tooth comb. So judge could kind of be asking it questions, validate would be some external validation, and then iterate,right? This isn't a once-off prompt and accept type scenario.

What did it getright? What did it get wrong? What can you do to improve that experience so that it starts getting more towards what you're looking for? You know, and then start it again. Frame it again. Generate, judge, validate, iterate, and keep going through that cycle.

I said there's a less academic version. I would like to think of this as kind of the nudge and iterate framework. So frame, generate, judge, iterate. Frame it up, generate some responses, evaluate those responses, and continue to prompt and nudge the model in such that it produces more and more reliable outputs.

This is kind of what I found at the heart of my reliable interactions. Not every interaction followed this. Not every interaction was good. But when I did, and I followed this frame, generate, judge, iterate framework, I tended to have much better outcomes.

So that's all good. I've got a mechanic. I've done some experimenting, but it still didn't explain why these things weren't, you know, intelligent. So I did some more thinking. I was like, okay, you know, I've done a lot of natural language processing work.

Coherence is a concept that sits there. It starts to feel like it makes more sense. Some work by Anthropic had come out around feature superposition and concept circuits, along with a lot of other research I was reading at the time.

And I was like, hmm, you know, maybe this is a coherence theory. You know, what would that look like? How would I actually explain what's happening inside the machine to achieve these outcomes if it's not thinking?

And I think the idea here is that, you know, it's—coherence is a system property. It's not a cognitive one. It's a precognitive one. It's the infrastructure that thought navigates. My thoughts, particularly. I'm giving it to the infrastructure of the machine.

So what is coherence? Let's talk about that briefly. So for me, coherence has, you know, four key properties. It's relevant. It means the model's output feels topical, connected, and purposeful to the conversation. It's consistent. The model maintains, you know, a singular tone terminology and structure across multiple turns.

So if I'm, like, the 10th time I'm having a response with the chatbot, it still knows that MCP is model context protocol. It's not something else. And I think the single most important property of this is stability,right? The model's able to withstand pressure.

So you can question it about its outcome or, you know, give it a competing theory or, you know, whatever you're going to do to kind of judge what's happening there. It doesn't fall down. It doesn't collapse under that interrogation.

It maybe firms up or course corrects. I feel that's a lot of what was missing from the ChatGPT 3.5 days and other earlier models. And most intriguingly, it's an emergent property. We see this specifically. An example would be GPT-4.0 has never been trained to detect swine disease, yet it can diagnose that through what I'm calling coherent pattern alignment.

We see the same thing with certain types of cancer diagnosis. It's able to pick up that pattern before we can even see that there's a pattern there to be detected.

So how does this operate? So talking about coherence in the machine, let's get into some of the mechanics here. So typically we think of neural networks as storing a concept in a single neuron. So a neuron holds on to a cat, and it knows that when it encounters a cat, that's the neuron that's supposed to light up.

Well, some research by Anthropic is saying that in these large language models, that's not exactly the case. There's something called superposition that allows the network to represent these complex ideas with fewer parameters, kind of packing more nuance into the same space.

And as the context accumulates, the network teases apart the relevant meaning and collapses that ambiguity into a coherent output. So cat, for example, might store feline pet animal,right, across a few different neurons. And those neurons could also store the essence of a dog, canine pet animal, and so forth.

Like, an idea isn't distributed just across a set of neurons. A set of neurons can hold multiple ideas as well. So meaning isn't retrieved. It's constructed on demand from distributed sparks of possibility. Okay. Think of it this way.

Your prompt is a force vector. Prompts act with a directional force in the high-dimensional latent space that an AI model operates in. Each prompt sets a specific direction, which the AI then aligns patterns to. So the concept of a cat or a dog kind of wears grooves in that latent space, and it'll generally follow that space.

Anthropic's tracing thoughts. Ethan Mollick's got some thoughts here. Stanford CRFM also has some thoughts here. There's a lot of research to kind of, I think, justify this perspective that, you know, prompts are force vectors navigating latent space. Okay.

So if they're force vectors navigating latent space, how does that actually happen? Well, when you prompt a model, you're going to give it some external context. You're going to give it the prompt, the question, the thing you're trying to figure out.

And it's going to go into the latent space. So let's say we got storytelling and freedom here. You know, these are conceptual clouds. They're going to light up around that idea at execution, at inference time. So, you know, some parts of the model are better at these things.

Storytelling is obviously encoded in one region, freedom in another. You know, maybe everything it knows about household pets in a third. And they only light up when they're needed. So they're not pre-wired. They emerge during training, but these specialized patterns activate when the context calls for them.

So let's say storytelling and pets. Now, it's not going to always cross over on the same, like, set of concepts,right? Like, maybe I'm a little more detailed about the type of pet. Maybe it's a bird, and I'm telling a lot of detail about the bird, but I still want it to tell me a short story.

So it's going to merge those two concepts into a new idea. That force vector is going to cause something new to emerge based on the subnetworks that light up.

Not just retained compressed information. So what does this lead to? Well, this leads to a new kind of utility. Most theories focus on compression, how efficient models store and retrieve knowledge. But I think the real value of LLMs is in their ability to recreate the essence of an idea, to take multiple essences and to create something new.

So this is why when they hallucinate, it still feels correct. They're creating a compelling pattern. They're not fact-checking. They're not being intelligent. But that essence reconstruction, there's a lot of utility in there. So what does this mean from an engineering standpoint?

Well, we got to frame systems as coherent, not intelligent. And when we do that, we start to see things slightly differently. The first thing that we're going to see differently here is that hallucinations are an indicator of coherence.

They will fill in the gaps if you don't give it enough information not to. You know, models hallucinate, generally hallucinate on the same type of prompt in predictable ways, following an internal logic and coherency vectors. I see it as a system feature.

It's kind of an emergent behavior of this emergent behavior, less a bug of the system. It's trying to complete the pattern. How do we fix that? Well, we all know RAG,right? Retrieval Augmented Generation. So we think about those RAG fragments as factual anchors,right?

And if they get large enough, there's enough contextual gravity there, it's going to really pull that concept in theright direction. Think of it as the structural scaffolding that shapes how coherence unfolds. It's the infrastructure for thought. I take my thought, my intent, and I give it to the machine to navigate that latent space with.

Another way to think about this is my three-layer model. We've got layer one, the latent space. That's the internal model structure, concepts, weights, activations. It's just sitting there. Layer two is the execution layer, the tools, the APIs, and the retrieval mechanisms.

Its goal is to bring extra context for layer one. And layer three is the conversational interface. That's where the intent passes from me. The thought passes from me to the machine. There's grounding and human intent that need to come through for layers one and two to be of value, to be actionable.

So how do we actually build for coherence, not intelligence? Think of prompting as interfaces. They're not once-off. They're a component in the system. You know, RAG to ground as coherency anchors to steer the generation. Dense relevant context works like gravity pulling output towards a reality.

Design for emergence, not control. It's not deterministic. You know, build around that frame, generate, judge, iterate loop. Avoid fragile chains. Long reasoning chains often break coherency. Keep structured, keep your chains modular and reinforce the context at each point.

And watch for breakdowns in tone, structure, or flow. They're early signs that the model's losing its grip on context. It needs to be brought back. Use that as a jumping-in point to kind of debug that part of the system.

Maybe it means adjusting your, you know, chunk size in your vector DB. Maybe it means bringing in other tools outside of just a vector database to help that problem. So in the end, I like to think of LLMs as a high-dimensional mirror.

We're not talking to a thinker. We're standing in front of a mirror with 8,001 dimensions. The machine doesn't understand you. It resonates with you through structure, not thought. Sometimes what it reflects back is sharper than what you gave it.

LLMs are not intelligent. They're coherent. And that's their superpower. The magic happens in the collaborative dance. So stop chasing intelligence and start designing for structured resonance.