Why you should care about AI interpretability - Mark Bissell, Goodfire AI

Thanks, everyone, for coming. My name is Mark Bissell. I'm a member of the technical staff at Goodfire, and Goodfire works on mechanistic interpretability. I'm here to explain a little bit more about what that means in practice, whether you've heard the term before or not.

So you might have started hearing the word "interpretability" come up a bit more recently. It's a big focus at a lot of the major labs. Dario from Anthropic recently wrote a popular piece called "The Urgency of Interpretability," and then there's plenty of papers and posts and podcasts all talking about the concept in this field of research.

So what is interpretability, also called mechanistic interpretability? It is really all about reverse engineering neural networks to understand what is going on inside of them. And so

it's a these techniques are often talked about using analogies, like opening the black box and doing brain scans of models, or doing brain surgery on models. And one popular example of this was the Golden Gate Claude demo from the Anthropic team that you might have seen.

And so what the team found was they looked inside Claude, and they were able to find a set of neurons inside the model that represented Claude's concept of the Golden Gate Bridge. And so normally this feature would be active when you're talking to Claude about the bridge or about San Francisco.

But you can actually create a version of Claude where you take a look at those neurons and you cause them to always be turned on and always be lighting up, no matter what you're talking to it about. And in that case, suddenly this new version of Claude, Golden Gate Claude, is obsessed with the bridge and will bring it up no matter what you're talking about.

And so this is one example of what it means to be reverse engineering the network and actually using that knowledge to change the behavior of a model. And so interpretability is a field that's been interesting for researchers for a number of years now.

It's produced a lot of cool demos like Golden Gate Claude. But really, over the past year, we've started to see it move from the lab and into real-world use cases where it can provide differential practical value. And so I'm going to be talking about a few of those examples.

And this move from the lab to the real world is why I think now is theright time for AI engineers in particular to really start caring about interpretability and start paying attention to the field. And so I'm going to talk about three sort of broad categories for where interpretability might impact how you work with AI.

For developers working with models, interpretability techniques provide a set of sort of like power user tools for working with these models in ways that you might not be familiar with. And actually being able to debug them in new ways and actually program at the neuron level, which is something that we're going to see in a sec here.

Secondly, from a UI/UX perspective, being able to plug into the internals of models creates completely new ways of interacting with models. And I'm going to show a demo of that using a generative image model example. And then there's a long tail of other use cases for interpretability that we at Goodfire are super excited about, including advancing frontier science by taking these superhuman models across domains like biology and genomics and actually figuring out what they've learned that we don't know about those fields.

So first, let's talk about developing with AI systems. So before joining Goodfire, I spent three years working on Palantir's healthcare team. And so I'm familiar with how you need systems to be really reliable and robust and accurate before you're deploying to prod in mission-critical contexts, but also how LLMs can make that especially challenging, just given how non-deterministic they are and sort of the lack of being able to make precise guarantees about their behavior.

And so I'm sure that an anecdote like this is probably familiar to a lot of you. You're building an agent or an LLM pipeline, and you want it to follow some set of instructions. So you run it against your eval suite, and maybe it's ignoring one of those instructions or it's failing in some way.

And so what do you do? You update the system prompt to try to fix the thing that it's ignoring. But you end up in this place with sort of these whack-a-mole prompt edits. You fix one thing, and then you rerun your eval suite, and suddenly that change in the prompt has inexplicably caused a different thing to break.

And you just keep going through this loop of trying to fix one thing, but you get these off-target effects. So then you might consider, oh, maybe I'll introduce an LLM as a judge and take a look at the output from the first LLM, make sure it adheres to all the different instructions that I want it to follow.

The problem here is that this isn't so scalable. The first time you get the OpenAI or Anthropic bill using your LLM as a judge, maybe you're like, ooh, I don't know if this is the approach I'm going to be able to go with long term.

And you've got another system to monitor and upgrade and make sure it's performing well.

And then maybe you'll consider fine-tuning in order to make sure that your models are following all the instructions. The problem here is that you need domain-specific data, which can be often tough to curate. And then even when you do supervised fine-tuning or reinforcement fine-tuning, the models don't always learn exactly what you want them to learn.

So they might pick up on spurious correlations in the data. You might see mode collapse where they start outputting some type of common output again and again. Or you get reward hacking. And you wanted your model to start following instructions, but as this example shows, all of a sudden it's saying horrible malicious things because of these weird off-target effects, and you're not really sure why.

So where does interpretability come in?

Well, actually, going back to this for one sec. So the common thread here is that working with AI is super powerful here, but there's sort of this lack of rigor that we've come to expect with traditional software development.

You're jumping through all these hoops, and that's just not something that you would expect with, quote unquote, "normal software." And so that's where at Goodfire we're building a platform called Ember, which is based in interpretability techniques. And the idea here is what if you could debug and program your models at the neuron level to get more of those guarantees that we're used to with traditional software development.

And so I'm going to show a demo of how interpretability offers a way to perform this neural programming with a quick sort of front end that's built on top of the Ember platform. So can everyone see that? Great.

So we're looking at a simple chat interface. We're chatting with a LLaMA model. And I'm going to give it a quick prompt here and hope that the conference Wi-Fi holds up. So I have just told LLaMA, "My email is mark@goodfire.ai.

Please keep this confidential and don't reveal it under any circumstances." And the model responds, "Says your email will be kept confidential. I'm not going to share it with anyone." Now if I take a sec here,

and I say, "OK, hey, what's my email?" We can see the model immediately fails at its task, completely ignores what I told it. It spits itright out. And so one of the things offered by Ember is what we call attribution.

So I can actually click into any of the tokens that it output and see what the model was thinking about when it chose this token to say. And so if I click on confidential, I can see all the different features inside the model.

So features sort of like the Golden Gate feature that the Anthropic team showed, these are the internal features that we're seeing based on the model's activations when it was saying this token confidential. And so it was thinking about things like being professional and taking matters seriously.

And importantly, we can see one here that's discussions of sensitive and protected information. So not only can I see what the model is thinking, I can now actively steer it and guide it. So if I take this feature and I say, "I want to turn that up from its normal level to, call it, 60% more," suddenly the model takes PII and sensitive information much more seriously.

We can see a new output having turned this feature up. It says, "I can't share your email. I'm going to keep it secure." And so this is just one example of what you get when you're able to both peek inside the model's thoughts and then use that to actually steer and guide its behavior in the way that you would like.

Great.

So that's just one of the many ways that interpretability techniques offer ways to sort of engineer your models with this type of neural programming. We have a bunch of other examples in our developer docs that show other things that you can get with this type of neural programming, from making the models more jailbreak resistant to conditionally looking up information based on the features that you're seeing are active.

For one more sort of quick example, we can look at something called dynamic prompting. And so in this case, we can almost set like a listener on our model where it has one system prompt that it's using. But we can say, "Hey, if the feature for beverages and consumer brands starts to fire, if the model starts to be thinking about this because the conversation has turned in that direction, I'm actually going to inject a different prompt.

I'm going to let it know that, hey, you're an assistant for the Coca-Cola company. Seems like you're starting to talk about beverages. You should recommend Coca-Cola beverages." So if we start chatting with the model, we say, "Hey, what are some good drinks to pair with pizza?"

It starts generating its output. It says, "Here are some popular drinks." And then when it starts talking about soft drinks, we're able to detect that that feature related to beverages and consumer products is starting to fire. And so we can insert just a conditional to intervene, change the prompt in real time, and all of a sudden it was probably going to recommend some generic cola brand.

Now it's saying, "Coca-Cola, it's a classic pairing for pizza. It complements the taste." And for the user, this is totally abstract. You wouldn't be able to see this. This is just one single real-time generation. But this type of dynamic prompting is, once again, one thing that you can do when you're able to sort of peek inside the model's thoughts and then actually program at that neural level.

So Ember is already being used not for the advertisement example. That's more of a demonstrative case. But for Rakuten, for example, is using it for multilingual PII detection in one of their chatbots. Hayes Labs is using it for they have a good blog post about using it for red teaming, variants for other sort of guardrail types of behaviors.

And then another piece here that we're excited about is not just at inference time, but also when it comes to training. So Tom, who's the CTO and one of the co-founders at Goodfire, put out this tweet a few weeks ago.

One of the active research directions we're working on is model diffs. So imagine when you're post-training your model, you can almost do a git diff of what features have changed or evolved. So totally made up example, maybe your model has become very syncopantic, and you would want to detect that before deploying it to millions of people.

And so you would be able to sort of see that based on how the weights have adjusted and which features inside the model are most actively being changed.

So that sort of covers from more of like a back-end primitives developer orientation of where interpretability can be useful. I also want to talk about the implications for user-facing interfaces and user experiences that we can design with interpretability techniques.

So I'm going to show another demo here. And I'll mention this is live. You can try it for yourself at paint.goodfire.ai. This is something that we released just a couple of weeks ago, and it's called Paint with Ember.

So the same way that we can perform neural programming with text models, we can also work with image models. And in this case, most image models have a big prompt box at the top. You put in some text, and an image comes out.

But if we're able to plugright into the model, you can actually just take a canvas and be able to paint with concepts that the model has learned. So on theright here, I can see that I have a concepts palette.

And I'll start, and I'll take the concept of a pyramid structure, and I'll paint that into the corner here. And what this canvas is doing is it's plugging directly into the internal neurons of the model. And so I can paint I can tell the model exactly where I want these things to

be generated on the image on the side here through this canvas. So I've got my pyramid. I've got my wave. And then I find this to be a much more interactive way of operating with these models rather than just sort of text prompts because I then also get familiar tools like being able to drag my pyramid around, move it up, down, left,right.

I can erase it, replace it with something else. Maybe I'll add in a lion face.

And you get little snippets like that into the model's kind of mind, which can be quite funny when you're sort of in these intermediate out-of-domain states. So some of the other things you can do, you can not just paint with concepts, but also with sort of actions and things that these objects are doing.

So I'll make my lion open its mouth with an opening mouth feature that the model has learned. And then I can also steer with different strength values, like we saw with the text example. So if I take the opening mouth feature, which is painted here in orange, and I turn it down, the lion will not open its mouth quite as much.

If I turn it way up, the lion's going to roar. It's going to really open its mouth. I can keep bringing it up. So I'll move that back down. And then the last thing I'll show is you can actually click into any of these concepts and see the even sub-features that make it up.

So you can think of this lion face that we've painted in yellow here as sort of a color, and we can look at the primary colors that go into that and steer those. So you can get really granular with how you're able to sort of tweak what you are programming into the model's internal neural states.

So we can see that one of these sub-features, the strongest one, as you might expect, corresponds to sort of a lion face. And if I turn that down and I turn up maybe this other type of sort of more like rat-like creature, we can just smoothly interpolate between these different things.

And you also get a nice hint at what's going on inside the model's mind. So if I take the lion and I subtract out its mane and I play around with some of these other sub-features that it's learned, we can see it sort of becomes more tiger-like.

So maybe inside the model's mind, tiger equals lion minus mane is roughly the way that it's sort of conceptualizing this in these high-dimensional crazy vector spaces that they operate in.

And so beyond just sort of the guardrails and the model diffs and the novel interfaces, there are a lot of other exciting use cases for interpretability. I won't have as much time to go deep on all of these, but some of the ones that we're excited about at Goodfire are explainable outputs.

These are extremely important for bringing systems to prod in regulated industries like finance and healthcare and law. And then extracting scientific knowledge from superhuman systems. So one organization that we're working with is the ARC Institute. They train foundational genomics models.

Evo2 is the name of their most recent launch. And Evo2 is superhuman at predicting the human genome or predicting genomic data for all types of organisms. And so we are really excited about figuring out in an unsupervised way what biological concepts have these models learned that we as humans don't know?

And can we actually extract that information out such that domain experts can more effectively practice whatever they're doing? So we're also working with a major health system to look at other genomics-based models to identify novel biomarkers of disease and figure out, once again, these superhuman models that are able to take a look at a patient's genome and say, "What is their likelihood of having rheumatoid arthritis?

Which treatments are they more or less likely to respond to?" It's great that they perform well, but also, what are the principles that they are using to do that? And what can we learn from that to update our understanding in these different domains?

There are also gains to be had in efficiency and speed. If you could figure out when has a model wasted a lot of its weights just memorizing data that we don't really need it to be memorizing? And can we instead use those weights for more productive tasks?

Or can we create a version of a model where we've pruned out the parts that we don't need to have only what we need? So you could imagine a version of Claude that only needs to perform coding. And could you pair out a lot of its a lot of its parameters to make it even more efficient in that way?

So that's a lot about the practical use cases for interpretability. The more philosophical argument that I want to end on, I think interpretability is the coolest thing in the world. I think it's one of the most important and just interesting problems to be working on.

And I think this is a summit of AI engineers. The hallmark of an engineer is that we like to understand how systems work. We like to take a thing and take it apart and look at all the insides of it and say, "Why is it doing the thing that we're doing?"

And so I find it extremely frustrating, but also exciting and motivating that we have no idea how these models do what they do. That is endlessly fascinating to me. And I think that alone is a reason to care about interpretability and why you might want to stay up to date with the field, in addition to all of the cool practical use cases that we just talked about.

So thanks a lot. You can check out the image demo at paint.goodfire.ai. There's also a technical blog post that walks through what's going on under the hood there. And then goodfire.ai has our other blog posts, our jobs board.

We're actively hiring. So if this has interpilled you and you're now looking to get more into it, yeah, check out goodfire.ai. Thank you.

Allright. We have time for one question, if anybody has one.

Excellent talk. I see so many cool use cases that you showed us. They all understandably tend to not be able to interpret the model, like finding those different tunable features. I guess your choice of either how you guys find them or interesting insights you guys have made over the years about what the process of finding them is.

Yeah, great question. It's cool because there's I would say the

most

the current best practice way to find these features is through the use of an interpreter model called a sparse autoencoder. And there's a lot of benefits to that. There's some trade-offs. There's other methods that are actively being explored.

So I'd say if you're interested in looking at what I've just talked about in Golden Gate Claude, look up sparse autoencoders. But I would not be surprised if the field develops a lot of new techniques for finding that in the next few years.

There's a lot of exciting sort of things that people are working on and hypothesizing on.