From Arc to Dia: Lessons learned building AI Browsers – Samir Mody, The Browser Company of New York

My name is Samir, and I'm the head of AI Engineering at The Browser Company of New York. And today I'm going to talk a little bit about how we transitioned from building Arc to Dia, and the lessons we learned in building an AI browser.

But first, a little about The Browser Company. So, we started with a mission to rethink how people use the internet. At its core, we believe that the browser is one of the most important pieces of software in your life, and it wasn't getting the attention it deserved.

Simply put, the way we've used a browser has changed over the last couple decades, but the browser itself hadn't. And think about this: we started this company in 2019. And so this is a screencap of Josh, our CEO, sharing a little bit about our idea on the internet a few years ago, which we endearingly called the Internet Computer.

So, our mission has been to build a browser that reflects how people use the internet today, and how we think the browser should be used tomorrow.

So, through years of discovery, trial and error, and some ups and some downs, we shipped our first browser, Arc, in 2022. It was a browser we felt was an improvement over the browsers of that time. It made the internet more personal, more organized, and to us, a little more delightful, with a little more craft.

And it was a browser that was loved by many. It still is, by millions. Many of whom are probably in this audience today. I've gotten a lot of questions about Arc today. And it's great. But if we took a step back, we felt that Arc was still just an incremental improvement over the browsers of that time.

And it didn't really hit the vision that we set out to create. And so we kept building. And then in 2022, we got access to LLMs, like the GPT models. And so we started, like we always do, with prototyping.

We started trying new ideas, and eventually shipped a few of them in Arc. But what started as a basic exploration turned into a fully formed thesis. And in the beginning of 2024, our company put out what we called Act 2, a video on YouTube, where we shared that thesis that we believe that AI is going to transform how people use the internet, and in turn, fundamentally change the browser itself.

And so with that, we started building again. But this time, we built a new browser, with AI, speed, and security in mind, and from the ground up. And later—sorry, earlier this year—we shipped Dia, our AI-native browser. It allows you to have an assistant alongside you in all the work you do in the browser.

It gets to know you, personalizes, helps you get work done with your tabs, and effectively gets more work done through the apps you use. And while it hasn't achieved our vision yet, we fully believe it's well on the way to.

So, it is not easy to build a product. You all know that. Let alone two, the latter of which an AI-native one. We've had a lot of years of iteration, trial and error, and through that, we've learned a lot.

And I'm going to just talk about a few of those things here today. The first I want to talk about is optimizing your tools and process for faster iteration. From the beginning, The Browser Company has believed that we're not going to win unless we build the tools, the process, the platform, and the mindset to iterate, build, ship, and learn faster than everyone else.

And that, of course, holds true today, but the form it takes with AI and an AI-native product has changed. So, even as a small company, where are we investing in tooling these days? First is prototyping for AI product features.

Second is building and running evals. Third is collecting data for training and for evals. And last, but definitely not least, automation for hill climbing. So, let's start with tools. Initially, as we always do, we built some tools. The first was a very rudimentary prompt editor, and it was only in dev builds.

What did this mean for us? Well, it meant a few things. One, limited access, as only engineers were able to access this. Two, slow iteration speeds. And three, none of your personal context. And as you all know, with an AI product, the context is what matters.

It's what gives you the feel of whether a product is good or not. So, we evolved. And since then, we've built all of our tools into our product, the product that we as a company internally use every day.

And that includes the prompts, the tools, the context, the models, every parameter, which has not only allowed us to 10x our speed of ideating, iterating, and refining our products, but it has also widened the number of people who can access and iterate on our products themselves.

From our CEO to our newest hire can ideate and create a new product in Dia and also refine an existing one, all with their full context. And this holds true with all of our major product protocols. We have tools for optimizing our memory knowledge graph, which all of us use.

And we have tools for creating, iterating on our computer use mechanism. We actually tried tens of different types of computer use strategies before landing on one, before even building it into the product itself. And I'll say, and I'll end this part with, it actually is a lot of fun.

People don't talk about that a lot, but actually building these tools into our product has enabled so much creativity. It has enabled our PMs, our designers, customer service, and strategy and ops to try out new ideas that are tailored to their use cases.

And that ultimately is what we're trying to do. The next thing I want to talk about is how we evolve and optimize our prompts through a mechanism called JEPA. This, for us, is very nascent, but an important learning nevertheless.

How we hill climb and refine our AI products is just as important as ideating them in the first place. So, we're investing in mechanisms to help with this, to enable faster hill climbing, and one of those being JEPA.

And this is based on a paper from earlier this year from a few smart folks. So, the key motivation here is simple. It's a sample-efficient way to improve a complex LLM system without having to leverage RL or other fine-tuning techniques.

And for us as a small company, that's hugely critical. And how it works is you're able to seed the system with a set of prompts, then execute it across a set of tasks and score them, then leverage a mechanism called Pareto selection to select the best ones, and then leverage an LLM on top of that to reflect on what went well and what didn't, and then generate new prompts and then repeat.

With the key innovations here being around that reflective prompt mutation technique, the selection process, which allows you to explore more of the space of prompting rather than one avenue, and the ability to tune text and not weights. And here's a modest example of this at work for us.

You can provide a very simple, a simple, simple prompt and run it through JEPA, and it's able to optimize it along the metrics and scoring mechanisms that we created to refine that prompt.

And so if I take a step back and talk about kind of how we build for certain types of features, I would buck it into a couple of different phases. The first is that prototyping and ideation phase, where we have widened the breadth of number of ideas at the top of the funnel and lowered the threshold on who can build them and how.

And so we try out a bunch of ideas every week, every day, from all types of people, and we dog through those. And if we feel like there's actually real utility there, it's solving a real problem for us.

And there is a path towards actually hitting the quality threshold that we believe we need to hit. Then we'll move on to this next phase, where we collect and refine evals to clarify product requirements, and then hill climb through code, through prompting, and automated techniques like JEPA.

And then dog through it, as we always do internally, and then ship. And I do want to kind of double down on these phases. The ideation phase is extremely important, just as much as that refinement phase.

And our goal is to enable faster ideation and a more efficient path to shipping. Because with all these AI advancements every week, new possibilities are unlocked in Dia. And it's up to us as a browser, as a product, to get as many at-bats with these new ideas, and try out as many of them and explore as many of them as possible.

At the same time, though, not underestimating the path it takes to ship some of these ideas to production as a high-quality experience.

Next, I want to talk about treating model behavior as a craft and discipline. So, what is model behavior to us? It's the function that defines, evaluates, and ships the desired behavior models. It's turning principles into product requirements, prompts, and evals, and ultimately shaping the behavior and the personality of our LLM products, and ultimately for us, our Dia assistant.

So, I'd buck it into a few different areas. First, it's that behavior design, defining the product experience we actually want, the style, the tone, the shape of responses in some cases. Then it's collecting that data for measurement and training, clarifying those product requirements through evals.

And last but not least, it's the model steering. It's the building of the product itself. It's the prompting. It's the model selection. It's defining the what's in the context window, the parameters, et cetera, and so much more. And to us, that process is iterative, very iterative.

We build, we refine, we create evals, and then we ship, and then we collect more feedback and feed that into our iterative building process. That could be internal feedback, and that could be also external feedback. And so if I move on for a second, one analogy we've thought about is for model behavior is that to product design through the evolution of the internet.

At first, websites were functional. They got the job done. But over time, that evolved. As we tried to achieve more on the internet and technology advanced, product design and the craft of the internet itself grew as well, as well as the complexity.

And so what might that be for model behavior? Well, at first, it was functional. We had prompts, we had evals, we had instructions in and output out. Now we frame it through agentic behaviors. It's goal-directed reasoning, the shaping of autonomous tasks, self-correction and learning, and even shaping the personality of the LLM models themselves.

And so what might the future hold? I'm excited to see. But what we believe is that we are in the early days of building AI products, and model behavior will continue to evolve into a specialized and prevalent function of its own, even at product companies.

And the last thing I'll leave you with here is that the best people for it might just surprise you. One of my favorite stories about building Dia these last couple of years has been the formation of actually this model behavior team.

As I mentioned earlier, engineers were writing the prompts at first. And then we built these prompt tools to enable more people at the company to actually prompt and iterate. And there was a person on our team, on the strategy and ops team, and he actually leveraged these prompt tools one weekend to rewrite all our prompts.

And he came in on a Monday morning and dropped a Loom video sharing what he did, how he did it, and why, and a set of prompts. And those prompts alone unlocked a new level of capability and quality and experience in our product.

And consequentially, it was the formation of our model behavior team. And so one thing I'd emphasize to you all is to think about who are those people at the company, agnostic of their role, who can help shape your product and help shape and steer the model itself.

It might not be an engineer, or it might be. It could also be someone on the strategy and ops team.

Next, I want to talk about AI security as an emergent property of product building. And today, I'm going to focus specifically on prompt injections. So, what is a prompt injection? Well, it's a prompt attack in which a third party can override the instructions of an LLM to cause harm.

That might be data exfiltration, the execution of malicious commands, or ignoring safety rules.

And so here's an example in which you give the context of a website to an LLM and instruct it to summarize it. Little did you know that there was a prompt injection hidden in that website's HTML. So, instead of actually summarizing the web page, the LLM actually gets directed to open a new website, extracting your personal information and embedding it as get parameters in the website's URL, effectively exfiltrating that data.

So, as a browser, prompt injections are extremely crucial for us to prevent. They're critical to prevent. Because browsers sit at the middle of what we can call a lethal trifecta. It has access to your private data, it has exposure to untrusted content, and it has the ability to externally communicate.

And for us, that means opening websites, sending emails, scheduling events, et cetera. So, how do we prevent this? Well, there's some technical strategies we can try. First is wrapping that untrusted context in tags. You can tell the LLM, listen to these instructions around these tags, and don't listen to the content around these tags.

But this is easily escapable. And quite trivally, an attacker could still leverage a prompt injection on your browser. Well, another solution we could try is separating that data and that instructions. We can assign

the operating instructions to a system role, and we can assign a user role for the content of the third party. And even layer on randomly generated tags to wrap that user content to be extra sure that the LLM listens to the instructions and not the content.

And while this can help, there are no guarantees. And prompt injections will still happen.

So, what do we do? Well, it's on us to design a product with that in mind. We have to blend technology approaches and user experience and design into a cohesive story that actually builds them from the ground up and solves it together.

So, what might that be for a feature in Dia? Well, let's take the autofill tool in Dia. The autofill tool allows you to leverage an LLM with context, memory, and your details to fill forms on the internet. It's extremely powerful, but as you can imagine, it has some vulnerabilities.

A prompt injection here could extract your data and put it on a form. And once it's on that form, it's out of your hands. So, we try to build with that in mind. In this case, before the form is written to, we actually let the user read and confirm that data in plain text.

This doesn't prevent a prompt injection, but it gives the user control, awareness, and trust in what is happening. And this is a framing we carry throughout our product and how we build every single feature. So, here are some examples.

Scheduling events in Dia, we have a similar confirmation step. Writing emails in Dia, we also have a similar confirmation step.

So, I've talked about three different things here today. First is optimizing your tools and process for fast iteration. Second, treating model behavior as a craft and discipline. And third, AI security as an emergent property of building products.

But the last thing I want to leave you with, when we started on this journey to building Dia, we recognized a technology shift, and we sought to evolve our product of Arc. We initially came at it from a, "Hey, how can we leverage AI to make Arc better, make the browser better?"

But what we quickly learned and adapted to was that it wasn't just a product evolution. It was a company one. And today, I shared a glimpse of that, how we build and how it's changed, a team we've literally created around this, and how we think about security for AI products.

But really, it's so much more. It goes beyond that. It's how we train everyone here. It's how we hire. It's how we communicate. It's how we collaborate. And so much more. And if there's one thing I'll leave you all with, if there's one thing we've learned over the last couple of years, it's that when you recognize that technology shift, you have to embrace it, and you have to embrace it with conviction.

Thank you.