Real-time Experiments with an AI Co-Scientist - Stefania Druga, fmr. Google Deepmind

My name is Stefania. I'm so glad you made it until the, yeah, last day of the conference and came to the robotics track. So we're going to start with a live demo, and then we'll switch to the presentation, just like, to kind of, like, swap things around.

So I'm going to try to connect a microscope over here, and let's see, the other camera. And some sensors. So my talk is about real-time science ex- co- co-scientists. So think about pair programmers. How many of you use any form of co-pilot for coding?

Right, so it's just like that, but for doing things in the real world, like science experiments. So you can see here, now I have this board, which is a micro:bit board with Jack DAC connected, and it's measuring the temperature.

And I actually have a heat pad, so I can increase the temperature and make it very, very hot, and hopefully it's not going to melt the board. And then I can send that to my science assistant. It's going to analyze the data and kind of give me an answer in real time.

So it's telling me, like, okay, the ambient conditions indicate, like, stable, dark, quiet environments, because it has, like, it knows it can measure all these different sensors, no buttons, but it sees that the temperature is 26 degrees. And then I can actually give it more context.

I can go and create a protocol and say, like, this is the type of experiment that I want to do. So whenever you give me feedback about the data or the images that I'm sending from the microscope, which I can also do, it's going to be in the context of that experiment, with those conditions and those constraints.

And of course, like, if you are actually doing that experiment several times, you can go ahead and create a custom page for your experiment, and it will monitor the data from the, we're going to connect again the Jack DAC.

It's going to monitor and plot the data in real time. And the cool thing is also that if you want to leave your experiment running and go away,right, like, you could actually have cameras that are autonomous. They go back to the previous talk, they're not fully autonomous, but they can track objects.

So this camera is called ReCamera. Also, by the way, all the hardware I'm showing you and all the components are open source. Soright now, I'm going to set it to track me as a person

and start the tracking.

And it's running a model on the camera itself, so it's going to start, like, moving around to see me and see where I'm going. But you could train a custom model, like, to track, like, crystal growth or specific objects that you want to monitor, and in real time.

And this runs on Wi-Fi, so it can, you can place it anywhere and then control, like, the conditions in the experiments, like increase the temperature or decrease the temperature. So, oh, it sees all of you. That's awesome.

So I'm going to stop this for now. So that's, like, a short, and here we're seeing, like, our temperature kind of increasing. I think my heat pad stopped. But that's just, like, a simple, like, demo of, like, without having a specific experiment in mind.

And now I'm going to switch to, oops, my slides, and actually show you what happens when you record much longer experiments and what it can do. Why do we care about co-scientists? We're not talking yet about real-time co-scientists, just co-scientists in general.

Like, why should you care about it? We know that there's a data overload in science, and there's a lot of complexity, a lot of things to parse through and analyze. And AI can really help with that. We also know that beyond the analysis of data fast at scale, they can help with generating new hypotheses and maybe identifying blind spots in prior work and in your own thinking.

And we also know that it can, they can speed things up,right? So instead of testing one hypothesis at a time, you could test 100 hypotheses. So I got inspired to build this demo. And by the way, the cost of all the parts is under $300, and it took me two weeks to build it, and it's open source, so you can play with it and hack it.

So I got inspired to do this because of this demo from DeepMind. So this paper got published two months ago on AI Co-scientist, and it was actually showing, not in the real life, but it was actually showing what happens when you're analyzing papers and data.

If you have a multitude of agents that can perform the different roles that we do as scientists, what are the results? So what are the roles that we do as scientists? Like, we analyze papers, we summarize them, we look at data, we rank the options, we rank the different hypotheses.

So they actually created an orchestration of different agents. Each of them was actually working on Gemini 2.0. And then with this orchestration of, like, ranking the different results, the different hypotheses, doing search online, doing search on papers, come up with a plan for the researcher to use.

And not only that they did that, but they actually tested it against prior discoveries. And this was super interesting. So they tested it against prior discoveries on gene transfer mechanisms. So a discovery that took scientists 12 years, it took the AI scientist two days to come up with, without having seen the data.

So it was not aware of how the gene transfer works. This was not part of the training. So, and that was one verification by trying to replicate past results. But another one was, like, to come up with completely new hypotheses.

So they used it for liver fibrosis target treatment discovery, and the AI co-scientists came up with target drugs that were actually efficient in the lab, in wet lab. They were tested by experts. So it's not a science fiction, it's not, like, something in the future.

It's actually happening now. And for me, what was inspiring about this is, like, when we can make real discoveries in drug discovery, like healthcare treatment, bacterial resistance, new type of materials, I think that can be accelerated doing it in real time with a scientist.

This clicker doesn't work very well. So the vision was, instead of just doing it, like, on the data, like, not sync, like, async, and giving the researchers a plan based on the data that exists, what if we do it in real time,right?

And we formulate this hypothesis based on the empirical data we're observing in the lab in real time. Or it could be, like, when you're observing the robot, like, breaking down in real time,right? So what motivated me was the results from the AI co-scientist, but also this vision from Silver & Sutton, which is, they just published this paper, Welcome to the Era of Experience, which is fantastic.

It's very short. If you haven't read it, I highly recommend it. And they really talk about how we're going past the era of human data, where we're only indexing and making predictions based on the data sets that we created, and going to an era where the AI learns from the continuous environment in which we operate,right?

And especially with multimodal, when we have real-time data from images, from sensors, from audio streams, like, that is possible. So I hope I convinced you by now that real time matters. And I wanted to show you, like, how I use my system with longer experiments.

So you already saw, like, the chat image, and I had to find experiments that I could actually do at home. So this is the overview of the system. So it's a very simple React app, and it has, like, all the input sources that you've seen live in action, like the Jack DAC sensors via USB, different webcams, you can add as many webcams as you want, text input.

It actually works with voice. I forgot to show that. You can talk to it, and it talks back. And then all of these, like, inputs become webhooks, and I can, I'm going to go into more detail how I optimize each and one of them.

And that gets sent to a backend, which communicates with a real-time AI API. I use the Gemini in this case. And I had to, this is the information flow. So you have the physical sensors, and for the Jack DAC, I'm actually using the web USB API.

That goes to the frontend hooks, which gets pulled by the context assembly. And the context assembly actually is always checking what sort of modalities are present. Do we have text? Do we have voice? Do we have image? Do we have a chat history?

And depending on what modalities are present, every single time I'm sending a message, it builds this context that gets sent to the API that gives me back a response. And this is how the unified context assembly works. So I guess, like, the very important piece here is how you're doing this context injection, depending on what sensors you have connected and the type of experiment you're defining, and making this dynamic on the fly, like, when you create the protocols.

And

that was the, those were the ingredients for the code. Now let's talk about the ingredients for the hardware. So when I start doing the experiments, I had to list and put out all the parts that I had at my disposal, kind of like cooking,right?

So it's like, what can I make with all these sensors? Like, these are my inputs, these are my outputs, these are my, all my cameras and all my cables, like, and how many boards I have. What sort of experiments could I create with that?

And I also had the constraint of doing experiments that I can measure in real time for this talk, and also that are safe for me to do at home, and I can travel with,right? So that really narrowed down the list of things that I could play with.

So one thing that I did was exploring crystallization. So you can see there, like, the crystals that I was growing, and that took longer, and I recorded the whole thing, and fermentation. And let's dive into each and one of those.

So this was my lab in a box in our apartment in Tokyo. The people coming to the house were very confused every day.

There was something, no explosions though, so that was good. So for crystal growth, how many of you remember this from chemistry? Okay, okay, that's awesome. So I don't need to go into great detail, but basically the principle is quite simple.

Like, you oversaturate the solution by adding salt in hot water, and then you're, like, slowly cooling it off, and that creates this process, this nucleation and the growth of crystals. Now, the trick in having beautiful crystals is how you do the cooling off,right?

It needs to be gradual. You need to not move, like, the liquid or the object. So the gradual cooling and the level of humidity is what gives you beautiful crystal formation. So the main things we want to measure in this example is what are our curves for how fast the salt dissolves.

Like, where are the places where the crystals are being formed? Those are called nucleation sites. What's the growth rate of the crystals? And that is affected by temperature concentration. And, oh yeah, I forgot to play this. This was, like, a sped up.

So this was a recording from the microscope. It's moving because it actually has a fan

blowing cool air fromice. So that's why, like, the camera is moving. That was how I was, like, cooling it off. But you can see the data in the sensors changing on the side. And this is, like, being recorded for an extended period of time.

And then I get a CSV from the sensor values, and I can go and analyze that and kind of get a sense for, like, what happened. Was I cooling it off too fast or too low, too slow? Were my crystals, like, growing or not?

And then I also played with different samples. So I had samples that I would put in the fridge and in the room and, like, at different temperatures, so I could also measure and have, like, control groups for that.

And then once I have that CSV data, I can plot it,right? So I can actually see what was my crystal growth rate and if my temperature was, like, the best. And I can plot that for the different samples.

Andright now I don't have that integrated live on the platform, but that's coming. So I had to write a separate script to get the CSV data and create these visualizations. And the insight I got from this is that the crystal formation is actually not gradual, that it happens in bursts.

So that was something that surprised me when I looked at the data. So once the critical saturation is reached, there's, like, sudden crystal growth formation. So it's not, like, something that it's kind of, like, slowly growing, but it grows in bursts.

And here's, like, 40 minutes recording of the crystal growth, like, super sped up. So you can kind of see how the humidity and the temperature was changing. And

the next one was, how am I doing on time? Because I have a lot of things. Okay. I might skip over fermentation just in interest of time, but I'm sure all of you have dealt with fermentation in some shape or form, even if it was liquid, having to benefit

the end result. But the insight here is to actually control how much salt and sugar we put into the different dough and then measure, and then also change the temperature and measure the growth rate and the CO2. And that was the fermentation.

And I also have, like, a sped up recording of the data collection. And I already showed you the recamera and the fact that it's mobile and it can track objects. This is the education version. If you want to play with some of these things and just kind of do your own experiments, please, like, the lab mind that I showed you is not deployed yet, but it will be, and the code will be open source.

But this you can play with. You need to put your own API, but it runs on the phone, and you can test the camera for now. And if you have a micro:bit or you want to connect mine, you're welcome to try it.

And let me know if the QR doesn't work. And yeah, this is a version kind of, like, to teach it to kids. And I wanted to end by talking a little bit about the open source ecosystem that can support this type of initiative to go beyond a demo and kind of, like, a cool experimentation to a real solution for scientists and builders.

So there's an entire open source ecosystem for recreating all the lab equipment, but then also for creating machines that can support the automation of pipetting and, like, doing all the

analysis that you need to do in a lab and all the manipulation that you need to do in a lab. The Jubilee Motion platform, it's actually coming from the University of Washington where I did my PhD. It's open source.

And there's an open bioreactor, and there's actually a workshop that just took place in April at UW where they had people for a week hacking on creating different solutions for automating

scientific experiments in the lab. So anything from, like, droplet manipulation to robot handling liquids and mixing, like, vials and things like that, and other applications with Jubilee. So definitely check them out. And for the future, and this is my last slide, you can imagine that based on the samples that we're collecting from these cameras and sensors and voice, it's much easier to create simulations.

So we don't need to be limited by the experiments we're doing in real life, but those experiments that we're doing in real life are going to inform realistic simulations,right? So I could actually create a simulation of my crystal growth, but more useful stuff,right?

Like bacteria colony growth, and run those simulations with the lab conditions, and then identify what are the best conditions for the experiment and create those conditions in real life. So very excited about integrating simulation, and I think that's where this is going.

That was me. If you want to read more about my projects and the paper and all the open source projects, I do a lot of work in education as well. It's all on my website. And as Ben mentioned this morning, we are going to do an AI Education Summit, and I'm very excited about that.

Some of you who saw my talk, like, in New York or last year know how much, how passionate I am about education. So I hope you can join us for the summit. I really appreciate you coming to the talk today, and I'll be around.

Thank you so much.