The Multimodal Future of Education: Stefania Druga

Hi everyone, I'm Stef. I'm going to talk about the future of education with multimodal AI. We are here at AI Engineering Summit, and AI engineering starts very early. So I'm curious, how many of you have kids? How many people in the room have kids?

Okay. Wonderful. How many of your kids have played with generative AI so far? Okay. So you won't be surprised to see the next slide. Basically, 70% of generative AI users are from Generation Z. This is a recent study from Salesforce.

So it starts very early. And the reason I care about the future of education with generative AI is because education needs a wake-up call. So we know that early literacy rates need to be improved around the world. Only 70% of 10-year-olds actually can read and understand a simple story.

At the same time, we've seen a big gap in learning that happened during COVID. 60% of children and teenagers are left behind, in particular in math and reading. And it's not just K through 12. For older people, for adults, we need to do a lot of reskilling.

So

multimodal AI has a potential to transform education. And we know that students are using these tools for their homework. They recommend it to their colleagues. And we also know that they prefer using these tools over a human tutor.

So this is not new. We areright now dealing with the first AI generation. These children have been growing up with AI. Since 2015, half of households in the US had some sort of voice assistant. And I started researching this in 2015 at MIT and basically showed that youth perception of voice assistants, chatbots, smart toys really influence how they interact with these devices, how they learn from them, and how they perceive them.

So overall, they perceive these AI devices friendly, but they also have a different perception of the intelligence of the devices. And age plays a huge role. So younger kids, we're talking 4 to 6 and a half, are much more skeptical of how smart Google Home is or Alexa.

And older kids, the moment they start going to school, they all say, voice devices are smarter than I am. And we're just talking about voice assistants. We're not talking yet about generative AI tools. So how can we change that?

And why does that matter? It matters because their perception of how this technology works influences what they expect. And their mental models are influencing the type of queries they're going to ask, how much they trust the answers. So we need to cultivate AI literacy and critical understanding of this technology.

To that end, I built this open source and free platform called Cognimates in 2016. It expanded Scratch, which is the largest platform for coding for kids. And it basically allowed children to do programs like this. Can someone guess what this program does?

Yeah, it's a game. It's a hide-and-seek game,right? So she's like, she programmed a hide-and-seek game with the robot. And if she puts a loop, she can run around the room, and the robot is constantly going to try to find her.

Now, this is the first step. It's using, like I mentioned, a block-based programming language expanding Scratch. And at the time, it allowed kids to not only program their smart lights, their voice assistants, but also train their own custom models.

So they can train models with examples of images or examples of text, and then use those custom models in their own games and applications. So for example, here, this student trained a model to distinguish between unicorns and narwhals.

And then not only does it get a prediction when it plays with a game, but it also gets the confidence level. How confident is his custom model that the drawing is a unicorn? And we see the confidence is pretty low.

So they made all sorts of things, like looking at what's in their food, trying to program games like rock, paper, scissors, get the robot to talk like Shakespeare. And this was used all over the world. It's translated in more than 30 languages.

And the good news is that we evaluated this to see how it increases that critical understanding of AI and how it helps with AI literacy. So to do that, I did a longitudinal study in public and private schools where we asked questions of what kids think about AI before.

Then we allowed them to engage in AI learning activities. And then we asked the same questions at the end. And what we found after they learned how to do text training, image training, smart home programming, is that they became much more skeptical of the AI smarts.

In the beginning, they would say, yes, Google Home is smarter than me, or this model is much better than me. And after they learned how it works and how to train it, they were not so sure it's smarter than they are.

And I'll show you a quick video to see how that went.

So we were programming robots. You could play rock, paper, scissors.

I did it rock, paper, scissors into the camera. And on the shoot, you did it one of the motions. And the camera did one of the motions. And it said, rock, paper, scissors, shoot. The computer gets better as you play the game.

Because we might not know everything at first, but if you keep trying, you get better.

Everyone has heard about machine-based learning or deep learning. And there was sort of no question to ask for a lot of the more tech-savvy parents. It was like, go for it.

Technology is going to be a huge part of their lives, much more so than my life. If it's scary for some people, this AI technology, I totally get it. But as a parent and as a teacher, I thought it was really important because these are skills that 21st-century kids need to have.

When my dad was young, he bought a car and took it apart to see how it worked. So you teach people that young how these things that grown-ups mostly program, how it works.

So as I was saying, AI engineers in the making. And this is the significant difference to their perception of the smarts of AI before and after doing these learning activities. So how did they why did that happen,right? Why did they become more skeptical, more critical, and also more literate in how to read and write with AI?

It's because by providing this platform and allowing them to tinker and form hypotheses and test them, we basically allowed them to engage in the scientific process, just like researchers do, just like we do,right? But we needed to have theright sandbox, theright platform for them to be able to quickly tinker and quickly iterate.

So kids are not alone in learning this. Parents need to learn too. Teachers need to learn too. And we've seen during the pandemic, when kids were stuck at home with parents, a huge opportunity for them to learn together.

So I'll show you one of the early demos of Cognimates.

Oh, the audio is not working on this one. I'm not sure why. Basic.

There you go. You did it. No, I need you to help me ask a question. For that, we'll need the ask block. See if you can find it.

Awesome.

So the thing that you are programming is kind of collaborating with you to teach you how to program it,right? Just imagine applying that to any of the chatbots we have today,right? When you're not happy with the answer, or maybe the answer is not age-appropriate, or you also want to teach something to the model about your language, your culture, weird facts that you're interested about, how do we do that,right?

So I did another study where this was with kids and parents in 10 different states in the US over multiple weeks, where we wanted first to learn how do we design a copilot for programming for families. So before we start and build it, what do they want?

What works and what doesn't? So what we found was that some of the things that kids and parents liked the most was to generate coding ideas with an AI friend, like if they had a copilot in Scratch. And this was very, very helpful.

Here are some quotes. Because here, one of the participants says, "Most people would like coding with AI friends because one of the hardest parts of your project is when you start, you run into a wall because you're out of ideas."

So the AI friend helped with that. It also allowed them to express and elaborate their ideas in code. So if they had an idea for a game, like I want to make the bear kind of jump over the hedgehog, but they didn't know how to do it, it would kind of help them find theright code constructs to do it.

And more importantly, it supported their creative coding identity. So it wasn't the bot that was making all the coding. They were doing it. The bot was just helping them when they were stuck. So this was very, very important.

It encouraged kids and parents to work together, which is not always easy,right? One of the things I discovered, I've been working with kids and families for a long time now, since 2015. It's not always easy. So actually having a third moderator to be like, "Oh, what does mommy say?

What does daddy think? Take turns. Try this," really helped with family joint engagement. Also, it doesn't always work,right? Sometimes it's too distracting. And it was very important to enable families to shut it off. Maybe they want to do the game alone.

They want to do the coding alone. So they could stop it whenever they wanted. If you have multiple siblings that fight over the laptop, it doesn't really it cannot help with that. Or if the concepts were too complex, it was not able to scaffold it always, like break it down.

So parents were very helpful to help. So after understanding what are the core things that families want from co-creating and learning how to program with an AI friend, I went and basically evaluated all the generative AI models to see if they could do that,right?

So for Scratch, top generative AI models are pretty good at generating explanations, giving ideas or questions to help kids and parents explore and test new games. And this was published. We created a benchmark as well for measuring this.

And this is just an example of what the future of education with multimodal AI could look like. If it's applied to Minecraft, to games, to physics simulations, science simulations, it can become a creative sidekick,right? There are a lot of people who love to build things with their hands.

What if I could get ideas by taking pictures of flowers I like and colors I like? And it gives me ideas and helps me generate 3D models that I could afterwards print and paint. Or I'm into knitting, and I want to use a generative AI model to inspire my knitting projects.

It can also be a learning companion and a coach. It can help with math. So together with Nancy Otero, we created the first benchmark for math misconceptions to show what are the most common math problems that kids have in K through 12 and evaluate how good are top-of-the-art generative AI models in identifying these misconceptions when kids talk with a chatbot.

And I put a link to it if you want to download it. So I am here to invite you to think about AI engineering and AI tinkering for all ages and how do we go from my experiments to Cognimates to things that people are doing and tinkering with Hugging Face and make sure we open up the space.

So we use AI not just to teach, but we actually use AI for people to learn how to tinker and learn by playing and learn by doing. So I like to do what I preach. And I'm going to show what I tinkered with AI last night.

These are very fresh demos. So this is using the latest Gemini API. And I have three demos. Let's hope they work. Let's start with the science one.

And

I was hoping to draw in real time, but I don't have a table. So luckily, I have some drawings. And we'll see how well this works. So I have a drawing. Oops.

A scale with a weight on each side. What would happen if you add another 5 kilograms? So it's asking me questions based on my drawings. And then I can make a new drawing that has 10 kilograms and 10 kilograms and see if that gets better.

Let's try another one.

Water and CO2, what happens if it gets mixed? Oh, I need to so imagine I have a webcam and I'm like a table and I'm drawing in real time. And we could play with it. But it's very interactive.

Let's try this one.

The earth is being hit by something. Hopefully not. Let's add one more arrow and see what happens if we do that.

Ah, that was fun. So it finally understood it was the moon. Let's play with the math one.

And

solve the expression inside the parentheses. OK, so I have one where I did

OK, solve the multiplication with the parentheses. And let's assume I've done that too and I have the next question.

I need a better background for this demo. That's for sure. The first step is to simplify. No, no, no. Go back. OK, so any number divided by itself equals 1. But you see, it doesn't give me the answer.

It just gives me a question so I can keep trying,right, and learning. Let's try one more, more complicated.

It's OK, 4 on the axis.

I was hoping it would give me a better question. Yeah, so the last one is the one that is encouraging curiosity. So this one

what is the lady doing? OK.

What are the colors on the flag? What shape is the star? Oh, it asked me things about Jordan.

Let's see what it does with the apple.

It has that.

Do you know what apple this is?

Let's see.

What does the apple smell like?

I had a nice origami thing as well.

OK,

I don't know where the origami went, but

so you sort of get the gist. These are like I don't know if you want me to draw something or do you want to ask one of the questions of the science or math or objects? Any requests from the audience?

Don't be shy.

Yeah? What should we ask?

No? OK. Well.

What's wrong? You didn't ask.

Did you ask it to solve a system equations?

Excuse me?

A system equations.

A system equation, yeah. Can you tell me what to write?

Sure. 2x plus 7.

2x plus 7.

Equals 2.

Equals 2. Thank you. Let's try it.

Let's see if it does well with my

subtract 7 from both sides of the equation. Not bad. And now if I do that

divide both sides by 2 and so on and so forth. Now, the cool thing about this, like I made the code open source and template, so you can play with it too, is less than 100 lines. You just need to create an API key, which is free.

And you can create your own instructions. And hopefully, I inspired you to think beyond chatbot interfaces and delegating instructions and delegating questions and think more like a tinkerer and think about how we could put these tools in the hands of young people because they are the future and they need to learn about this technology as well and how it works.

I think that's my time. All my research is on my website. And I put a QR link for that as well. And I look forward to your questions afterwards. Thank you so much.