OpenClaw in Your Hand: Building a Physical AI Terminal - Lech Kalinowski, Callstack

Probably all of you once planned the cleanup activities, and you can imagine I also plan one weekend's cleanup of my basement, and I just went there and want to clean up all of the stuff there, you know, throw up an old washing machine, and I just look at the corner there and realize there's something weird there lying on the floor, so I just grab it, look at the strange device, looks a bit familiar but a bit old, like from '80s, with keyboard but with a weird displace, of course covered with dust, I just take it to my hand, look at it, and I was completely surprised that device have a shell and it's on,

so I used my superpowers skills in a terminal and I asked, I provided there a simple question, "Who am I?" and the device responds in the weirdest way I could even imagine, in the natural language, so the story I just described to you, it can be a great story for marketing,

but honestly I just wanted to build a device which is physical and AI native, like the device which comes from the future. Hello, my name is Lech Kalinowski, I hold a PhD in physics, and today I want to present you a device and the backend and the system I built over last couple of months, and the whole story begins because I just simply wanted to build a remote controller to my OpenClaw instance on my DJX Spark.

And the story is simple, because when you are just first think about LLMs, then you're not thinking about the audio, about the generative

videos or pictures, first you're thinking about the text, and then I just get a bright idea that maybe I can use some kind of better display to read and type with my LLM.

Of course, on the market there is electronic paper, but I realized electronic paper can be a bit slow, of course it's perfect for reading, but if you just want to provide dynamically text to it, you probably need a faster display, and then I just used and in my design

a simple connection of the OLED display, the small one, one color, with the electronic paper display, and then with this simple, let's say, dual display approach, I realized I can build quite a powerful and energy-efficient terminal to

play with my claw.

And of course, behind it, there is some kind of lot of complications, because from one side I just wanted to build a remote control, but from other side there is such a niche on the market, and this is the AI native operational systems, and when I just started to think more clearly about it, I just realized I can build such a system for my purposes, but no with a super powerful computers, but with a small microcontrollers, and I just design a system, build in, let's say, with a, say, four classes, one is the internal shell to control the terminal itself, you know, system settings, basic configurations, Wi-Fi connection, and within my idea to build an AI native device, I build an assist, control, and one thing which really surprised me, and I will explain you

later, RPG mode.

And

here it is, I started to prototype DGX Spark at the

background here on the picture, two displays, one it's a live surface, that means the dynamic part, you can type, the whole text provided is written there, and when you just trigger action, hit enter, then the whole render also happens on the second bistable electronical paper.

But there is a lot of complications and a lot of tech and a lot of steps you need to do to get an clear user experience and clear interface, so I just found out a lot of the different approaches how to render properly on the two displays, properly render the interface, I just interact to control my LLMs and to get in responses and to do whatever I want with my DGX Spark through the OpenClaw, and this is one of the approaches that you can just have fixed static buffers and render one bit images stored in the memory, that means the pages live in a pre-allocated memory, there is no markdown engine and no malloc on the MCU side.

I just wanted really to show you how the system is really complicated here, and this is not because I just

wanted to use that much components, it's because there is a huge need to build a power management system, because I just blow up two displays over my prototype build, and

I just needed to be sure that the current and the voltage is stable as possible, so here inside you have an MCU, which is ESP32, dual core microcontroller, here you have an OLED display, keyboard, and an encoder, and of course the mention power supply.

But the AI native end-to-end device is not, let's say, that simple, because of the, you know, high power consumption and the hunger for the compute capabilities of your device, so here is a full backend created, that means I just created a full backend, and here's the presented architecture of it, that first we have a Vault firmware deployed on the terminal, and there's a huge backend which handle all of the agentic work with the OpenClaw and handle the LLM.

I just want to point out here within my today's presentation, I just want to demonstrate you the device connected to an open-source GPT-120 billion parameters model, and I just serve that model with the, let's say, most possible optimal way using, and I use their TensorRT serving system, and I just exposed OpenAI style to also like LLM proxy, because I just hit a lot of walls with the other open-source models, because not all models match the style of the OpenAI API.

As I said, the first idea was to build and play with the OpenClaw through that terminal, and this is the stuff I just do with this, here you can look at the picture and see an example, I was possible to hit a command, like write a Java example, and store it on my local machine, and OpenClaw with the LLM sample just did work and make it happen.

Okay, and here's my field notes, the things that doesn't go as expected, lessons learned, first of all the software I2C, because I just wired up things there and it doesn't work correctly,

then there's a way to do proper control over I2C without any additional physical pull-ups mounted. From other side, GPIO-13, there's a silent failure, there was a need to move to the other port to be sure that everything works correctly.

From other side, I need to build up the regulator, the power supply, the whole unit of the power supply, because the regulator kills the OLED and fragile parts of the display, and it cost me a lot of time and

getting the replacement parts from the market took so long, a couple of weeks. And the last part, quality of your parts, that means encoder, cheap and low quality, give me a lot of rotational noise, and there was a need to build up the pull-ups and to

wired up additional condensators there, but this is the funniest stuff, my favorite one, because this part really surprised me, you already know there's a huge part called RPG, and behind it is a bit sub maybe story, because I just never played RPGs game like on the paper, that means with a lot of friends, you can go to an apartment, you know, open a book and make someone a game master and play in a real text-based RPG, and after a lot of those years, I just never tried it, but I built

an RPG game and a console

which give me a pure experience of the text-based RPG games, role-playing games, and actually it's a bit funny, because those device, it's really like perfect fits for that kind of gaming, I built out an NPC and memory around that, I just created the mood of the world, the contents, omnius, and I just used all of the, you know, the LLM advantages to build the best role-play game experience ever.

I just created four different worlds, from one side I just wanted to build something cyberpunk related, from other side I just wanted something the Witcher part related, that means some kind of fantasy world with the dragons, but also my favorite one is the void in a deep space somewhere in the cosmos.

And

this is really

a really good example how we can just use generative AI to build computer games, from one side I just generate the characters, their worlds, maps, skills, and all of those converted with the mechanism I just described at the beginning of that presentation with the one bit memorial location, that means all of the pictures transform to the matrices, to the maps.

And

one important thing, actually the device is really bulletproof, if the OLED doesn't work, the e-paper works, if the cardboard doesn't work, that means the keyboard, then you have an encoder, Wi-Fi falls, you always can use your local shell.

And

funny stuff, the area of the intellectual property there, I just do some kind of research in the field, and everyone just want to build the devices around the audio interfaces, the video capture, you know, but for the quiet places, when you just want to, you know, sit down, keep calm, and play your RPG game, you don't need any of the, you know, colorful, powerful displays, you know, OLEDs, you know, popping up with the colors and all of the distractions, commercials, web pages, and other things, sometimes you just want to sit down and, you know, do theright job with the OpenClaw, or, you know, just chat with your LLM, read and write in the quiet environment without any of the distractions, and there's a niche on the market, so I also decided to build a provisional

patent here in that field, and maybe it will bring commercial device in the future to a market. Some numbers,

in my project

I just pushed 130 comets, around three months of work, two displays, each of the display do other job, four modes, four worlds for gaming, in general 16 classes, that means you can generate RPG worlds inside of it, firmware is really tiny, really small, and it works extremely fast, and everything is powered by one single

polymer cell, lithium polymer cell.

But maybe some remarks and takeaways, much of these plays to the job, there's a plenty of them on the market, keep the model of the metal, because it's really heavy, that means for now we don't have such a models which runs really on the super tiny MCUs, it means LLMs, and track a narrative, the context really matters in LLMs, and in general not

numbers.

So here I also record the demo, the usage, how the device work, and some kind of

last

session, please enjoy, and I will place a couple of comments there.

Device is booting, connecting to a local Wi-Fi network, here's the welcome screen and the main screen,

small display,

keyboard, first simple

help command,

here you have all of the things you can do with the device,

there's also a tutorial for gaming and tutorial for the device itself, and when you just switch it on for the first time, then you need to, or can go, you don't have to, but you can go through the tutorial itself to learn how to use it.

Usage of the OpenClaw agent, that meansright now I just checking my DGX Spark and want to look at this space,

here's it,

let's maybe now look on the. My favorite part with gaming.

Fair world, Alder Gun, Neon Abyss, the Hollows, and Void Witch.

You can select your

character,

and the LLM just generates the stuff, everything is generated there, pictures,

personalities of the people, and here there's a mood, there's a breach, you're in the deep space and can't deep dive into the other world.

Yeah.

So that's it,

that was the video, funny part is that the game like this, without the three-dimensional graphics, use the most advanced NVIDIA processor mounted in the DGX Spark, that means this is the backend currently, I just prepare for this demonstration, so the quiet game with the, you know, text interface requires the most powerful computer graphics card in the world, and maybe another thing is that the presentation was about the handled device to control agent and OpenClaw with the local LLMs, but in general it was about the Game Boy, the device to play with LLMs.

Thank you for the attention, thank you very much

for the Callstack support, because I'm just working at Callstack in the technological incubator, and the whole work was supported by

Callstack company. Thank you very much.