How does the world's first open-source video editing agent work?

The agent, presented by Muhteşem, starts a browser session using Playwright and opens operator.diffusion.studio, a video editing UI for agents. It uses three main tools: VideoEditingTool generates LLM code based on user prompts; DocsSearchTool uses RAG for additional context via llms.txt; and VisualFeedbackTool provides multi-modal feedback on sampled frames. File transfers occur via Chrome DevTools Protocol, with optional GPU-accelerated remote browser sessions via WebSocket.

How does the VisualFeedbackTool decide if the video is ready to render?

After each execution step, the composition is sampled at 1 frame per second and analyzed by the VisualFeedbackTool via a multi-modal model. Muhteşem likens this to a generator and discriminator in a GAN architecture: the tool acts as a discriminator, giving feedback to either proceed to rendering or refine further. This feedback loop ensures the final video meets quality standards before rendering.

This video was edited with AI agent. But how?

Q: Why does the agent use code generated by an LLM instead of JSON for tool calling?

Muhteşem explains that code is the best possible way to express actions performed by a computer, as it allows the LLM to write its own action directly. Multiple research papers have shown that having LLM tool calling in code is much better than in JSON. This approach is a perfect match for the Diffusion Studio Core library, which already uses a programmatic interface.

Paper Tease0:30

Guest0:30

Today we're tackling a paper that's basically legendary, at least in the world of natural language processing—NLP, for those in the know.

Guest 20:38

Right. Attention is all you need. It's not just a catchy title.

Guest0:42

Definitely not.

Guest 30:52

OCAML. A general-purpose functional programming language that is also an imperative language and also an object-oriented language. It is what Haskell, which is, it could be.

Agent Introduction1:04

Muhteşem1:04

Hey everyone. My name is Muhteşem and I'm excited to talk about the world's first open-source video editing agent.

Motivation1:13

Muhteşem1:13

Backstory is that we needed some automatic tool to edit videos for re-skill.io, a platform for personalized learning. While doing so, we quickly realized limitations of FFmpeg and started looking for more intuitive and flexible alternatives. ReMotion was nice, but it had unreliable server-side rendering.

After trying out Core, we really liked the API as it did not require the separate rendering backend. We met with the author of the library and decided to collaborate and build this agent together.

Core Discovery1:34

Muhteşem1:48

The Core library from Diffusion Studio can do complex compositions via JavaScript-TypeScript-based programmatic interface, meaning we can use LLM to generate code to run this. And if we take a step further and let our LLM write its own action in code, it's a perfect match, simply because code is the best possible way to express actions performed by a computer.

Code Generation1:48

Muhteşem2:14

Lastly, multiple research papers have shown that having LLM tool calling in code is much better than in JSON.

Architecture2:25

Muhteşem2:25

Now, let's take a look at current architecture. Agent starts a browser session using Playwright and connects to operator.ui. This web app is a video editing UI designed specifically for AI agents. It renders video directly in browser using WebCodecs API.

It also has helper functions for transferring files from Python to browser and back, via Chromium DevTools Protocol.

Agent Tools3:00

Muhteşem3:00

This is a typical flow of agent. We have three main tools: VideoEditingTool, DocsSearchTool, and VisualFeedbackTool. First, a VideoEditingTool generates code based on user prompt and runs it in browser. If additional context is needed, DocsSearchTool uses RAG to pull the relevant information.

Visual Feedback3:22

Muhteşem3:22

After each execution step, the compositions are sampled, currently at 1 frame per second, and they are fed to VisualFeedbackTool. VisualFeedbackTool can be thought of as a generator and discriminator, like in the famous GAN architecture. After the VisualFeedbackTool gives green light, the agent proceeds to render the composition.

llms.txt3:53

Muhteşem3:53

We also shipped llms.txt, which is essentially robots.txt but for agents. You can see a sample in the screen. llms.txt, in addition to specific template prompts, will take you far in your video editing journey.

Scaling4:12

Muhteşem4:12

While you can bring your own browser and run the agent, the current setup is also flexible enough to let the agent connect to a remote browser session via WebSocket. And each agent can get a separate browser session, which is GPU-accelerated.

And of course, there is a load balancer behind this.

Of course, the first version of the agent is in Python, but TypeScript implementation is underway. As the famous saying goes, any applications that can be written in TypeScript will be written in TypeScript. Thank you very much. This was a collaboration between Diffusion Studio and Re-Skill.