# This video was edited with AI agent. But how? AI Engineer · 2025-02-22 Muhteşem from Re-Skill presents the world's first open-source video editing agent, built in collaboration with Diffusion Studio, which uses LLM-generated code to edit videos via a browser-based UI. The agent starts a Playwright browser session and connects to operator.diffusion.studio, a web app built with Diffusion Studio Core that renders video directly in the browser using WebCodecs. Three tools drive the process: VideoEditingTool generates and runs JavaScript code based on user prompts, DocsSearchTool uses RAG to pull relevant information from llms.txt, and VisualFeedbackTool samples one frame per second and decides whether to proceed with rendering or refine further. File transfers between the Python agent and browser happen via Chrome DevTools Protocol, and for scalability, the agent can connect to a remote GPU-accelerated browser through WebSocket. The project emerged from Re-Skill's need for automatic video editing for personalized learning, where they found FFmpeg too limiting and turned to Diffusion Studio Core for its intuitive API and reliable in-browser rendering. ## Questions this episode answers ### How does the world's first open-source video editing agent work? The agent, presented by Muhteşem, starts a browser session using Playwright and opens operator.diffusion.studio, a video editing UI for agents. It uses three main tools: VideoEditingTool generates LLM code based on user prompts; DocsSearchTool uses RAG for additional context via llms.txt; and VisualFeedbackTool provides multi-modal feedback on sampled frames. File transfers occur via Chrome DevTools Protocol, with optional GPU-accelerated remote browser sessions via WebSocket. [2:25](https://aie.addtry.com/6e556e8c-300b-411d-b92c-c8fea8b79b5d?t=145000) ### Why does the agent use code generated by an LLM instead of JSON for tool calling? Muhteşem explains that code is the best possible way to express actions performed by a computer, as it allows the LLM to write its own action directly. Multiple research papers have shown that having LLM tool calling in code is much better than in JSON. This approach is a perfect match for the Diffusion Studio Core library, which already uses a programmatic interface. [2:14](https://aie.addtry.com/6e556e8c-300b-411d-b92c-c8fea8b79b5d?t=134000) ### How does the VisualFeedbackTool decide if the video is ready to render? After each execution step, the composition is sampled at 1 frame per second and analyzed by the VisualFeedbackTool via a multi-modal model. Muhteşem likens this to a generator and discriminator in a GAN architecture: the tool acts as a discriminator, giving feedback to either proceed to rendering or refine further. This feedback loop ensures the final video meets quality standards before rendering. [3:22](https://aie.addtry.com/6e556e8c-300b-411d-b92c-c8fea8b79b5d?t=202000) ## Key moments - **[0:00] Intro** - **[0:30] Paper Tease** - **[1:04] Agent Introduction** - [1:04] Muhteşem introduces the world's first open-source video editing agent, built on Diffusion Studio Core and Re-Skill technology. - **[1:13] Motivation** - **[1:34] Core Discovery** - **[1:48] Code Generation** - **[2:25] Architecture** - **[3:00] Agent Tools** - **[3:22] Visual Feedback** - [3:22] VisualFeedbackTool samples composition frames and acts as a GAN-like discriminator to approve rendering or request refinement. - **[3:53] llms.txt** - **[4:12] Scaling** - **[4:39] Closing** - [4:39] "Any application that can be written in TypeScript will be written in TypeScript," says Muhteşem, previewing the agent's TypeScript version. ## Speakers - **Muhteşem** (guest) ## Topics Browser Agents, Agent Engineering ## Mentioned Diffusion Studio (product), FFmpeg (product), Playwright (product), Re-Skill (product), Remotion (product) ## Transcript ### Paper Tease **Guest** [0:30] Today we're tackling a paper that's basically legendary, at least in the world of natural language processing—NLP, for those in the know. **Guest 2** [0:38] Right. Attention is all you need. It's not just a catchy title. **Guest** [0:42] Definitely not. **Guest 3** [0:52] OCAML. A general-purpose functional programming language that is also an imperative language and also an object-oriented language. It is what Haskell, which is, it could be. ### Agent Introduction **Muhteşem** [1:04] Hey everyone. My name is Muhteşem and I'm excited to talk about the world's first open-source video editing agent. ### Motivation **Muhteşem** [1:13] Backstory is that we needed some automatic tool to edit videos for re-skill.io, a platform for personalized learning. While doing so, we quickly realized limitations of FFmpeg and started looking for more intuitive and flexible alternatives. ReMotion was nice, but it had unreliable server-side rendering. After trying out Core, we really liked the API as it did not require the separate rendering backend. We met with the author of the library and decided to collaborate and build this agent together. ### Core Discovery **Muhteşem** [1:48] The Core library from Diffusion Studio can do complex compositions via JavaScript-TypeScript-based programmatic interface, meaning we can use LLM to generate code to run this. And if we take a step further and let our LLM write its own action in code, it's a perfect match, simply because code is the best possible way to express actions performed by a computer. ### Code Generation **Muhteşem** [2:14] Lastly, multiple research papers have shown that having LLM tool calling in code is much better than in JSON. ### Architecture **Muhteşem** [2:25] Now, let's take a look at current architecture. Agent starts a browser session using Playwright and connects to operator.ui. This web app is a video editing UI designed specifically for AI agents. It renders video directly in browser using WebCodecs API. It also has helper functions for transferring files from Python to browser and back, via Chromium DevTools Protocol. ### Agent Tools **Muhteşem** [3:00] This is a typical flow of agent. We have three main tools: VideoEditingTool, DocsSearchTool, and VisualFeedbackTool. First, a VideoEditingTool generates code based on user prompt and runs it in browser. If additional context is needed, DocsSearchTool uses RAG to pull the relevant information. ### Visual Feedback **Muhteşem** [3:22] After each execution step, the compositions are sampled, currently at 1 frame per second, and they are fed to VisualFeedbackTool. VisualFeedbackTool can be thought of as a generator and discriminator, like in the famous GAN architecture. After the VisualFeedbackTool gives green light, the agent proceeds to render the composition. ### llms.txt **Muhteşem** [3:53] We also shipped llms.txt, which is essentially robots.txt but for agents. You can see a sample in the screen. llms.txt, in addition to specific template prompts, will take you far in your video editing journey. ### Scaling **Muhteşem** [4:12] While you can bring your own browser and run the agent, the current setup is also flexible enough to let the agent connect to a remote browser session via WebSocket. And each agent can get a separate browser session, which is GPU-accelerated. And of course, there is a load balancer behind this. Of course, the first version of the agent is in Python, but TypeScript implementation is underway. As the famous saying goes, any applications that can be written in TypeScript will be written in TypeScript. Thank you very much. This was a collaboration between Diffusion Studio and Re-Skill. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.