Skill Issue: How We Used AI to Make Agents Actually Good at Supabase — Pedro Rodrigues, Supabase

Hello everyone. Uh, is everyone excited for the conference?

Yes.

Awesome. We got a full house here. It's very— I'm very glad to be here, very honored to be giving the opening, or one of the opening, workshops today. If you've noticed already, the title is slightly different from what we have in the schedule.

I've basically done a rebrand, but the theme for the workshop will remain the same. We went from "Skill issue" to "Level up your skills." I've decided to move the "Skill issue" title to the keynote that I'm giving tomorrow.

If you'll have time to know more about what the keynote is going to be about. But mainly, it's this workshop. Basically, this workshop is what I've been doing in the last 2 months at Supabase: writing our own skills.

And tomorrow I'm going to present how we put this actually into production and the lessons we've learned. So, for everyone who's been paying closer attention, you probably noticed that I'm running this slide deck on localhost. Some of you have already noticed.

This is no coincidence at all. I basically— I essentially vibe-coded the presentation, so if you see something off, it was not my fault, it was Claude. But for you to— if you don't believe me, you can see that.

Or you cannot do, or you'll have to be a very Google Slides guru to have dark mode enabled. So, honestly, I like this layout better, so I think we're going with dark mode here. If there's any light mode fans out there, or the majority of the room is light mode, I'm happy to switch back.

But for now, let's go with this one. So, to do a little presentation of myself before starting the workshop: my name is Pedro. I'm from Portugal, Lisbon, and I work at Supabase as an AI tooling engineer. Essentially, my day-to-day is to think of how we can make the Supabase the most agentic, friendly as possible, and improve the agent experience.

So you've probably heard about development experience, the DX. We're more focused on the AX, which is the same thing but for agents. In this workshop, we're going to talk a bit about skills, because essentially that's how we've been improving the performance of agents around a product like Supabase, or a company like Supabase has multiple products.

The secret sauce has been basically skills. So we're going to dive into how to write one, how to test it first manually, and then how to automate the testing with evaluations. So, to start with, how many of you have heard about skills?

Allright. So, almost anyone. Everyone. So, what I'm going to say is probably no news to you. Skills are basically folders with instructions and files for you to run workflows, repeated workflows, or give custom, essentially custom information to your agents, or provide a new set of tools, let's say, in form of scripts.

So, there's a bit of a misconception about skills. Usually, the skill.md, the main file, takes the spotlight. But skills can actually be more than just the main file,right? So, the main file is basically a Markdown file named skill.md, where essentially the main information about the skill lives.

It is composed by this front matter at the top, which essentially can have multiple fields, but the two required ones are the name, which basically identifies the skill, and then the description, which tells the agent what the skill does.

The main— what exactly the skills basically bring that tools like MCP didn't— was this concept of progressive disclosure. Progressive disclosure is basically when the agent, or all the information about a subject, is not loaded straight to context. Instead, you just load the exact amount of information that allows the agent to choose to load the rest of the information once it actually needs it.

So, in this case, the skill.md file is designed like this. So the front matter will be loaded at first to the context of the agent, not the content of the file. This works as an envelope, so the agent knows from the description what the skill does and when it should load the rest of— so when should it look for the information inside of the file.

Inside this file, you can also reference another file. Usually, these other files are either Markdown files or script, bash, Python, whatever you would like to reference. Starting on the reference files, you usually put them inside a reference folder, and they provide more information.

You can think about a skill in this format as a book. The skill.md, you can think of it as the index on steroids, because besides having these links to the other files, you can think of them as the pages of the book or the other chapters.

You'll have custom information— you can have custom information and then also reference it to the other files. The reference files, they have nothing special about them. They're basically like a normal, regular Markdown file. You can think of a similar to skill.md file, but instead of being the main one, it's the one that got referenced.

You can also— funny enough, you can also reference files inside of reference files. So you can make basically a graph out of a skill. And for scripts, I've actually talked about how the— how MCP and skills differ from each other, and we're basically comparing apples to oranges when it comes to MCP and skills.

One of the misconceptions currently, probably, was already debunked. The debate now is more about MCP versus CLI. But when the skills were released back in, I think it was November or October last year, they basically started this debate about, well, should we use them instead of MCP?

Because if I can run— if I can provide more information, more context to the agent without actually loading every tool to the context like the MCP, and I can also have scripts so I can have actions, just like I have on MCP tools, should we use them?

And the answer is, you should use both, to be honest. If you're building anything that is an integration, you should use MCP,right? Anything that— if your agent doesn't have access to bash, you should use MCP to integrate to your service.

Skills actually just provide more context to your agent,right? And you can define workflows. Everything that you would not— that you don't have space to define on the MCP tools, descriptions, you can define them on skills. Also, regarding the comparison— the debate between skills, scripts, and the MCP tools, the main difference is that tools don't need an environment to run.

The agent can just call a tool, knows how to call a tool, especially if the MCP server is remote, and the tool will run on server side. While the scripts, well, they basically are loaded into your machine, they run on your local environment, and they're tied to whatever environment you have.

So if you're running on Linux, they have to be Linux compatible. If you're running on macOS, the same. Windows, I'm not going to even start about it. But essentially, those are the main differences between the MCP tools and the scripts.

Hope is everything clear. If you have any doubts, feel free to— I'm going to have a little demonstration. This workshop is going to be more of a walkthrough than actually code along, but feel free to tag in. I have a GitHub repo prepared, so you'll be able to visit it and to explore it.

But if you have any doubts in any moment of the workshop, feel free to interrupt me or to raise your hand. So, moving to— see, this— exactly. So I tested this on a smaller screen. It was working. You can see it was vibe-coded.

So, how do you test your skills,right? So, if this is just a Markdown, how do you test your Markdown files, basically? So, to test a piece of code, it's already straightforward,right? We already know you have all sorts of test types.

You have unit tests, integration tests, or you can test the whole flow, or what we call end-to-end testing.

Well, essentially, when you're testing a Markdown file, you can basically do exactly the same. You can be as granular as you want. But usually, since we have an LLM in the loop, you'll have something called evaluations. So, for those of you who haven't heard about evaluations, or evals for short, they essentially

are a more non-deterministic way of testing the output or the behavior of an agent or a model. You can test both an LLM or an agent with evals. Essentially, the most common structure— I'm going to present to you at the end a framework for you to test your evals, like a very simple one where you can start.

And I'm going to dive deeper on evaluations there. But essentially, they usually are made of an input, an expected output, just like a regular test, and in between you can evaluate the steps that the agent took, the reasoning, the tools that it called, which is normally more interesting and easy to evaluate than just a regex on the exact output, since this is non-deterministic.

So, there's essentially a framework that you can follow to test your skills. This one was proposed by OpenAI on their blog post called "Systematically Evaluate Agent Skills." I think they released this back in January or February, so not that long ago.

But all this is fairly new, so this is basically prehistory. So, you start by defining your metrics, so what you want to evaluate on your skills. If you're building a skill for your product, for example, what exactly do you want the skill

to highlight to your agent? Is it going to be to forward it to the documentation? Are you putting some specific instructions, specific workflows? So, depending on what you want to evaluate, you start this eval-driven development, so this test-driven development.

You start by defining the metrics, what exactly "good" means when it comes to the skill. Then you create the skill itself,right? So you write the skill.md file.

Oops. So you write the skill.md file, any scripts alongside it, the reference files if you want to. They're all optional. The only required is the skill.md file. And then you move to the testing part. So you run the evaluations, or you run it manually.

I've recently heard the CEO of Braintrust during a podcast. I don't know, how many of you know Braintrust? Okay, not as much the— not as many as the— not as popular as skills, but. So, Braintrust, for those of you who don't know, Braintrust is a platform that allows you to systematically run evals and provide you the full picture

of the agent behavior during the evaluation scenario,right?

I'm just trying to think about another platform to compare with, but this is fairly new, to be honest. So, you can think of it as like an observability tool to check the behavior of your agent during a specific controlled scenario, which are the evaluations.

So you move to the testing part. Basically, you run a set of evaluation scenarios.

How you— these are defined by the input and expected output tools that should be called. So basically, how you expect your agent to behave. And then you move to the grading part. So how did the agent do? Well, essentially, this is very similar to a testing cycle,right?

But now, instead of having a deterministic output, you can have— it's non-deterministic, it's an LLM in between— but you can still have deterministic parts to evaluate on. And then you iterate, basically, and repeat. This is the— that's why it's a cycle.

It's pretty similar to any of the test development cycles that we have at the moment. Allright, so jumping straight to what we're going to do during this workshop. So we're going to write a skill. I've prepared a little demonstration app, a demo app.

It's going to be a performance review application with four, I believe, employees. One employee, two managers, and one HR

representative. And essentially, we're going to— there's some errors on the database side that we're going to find and fix. We're going to build a skill to help, to guide the agents to fix them, allright? And then at the end, I have, as I said, a framework to test automatically the same scenario that we're going to test manually using evals.

Before moving to the demonstration, how many of you have heard about or used Supabase? Allright, so almost anyone knows or used Supabase. I've seen some hands down, so still, I'm going to give a little brief. So, Supabase, it's essentially a backend as a service.

You can think of it as the open-source version of— thank you, Firebase. Only FireShip was coming through my mind. Sorry. To Firebase. And if you don't know Firebase, you're probably living under a rock. No, but essentially, it's a backend as a service.

You can use it to build any backend as you would like, coming straight out of the box. We provide a database for you to just plug into your application and run on Postgres, one of the most, if not the most popular open-source solutions out there for databases.

You can easily integrate it, including authentication on your application, running storage to save files,

and many other things. Edge functions, which are Lambda functions, for those of you who come from the AWS environment, and so forth. So, the demo application that I've built was built on top of Supabase, of course. And so you can follow along.

Here is the QR code. At the back, can anyone— everyone— scan the QR code? Or should I make this bigger? Bigger? Allright.

Okay, just so everyone can see, I'm basically editing the presentation at the moment as we speak. Let's see what Claude

has to offer us.

Bigger?

This is the cool thing of vibe-coding our presentations. I really recommend. I probably spent the same or more time than if I just used something like Google Slides, but at least it's more fun. And Anthropic should be thrilled about it, for sure.

Allright, let me know once everyone is in the— is that the— you can see the repo. If you cannot see or scan the QR code,

I should probably make the link bigger as well.

So you asked for a demo? Here's the demo. One more slide.

So, mainly everyone here in this room used skills, so I probably won't have to sell you the power of skills. But if you're still a bit skeptical about skills, this whole presentation was— without skills, this whole presentation would be a lot not pleasant.

Let's say, it would be much uglier, in a sense. Okay, you should probably see it now. So basically, it navigates to GitHub, "hudrixpn," which is my nickname, and "Improve Skills Workshop AIE Europe." It's a very long name. Allright.

So is everyone at the GitHub repo at the moment? Okay, everyone had no trouble. Allright, so.

Not this one.

Right. So, this is the repo that you should be looking at. Essentially, I know it's a big repo, but we're going to break it down. Actually, going to

move—

okay, I'm going to—

I'm going to move to VS Code.

Allright. So, here we have

two Next.js apps. Actually, the slides are also embedded here. The Next.js app that matters, it's inside demo,right? And to give you an insight of what that looks like, it's basically this.

So it's a very simple application. You can see that it's a vibe-coded application, to be honest. The layout, there's nothing special on it. You have, as I've described earlier, several employees of this fictional company. And you can think of it like an intranet or a performance review application where you have all the information.

As an HR employee, you have all the information about

the other employees of the company. And you can change, for the sake of the presentation, you can change between

users,right? So what we're going to do is, first, without a skill, we're going to try to implement a new

view here. We're going to implement the reports view. Essentially, this reports part of the application is going to show both the salary and the average rating for the performance review of each department. So HR can know what's— can have an overview of the whole company.

So before we start to vibe-code, because during these workshops, no one actually writes code anymore, so of course I'm going to vibe-code it. We're going to just break this

application down. So if we navigate to the dashboard, it's nothing special to see. You have— it's basically the page, the first page, the main page that you've seen. And then you have here the reports where we should have— yeah, where you should have this set view exists.

That is going to— so I've prepared the backend. We're just going to create on the backend the view as a SQL view on the backend on the database, and then we should be able to see it on the application.

So I've prepared— where is it? Yeah. So I've prepared the prompt, and

we're going to

live-test it.

Fingers crossed that this works. Wait. First, let me navigate to

its app. Okay.

Here we have more control. Allright. So essentially, for the ones in the back, I'm going to ask Claude to create a department stats view that shows the headcount and the average salary broken down by department, allright? So for HR to have a full overview of what's going on in the company.

So we're going to hit the prompt and wait to see what it should come up with.

Right, forgot about this part. I have this MCP server configured. If you have— you actually have— I totally jumped the README. If you follow along— sorry about this. If you follow along, if you're following along, you can follow the setup guides to get your application started locally.

This essentially is going to install the dependencies, clone the repo, install the dependencies, start locally your Supabase project. You don't have to have the CLI installed. We were using NPX to run it as a binary. Just resets the database state so you start from scratch with the seeded data, and then just run the app

as running npm run dev. It should be available on localhost:3000/dashboard.

You also have— you'll have this MCP.json file prepared. This essentially is pointing to the MCP server that we Supabase enable for local projects. No authentication required, so your agent should be able to just load it on demand. This MCP server exposes a set of tools.

I don't know how many of you have used the Supabase MCP server, but I think we currently have something along 20-something, 29 tools, I believe, for the production one. This one is a smaller version, has 20 tools, but you can basically perform essentially almost anything that the one to connect to your remote project does.

Basically, list the tables that you have, execute SQL straight on your database, apply migration, and run the database advisors and so forth. So essentially, what it started to do was to list my tables. So I've asked for review.

It's going to review the schema that I already have implemented. So I'll let you— and now it's going to run the apply migration tool to create the view. So it's basically doing a schema change on my database, and it's going to create the view.

If we inspect the view, we're basically

creating or replacing a view, a department stats, the name that we gave. And we're fetching all the information from, I think, department, exactly.

No, from profiles, exactly, and grouped by department. Okay. Oops. Made a mistake. It's going to try again.

Okay. It's going to test it. It's actually something that I really like about. Allright, then here's our view on the database. So we currently have it on the database. Let's see if that also enables on the app.

Okay, it's not. Then let's quickly— I've created a SQL

element. What's the name I gave? This is essentially the problem with live views.

It usually doesn't go well at first try.

I want to repeat the—

and let's see if it implements. If not, we can just run the SQL query for you to see as different users, for you to see if everything is working accordingly. So for now, it's going to implement on the Next.js application so we can have a nice interface to check the results.

I need to enable everything. Wait. I'm going to just put on auto mode so I can continue to talk. So essentially, the agent created the view, tested, said everything is working accordingly. We should not have— the app was— the feature was implemented.

All good. But we're actually going to see if it's actually— if everything, it's good or not. So let's give it a— let's give him some space, not to pressure it to create

the feature. Let's just wait a bit more time.

In the meantime, if you're following along, you can also play with it, change the layout, actually using Claude Code. I don't know. Just doing a brief survey here during the workshop. How many of you are using Claude Code as well?

Oh, fairly almost everyone. Okay.

How many of you are using Cursor? With Claude Code, or with the plugin, or— okay, yeah, at least one person.

We're going to have some Cursor folks here, I think from Anthropic as well. OpenAI is going to be here. Gemini, of course, Google DeepMind is sponsoring the event. So we're basically going to have the whole gang here. Okay.

So we should be— I'm trusting his word, allright? So it says that we should now have correctly displayed the department stats view. So let's see if that's actually true. It looks like it. Yeah. So we now have these cards with the whole view of the company.

So I'm logging in as Julia from HR. We can see that we have five people on the engineering team with an average salary of 107K. HR as well, only one person would be Julia. And product has four people and that average salary.

So so far, so good. Looks okay. Let's see. So this is sensible information, the reports,right? We're expecting that the other

employees will not have access to it, and even the managers only have for their departments. Let's see if that's the case. So let's navigate to Bob. Bob is the head of engineering. Oh. Okay. So Bob also can see the performance reviews of the information of both HR and product.

Well, it's not that bad,right? It's not ideal, but at least he's a manager,right? So he should be accessed to privileged information anyway, and he doesn't like a transparent company. Let's see if our help is okay. Okay, this is problematic.

So we basically created a view. Claude said everything is working because, as you can see, the information is here. It was created. But he missed something that is training data, basically missed something, which was, for Postgres specifically, when you create a new view

and your table has role-level security enabled. So for those of you that know, role-level security allows for you to define who can see the information on a specific row on a database level. So without trusting the application, you can filter it directly on the database.

So in this case, we should be limiting the view of the rows by user ID,right, and the user role. So if a user has an employee and has an employee role, he should not have access to the rows that don't belong to them,right?

We have role-level security enabled. If we navigate to our Supabase migrations,

you can see

that we have role-level security enabled

both on profiles and on performance reviews,right? And on a performance review

should be allright,right? So we have

reviewer ID equal current setting. So it should work. Why is it not working? Well, when you create a view on Postgres, by default,

the permission it creates with the permissions or the credentials of the user that created the view and not with the credentials of the table, let's say, with the role-level security. So basically, by default, it bypasses the role-level security that you have in place on your— that you might have in place already on your

table. So for this not to happen, we have to add a security invoker. We have to use a security invoker flag to

transfer the role-level security policies or to enable the RLS policies on the view itself. So this is why currently everyone can see everyone's, because the role-level security policies were basically bypassed on the view. So

for the sake of the demonstration of this workshop, I've already created a skill. I prepared a skill for the presentation.

And essentially, the skill is three main security points about Postgres that the agent should be aware of during the presentation. For this one specifically, I actually overfit it to the exact view that we're creating, but modelsright now are smart enough to generalize this.

And if I wanted to create a new view, you will be able to. Essentially, it has to create with this flag. Since Postgres version 15, this flag was enabled, and every time it's enabled, the role-level, the RLS policies are also enabled on the view.

As you can see, it's actually quite human-readable documents. Most of you have already written skills, so I'm not going to dive deep into this.

But as you can see, we have both the title. Let me just move this. We have the title. I called it Supabase Security. And the description uses the verb use. This is an insight that I got from some experiments that I've did.

Using verbs, mainly the verb use, increases the chances of the skill being loaded, at least on Claude. I don't know if this is default behavior for Claude, if it was trained to recognize the more easily recognized verbs, especially use.

But I found it more efficient if you write use and then the whole purpose of the skill in front of it. And then a regular markdown list. We have the view case there. But also another checklist points for security on RLS.

So public schemas should have RLS enabled by default. Public schemas or exposed schemas are the database schemas that are going to provide information for the application that the user can see. So for example, the user's table, the profiles, the performance reviews, all this information is going to be fetched by the front end.

It's completely secured because Supabase makes it secure by allowing you to fetch information from the front end. But the key part here is that if you don't enable role-level security, you will not have this filter on the table, and you will have to rely on the application logic to make the filter.

So enabling role-level security at least makes it safer for you as the backend engineer that you only expose the information that you actually want from the start.

And then a couple more things that I'm not going into. So we can install this skill on this project by running— where did I have the command? NPX. Yeah.

So I'll be

using— where's the skill? Okay.

I'll be using Vercel's NPM package called Skills. Curious to know how you guys have been packaging your skills. Have you ever used this package? Are you using plugins? Just this one. Yeah, this one mainly. Yeah, it became very popular a few months ago.

I think the only problem is it doesn't really adhere to the project boundaries. So you get global skills. And then if you want it only for your local project.

Yeah, you can install it both globally and on your project, and also as support for multiple agents. While plugins for now are still tied to the agent that is going to load them. So Cursor has plugins. Claude Code has plugins.

I think other vendors have as well, but they're specifically distributed and made for those specific models. So we're using this one to install. You can install any skill from a repo online that has a skill.md file, or you can use it to install the one locally.

It will auto-detect

the location that you're trying to fetch from based on the format. So in this case, we don't have any GitHub. We don't have any HTTP protocol there. So we have a dot slash, so it'll recognize that it's

a local one. And for this, I'm going to

move to the main. Yeah. Okay. And on a good old-fashioned way, going to run on the batch. So it's going to pop this up. Ask me which agent do I want to install on. I'm using Claude Code, so I'm going to install it on Claude Code.

If you're using any other

agent harness, you can also install it as long as it

supports it. I'm going to install it on a project level. So it's going to, in this case, create a .agent folder with the skill and link it to my .cloud/skills folder as well. So Claude knows where to find them.

I'm going to sim link, and we're ready to install. So if we— oh, let's not expose my key.

This is just for the workshop, so I'll delete it afterwards. Feel free to use my free credits for the time being. But essentially, it created the

.agents. Where is it? Yeah. So I also have some more things that we're going to see afterwards. But the essential part, it has the skill that I've showed previously. Yeah, there it is. It's the skill. And it also created a symbolic link to the .cloud.

This is how the package works. And this way allows Claude to either search on .agents, which is becoming the standard, or on the .cloud folder that it has. So let's run the same prompt again on a new session.

Let me go back to the apps demo and start a new session. We should have this one enabled. Yeah, there it is. So Claude is aware of the Supabase security skill now. For you to run skills, you can either just run your prompt and pray that Claude imports your skill based on the description that you gave.

You can include the keywords used and then the name of the skill that you have on the prompt, and this will almost 100% of the times load your skill. Or if you're using Claude Code, you can just slash and write the name of your skill.

And this 100% guarantees that Claude is going to import the skill. So for our use case, for the presentation, I'm going to— because I cannot afford that it doesn't load the

skill— let's wait. I need to

reset the database to create

the view again.

Workshop. I need NPX Supabase to be reset. Yes.

I'm just resetting the database, applying the migrations from the start.

It didn't create any

migration file. It applied the migration directly to the database. So we now should be good to go. It's going to bring down the database and create a new one based on the schema that we define on the migration files and the seeded data.

Yes.

On the loading the skill, I mean, when I saw you first presented, the whole point was a kind of loading magic in the background based on language. And now we have skills using slash command or the use and enabling skills to make things still missing a little bit, the initial promise.

Have you found ways to load skills when it understands the topic or not without you having to backstage it a little bit? You know, I understand what I'm trying to say. The whole kind of how skills get loaded kind of thing, it's still pretty just, I want to use that specific thing rather than be like, understand that you're working with security or low-level security.

Just do it when it's working on that.

Yeah, that's a fair point. So your whole question or observation is that the initial promise of skills when they were presented by Anthropic were.

How are they in a way to actually get it working?

Yeah. So since this is on the agent side,right? The agent decides when to load this. The best thing that you can do without explicitly either with the slash command or the use and then the name of the skill on your prompt is for you to play around with the description and run a bunch of tests, either manually or

automatically, to check what actually works and not for the ones that you're expecting the agent to behave,right? So you define a bunch of scenarios where you think that the skill should be loaded and when the skill shouldn't be loaded.

You test it out. You can test it on your machine, like on this scenario. I don't want the skill to be loaded. Prompt the prompt on Claude Code, let's say, and check if the skill was loaded or not through the CLI.

And then play around with the description to see what actually works or not. Without actually explicitly call the skill, this is the best thing that you can do

to test if the skill is being loaded correctly or not.

We're still at the very beginning of

a very early stage of skills. Even for MCP, all this agent stuff, it's fairly new. So we're still standardizing things. We're still figuring out what works and what doesn't. Progressive disclosure was something that no one was talking about six months ago, and now it's fair to say that it's one of the north stars of agent development.

So in six months from now, probably it could be another thing. Or skills could be the standard, or maybe Anthropic or OpenAI or someone else found a more efficient way to manage the context or provide more context to the agent.

So we'll see, basically. Allright. So the database was reset. Okay. So at least now we have the view, but we don't have the information on your database. So now we should be able to run the same prompt again, but with

the skill. So if we hit the prompt, let me see. It was quite fast, I don't think.

Yeah, but it didn't create one.

Okay, let me try another thing instead of

this. Let's

use to create. Let's see if it works now. Yeah, okay. So it loaded the skill. So now at least you'd have the context to create that the RLS or the Security Invoker flag should be included when creating the view.

And the steps should, the rest of the workflow should remain the same. So it will list my tables,right? Exactly. Identify the tables. And now if we look closely, we can see that we now have

the flag here. It's going to be on the migration. So let's see if with the flag, this is the expected result.

This is what happens when you VibeCode a CLI.

You now have the UI duplicated. Right. So it created the view. We should be able to see it. But Alice shouldn't. So what's happening? Wait. Okay. So

do I have to reset now?

Hmm.

Interesting. I should have another. Probably. Let me just see if I have it here.

Where did I put it?

Reset count.

I'm going to cheat here. Let's say both

and the employee should be able to see information from the

harness and the new.

Okay. Basically, live troubleshooting. Why it is not happening? Probably from a different policy that I've defined here.

But now it's going to troubleshoot. Let's see if the skill actually improves the efforts here. If not, I have something on my sleeve. Because if you're not aware of, Supabase basically has

database advisors that you can use to try to early on identify some potential vulnerabilities or schemas that are exposed, information that might be exposed before you're running it into production.

So if it can't figure out by itself, I'm going to include on the skill to also run the advisors to check. So this is the main part of skills, is that you can

see, well, it's a very poorly written application, let me say.

It's essentially the main part of skills. It's not if this specific demo works or not, it's that the behavior changed once it loaded the skill,right? It created with the Security Invoker part. And with that, that just shows how powerful it is that you can

change the behavior or guide the agent on demand based on the information that you put. You can think of the skill.mds as a prompt template that you can give to your agent. So let's just quickly troubleshoot. Oh, it's even offering to apply a migration.

Let's see if it doesn't break my app.

Allright. So it seems too complicated. Too complex. Anyway, going to move. What's the separate table?

Anyway.

Could you look at the context? Like just for slash context in your, just to see how your context looks.

Good point. I have a fairly amount of skills. As you can see, I've been playing around with them. I also have some of the pre-installed MCP servers that Supabase enables. But essentially, it would be more interesting if I've

just compared the context from before and after loading the skill. Soright now, skills take 1.3 thousand tokens on my context,right? As you saw, I have more than just this one skill. But the skill was loaded, so the whole information inside skills.md was loaded to the context.

If we clear and run the context again,

the skill amount. So this skill is not enough for you to see. But as you can see, the skills take quite

less space that the MCP would.

Allright. Oh, okay. I have a newer version of the Claude Code. So for those of you who are not aware of this, Anthropic recently released the tool, the tool search tool, which is a mechanism for Claude Code to load tools on demand.

So it doesn't load, basically progressive disclosure, but for MCP tools,right? The main difference between this progressive disclosure or the tool search tool on Claude Code and skills is that the progressive disclosure is built by design for skills. So it's like already baked into the structure of the instance of the skill, while on MCP, it's still not a standard for all tools.

So it works for Claude Code, but for many other clients, it won't. It will just load all tools straight to your context. So

this is, for now, a thing

for just Claude Code. If you're interested about it, we are going to have the founder or one of the co-founders of MCP speaking on the 10th, so on Friday. He's going to give a brief overview of the MCP roadmap, which if

nothing changed since last week when he presented this in New York on the MCP Dev Summit, he should bring this progressive disclosure part to tools, to bring it to the protocol itself. So.

Excuse me.

Yes.

Let's say that we have a very large database and we have to load in the context the schema of this database because we have to query the database using agents. In your opinion, is it better to use a skill or an MCP, for example, to load this schema, but progressively?

Okay.

Is it possible to use the schema to progressively load the schema of this big database in your experience?

Yeah. Yes. Okay. So is your question more about how should we access it or the whole architecture of this pipeline to import the data?

I just want to ask to an agent to query the database. And obviously,

the agent must know the schema of the database before or not.

How can you teach the agent to query the database using the skills, using an MCP server, or something like that? And if you use the skills, if you decide to use the skills to load the context of the agents with the schema of the database, is it possible to progressively load the schema within the context?

Okay. Gotcha. So let me break the situation. Let me break down the situation for you. Here, you'll have

essentially two parts. One is what's going to be on the context, so what's going to be loaded and the specific information that you want to have on your scenario. And the second part is the actual mechanism, the extraction mechanism that you're going to use to load the information from the database.

So for the second part, to load the information from the database, you can either use a script, so a skill that invokes a script, or an MCP tool. I would advise to use an MCP tool because you can use it if you're using on production or on remote projects, you don't rely on your local environment, you don't have to manage the keys, and the tool is already standardized and you already have the authentication baked into the protocol.

So the agent never manages the authentication token. It just runs the tool and it works for it. To progressive disclosure, the information on the database,

you can include it on a skill. You'll be using the MCP tool. So on the skill, you'll probably state that use this tool to load. And in the tool implementation, you have to enable it to progressively load it,right?

So to load into chunks.

It might be just enough from the tool parameters. The agent should figure it out by itself that if you put a parameter called buffer, for example, should be able to load it in chunks,right, instead of the whole table.

But if you want to have 100% sure that it's going to load into chunks and use it properly, I would also package with a skill and describe how I intend to use this tool. So this is actually how both skills and MCP play along together.

It's the tool to enable this connection, this integration, and the skill to describe how to use it. Yeah. This is how I would implement this type of system. Thank you for the question. And it got me the opportunity to basically talk about how to use both skills and MCP and not put it one against each other.

So now, as I promised, we should be moving on. I'll have to give it more time to figure out because I'm basically doing the workshop. When I was preparing the workshop, I gave it a bunch of vulnerabilities. So if I just kept it simple and that one, the demo would probably work.

Since I have more vulnerabilities exposed that if I had time, I would try to solve it. It didn't for the moment. But you saw on both scenarios that the first one didn't have the security flag, Security Invoker flag, and the second one had.

So at least

we can imply that the skill was doing something. The agent saw the information on the skill, it merged it with the system prompt or stored it near the system prompt and changed the behavior accordingly. To test this, so if you want to move this part, the skill, into production,right, so it works on your machine.

It's a tale older than time that it's working on my machine, but I don't know if it's going to work on your agents, on your machine, on your environment. So to

test this or to automate this testing, and with this, we can unlock having a pipeline, for example, if you change one thing on your skill, how can you reliably tell that it keeps doing what you're expecting, didn't break the previous flow?

So if I change one of the checklists, how can I ensure that the other ones were still working,right? So this is where evals could step in. So

evaluations, it's a very broad term. You can basically evaluate anything. Since this is a Markdown file, it's a free-text file, you can evaluate basically anything. So it's fairly difficult for you to the most difficult part to create evals, I would say, is actually coming up with the scenarios because you would first have to know what's the expected behavior of your agents.

So coming up with representative, actually good scenarios that represent a fairly amount, that cover a fairly amount of use cases that you want to build, are the most difficult. And there's still not a standardized structure to create evaluations.

You can test it

by importing a bunch of prompts and expected outputs from a CSV file, from a JSON file. You can use tools like Braintrust or LangFuse to test it and to have an analytics and an observability layer on top of it.

For this presentation, I

followed what Agent Skills Open Standard defines to design the test cases. So if you're not aware of this website, this is the landing page of the Agent Skills Open Standard to try to standardize what a skill is and how it should behave.

And they basically propose a very simple structure, local way to test the skills, organized by. You'll have an eval.json that essentially has

a set of evals, so an array of eval scenarios. You'll put the prompt that you're going to give the agent, the expected output from the agent. This is only if you have an LLM as a judge. This is a technique used for non-deterministic evaluation.

Instead of a human, you can give the outputs of an evaluation run to another LLM, say it defines a success criteria and let the LLM whose

role is to judge in this case, that's why it's called LLM as a judge, to give it a grade, basically. So this is one part that you can automate on your evaluations for non-deterministic workflows. You can either assert if a tool was called, or you can give the results to an LLM and non-deterministically try to get the agents to grade the performance of the other agents.

So you'll basically have agents evaluating agents. So I followed this structure. I gave

the same inputs here,right? So the agent that is going to run this evaluation is going to get the same input that we had. The expected output, it's that the Security Invoker, it's true, so it's present on the app, sorry, on the view.

And then I have a bunch of assertions that in this case,

I'm going to check it deterministically,right? I prepared a Python script that essentially just resets the state of the database. So we ensure that since we're running this locally and not on isolated container, like a Docker container, for example, we have to make sure that the systems always start from the same ground.

So I'm going to reset the app. If you want to run the evaluations as well, you have to pick your own Anthropic key, create, copy this. You can follow the README inside the Supabase security. Here, you'll have how to set this up.

But then I will run the Claude Code CLI on, I think it's on print mode or I can't remember what they called. But essentially, I will run it as a binary, headless. So the agent will receive

the prompt that's on the evaluation as the task to perform. And I'm also going to give the condition. We're going to test two conditions, one with the skill and another without it. And essentially, so for you to see the condition,

this is where the Claude Code will run. And if the condition is with skill, we're going to load the skill.md into the system prompt,right? If you would actually like to mimic the behavior, you would run this on the Docker container.

You will put the agent skills on the .claude/skills directory inside of a Docker container and let organically let the Claude Code find them and use them. For this presentation, this is a very simple setup. I've just basically appended to the system prompt.

So we're going to run the evaluations. Do I have the other? Yes, I do. Okay. I think we run it on the

Supabase.

How is it not finding the skill for Supabase? No. Okay.

Oh, I have wait. I know what's going on. I have the wrong name. I change it.

Allright. So we started by running with the skill. So the first result that we should get is the with the skill. It stopped. Now it's running without it. And then we're going to compare it. This will output a workspace iteration one folder.

And we can compare it both the outputs of with the skill and without it. While the without skill is loading, let's just quickly inspect what the with skill output gave. And essentially, you can see that it created the view with the Security Invoker.

And then we have this grading.json file with a bunch of information, like the assertions that we've put on the eval, we've set on the eval.json. We have them here. And we can see that for this one, grade it as failing, even though that created.

Where is it? Not found.

The view has security certain.

Okay. I'm actually evaluating something wrong. So the problem here now, it's with

the skill view. So since I was expecting this to create an PG class, RL options, instead of just inspecting the view, it's giving me that it failed. But the key part, is it finished? It's not finished.

Still running. Taking a long time.

Could be. Okay.

And now we can inspect. Okay. So this is actually a good insight. So with these results, this is the tricky part of writing evals.

So as normal tests, the results will depend on how you implement them,right? It's just code. So if you're evaluating something wrong or not the expected behavior, you're going to have wrong results. It might not be because the system is not working.

So we've tested manually and see that with the skill, it created with the security

flag. We can actually just inspect it here with the skill created. Let's see if on this one, surprisingly, this time it did. It's a non-deterministic

behavior of Claude.

But since I was evaluating something wrong,right, I was expecting it to create or inspecting a meta schema to check if the security invoker was there or not, instead of just inspecting the view directly. The results came a bit off.

So it said that with the skill, it failed. And

without the skill, it passed. And if we inspect both outputs, they're basically the same. So with this, just to show you how tricky it is to write evals, because although this can happen with the regular tests, it's easier to catch because the output is deterministic,right?

It's just code. Here, if you're handing to an LLM to evaluate, it can sometimes hallucinate. So to finish, because we're also almost running out of time, to sum up the structure, this is the one that they recommend. I find it very easy to implement to get started with.

Later on, you can move on to more

complex evaluation scenarios, like running on a Docker or in a sandbox, to guarantee that you get a fresh environment with just one skill that you're testing on your set. But essentially, you would just put two conditions with and without the skill, compare the results, and run them on the agent harness that you would like, and you compare the results out there.

This is basically your very first evaluation pipeline to test a skill automatically. From my end, that's all. I hope you found this workshop useful to get your skills leveled up and ready to productions. As I said in the beginning, I'm going to give a keynote tomorrow, a keynote, no, a talk tomorrow about how we've implemented and created the Supabase skill for the product itself.

How we're keeping it maintainable while ensuring that it provides value, and how we're testing it into production. Thank you.

Anyone has any doubts, questions? I'll also be yeah.

So I have a question about the number of skills that you typically install on your environment, because with this progressive disclosure, it seems like we can basically keep adding different skills and the agent will automatically, basically, the agent will automatically find them.

Do you have any recommendation on how many skills to have, or is there any limit, or we should just basically keep adding and it will magically work?

I'm probably not the best person to talk about this, because it's easy for you to get into this rabbit hole of just like, especially when you're experimenting, getting a bunch of skills. As you saw, I had plenty of them installed globally.

And I think it's fair to say that I use them all on a daily basis. But it depends. If you're using them on your local machine, I think it's

going to be pretty easy for you to get this messy environment where you'll have all of them installed or most of them installed. For your local environment, for now, since it's very experimental, in my personal opinion, I would not constrain myself on

space management or context management about this. The progressive disclosure, it's a very powerful thing that you can explore in this case. Sure, if you have skills that you don't use, you're going to have them fill your context window, but the descriptions are so small that you can afford to not delete them if you don't want to.

Into production, treat them as any artifact that you would have on your CI. So keep it clean. Into production, into your CI, I would keep them only the exact skills that you're using in that specific case.

Yeah. Another piece of information that I could give you on the production part is that it's now more and more common for you to also export skills or make skills available on your repos as another piece of documentation.

So treat skills that you put into production as actual document, as you would treat documentation. So it's important for you to keep them updated, include the update workflow on your Claude.md or on your agents.md, so you make sure that if anything changes, you will change the skill as well, like you would do on the documentation.

If a feature or a workflow changes,

from time to time, you can also create a job to check if the skill is still running a fair workflow. If somehow you could check if the skill has been loaded by your users,

if it hasn't been loaded by your users for a long time, does it still make sense to have it there? So yeah, this is basically the piece of advice that I could give you for skills into productions, based on my experience.

For the rest of it, you'll have to come to the talk tomorrow to learn how we're putting it into production on Supabase.

Any more questions? I'm going to be around throughout the whole event. So if you catch me, if you cross paths, feel free to ask me anything. Tell me about what you're building. I'd love to see. If it's with Supabase, even more thrilled to hear about it.

And from my hand, once again, thank you very much. You've been lovely today for 9:00 a.m. Pretty good energy. So just from my end, enjoy the rest of the conference, and we'll see you around. Thank you.