10x Development: LLMs For the working Programmer - Manuel Odendahl

Hello.

Hello!

Are you all awake and do you just have your coffee? That's great. Welcome to my workshop about LLMs for programming, or like how to become a 10x engineer, which I'm like, actually pretty serious about. I put some handouts out, so I didn't print like 200 of them.

But if you can, sit next to someone who has some. There's like a couple lying around in this part of the room. And in general, I encourage you to have a partner, two partners, because it's a workshop and it's like nice to share ideas.

There's a URL that's to a GitHub repo, which has nothing in it, but it has the handouts as PDF and it has like a list of links or something like that. I encourage you to join the Slack channel, because of the size, I won't be able to really interact with individuals, but I will be able to through Slack.

Allright. The only prerequisites you need are an LLM. So I use OpenAI, just like the chat interface, or like Claude Sonnet is great. But you can use, like, if you want to have fun and use like a small one on your laptop, you're welcome to.

Find.com is also great, like anything that's all you need. No code, no Git fork, no nothing.

And with that, I'm Manuel. I'm like a software engineer by heart. I knew I wanted to be a programmer when I was six. And when gpt-3 came out, or like Copilot, I was like, what the, and started just like doing everything I do usually after 25 years of being a professional software developer with LLMs and just like keep finding new techniques until today.

I kind of am not on social media. I don't really follow what other people are doing, so this is kind of like all homegrown kind of stuff, but always with the focus of like actually doing like e-commerce work, PHP programming, just like down to earth stuff.

And so I'm going to show first just like a few general concrete techniques. Like I think a lot of programmers don't really

vibe very well with the technology, especially because the hype around it is like it's going to replace you, us, which is kind of true. But so I'll give like a bunch of techniques just like to ease in getting into it.

So who of you are programmers?

Cool. Is there someone who's not a programmer? Because you can still stay, because it's everything will still apply, actually.

Who's using LLMs, Copilot? Well, I'll put Copilot a little bit apart, but like LLMs code generation for their work. Allright, cool. I guess that's why you're coming to the fair, huh? So after a bunch of general concrete techniques that I use, which I feel are pretty important, I'll go into the main concept of how I approach programming with LLMs, which is like treating them as translation engine, not like as something that reasons or something that like writes code and can figure things out with agents.

It's just like, oh, they're good at translating one type of language to another, which is honestly what we do as programmers as well,right? We take GitHub tickets or whatever and we convert them to code. Or the other way around, when the code doesn't work, we create GitHub tickets.

Then a pretty important, this is like a new concept I haven't really played with, but it's so massive, is treating LLMs as like word simulators and what we can do with that

in the context of just like pragmatic programming. And finally, a bunch of like what I try to distill is like, what are the skills that we need in this new age as software developers to become really effective, because they're very, very different from what they were in the past.

But some of them stay the same, which is also good. So what I'm going to do is like I'm going to go through these pretty quickly. I wrote up, I wrote them up on the handout, and the handout, you'll see these gray blocks.

They're concrete examples of just prompts. Like I don't want to paste 20 pages of GPT transcript that no one cares about. But what you can do and try to maybe it will work, just like paste these prompts into ChatGPT and see what happens.

And like there's some sequences which might or might not work depending on what the model outputs, but hopefully it can follow your way around. And there's links, which obviously don't work on paper, to some of the transcripts I just did yesterday.

But really, it's about you trying them out. Like don't try to, these are not examples you should follow. This is not the way to do it. It's just like an example of applying what I really want to get at.

And I'll give you about 10 minutes after showing you these things to basically communicate with your neighbors, try to solve some problems that you have, come up with some, just use some of the examples I show to get a sense of

how these things work. So first technique, which I see a lot of programmers not doing, is not regenerating. Right? You'll come in, you'll put some prompt in it, and like either it outputs something great and you're like, holy crap, or it outputs something that's buggy and you're like, ah, it doesn't work.

But if you regenerate 10 times, you'll see like, oh, it maybe works three times. And five times it outputs something that's like almost working, and then two times it's outputting something that's completely random. The API works better for that because you can up the temperature.

Like I realized Chat OpenAI used to be much more wide in its outputs. Now when you regenerate, it often comes out with the same stuff.

Second one is like instead of trying to correct the models and like, no, this is wrong, this is not theright method and stuff like that, just go back and edit either your prompt or if your UI allows it, like LM Studio or LibreChat, like other front ends allow you to edit the LLM's response.

So if the LLM outputs bad code, just go back, fix the code, and then the LLM thinks it output theright code. So it's really useful. Then a next technique is to often clear the context. Like I don't think I do more than three, four prompts in a conversation.

And then I just take the output at the end and just start a new conversation. That tells me if, you know, my technique is robust. Because I can't, you know, if I have a 20-page transcript, I don't really know what's going on.

I don't know what the context has been summarized to. And usually by the end, you're just like in la-la land. It works much better with the bigger models now. So, you know, maybe this doesn't apply anymore. But it's good practice anyway.

Experiment with different models, because they all kind of have different vibes. And you get to feel these different vibes and how they change over time or like how smaller models might just need more coaxing but can actually do the same stuff than the bigger models.

And this is all through practice. So like practice, practice, practice, practice. I have the trouble that I just get rabbit holed when I have a task to solve. I try to write the prompt that solves the task, and then I try to write the prompt that generates the prompt that solves the task.

But, you know, practice.

And then this is some very concrete stuff. Just make like custom system prompts that you can put into like Chat OpenAI or whatever. Don't worry about changing them. Like if you have a session today where you're going to write PHP, just change the system prompt to be PHP oriented, like which libraries you use, maybe what code formatting you use, just because it saves like a ton of time.

That's a very concrete technique. There's a prompt on how to do that. And I want, once I get to this slide, maybe we can try it out. Generate helpers. Like every time you have a problem, just ask the model to write a script to solve it,right?

Like often I put stuff on my clipboard and I have to like edit it to put it back into my code,right? Like maybe the imports. I don't want to always like have to remove the imports when I paste it into my IDE.

So I can ask like A GPT, write a shell script that just opens the clipboard and an editor, or open the shell script and just open the clipboard and remove the imports at the beginning. It's like a simple said three-liner.

Just generate it, and at the end of the day, you can throw it away or save it. And this might be the most important technique summarizing your transcript, and I want you to do that today to exchange with your neighbor.

When you try something at the end, just like summarize what I did as a README, as a how-to, as a wiki entry, as an RFC, and then share it with your neighbor so that they only have to read like one clean document and not like 10 pages of whatever.

So I really want you to do that today if you're like working with your neighbor, which I really encourage. At the end, summarize the transcript and just send them that. Even if you do like rabbit holes, I'm fine trying to solve a problem,right?

Like, oh, what's this TypeScript bug that I have? At the end, you can summarize your search and just send it over and say like, look, this is how I solved it. Here's all the links if you want to look at it.

So that's something you would never do before when you were reading Slack overflow and like Google searching. You would never send someone your Google search history. But now you can't. And a final technique is like when I ask it to, when I try to figure out how to do something, I use ridiculous domain examples.

Like instead of working on my app, which is often going to be kind of a meta app,right? Like if I do infrastructure, then it's going to be like programming jargon, and the model might get confused because I want to, you know, create a script that creates scripts or manages scripts.

So instead, I always ask it to do like, you know, manage dinosaurs or like zoo animals or like write an operating system about Tolstoy. And then you see in the output, what is Tolstoy relevant is like what it fills into the pattern.

And then when it says like kernel driver, you know, okay, that's kernel driver. But if it says like kernel driver system function interface, you're like, you're not sure if it's like a driver about functions, if it's the function of the driver.

So I have an example here, like write an Arduino R task for remote controlled T-Rex.

And I'll show that on screen. And while, you know, for the next 10 minutes, I want you to try out some of these things, like, you know, write some helpers, like create some system prompts or summarize some transcripts that you have, research how to, how bourbon is made, and summarize, you know, your research to share with your neighbors.

Like just try different things. And while you're doing that, I'm going to watch the Slack. If you have cool examples, you can also open PRs on the GitHub repo so that everybody can see them later on. And I'll answer questions.

And if no one has questions, I'll just do my own stuff in the browser that you can look at for inspiration. So before we get into that, like maybe I'll just show you

one of these prompts. And, you know, the code doesn't need to work, but it like gives me an idea of like different concepts. Can you read this? Is this big enough?

Yes.

Real-time preemptive

prey hunting. I don't know. Just mixing up words.

This is amazing. The artifact stuff

is just amazing. And here you can see,right, like the tasks are prey detection. I know, okay, like it understood that the task name is something different than the task thing itself. But so here I can tell that it used free RTOS and it's like, no, implement an RTOS from scratch.

And then you can just like rabbit hole,right? So I'm going to implement it from scratch and I'm going to summarize the transcript and you can see how that works and how useful it is. So this doesn't necessarily need to work,right?

But it gives me a bunch of stuff to look at and try to Google maybe. You know, I can just try it out and run it. But then I'm like, okay, well, where did it kind of fetch this from?

You can probably Google it because it's in the training corpus. And that gives you like a good start of either a lot of boilerplate that you want to change,right? And then maybe you're like, okay, well, I know how to do a timer interrupt.

I will replace that with my own. You can edit the LLM's response to put in your timer interrupt and regenerate and it will be like, oh, this is how you write interrupts. I'm going to write interrupts that way as well.

And you have a really efficient thing because it's still two interactions with the LLM,right? Like fast inference and stuff like that. Summarize this as a README for my colleagues. And then

this is like high value when you work on a team and then you can just be like, look, this is how my RTOS works. I would never have written it by hand. Documentation of that quality,right? You can say like add examples, list of caveats, only output new text.

I will make many typos. The model doesn't care.

Allright, your turn. I'm watching the Slack.

If you want to share ideas of stuff to build, like feel free to do it so that people who are, you know, early morning. Yeah, but can you later on ask in the Slack? That way I don't have, more people can do it.

But go ahead.

I just wanted to say that it works nice for mermaid charts. You can create a chart of your code. It also shows on the artifacts.

Yeah. Oh, nice.

One more generic technique. When you talk about doing something, just type that into the LLM and then we'll do it. It's like programming at the speed of mouth.

And then.

Hi. Could you spend time to join the Slack channel?

Oh, yeah, sorry. So the Slack channel to join it, the Slack channel is called LLM Working Programmer Workshop, LLM Working Programmer. Sorry about that. Maybe I'll just leave this slide on. You have the handout maybe or, and I'll just mess around in a different tab.

Allright, one minute and then maybe I'll spend two minutes showing what I did.

Allright. I hope you had fun. Feel free to summarize what you did, you know, in like a README or a little message and then share the chats

in Slack so that everybody can see them or make a pull request to the repo. And for the sake of time, because we have 50 minutes left and I have a lot to show, let's just move on. Is there any blocker?

You all good? Allright, no Wi-Fi issues, no model issues, anything? Allright, cool.

Allright, so this is maybe the main concept I use, which is no one knows how LLMs work and we never will because LLMs are basically these weird computational artifacts that have been trained on like whatever gigantic corpus we've trained them on, which no one has the time to read, let alone understand,right?

So no single human will ever understand what is in a model. You can just like kind of poke at it and connect it to what we know. And so in a way, it's not like, you know, it's not like a computer technology or a math technology or whatever we want to call it.

It's like a cultural technology. It's like based on human culture. And if you know your way around human culture, you'll be able to squeeze out more of the model, not just programming,right? So it's all about language. The model really doesn't care if it's programming language, if like a formal language, or if it's like German or, you know, symbols or whatever, because the only thing it does is transform tokens and tokens and it looks back at its own tokens doing so.

So even if we say like, oh, please output JSON, it's like, yeah, I'm going to output numbers or probabilities of all the numbers that exist. But so if you make the switch away,right, like you're writing code, but like there's no reason that what is in the code isn't actually also like language.

Suddenly it makes a lot of sense of calling a variable like year over year revenue and not call it A, even though as programmers we think like, ah, that doesn't really matter,right? But actually to model, it matters a lot because one is like the token A and the other one is the token year over year revenue.

And then suddenly it's able to access all the like finance literature to do to write your code, even though you think that's like just, you know, a stupid dashboard. So understanding that means that suddenly things like theater, poetry, marketing literature, like textbooks, mathematical formulas, all of that is like fair game to write code.

And so for example, theater, I bring that up as the first one. The models are trained to be nice,right? Like they're re-instructed to be like, yeah, I understand, like please, please tell me what to do. I'm happy. And you lose a lot of conflict, which is useful.

Like when you do a code review, I don't want someone to be nice in the code review and give me platitudes or like, you know, like shit sandwiches. I want something that just like challenges every line I give it.

And so if you say like make a theater play code reviewing my code with like six different people just like arguing about it, then suddenly in one inference you have like a really good code review instead of someone telling me to handle my errors.

So that's kind of the main thing,right? It's like thinking of them as culture, not as technology or as cultural technology. And so the way to approach is I think of them as translation engines,right? Like there's many, many different kinds of languages.

Like when I talk to my mom, I have a different language than when I give a talk. I have a different language than when I write Python. I have a different language than when I try to write like a math paper.

And so some of them are like from less formal to more formal. Once you step into the formal language world, you can use a computer to interpret it or do stuff with it more easily. But you can also, you know, transfer from one domain to another.

You can go from the T-Rex domain to the real-time operating system domain or mix both together. It's just tokens,right? But so what that means is writing software now is about decomposing the problem that you want to solve into language translation steps and then let the model do these language translation steps, understanding which of these steps are too much for a certain model or how to do them, like how to prompt it to say transform, you know, when it outputs JSON, it doesn't output theright JSON.

You might put in like, this is important for your career. That's like kind of a cultural technique to get it to output JSON, which is a little bit weird as a programmer. So here's a bunch of examples and like I'm not going to go through every graph, but like feel free to look at them in the handout.

This is, for example, an example of how to transform meeting transcripts,right? Like you're on Zoom or you have like your software planning meeting. Everybody interrupts each other. It's like a big chaos. No one's going to go back at that transcript and try to understand what was discussed.

But now you can,right? And so the steps to do that, because if you just asked like, tell me what we discussed last week, they're getting good at that. Like it's a little bit scary, but also it used to really work not very well because you have like 28 pages of transcript with a lot of like just people interrupting each other and using colloquial language.

So the different steps to do it that I like doing is like first I want to get a summary because Claude has gotten really good at like figuring out actually most of the information that's in there as a bullet point list.

It's not good at details. I just want like which topics did we discuss? Bam, five bullet points. Then I take this translation step,right, from transcript to five topics we discussed. And then for each topic, I go back and I say like, for each of these topics, tell me exactly what we did.

And now it has like a direction to know what to grab for in this huge transcript that's only relevant to it. So you get this like pretty good result. People call that like there's plenty of papers about it, I think,right?

Like with complicated names or whatever. The technique is really just translation steps. It's like from a topic to get all the details to that topic. No need to have like really fancy names about it. No need for fancy prompts.

You just go back, paste the summary and then for each point in the summary, like give me the details. Then the next step is once you have these technical details and you take the transcript again, you say like, oh, give me action points.

Like who's going to do what, which we discussed in the planning meeting and just give me a list of like, oh, Enrique is going to do that and Sarah said that maybe we should do this. You know, and you get like a clean list of action points for each of these like fairly detailed technical summaries of each transcript point.

If you do that one chart, there's no way that that will work. But if you decompose in these like little translation steps, then you get it. And at the end, you can just take the action points and the technical details we discussed, maybe paste some of your code base in there and create the GitHub issues, which already have like a plan laid out.

They're assigned to theright person. You can ask to add tags to it. The way you do tags, like name them correctly with the whatever ticket naming convention you use. So that would be an example of like a workflow for transforming transcripts into GitHub issues.

No need to really automate that. Like it's just a list of things you can paste. And then in the conversation, if you're seeing it veering off a little bit, you say like, oh, no, please tell me what Enrique said about XYZ.

I remember it. I don't want to script and like kind of formalize that. I'll just put in transcript tomorrow. I'll forget about it. So

I have a couple of scripts to help me do prompts and I'll get back to those, but I zero shot everything. Like I haven't done agents, whatever people call agents. So that was an example. You can do other techniques,right?

Like take this meeting transcript and the existing docs and code and just say like, oh, we talked about refactoring this thing and now give me a concrete plan. Like tell me which API should be renamed to what based on what we discussed.

I don't want to do that by hand. It's like I need to look for the functions or whatever. Just paste all the names. Say like, how do you rename them? Write the script to rename them with said, you know, you can add in like the technical details we worked out,right?

From the summary, we had all these like, oh, we discussed this way. You can maybe put in like your RFC guidelines. You know, this is how we write RFCs and then you get an RFC out of the refactoring planning that we discussed in the meeting, but with theright function names, maybe with a script or an actual refactoring that you can put into a test branch, for example.

And then you have an RFC that people can discuss,right? And then you can go back, RFC comments, just put them back with the RFC and the technical details in the summary and just like update the RFC.

So this one's a little bit more meta.

And I have to see how much time we have. This one is like once we have the GitHub issues,right, from the transcript, I can formalize them because just like output them as YAML, like make me title, tags, assignee,

ticket number, link to your whatever. And then you have, now you have a formal representation that a computer that we can write programs for,right? So then I ask it like, oh, make a shell script to just take this YAML and create the issues for real.

Like just use the GitHub API. Models are usually pretty good at zero shotting that. If they mess up the GitHub API like they used to do, just paste the API in there. That will usually work. And then you can take the script, which is pretty simple, but iterate on it,right?

Like add a help, add a README, add more flags, maybe confirmation, add like colors and little emojis, make like an NPM package for it that everybody can install. That's all like little translation steps going from a JavaScript to the package JSON.

It's like not really reasoning. A model can do it great. But when I do it, like it's really annoying and tiring. And then I'll like forget that you have to write like ID uppercase and ID lowercase and it costs me the afternoon.

So what you can do in three minutes,right? Like because I was talking about it, but if the model, if I just paste it into the model, I would be there. I would have the script. It would probably work.

Now you can actually talk, okay, now that we have the script, how do we change our workflow within the team, within the company, within the open source project to just automate it,right? Like we have a meeting transcript where it's like create GitHub issues.

Boom. It's all kind of automated. You maybe want to look at the YAML before you press fire,right? But there's a decent chance that at least you get a lot of boilerplate and you can replace like a couple of LLM fringe speak.

What you can do now is just like take the script and the Slackbot API and say like, make a Slackbot for this. Notify the people that have been assigned the tickets. You take your GitHub transcript, paste it to the bot, and then everybody gets notified of their GitHub tasks that come out of it.

And then I did some meta, meta stuff, which I'm going to skip.

If we go back to this workflow now,right, which probably took us like maybe an hour to build all of this, a good chance that, you know, if it's annoying to get like the Slack access API token or if you like some OAuth stuff, you just like write me a script to get a Slack API token.

I really don't want to look up how to do it. Just give me the URL.

But what you can do now is that as part of these prompts that we give it, every individual developer you have on your team, they can like put in their preferences,right? They could say like, well, I don't know about this library, so explain a little bit more when it goes about this thing.

I like bullet points. You know, my background is this and this and this, but currently I'm busy with this other part, so only tell me about parts that overlap. You can, that's just like a TXT you can have on your GitHub,right?

Or somewhere. But every time the Slackbot, for example, is going to create the GitHub issues assigned to you, it's going to put this thing into the context and then you will have like tailored documents that explain how to do something.

So, you know, if you have the intern and you know the intern doesn't really know much about APIs or like HTTP, you can just put in their prompt, like explain HTTP as it relates to this ticket or explain our internal infrastructure deployment script.

Onboarding, super easy. Just have like an onboarding TXT that's ready to go explaining like the architecture of your code base. Just put that into the new developer's GitHub issues and make them match,right?

Easy. It's just language. So for the sake of time, you can go through some of these. Maybe I'll show one that's really funny is like, do you want to write API clients? I just don't even look at the docs anymore.

It's like I record, say I'm in a browser and I click on things and I want to write like an API for the Claude UI, which they don't want me to do, but they have. I just record the entire session, clean it up a little bit, and then just say like, well, write the documentation for this

mass of XGP requests that I have. And once I have the API documentation, I'm just like, well, write me like a proxy. And then I have a proxy for Claude AI's newest thing.

And

let me see. This one's great for learning. You have a topic you want to learn. Get the search result, existing knowledge that you have, maybe some code base of preferences that you want, create a wiki entry for your personal knowledge base.

So you want to learn about advanced TypeScript, Lambda things. Just look for it. Take your existing knowledge saying like my background's in Ruby. Give me some exercises and project ideas to build. Maybe use the guidelines from my company to format these exercises so I learn two things at once.

Create a prototype. And then maybe once you have a prototype, you just like paste your actual API and say like, well, just make it real,right? You get the actual implementation. But those are all, if I just said like, you know, solve the problem and teach me how to do this, it's not going to work.

But if you decompose it to these little steps at each point, not only do you get value of doing it because you see the output, but you also get like artifacts you can share, for example,right? Like once you explain how Lambda TypeScript stuff works and your colleague also wants to learn it, you just give them the exercise or transcript, you can help them.

Maybe you put their personal prompt into it and their background's like in Lisp and then it will just update it. Allright. So you can take some of these graphs in the transcript. I'm going to give you like 10 minutes really sharp so that I get to the mind melting stuff.

There is a section about DSLs. If you know about DSLs, if you don't, you really should look it up. LLMs are great at creating their own language. So you can actually create the target step itself. You can say like, well, I want to write reports, just like create a language to write reports.

And then you can target that and, you know, implement the interpreter for that language. So I really, there is an example set of prompts in here about DSLs where I create like a text adventure game. I did that, you know, this morning at breakfast.

Try those out. It's like on page, page no number, but you'll find it. Creating adventure games, three lines of prompt and I create like an alien-inspired adventure game with like a full source code and stuff. So yeah, try stuff like that.

A good target is also self-contained HTML plus JS plus whatever your favorite library is.

There's a couple over here. And then if you have neighbors, I recommend just working with your neighbor as well, like sharing the handout and communicating. It's like,

yeah, I think that was the last handout.

It's also as a PDF on the repo. If you want to see the repo again,

I guess it's in the Slack channel. So the Slack channel is LLM workshop working programmer.

I'm going to accelerate a little bit, like four minutes. You know, you should be able to write like five programs in that time.

I'm serious,right? Like try out my prompts and just replace, create a YAML DSL for the program you want to write.

Where's the Slack workspace again? It's called Workshop LLM Working Programmer. If you're in Slack, are you in the AI Engineering Slack? Yeah, just look for Workshop and then you should find it. Yeah, it's the channel.

Yeah, actually, let me put that back. Here's the Slack. Here's the GitHub repo.

Allright, I'm moving on because I've got some really important stuff to show you. And one is, so Josh, no, Brendan just asked, like, have you found LLMs able to work with PCAP files? I'm like, not really, but like create a program that outputs the PCAP format in a format that, you know, in a language type that the model can understand.

So I said, write a YAML DSL to represent interesting DNS traffic out of the PCAP file. That looks pretty useful,right? Like this is language that the LLM can do well with because it's called DNS traffic, destination IP. It's not called like byte0x5 or like DNS underscore, I don't know, whatever.

And then I'm just like, write a Python file that takes a PCAP file and outputs that YAML. And now I have a script that I can give every PCAP file, outputs that into something that the LLM can use, and I can put it into my LLM agent loop or whatever.

That's what I mean with 10x engineering,right? Like in the three minutes here, I wrote another one because, so

what Josh was asking,right, is like, oh, when I put in code base in my thing, like what does that mean? It's like huge. But usually when you interact with a code base as a human, you either interact with a small certain part, you read some docs, you maybe look at the function APIs.

So just create a YAML DSL to like grep your code base for interesting stuff,right? So maybe I want to get all the functions in Python files as well as the classes. So I said like invent a YAML DSL to create a program that finds functions and classes in my code base as well as the doc string.

So it outputs this. I'm just like, yeah, that's a cool idea,right? If I don't like the idea, just regenerate until I have a language that matches what I want. Here it added like recursion, which I like. And I was like, well, I also want like the Markdown files.

So it like made up, you know, two things for the Markdown. And then my third prompt is just like implement it, please. And then now I have a Python file with a YAML that I can tell how to look for stuff in my Python code base.

And then while now I can run it on my massive code base and I just have the function titles, the doc strings, the Markdown document titles, and that's a good context to give to my DSL,right? Like took me 30 seconds.

Might maybe not work too much,right? Like in real life, I would say like this is an afternoon project maybe to make it really nice with a README with unit test with like examples and install package, whatever. There is a graph on how to do this.

Like this is maybe one of my main techniques is like writing these little fragments that I know are good context to do a certain task. So some of them could be like, oh, write a new widget to do something and then I'll actually add like grep for these widget names, maybe add the documentation out of the Git log of last week.

I'm going to create a how-to and then that's going to be the how-to that I'm used for prompting. And now I have like a little script that I can, every time I want to write a new widget, now I just paste the result of that script in so that it's the live status of the code base.

And then usually it does pretty well,right? Like if you give it like first write the props, write this and write this. So this thing, and I do this for third-party libraries,right? Which often don't have like good documentation. I'll go over that code base and just generate good documentation that I just need for, you know, for the task at hand.

And then it takes me like a minute usually. Sometimes it'll take me like the whole day. I'm like, what am I doing? But pretty good technique. There's a graph on how I do it. And now I'm going to quickly switch to the wild stuff.

So I went over DSLs. So word simulation,right? Like everything you tell the model is what it thinks reality will be or whatever it uses as patterns to grep its training corpus. If I tell it, you are a wizard, it will pretend to be a wizard.

If I tell it to write poetry, it will write poetry. If I tell it like time travel exists, it will say like, sure, time travel exists. So anything that's kind of in the training corpus, so things that humans can think up so that you can think up, the model will like usually know what to do with,right?

That also applies to like formal languages. If you say like your world is like def function paren, whatever. I was like, yeah, cool. Def function paren, that's my world. So one thing you can do, for example, for code reviews,right?

It's like I tell it, you're a reality TV show, Code Survivor. And like each participant in this reality TV show is one of my variables or functions in this piece of code. Now have them like fight each other and figure out who's the bad one,right?

And then vote them out, like who's got the most bugs. And

so this is based on the concept that whatever you tell it it is, I'm going to use what's in the training corpus, like reality TV drama. So I'm going to write some bad PHP code because all the code I write is great.

And I have a GPT for that that I shared. But really, you can like write it's the idea, it's not the prompt. I don't really do any prompt refinement engineering except when I want to repeat something in a loop with a small model.

I make typos and all that stuff. So now I have this bad PHP,right? I just pasted it into my Code Survivor thing. And this has multiple contestants or like different variables. And I told in the format in the prompt, you know, just like each one gets to say one thing and just like attack someone else.

Identify the conflict and then there's some kind of format in there. And one of the advantages is like A, I get conflict,right? Even though the model has been trained to be nice, this is like standard red team tactics to just exploit LLMs.

This works really well to do this stuff. Just have it pretend to be a different world. It allows me back when ChatGPT was like really annoying and said like, please fill in the rest, dot, dot, dot, which it doesn't do anymore.

This would allow me to select, make six rounds and it would do six rounds,right? Like it was so strong in the prompt that I could in one inference,right? Like compare that to code review my PHP. This has already a lot of concrete information in it and I can, if I don't like it, I still like write three more things and there is the comeback participant.

And it doesn't need to be a variable,right? It's like

SQL injection. It can be a concept. Like concept can be a character. You can give them like a personality.

And then SQL injection will start to be like dissing other people, which is pretty funny.

But it's useful,right? Like this is proper engineering. It's not like it's not fun. It will actually tell me like you should use MySQL close when you finish this and you forgot like I'll vote you out.

And so once you have all this garbage, which I don't even really need, it's like funny to look as it scrolls by because they're like, hey, we're just sanitation is not our job, whatever.

And the, oh my God, this just goes on for pages and pages, which was really hard for GPT to do back then. Like it would be like, oh yeah, I think this variable is bad. Like please handle errors and I'd be like, what?

Then you can do like write a sober code review report at the end. Just based on everything that's in the context, like now strip the reality TV part out of your language and just keep

that stuff,right? And actually suggest fix and I could then turn that into GitHub issues or whatever. You get the idea. But so this is the world simulation thing is that you can really make everything up. Like one technique that I have is time travel because when you correct the model, it's like useful information,right?

It tells the model what not to do. But if you have 25 messages telling it not what to do, it will confuse it a little bit. So I'll do my 25 messages. I'll get to theright code at the end.

I'll say like, summarize what you did and then, thank you.

It will be like, summarize what you did. Then I'll rewind the 25 things and I'll paste in, like this is you from the future. This is what you tell your past self. And now I have like this pretty good, and it knows what to do with time travel, people coming back from the future telling like your young version what to do.

And like it's able to generate better code out of that.

Weird engineering concepts.

So try it out. Like this is not, like this is not fun. This actually works,right? Presidential debate is great. Greek drama is pretty funny.

But where it gets really wild are prompts like, you are the application I want to build. Please start,right? And now suddenly like going to pretend to be what you want to build. And so you can tell it like, please output your UI as like a concise DSL.

I was like, well, I've got like a main widget. I've got like a text area and I've got this and I've got this. That's formal language that you can already like visualize. But also it can be like, well, now I want a sidebar.

And it's like, yeah, sure, you have a sidebar. So like click the third element of the sidebar and we'll pretend that I click the third element of the sidebar,right? So just to show you,

you are an application to plan flight, travel, and sightseeing for me and my kids. Output your UI as a concise DSL YAML. Start.

So, you know, regenerate if you don't like it. If you want it to be like React widgets instead of like this high-level description of what it is, feel free to do that. Like you can steer it,right? It can be like, no, I don't want like my data.

I want like actually the UI here. I don't want to rewind it. I'm just going to correct it. And then once I have theright thing, I can like gaslight it and say like, this is what you generated. Like, no, I want some HTML markup kind of YAML DSL with the actual UI.

Zero prompt engineering. Like I don't know what I'm doing, but you know, this is much more closely to what I had in mind. It's like, oh, I want the actual widgets.

And if I were to, you know, write a JavaScript that parses this and then as callback handlers, just like output, press, I press the button X,right? It's like if I want to do like open the tab for day two, I don't think there's even tabs for it, but it'll just be like, oh yeah, I've got tabs.

Like, of course.

And so you can interact with it and it's kind of like, is this code? Is it not code? But at the end, you can say like, oh, write a spec,right? It's going to write like a concise README kind of spec and I can be like, implement yourself,

which this is new. I haven't really played with it too much. I came up with this two weeks ago, but it's kind of wild because I'm into film photography. I wanted an app with like three timers, tell me which chemicals to mix and what.

And I built it in five minutes. I was like, this is wild. Like I just said, like I want an app to help me develop film with this process and like fill it in with theright chemicals and all.

And I was like, no, I want like two parallel timers that, you know, loop. I was like, yeah, sure. And at the end, I was like, write a concise HTML, JS self-contained prototype. And then I can pass that on to the graphic designer or maybe the graphic designer does that all day,right?

And he's like, no, I want a sidebar. Wouldn't that be nice? And then tomorrow he'll just send me the transcript and I'll be like, okay, I'll make this work for real.

So there we go. This is, you are an application. I showed you the Code Survivor. This is your turn now. Five, six minutes maybe. Just like invent an application, play with it. Like you're an application. Use a concise HTML markup, YAML DSL, start and then see how far you get.

I'll actually interrupt this. Like I think you afterwards, you're going to have fun this, but I had a really interesting question come up. It's like, how do I manage all of this,right? Like, because I don't know why every LLM company doesn't manage to make like a decent history browser.

Like it's really not that hard. I could generate one in two days probably. That's like has tags and allows me to search. But no one manages to do that. I don't know. I don't know why. So what I use and I would recommend not using my tool,right?

Like don't take my tool as the thing. I break it off and I.

Sampling?

Yeah. Like the tool to manage.

Moments?

Moments is like you're managing text files and files together or maybe manage shell scripts. So my most used tool is called Prompto. I literally generate it once. It's a 100-line script. It will go over all my repos,right? Like I have a config files where I say like, look into all these directories.

If it finds a text file or a script or a YAML template, it will just list these. It's horrible. It has no pagination. It's whatever. But so once I have like a context,right, for like my library, when I want to do something with parameters, I have like a little script that gives me the current parameter API and I can call it,right?

It's like if I do Prompto get glaze parameters, it will find, it will look into all these repositories like super inefficient, no indexing, no caching. It will find this parameters MD file and it will just paste it. Or actually it grabs the Go thing and just outputs.

It's super ugly,right? Like there's no structure in this except dash, dash, dash. But so now what I can do, I have another tool that just, it's really just like a command line LLM prompter. It's like HTTP request with no format.

And I can pass it this as a context,right? I've got a couple of like pseudo things which I wrote two years ago, a year ago. They're not great. I can print a prompt that it's going to output and I say like, oh, create a parameter

for URLs. And so it will just paste everything together in like a big blob. And at the beginning it says like, you're a great, you're a great Go developer. Actually, I messed it up. It actually says PHP, so whatever.

And then it just like literally says like, create a parameter for URLs,right? In Go. And then if I actually run this, it will have, it will use my API. It will do a good job. Like this is how I just add stuff to my code.

It will have theright packages. I can literally just paste this and it will probably run. Once I have this prototype, then I have other scripts to say like, integrate this into the framework, like add the documentation style I want and all of that stuff.

But so this tool pasting strings together and just like finding scripts in your, in a directory, they're like nothing scripts,right?

So that's how I manage all my fragments. And we have eight minutes left, so I'm going to skip to the final part, which is more experimental. It's like, what are these, like you realize now that I'm like kind of not joking with this 10X, because during this workshop, I wrote like six, seven decent programs,right?

They maybe don't work too well, but like you can guess that they're not far from something that's of decent quality. And then if I want to add unit test and whatever, I can, you know, I can take an hour at lunch while watching Netflix to actually do it.

Sometimes it doesn't work. So this is like a big part of it is like no one to step away. Don't,right? If it doesn't work like within 10 minutes, don't force it. Either do it by hand like you're used to.

It's fun. Like we all love programming. Maybe go for a walk and come back when you realize which language step it didn't understand. That's a really important one. I still get into full rabbit holes where I just like don't get the task done that I actually want to do.

I just like generate programs that don't work to solve the task. And it's so knowing when to step away, because it's like breathtaking,right? Like I generated seven programs. Like what am I going to do with that? Like I can't mentally manage seven programs within a time span.

Like I can't do it. And I really have this problem. It's like I have like an experimental repo where I put in all my scripts to kind of, I don't even, I don't even bother putting them in,right? But I have like so many and some of them are like, are like 5,000 line code things with like crazy features.

And then I forget that I wrote them and I write them again and I go back and it's like, oh, you already did it. It's like Shopper approved customer management things with like parallel worker queues and I wrote it three times.

Oops, sorry. So what, however, doesn't go away, I think, is like fundamental and practical knowledge. Like I know how a real-time operating system works. So it makes it really easy to see if the model doesn't get it,right? Like I'll be like, no, don't do it in assembly.

Like do it, use this pattern or use this framework. If I didn't do it by hand or like learn about it, I wouldn't be able to do it. I would go into a big rabbit hole. I wouldn't get an RTOS.

But that's the difference between someone who knows RTOS and generates a new one with an LLM and can maybe do it in an hour and someone who will just like after a week just get nowhere. Nothing works, nothing compiles, the assembly is broken.

So fundamentals, super useful. What you can't totally forget is APIs,right? Like I really don't care about Amazon's CloudFormation API to do XYZ. Like I really couldn't care less. It's not knowledge I want to take with me into my retirement.

And that's gone. But however, the knowledge of how to deploy, you know, to do code as infrastructure for functions as a service, that's like really useful pattern knowledge. And I can take that knowledge and apply it to like DigitalOcean and to like Amazon and Google.

What I don't need to do is like figure out after 10-minute deploys that I forget to put like an IAM rule somewhere. Like I really, I really don't care. Some people do. Like and those are going to have a hard time, I think, in the next few years.

What also, while practice already went over it, but as soon as you think about humans in the loop,right? Like because language is always going to be useful when interpreted by human. I don't care what the LLM, it could generate like 10,000 programs that don't work and like what are you going to do with that?

So everything you generate with the LLM, either it's like to kind of help the LLM along to finally output something that's useful to humans or it's directly useful to humans,right? Like a README is useful for humans. It so happens that because LLMs are trained on human language, it's also useful for an LLM.

But if you make it nice for humans, it will work well with LLMs as well,right? Like if I have like, if I go on for three paragraphs about how error handling is useful, like that doesn't matter very much for my senior software colleagues.

So I can strip it out and now have a smaller prompt and it's like going to be more focused. So if you always think in this like language decomposition things of like how would it work for humans? How would it work for the user?

What do I actually want to do? That's like a good engineering skill. It was already before,right? But now it's actually in the small is also pretty good. Divergent thinking. I don't know how to teach creative video, whatever, smoke weed.

But like thinking about like these weird things of like, well, I'm going to make like a reality TV show about my code review. I don't know necessarily how I came up with it, but it's the best code review prompt I ever found,right?

It's, I don't know. I don't know how to come up with that stuff. And what's useful there is to look at people who are not programmers because they have amazing prompts. Usually programmers are like, we're like very focused on like, oh, we know how to do it and it's like it should write code.

And then someone comes along and it's like, you're an alien that writes code and it actually works. That's why people who have no programming knowledge are able to build like full apps while a programmer comes in and says, like, well, I forgot to do the for loop check and this is worthless.

And then in the meantime, the non-programmer is like, well, I've write an Android app that like allows me to fill my daughter.

Abstract thinking. I think a lot of it is like once you build something, knowing how to build the thing that builds the thing or find like the deeper abstraction of it and then focus that abstraction, I think is pretty useful.

My background is in common list a lot. So there's a bunch of languages like common list, Ruby, Haskell that are very focused on like creating abstractions, manipulating the language that actually solves the thing. I think that's like a pretty interesting thing.

Not that you should write common list for Ruby,right? But it's like getting familiar with the concepts that make up the community around that language of creating compilers or like creating like dynamic dispatch DSLs or finding like mathematical abstraction of how to do control flow or so that that's really useful.

And that kind of goes into language design. It's not just

like programming language design. It's like any kind of language design. It's like our head, like my favorite format is like title, one sentence, code example. That's like a language design thing. It's how I like to have my docs.

So that's it. We have two minutes. Maybe there's some questions. I guess I can take live questions and then repeat them maybe. Oh, there's a microphone. Great.

Thanks so much for the great session. It's amazing to be here in a room with like-minded people who will be probably geeking out on all these things for so long. And thank you for sharing your findings. Couple questions real quick.

One of them is, are you finding yourself working with other people in your team kind of spearheading some of the incubation portions of work following these techniques and then the cavalry comes behind you to build the actual thing?

Have you found yourself that kind of role? And then the second question is, I see you can push yourself going really far with all these techniques, but how much have you tried using some of the auto GPT kind of metaphors as well?

I'm curious. Thank you.

So I just recently joined a team I was like kind of solo for a long time and it's hard. I don't know. I don't know how to bring people on board with like how I do things where I'm like, allright, 5,000 lines of code an hour and then I throw them away.

I don't really know how to bring that into the team currently. And then for the second question, I guess that's going to be a last one because I think the time is pretty strictly enforced. What was the second question?

How much do you rely on auto GPT?

Oh, zero. I just like zero-shot stuff and then I'm in the chat API. So I don't use, I use Copilot for, you know, like just as Copilot is. It's a very different thing. And then that's it. It's a wrap.