Building with Anthropic Claude: Prompt Workshop with Zack Witten

Allright, good afternoon, everybody. Thank you all so much for joining us. We have the enviable position of being after lunch. I'm seeing some cookies on the table still, but thankfully you're not here to listen to me. You're going to be riveted by the prompt doctor who's going to come up in a second, so I'm expecting no sleeping on the table, but you never know.

Excited to be here. I'm Jamie Neuwirth, I lead our startup team at Anthropic, and I just wanted to say a couple of quick things before we got going here with, again, the reason you're here, the prompt doctor. We've had a lot of really exciting releases just in the last couple of days, some in the last couple of hours, and wanted to just put these up there to highlight some of the cool things that we're doing, but also share how a lot of folks, not only in this room, some of your peers, maybe folks back at the office, can work with Anthropic on not only some of the prompting, of course, work we're going to do here, but just helping you grow your business and the really cool things you guys are building with Claude, with

LLMs, on top of AI. My team is here specifically to help your company grow and scale the business, whether that's getting access to higher rate limits, getting access to folks like Zack, who's going to be up here in a moment, learning more about what we're doing from an early access perspective.

We want to work with you and empower the next wave of really exciting AI companies built on top of Claude. And we're helping, from a product perspective, with a couple of the releases you see here. Has anyone tried 3.5 Sonnet yet?

Love it. Very cool. Really excited. Yeah, thankfully Zack has tried it as well, so we're in a good place. Really excited about what we were able to release there as well. Also, just today, or excuse me, I guess Artifacts came out with that, with our Claude teams and Claude AI plan.

Artifacts is this really cool tool that some have been playing around with, making video games with a couple of lines of text that turns into code, but actually a lot of really cool business use cases from a diagram perspective.

I've seen a lot of cool pharma companies using this and thinking out, how can I put what's in my head on a gene sequencing kind of discussion onto something like an artifact, share that with my peers, all the way to prompting cookbooks, helping, again, folks like yourself.

I would imagine a lot of you hopefully will be featured on their website coming up with what you'd be able to prompt for these use cases in your companies. So just some ways that we've, you know, things that we've come out with over the last couple of days, couple of hours, when you think of Claude teams and projects, and then ways to connect.

Feel free to reach out at sales@anthropic.com. We have our head of dev, Raul Alex, here as well. So we really love this community, love staying engaged. We're really excited about what we've released over the last couple of days to be able to help you again in what you guys are building.

And so I'm going to leave all that to the prompt doctor as well to get that, to really make sure that we can help. And Zack, why don't you come on up here and see what we can do?

Thank you all so much for joining us today. We're really excited.

Thanks, Jamie, for the intro, and thank you all for coming. This is really awesome. I had no idea this many people were going to be here, so thanks. Okay, so not going to be much talk. It's mostly just going to be straight-up prompting from beginning to end.

Did make a couple slides, so mostly about what to bring. So I set up a Slack channel in the AI Engineer Slack. It's called Prompt Eng Live Workshop Anthropic. So that's where you can upload prompts. And what we're going to do on stage is I'm just going to sort of look at them.

I'm going to read them. We're going to test them in the console, the Anthropic console. We're going to see if we can get better results, and we'll just sort of, like, try to learn as we go. So this is something that I do, like, internally in our team Slack quite a bit, but I've never done it in front of this many people.

And it'll be exciting. It'll be fun. Might be some hiccups along the way, but hopefully you all have a good time too and maybe learn something. I know I'll definitely learn something. So what kinds of things should you put in this Slack channel?

So you can put a prompt template. So a prompt template is kind of like a prompt. Actually, I just realized I don't even need this mic. Okay, so you put a prompt template, which is like a, it's like a prompt, but with spaces where the variables are going to go.

And the variables, they're going to be denoted with these double brackets. So in this case, it's like this document part. If you don't have it in this format, that's fine. We can figure it out. This is just like the ideal.

So this is like the prompt template. This is like the kind of thing you'd put there. And then you can also have a couple examples of cases where it doesn't do what you want, and that will give us the, like, direction as far as, like, where we want to go with it.

I might also, like, ask you questions out loud if I have questions about, like, what kind of output is good or not, or I might ask questions in Slack either way because it's easier. We'll have to kind of figure that out as we go.

Okay, so that being said, we're going to use the console for iterating mostly, although I might use a couple other tools, like Claude for Sheets, which is like a spreadsheet where you can call Claude. Okay, so yeah, let's see what we've got in the Slack already.

Okay, we have something here. Thank you, Gordy. So you're an expert patient. So let's put this into the console, and then let's take a look.

Okay, and I'm just going to go through as many of these as we can get through in the session. And yeah, this is pretty much what it's going to be. So first of all, we can probably capitalize all the

sentences.

Does that matter?

Does that matter?

Yeah, sorry. Hi, I'm Gordy.

Thank you. Yeah, yeah, yeah, yeah, yeah. Perfect.

Yeah, so does it matter having capitalization?

I think so.

Okay.

A lot of things, like prompt engineering is like, it's very new,right? So like, we don't know for sure. Somebody out there might have done a study where they, like, conclusively show that using capital letters and, like, using, like, better grammar, fixing grammar mistakes help.

I have, like, anecdotally found this in a few cases. I also have read some, like, quantitative stuff showing that, like, typos do hurt performance. But I'm also just, like, pretty obsessive about this stuff, so I just fix it.

And I think, like, it definitely doesn't hurt. Okay.

Can you zoom in?

Can I zoom in? Great question. Is that any better? No. A little more. Okay, is that any better?

Okay. So first thing, let's put information in XML tags. So we can go like this.

Why not, like, Markdown?

Why XML? Why not Markdown? Another great question. So Claude was trained with a lot of XML in its training data, and so it's sort of seen more of that than it's seen of other formats. So it just works a little bit better.

So this looks like all the information here. So we have the medication review will be okay.

Can you run it before and after? So before the first iteration and then the final iteration and see how it compares?

Yeah, great call. Okay, actually, let me undo everything that I've done so far.

Okay, so we can run it here. And then now in the console, it's asking us for the user input. So do we have a user input?

Yeah, give me samples in Slack.

Okay, perfect.

So who are you? Let's do this one. Why do I need to, yeah, we can do them both.

Who are you?

Okay, so Gordy, what do you think of this?

It's way too long. This is a conversational agent, so no more than one sentence.

Okay, it's too long. We can probably fix that. So

we can also use this evaluate tab. So let's just, like, add all the test cases here. So this is the evaluate tab of the console. I'm also going to be doing, like, some showing off of the console features because I think it's a cool tool for prompt iteration.

There's also some secret new features that I might show. We'll see about that. Okay, so then we also have, why do I need to do this experiment?

Looks like it added a bunch of new lines. Let's definitely get rid of those.

And then we can get this next one. Can I schedule it tomorrow instead?

Cool.

So I hit run remaining. And this is all running through dev, sorry, through, okay, so we have, why don't you do this appointment? So this looks pretty long as well. And here we have this, like, I apologize, but I don't have information about scheduling availability.

Okay, so

is that true? Is it true that we don't have information about scheduling?

No, we are always available. Make it easy for that. So 24-hour availability.

24-hour availability. Okay, allright, so this is like the version one. Now let's make some changes. So first of all, I'll, actually, I'll try to do things like roughly in, like, some order of importance.

So maybe I won't make you all sit through the capitalization, even though I, like, would definitely do that. I'm also going to add a new line here just because I think that's, like, more normal, what you'd see in, like, a written document.

You'd have a new line. We'll close the information tab.

Claude can fix capitalization?

What's that?

Couldn't you get Claude to fix the capitalization?

Yeah, I could, actually. That's not a bad idea. The one thing I wouldn't feel completely confident of is that it would, like, exactly transcribe the rest of everything, like, word for word. I think it probably would. What I actually might do is just, like, have Claude write code to capitalize every first word of the sentence.

Then I'd be worried about edge cases, like, what if there's, like, ellipses. But that kind of thing is definitely useful. And, like, I definitely use Claude a lot in writing prompts. For instance, like, we have, like, a Claude tool that, like, helps complete code, basically.

And I do a lot of prompting in that IDE because, like, especially with, like, very nested XML tags, it helps a lot just, like, suggesting the closures of them, which is, like, pretty obvious, but still takes a long time to type.

So yeah, if you have any sort of, like, copilot type thing, definitely that's, like, a good environment for writing prompts. Okay, now let's do the same thing with this instructions.

And we can do this. It looks like this one, like, didn't get a number, so let's, like, do that.

Where do you know to stop with XML and leave it as plain text?

Yeah, so the key thing in terms of XML, I think, is just, like, really, XML isn't even that important. The most important thing is just clearly separating the different parts of the prompt.

It's using it for the semantics of what this section is and so forth.

Yeah, exactly. It's like, here's this stuff. Here's this other stuff. Like, if we wanted to, we could do something like, like,

like, I wouldn't do this, but, like, I think it would probably work fine.

Yeah, it does.

Yeah.

Okay, so this is all fine. Let's also do the same thing with user input.

Now we can run. We can go back to the evaluate tab, and we can hit rerun all, and it's using our nice new prompt. Still looks pretty long. We can also see how it does on the last case where

okay, so here it still said I don't have access to the specific scheduling information. So let's try and fix these two things. So first of all, we can make it shorter. So do we have anything here about, like, making it shorter?

What's that?

Rule seven.

Rule seven, okay.

Be concise and offer only relevant information. Oops, let's actually do this. Don't want to miss a number here.

And

be concise. So, like, and offer only relevant information. Each response should be

or let's be a little bit, like, less prescriptive to give Claude, like, a little bit more room. Like, if we say, like, every response should be, like, exactly three sentences, that might be, like, a little too constraining. I'm just guessing.

So we could just say, like,

So why is responses should be two to four sentences better than telling it to be concise? Concise is arbitrary.

Yeah, concise could mean a lot of different things to different people. Like, in some cases, like, concise might mean, like, literally only one word. In some cases, like, if you ask for, like, a concise book review, we might be looking at, like, you know, a single-page Word doc.

And that would be concise in the context of a book review.

So yeah, Claude is, like, trying to guess what you mean by concise.

What if it needs to be a long prompt? So we have to have a really long prompt.

Sorry, one sec. Go ahead.

I was going to say, he has a really long prompt. So if you have a long system prompt with a lot of detailed instructions, saying be concise, you're not going to get something super, super short.

I think that'sright. I think the tone of the prompt, so what he was saying if people couldn't hear is, like, the prompt is long, so the response might also be long. I don't think that's, like, definitively true. Like, you can have long prompts that give short responses or short prompts that give long responses.

But it's more like, if you don't say anything, it might pick up on some of those, like, context clues. You were saying something over here?

What if it needs to be readable depending on the type of question that's being asked? Like, keep it short for, like, things that because in human interaction,right, like, sometimes we give really short responses, other times we give long responses.

Yeah.

Have we trained LLMs to do that yet, or we're not really doing it?

Yeah, so let me actually get to that after we do this. So this two to four sentences, it looks like it's still pretty long. I think maybe that's actually, like, longer than necessary. So maybe we should make it, like, one to two sentences.

Let's try that. Never more than three. Okay, now we can try that here.

Okay, that looks better,right?

Yeah.

This is definitely shorter. Okay, and it also seems that it is giving variable numbers of sentences. So these were both two sentences, and then this one is three. So one of the questions over here is, like, can the LLM figure out that it should do longer responses in certain situations and shorter in others?

So it seems like it did that here. Okay, so then the next point was that in this case, it shouldn't say that I don't have access to the scheduling system or specific appointment times. What should it say instead?

It should say, sure, what time tomorrow. But intentionally in the prompt, I left out that it's 24-hour service. So this is a case where we're asking a question that's not present in the information it has.

Okay, so yeah, I mean, we could add something like, you're open for 24-hour service. But you're saying you want to test its ability to, like, figure out how to do it without that.

It's saying I don't know.

Oh, okay.

It is saying I don't know, which is good.

Okay, well, then we're doing great.

Allright, should we anything else that you wanted to get out of looking at this example, Gordy?

I think that, oh, about the structure. So does the order of the rules or the order of putting information then rules or rules first then information, does any of that matter?

Yeah, so does it matter what order we have these components?

Yeah.

I think it's better to put the information above the instructions.

Okay.

We've sort of found that instructions are more tightly followed the closer they are to the bottom of the prompt as a rule. This doesn't necessarily apply in all situations, so definitely test it out. That actually is, like, a blanket statement that applies to everything that I say, but particularly for that.

Okay.

Can I ask a question?

Yeah.

I don't know if you were asking all the questions or.

No, no, no, go ahead.

It's only bald men allowed, so you're good. Is there a significance to the exclamation mark? I noticed that it's.

Exclamation mark?

The exclamation mark, yes.

Oh, remember, I don't think I added that on or it looks like you added that, Gordy. Exclamation marks, just as they emphasize things for humans, they also emphasize things to the model.

Do you think that has more of an effect over the numbers or less, like, in terms of relative?

Ooh, yeah, I don't know at that level of detail, and I think it's dependent on context as well. But yeah, if you want to emphasize things, like capitalizing them or putting exclamation marks or, like, just saying this is extremely important, that all does do something.

So yeah.

So the tokenizer, though, I'm taking a look at some of your tokenizer code, but it doesn't seem like exclamation points actually, like, do anything really. Like, the tokenizer kind of combines them into the word.

It does something. That's all I can say. Just anecdotally, if you put exclamation marks in, it's different.

I guess, okay.

Yeah.

That's my analysis of it.

Allright. Okay, I think let's go to the next. How many got we got 60 already. That's pretty good. Okay, this is just a general question. I'll just answer this really quick. Is it in general for translations or multilingual output, is it better to instruct in English or the native language?

I think it's better to instruct in the native language if you speak the native language. If you only speak English and you're choosing between, like, a better prompt that's written in English versus, like, a worse prompt that's written in a language that you don't understand, I would probably default to writing it in the language that I knew super well.

But ideally, I think for the ideal prompt, you would find a native speaker of the language and explain your use case to them and have them write the prompt.

Can I ask a question about language?

Yeah.

So

for a language that Claude is not so familiar with, should we, like, maybe write the prompt in English, but let the Claude to translate the output into that language, or should we just write the prompt in that language?

So if it's I think is that not the same question that I just answered? Is it different?

Oh, so you said it's better to write in that language.

I think it's better to have the prompt in that language in general, if you can, if you can write a really good prompt. Allright, let's go to this next guy. So you'll be acting as a test reviewer. Let me pump up the size here too.

Okay, not sure if there's a way to I can hide this. Okay, great. Responsible for improving unit tests based on a set of requirements. Below is the project directory. Project path. Do not include any other explanation, conversion, or output besides the JSON.

Okay, this is great because it's going to let me show off prefills. So

let's make a new prompt here. Let's paste this in. We have to use double brackets instead of single brackets to get variables in the console. And then I think there's another but this JSON string who gave this prompt?

This is from Dan. The JSON string Dan, is that variable or is that, like, an example in this prompt template?

So in this case, that is just a template, and then we this is just the middle of a multi-agent workflow. So basically, another one of the agents is going to write the unit tests, and then that's going to be in JSON, and that JSON blob is what it's going to respond with, essentially, just using the earlier part of the conversation.

I put the sample output in red.

Yeah.

Test of your agent does not always return JSON only. Yeah. Like, almost all of this is just, like, workarounds for the fact that it doesn't always speak JSON,right? Like, you can see how many times we said that in the past.

Yeah. And then do you have an example input here?

So that first, if you go up a little bit, sorry, so that first comment is from the unit test writer. So that is the input. Like, the unit test writer writes a bunch of unit tests, and then the reviewer reviews it and makes them better.

So everything here is what I should put into.

Yes. And then you can see that second set. This is a good result where it writes JSON, which basically says, cool, update this file with these unit tests, and here's the modifications I made, that sort of thing.

Okay, so in this template here, where would the thing that I just copied go?

Well, so essentially, we don't provide it in line with the prompt. We just provide the conversation, and then this thing jumps in as a separate agent. So, like, the context window is going to have the unit tests in it, but we're saying respond in this format given the unit tests that are earlier in the conversation that you're picking up.

So the thing that you just pasted is, like, step three of a multi-shot conversation? Like, sorry, not multi-shot, multi-turn.

Yeah, yeah. The thing I just pasted was two shots,right? Unit test writer and then a unit test reviewer. And the reviewer is the one that's having the problem. It comes second.

Okay, so it would be something like this.

Here are some unit tests

written by a unit test writer bot.

Right.

Okay, and then we have this. Now, you didn't okay, so this is basically how it works.

Does this lookright?

No, it does. I mean, we just do this in sort of this larger conversation, not just as sort of a standalone prompt for this one.

Yeah. Okay, so then here I put the unit tests in.

Right, yep.

And then for project path, what sort of thing should I put there?

Anything. I mean, this is just, like, the local directory that's going to be modified, and so it actually has access to the files in that directory, and it'll fill in its own sort of what files to modify.

Okay, so then I'll just put.

That's fine.

Something like that.

Yep.

Okay, so let's see if it comes out with a JSON or not. Bated breath here.

Yep. We did do most of our tests on Claude 3 and not 3.5, so 3.5 is probably a little better.

Okay.

We haven't done.

Yeah, I mean, if it makes it more realistic, we could also switch the model version to use high conf.

Oh, no, I mean, we're going to upgrade, so I'd rather see it with this.

Okay. What was that?

I see the temperature set to zero and then intentional.

Is the temperature being set to zero intentional? Yeah, I usually, for knowledge work, I usually have the temperature set to zero.

Not 0.01. Like, I usually use that as a hallucination. I mean, I'm just a bit of a weirdo, but, like, is that, like, am I just totally out of.

I think using temperature zero, you'll probably get, like, marginally fewer hallucinations. Okay, oh, here we go. Okay, so it looks like in this case, it did output JSON, I think.

Yeah, that looks plausible.

Okay.

Very long JSON. I guess that explains why it was taking so long to looks like actually it even ran into the max output tokens because it didn't finish its JSON.

Aha.

Just to make this since this one is kind of going kind of slow, I will test it with haiku.

And let's also increase the max tokens to sample so that it doesn't run into that issue.

So what I'm really hoping is to get a case that doesn't output JSON so that then I can fix it and then it will output JSON. If not, I can still say, like, how I would fix it.

Yeah, that would be great. Just honestly, any comments you have just on how we structured things.

Okay, yeah. So I mean, this is, like, definitely, like, a big request from people is, like, how do I make sure the model outputs JSON? The most reliable way to do that, I feel, is using the assistant prefill.

So maybe some of you have used this feature before. Maybe some of you have, like, only used other models, such as GPT, that don't offer this feature. Something that you can do in the Claude API is partially prefill an assistant message.

So what you're doing there is you're putting some words in Claude's mouth, as we call it, and then when Claude continues from there, it assumes that it's already said whatever you told it that it had said, and that can help you get it on theright path.

So for instance, in this case, if we want to make sure so the classic, like, bad response from Claude when people give it prompts like this where they want to get JSON is Claude would say something like, I'll just, like, add another message, just have someone else to type.

It might be like, here is the JSON,right? Have people seen stuff like this?

And this partright here is, like, very annoying and difficult to get rid of. So, okay, so I have two strategies. Let me actually just give, like, the simplest one. They both, though, they both require a tiny bit of post-processing at the end.

So let's start by let's actually, like, take out all this stuff about make sure to only respond in JSON. That could be one way to get it to not do the to be bad. So we could just go like this.

Let's try to make it not do JSON. Let's get rid of all this stuff.

Okay, so here now maybe it will do the preamble thing that we don't want it to do. Perfect. Okay.

So an easy way to get it to not do that is to just take this and then put it in the prefill. So it thinks that it already said that,

like this.

So if we do that, just the JSON. So what we're doing here is you can think of Claude almost like a child who's just, like, misbehaving, and it wants to do something, and you're like, don't do the thing, but it just keeps doing it because it just loves preambles so much and it has this, like, innate desire to do them.

So one way is to, like, argue with it a lot, but, like, if you have a kid, sometimes you know you just have to, like, let them do the thing that they want to do, and then they'll get over it.

So in this case, that's basically what we did. We just gave Claude this prefill where we let it do the thing. So as far as it's concerned, it already did the thing. And then from there, what it's outputting is JSON.

Now, if you want to make this even more reliable, you can put this nice little bracket here. And then it's like, oh, dang, like, I'm really in JSON mode now. Like, I'm really my JSON has actually already begun.

So at this point, it's definitely not going to do the preamble. The only thing here is if you sample with this prefill, you will need to add the bracket back before you try to do your JSON.loadS or what have you.

Because Claude, since you told it that it had already said the opening bracket, it's not going to give you another opening bracket. Okay, so then another thing that you can do is

return the JSON in JSON tags. And then if we do this without the prefill, let's try it without the prefill.

You don't normally capitalize JSON?

I'm not a software engineer, okay? I'm a prompt engineer. I don't even know that it's capitalized. For all I know, that's just, like, an English word. But yeah, good, thank you. Okay, so here we see it did the thing.

So it gave its preamble,right? And then it gave the JSON tag. Everything within the JSON tag is JSON. And then at the end, it closed this JSON tag. So again, this requires, like, the tiniest smidgen of post-processing where you're saying, like, you just it's like a regex.

You're just, like, take everything within the JSON tags and then use that. You can even combine these two techniques. So you could say, here's the updated JSON, and now you give it the JSON tag. We can even put a bracket here.

And now what we'll see is it will just give the JSON minus the bracket, and then it will close the bracket, and then it will close with the JSON tag, the closed JSON tag.

There we go. And it also gave this so you can see it did at first, I was, like, a little bit panicked because I didn't see the closed JSON tag at the very bottom. But then I saw that it actually did include the tag up here, and then it gave this little explanation afterwards.

So this is another useful thing. This will save you some time and tokens and trouble. One thing we could do, like, it costs Claude, like, it costs you money to get Claude to output all this stuff, and you probably don't need it.

In most cases, you don't need the explanation. You just need the JSON. So one thing we could do is we could say, do not include any explanation after the JSON,right?

Does it help to yell in that case?

I mean, probably, but I don't know. Honestly, I don't yell that much. I'm just, like, this is actually meant to be my parody of, like, what a frustrated prompt engineer would write if they were, like, couldn't get rid of this.

But in practice, you might not need to do that. But the simpler way to do this, there's a and we're getting outside the realm of prompt engineering for a second and into the world of, like, API parameters, but that's okay.

There's a parameter that's called stop sequences. And if you set so we told it to return the JSON in JSON tags,right? So there's no functionality to do this in the console, so I can't show it off at this exact moment.

But in the API, there's a parameter called stop sequences. And if you add

this

closed JSON tag that I've highlighted with my mouse, if you add that to the stop sequences, then it will just hard stop after it outputs those, and you don't even have to worry about telling it not to continue from there because it just won't even sample from there.

You won't be charged. It's all good. So one of the things that I'm sort of hoping to impart with this talk is that a lot of times, it's cheaper and easier to do a little bit of work outside of a call to the LLM and not even worry about prompting because prompting can feel sometimes, like, non-deterministic.

You don't know what the model is going to do. So when you can offload stuff to code, especially if the code is really easy to write, it's like, just do that,right? Like, don't put a bunch of stuff in the prompt about you must output JSON.

Just use the prefill and then parse it out with the regex. You know, don't add a bunch of stuff about how you have to stop after you say a certain word. Just add it to the stop sequences. So, like, simple is better, and falling back on code is better than relying on prompts.

Yeah.

Is the prefill available through the API prompts?

Yes, the prefill is available through the API. What you do is you include an assistant message as the last message in the messages list. And when I say an assistant message, I just mean a message where the role is set to assistant.

And what would have happened if you had the text that you put in the prefill, you just put it into the last line of the instructions?

So in other words, if I said I'm actually not sure. That's a good question. So let's actually try that.

So it does work, but, like, you can't just say, like, literally define what the sentence is.

I genuinely do not know how Claude will respond to this, so let's see.

So it looks like what it did was it looks to me like what it did without looking at this JSON is it included an additional open bracket,right? Because it's supposed to already have started with an open bracket, but here it started with an additional open bracket.

So it kind of almost worked, but not quite. Anyways, I don't recommend doing this, but that was fun just out of curiosity's sake. Yeah.

Can we stop back? Can we just remove that? I'm after the sentence, like, yeah. Is it on?

Yeah. After the sentence, we can just write it, like, you wrote,right, return the JSON, return the response in the JSON format, and then in the next line, you can just write JSON: then leave it. I think it will.

Oh, so, like, if I said something like this, like this?

Yeah, that's it.

Yeah, I could see this working.

Yeah, it looks like it worked pretty well. I think there's, like, a lot of ways to accomplish this. I think the ways that I showed are the most reliable, so that's what I would, like, officially recommend. But yeah, like, definitely experiment.

If you were going to, like, try to use this for production or whatever, what, like, these exact kind of things you're playing around withright now, how would you think about testing that, like, at some sort of scale? Like.

How do we test it at some sort of scale?

Yeah. More than, like, the one-shot test we just did,right?

Yeah, yeah, yeah. To test it at scale, you need a bunch of test cases. And if you don't have test cases, okay, this is maybe a good time to maybe show off this thing, although I'm actually not sure if it will work.

So one, I guess, a more pointed question is, like, in this case, I think test cases are useful when I'm writing a prompt to deduce whether, like, does asking it to think step-by-step lead to this thing being more accurate.

But in this case, for formatting, I guess what I'm wondering is, like, could you have this prompt and then feed in the output and the prompt and then ask the model itself to evaluate, like, how good these various things are at following the instructions?

Yeah, yeah. Okay, so can we do model grading? Can we model grade the outputs?

Yeah, especially for formatting-related things.

For formatting? I would not model grade the outputs because formatting is something that I can check in code. So if I can do anything in code and I don't have to call the LLM, the LLM is like this crazy black box,right?

It's like, if I don't need to, like, make this pilgrimage to the Oracle and, like, ask it, I'd rather just do it, like, in code. So formatting specifically, we're kind of, like, in luck. It's easy to check. For something like the other the previous prompt we looked at where the outputs are a lot more squishy, we might possibly model grading could work.

Possibly, we might need a human to evaluate the answers.

Just to lightly push back on that, so I'm wondering, like, I actually put an example in the Slack channel. We don't need to get to it because we're talking through it now. But, like, for let's imagine I don't have or actually, maybe tags are the answer to this.

Like, imagine I'm asking for a summary or something, and then I want to deduce whether there's additional chat, like, content, like, before or after that. In that case, would you I would have my mental model would have been to use, like, an LLM as a grader, but it sounds like maybe would you encourage instead using the summary tags and checking, like, hard coding for additional text around that?

Yeah. I think that will be pretty quick and easy to do. Also, just having the summary be in summary tags is, like, generally a good practice. I generally have all my outputs inside tags to make it really easy to extract them.

I don't think there's really any downside to doing that. So and it might even be that by doing that, you effectively fix your entire issue and you don't even, like, need to do the test anymore. Or and you just put closed summary in the stop sequences, and you're kind of good to go.

Okay, cool.

But that also does sound like a problem that an LLM could grade.

Okay. Let's go to the next prompt here.

Just shout out your question.

Here's a very poorly formatted Excel spreadsheet.

I got a question real quick. So this seems like a really ridiculously, like, powerful attack vector. So can we test a prompt real quick?

I don't want to get into too much, like, jailbreaking stuff here. Sorry.

Okay, apologies. So that's kind of my specialty.

Okay, yeah. I'm going to go to the next prompt.

So what do we have here?

Here's a poorly formatted Excel spreadsheet/CSV.

Please extract all data into JSON. Okay, how can we so Jan or Yann?

Yeah,right here.

Is it Jan or Yann?

Yann.

Yann. What is the actual text that I should can you paste the text here? Because I don't know how to get the CSV into the console.

Oh, hey, thanks. I've just been just copying the entire CSV and putting that into the prompts. Again, I've been trying to use Claude to extract some information from spreadsheets, and it's always been very, very hard. It hallucinates a lot or it skips a lot of stuff.

I was wondering if you maybe more generally, how do you have Claude analyze really poorly formatted spreadsheets that sometimes have different clusters or multiple data sets in the same sheet and things like that?

Okay. I'll try to answer the general question of having analyzing, like, poorly formatted spreadsheets. The first thing that came to mind, especially when you're talking about how the spreadsheets are very big, is breaking the problem down into so give it, like, fewer spreadsheets at a time.

Give it fewer columns of the spreadsheet at a time. Only give it the columns that it needs to work with. And then make the questions sort of smaller and more bite-sized, and then tackle it that way by breaking it down.

So at that level of generality, that would be my answer here.

I'd also be curious to look at this one more specifically. Right now, I'm just struggling with how to, like, copy the text and put it into the tool.

Can we run a three-comma?

Oh, yeah.

That's what I did, but it keeps downloading it. I guess I can.

Yeah, I guess the next tab has it.

Sorry, the next tab.

The tab.

This is that other one.

Sorry, I don't want to click open my downloads. I'm scared I'm going to, like, reveal some private information. This is my work computer, so I want to just do it all in the browser.

Yeah, so ask it to write some Python code.

Sorry, you can go to the next one.

Yeah, let's do the next one. Okay, you are a social media ghostwriter.

Given the boat along for our article. Okay. Generally, we would recommend putting the I like this one is short. We can do some quick hits here. We would recommend putting the instructions after the document. That's similar to the question that was asked about should we have the information first or the instructions first.

Particularly with long documents, it's a little bit better to give the instructions at the end. Let's also put some XML tags here.

Let's just, like, clean up the grammar a little bit. Given the above long-form article, create five to ten tweets. Don't use hashtags. Don't be hyperbolic, and don't be cringe. Return a JSON array of post content. Probably good to give an example

of the format.

So we could do, like

I guess we can just do it. We return, like, a list. What did you originally have it? Is there a special reason that you wanted it to be a JSON array, or is it just to make it parsable?

Just to make it parsable for automation.

Okay, yeah. So let's say return in I'm just a huge fan of these tags, so let's do it like this.

Okay. So that is that's some stuff without adding examples. The other thing that I would want to do is to give some illustrative examples of what it means to not be cringe.

So how long are these documents?

I put in I used a recent blog post from a provider of an example.

Perfect, yeah. Let's actually let's run this as is.

You're missing the bracket at the end. Yeah, yeah. The last, very last, hard bracket.

Oh, yep. Thank you.

Now we can take this. Uh-oh, what if they write cringe tweets about our product? I'm going to be embarrassed.

Okay, this doesn't include any hashtags. Doesn't seem very hyperbolic. Is this cringe? What do we think?

New feature alert. That could be a little bit cringe.

There's no emojis, which is a good sign.

Okay, what do you your name was Charlie. What do you think of this, Charlie?

I think they are adequate but not engaging.

Not engaging. Okay, so yeah, yeah, yeah. So we can try to make it more engaging without making it cringe. So let's say don't even be hashtags, but cringe. Try to make the tweets engaging. Are these meant to be tweeted from the Anthropic Twitter account or from, like, the AI influencer Twitter account?

Sure. Let's say AI influencer Twitter account.

Okay.

Let's see how this goes. I'm going to switch back to Dove too. Sorry.

Exciting news.

Is it better to break it up into small sentences in the prompt, or can you use complex sentences?

Is it better to break up sentences to use, like, small sentences in the prompt or big sentences?

I think generally in English writing, it's better to use small sentences and small words. So I think it's probably also better to do that in a prompt. I think it's fine to use big words if you are really sure you know what you're doing and you know that it's the exactright word for the situation.

Sometimes I'll find myself using more academic language if I want the output to seem a bit more academic. Generally, I think

simple small sentences is better. Okay. So these are maybe a little bit more engaging.

Like, they have these questions here. Want to try it?

What do we think? It's got exclamation points and question marks. Is it better? Do we want it to be even more engaging or something?

Let's see, because okay, so I honestly think temperatures doesn't is a bit overrated maybe. We can see if how that differs though.

I'm not sure exactly how to distinguish these from the previous ones. They look kind of similar to me from the ones with temperature temperature one or temperature zero.

Mm-hmm.

It's better to emulate certain personalities. That'sright. Yeah, so what I was going to say is I think this is roughly as far as you can take this without examples. I think the best thing to improve this prompt would be either examples of the sort of tweets that you want or even an entire other document and example of tweets that go with that document and maybe, like, multiple of those.

So

if you're cost-limited, maybe you don't want to put in all those input tokens every time. But I don't know. The models are pretty cheap now, and we don't need to generate that many tweets. So if they have, like, any economic value to you at all, it's probably pretty cost-effective to put so basically, like, but it's more work on your part because what you're doing then is so okay, so the way that I would actually do this is I would start out with some document.

I would have Claude write a bunch of tweets. I would take the ones that I liked, and maybe I would write some more or get, like, my friend to write some more. Or maybe I'd have Claude oops, Claude generate 100 tweets, and then I would take the seven that I liked best, and then I would put that in as an example.

And then from there, I would sample, okay, now here's another document, and then write a bunch more tweets based on this. And what I would do is iteratively build up this list of documents plus example tweets, and then I'd put them all into the prompt.

And it would look something like this. So let's actually do that. So let's imagine that we had done this. So it could be, like, you know, system prompt.

You are an AI influencer who writes engaging social media content about

new models and releases. It could be, like, here are some example documents along with the tweets you wrote about them.

And here you would actually I'm going to write this, but you would actually put the literal text of the document here.

And here again, you'd put a literal tweet here. And this could either be something that you wrote or something that Claude wrote or, you know, something that Claude wrote and then you edited. Like, a lot of times, Claude might give you an example that's not perfect, but it's close enough, and then you'll change it a little bit to make it perfect.

I have honestly given multi-shot examples pretty short shrift in this talk so far relative to their level of importance. Like, I think that in reality, most of the gains most of the effort, most of the gains of writing a good prompt is literally just picking the perfect document that goes here, picking the perfect set of tweets that go here, altering and changing them to modulate the tone.

In some ways, that's more important than, like, everything else that I've said combined. Like, another way to do the whole JSON thing would just be, like, with examples of Claude giving the stuff without a preamble. The JSON one is maybe an exception because the prefill approach works so well there along with the tags, but for anything else, the examples are really huge.

Anyways, then we would.

For few-shot prompting, do you prefer to front-load those all in one response like this, or do you find further success with an exchange of messages between the agent and the user where you're putting your few-shot prompts in there?

Yeah. Really good question and something that I would dearly love to know the answer to, but I don't. The question is I don't need to repeat the questions. I think people can hear them. But I'll repeat it anyway.

So do we want to just put all the examples in one big giant examples block like this, or do we want to structure the examples as a dialogue where the human says something and then the assistant says something back and we're literally, like, putting a large number of messages into the messages list?

I typically do it this way with a big examples block, but it's mostly because it's less work for me, and I don't have any evidence that this works either better or worse. I did do some testing of this at one point on a few data sets, and I found that it didn't make much of a difference for my particular case.

But there's a lot of, like, little particulars that went into my testing that make me not very confident in the result that I got. So sorry for a bit of an unsatisfying answer here. I'll just say I don't think if it is wrong to do one giant examples block, I don't think it's, like, very wrong.

Do you use anti-examples too? So I can hear when you give it a thing and say, like, this would be bad because this is cringe?

Yes. Yeah. I think that is good. I think it's good to include negative examples, particularly around, like, the cringe thing where Claude might mess up. I think just negative examples on their own don't usually get you there. You want to have some positive examples too.

But I think it's great to have especially, like, contrasting pairs. So, like, here's a document. Like, here's a cringe tweet about this document. Here's an excellent tweet about the same document. And, like, set those up side by side.

I think that's pretty powerful, and I do that, and I think it helps Claude.

And then if you also include, like, the reasoning for it,right? So, like, if it was a cringe tweet, it has, like, a little reasoning of, like, why. Do you also do you trust that reasoning for the model? So, like, if you ask it, like, hey, give me a like, what were you thinking when you were writing this tweet and then write me this tweet?

When you're reading through your examples to choose the best ones, how much do you trust that reasoning and how much do you rely on that versus just, like, I just care about, like, the input output?

I don't trust the reasoning very much, especially if it's after something the model already said. Then I, like, really don't trust it.

Yeah.

But I mean, humans are not very good at explaining why we do the things that we do. We're really good at rationalizing and coming up with, like, fake reasons. But a lot of times, we don't even know why we do the things that we did, let alone be able to coherently explain them to someone else.

So I

there's a subtlety here. So something that does work pretty well is having the model think about its reasoning in advance and, like, go through different reasons or rationales for why it might choose one option or the other or think about what sort of things might go into a good response.

So if I had the model do some thinking in advance before it gave the response, then I might just trust or assume that the response would be better.

Having a bunch of explanation for why it did the thing after, probably I would not trust that. Sorry, you had a question for a while.

Do you give your own reasoning that explains the examples? And if so, how do you make sure that the model doesn't give reasoning itself?

Do I give reasoning to explain the examples? Yes. I do a lot of giving reasoning to explain the examples. So, for instance, just in this case, one thing that we could do here is, like, we could add something like

I was going to say tweet planning, but maybe it's, like, key points of document. And then here we have some key points. Like, the document

presents the launch of.

So you would have this after the example. So if you have 10 examples.

No, this is before the examples. Oh, sorry. So this is part of the exampleright here. So in this particular example, in this example block, I gave it a document. Now I'm doing this key points business.

Got it.

And then I would have these tweets. Now, these key points could be something that I wrote myself, or it could be something that Claude wrote and then I'm editing it. Or if Claude did a perfect job, maybe I could just include the thing that Claude wrote.

But now in order to get this get Claude to do this, we would also say something like, return in this format key points a list of the key points from the document.

So this is, like, a lightweight chain of thought where we're having the model do some thinking in advance, and we also gave it examples of it doing the thinking in advance like this.

Hmm.

Yeah.

Yeah, so let's imagine we, like, really want to give examples like this, but we have a problem, which is that our documents are, like, super long, and I'm greedy and want to save on input tokens.

Yeah.

Would you err on the side of doing, like, one document but a really good example or doing, like, truncated versions of more documents? I would that's a good question. I would err on the side of one extremely good example and not truncated versions of more documents.

But I would also want to look at the outputs and test that assumption because it's possible that with only one example, Claude would fixate on aspects of the exact document that it uploaded and start trying to transfer them to your document.

So I think it's one of those it would be case by case. But I would want to start with, like, having one extremely good example. Generally, I think that, like, less but, like, higher quality is a better way to go than, like, more and lower quality.

Cool. Thank you.

Okay, we have a lot of prompts here. Let's go to the okay, this is good. I was hoping we would get some, like, persona ones here.

Okay. So this looks like something where we're trying to get Claude to role-play in these different protocols.

So let's try this out, and let's see how it works. So this looks something where we're going to have, like this looks like it's, like, meant to be a multi-turn prompt,right? So this is, like, a conversation.

You are talking to assistant.

It said execute greater than p assist. Where's that?

So basically at the top, you see three roles being defined, and then you can decide who you want to talk to.

Oh, thanks so much. So you see three roles at the top?

Yeah.

And then if you do that p assist, you see down there that's highlighted in yellow?

Yeah.

You can just do that little arrow p, and then you can pick a different persona.

Yeah.

And then you can have them talk between themselves, or you can just switch we use these for designers in our shop to do synthetic interviews to synthetic users, basically.

Gotcha.

It allows us to switch back and forth.

And then what issues or troubles have you been having with this?

I'm guessing you have seen a lot of role-playing prompts out there. So I was just wondering if you see anything that's perhaps not as optimized as it could be or any other best practices for role-playing, particularly with multiple synthetic personas within the same session.

Yeah.

Okay. For single personas, there's one answer that I would give. This multiple personas thing actually I haven't worked that much with, but here's, like, off the top of my head, here is probably how I would think about it.

I would give all the personas to I would write a separate prompt for each persona, and then I would have the user's command trigger some coding logic where it would decide which bot to send the which prompt to send that reply to.

So this is getting back to the thing that I said before about, like, don't do it in a prompt if you don't have to. Like, this I mean, this prompt, like, there's a lot of thought that went into it, which is probably makes it work a lot better than it would have if you hadn't put as much effort into it.

But I think it's going to be easier if you just dynamically route the query based on what the user said. Does that make sense?

It does.

Okay.

You're talking about, like, if we were to use the API instead of just the chat.

Yeah.

Construct something like this.

Yeah, exactly. But you're doing this just in the chat.

This was just in the chat, but I appreciate I definitely appreciate the note there. So maybe related to that, one of the other things is how much have you dealt with having a second thread with the API that acts as maybe the entity that's capturing inputs from multiple ones into a single thread?

You know what I mean? Like, let's say that I build an app, and I have the user interact with these different synthetic personas, but then I have a second interaction with the API that's tying these things together into a cohesive whole.

I don't know if you guys have explored some of that. I'm a bit curious.

Yeah. I don't have a great answer for that one. Sorry. I do want to kind of test this prompt out, though, just to kind of see how it goes. So maybe here I would say

I can just say something like so how would I switch it? I could.

So do theright arrow p?

Yeah,right arrow p.

And then type Sam and then say, hey yeah.

Okay, so now I could.

You could say, hey, how do you what's your process to look for theright for the best medication pricing whenever you get sick or something like that? And then here in this particular case, if you switch to Joe, Joe is optimized more for convenience versus cost savings.

So you have two different types of users, and we can learn from.

Yeah. Okay, so Claude did the thing here that I want to show you all, like, how to get rid of.

Oh, yes. Yes.

As Sam, it's like that's not something that Sam would say,right?

Yeah.

So I don't know for sure this is going to work. I feel like a magician that's about to do a trick, but, like, I haven't practiced it. But generally, something that is pretty useful here is to we could say

prepend each response with the name of the current persona in brackets.

So one thing I'm going to do here is I'm going to change this, like, multi-shot a little bit also because if Claude sees itself not doing the thing that I told it to do actually, let's just redo the whole conversation, or we can take out this.

So let's just, like, run that back. You are talking to assistant.

Nice.

And now we could say

and now we could say the same thing, like, what's your process for finding the best prices for medication?

Oh. Okay, so I guess we need to do this. We need to, like, change in a separate call.

Yeah.

Okay, great. Now

it's going to work. Totally. It's going to totally work.

Okay, it's a little bit better,right? It didn't say as Sam. This is, like, something that a human might maybe say. Like, as someone who's okay. I don't know. It's better than it was before,right? Maybe we could say something like.

Can you just redo that?

You don't need to

say too much about your persona and your responses. Just stay in character.

Hey, quick question. What are your thoughts on using things in the negative sense versus.

Yo, check it out. It worked a lot better. Sorry to interrupt.

Oh, yeah. Very nice. Very nice.

Yeah, so yeah, what are your thoughts on using, like, negative stuff, like you don't, versus the positive sense? Yeah, I think positive is, like, a little bit better. In this case, I don't really have a good answer for why I phrased this negatively.

I guess I did a combination. I was like, you don't need to say too much. Just stay in character. I guess I think it's better to use, like, a light touch, like, if you're doing negative prompting. I think there's, like, a little bit of a thing going on with reverse psychology where if you tell the model, like, don't talk about elephants, don't definitely know elephants, definitely don't say anything about elephant, it might make it more likely to talk about an elephant.

So if you do use negative prompting, I think it's better to have, like, a light touch where you just kind of say it once, but, like, don't dwell on it too much. Also, something similar with parenting. It's like if you don't want your kid to eat prunes, you're just like, oh, we're not having prunes today, and then you just change the subject.

But if you really, like, emphasize that there are, like, no prunes to be had, then you might get more pushback.

Hi there. I noticed you're not using the system prompt much. Like, is there a reason for that, or what do you think the biggest value items for a system prompt are?

Yeah, system prompt. Personally, the only thing that I ever put in the system prompt is a role. So I might say, like, you are this, you are that. I think generally, Claude follows instructions a little bit better if they're in the human prompt and not in the system prompt.

The exception is things like tool use, where maybe there's been some explicit fine-tuning on, like, certain system prompts specifically. For, like, general prompts, like the ones we've been going over here, though, I don't really think you need to use the system prompt very much.

Yeah.

Yeah. One thing we found when using the user prompt thank you.

One thing we've found when using the user prompt, I guess, sometimes is it makes it more prone to hallucinations because it thinks the user is saying it. And so we migrated things to the system prompt more. I don't know if you have any experience with that.

Yeah, yeah. I've actually heard that before. So it's possible I'm missing something. I've heard this from enough people that I could just be wrong. So I'm unusually likely to be wrong when I say this. I think that if you just put a bunch of context and you're like, here's the message from the user, open user bracket, and then put the message and then close the user bracket, it will work, and you won't have that issue anymore.

That said, like, I don't know, maybe it does fall over sometimes, but that would be my default is just to, like, specify even more clearly in the if you're having this issue, be like, here's the message from the user.

Here's the stuff that I want you to do. And I think it probably won't get confused by that.

I have a question about the counterexamples. So before, in order to get it to say not cringy things, you were saying provide it with a counterexample. So but here in the case of where you're doing this character bot, you haven't provided it any counterexamples.

So this is sort of like a generic question. So if the model is trained on preference optimization with examples and counterexamples, do you get a better result in the prompting?

Oh, I don't know that the details of the ROHF have that much bearing because I think when the model's trained, it doesn't usually see those both in the same, like, window. It's more that it's, like, some stuff that happens with, like, the RL algorithms.

So I don't think that's necessarily theright way to think of it. With counterexamples, I don't feel that I have to include them in every prompt. It's just a tool that I have in my toolbox that I'd use sometimes.

In regards to, like, negative prompting I'm over here. Hi.

Oh.

Do you think that it would be better to do negative prompting using control vectors, like what you talked about in your scaling monosymmetricity paper?

Yeah.

And maybe, like, having, like, a negative version of the vector as your kind of negative prompt instead of mentioning it in the prompt outright?

Yeah. Steering is still, like, super new. We don't know how well it works relative to prompting. I'm, like, a, you know, a die-hard prompter till the end. So

I've played around with it a little bit. I haven't found it to work as well as prompting in my experience so far. That said, there's, like, a lot of research improvements that I won't get into in too much detail, but there's a lot of stuff that could make it work better than so, like,right now, it's, like, finding these features, and then you're steering according to the features, which are sort of, like, these abstractions on top of the underlying vector space.

Yep.

There's other possibilities for how you could steer, and there's, like, academic papers that you can read where you're steering according to just, like, the differences in the activations versus, like, trying to pull it out to this feature first.

So maybe that would work a bit better, like the control vectors thing.

Yep.

I haven't played with it enough to know for sure, but I think there's definitely, like, something along that line. Those lines will work eventually. I can't say in the long term if it'll work better or worse than prompting.

Right now, I still think prompting works, like, a lot better.

I mean, from my experience with, like, smaller models and trying to work with control vectors, I've seen that it's better when it comes to style than it is for, like, actual deterministic prompting.

Yeah, pretty interesting.

Yeah.

Sometimes I feel like stuff from smaller models transfers. Sometimes it doesn't transfer. I don't have a great intuition for what does and doesn't transfer between small and large models, but yeah, it's good points.

Thank you.

Okay, I think we've gone over this role-play stuff enough. Let's go to the next one. I'm going to upload a few screenshots of my dating profile. Okay, this is our first image one. Are there any screenshots? No screenshots.

Okay, actually, somebody responded in the very first in reply. So since we're doing images, maybe I'll start there.

If you scroll down, it's that. That was me.

That was you.

Let me try and find my message here. You said it's at the bottom of the.

Yeah, it's in that panel ID posted there.

Riveting to watch me scroll through this channel, I'm sure.

Huh, there's a lot of messages here. Okay, here we go.

Okay, so let's

can I copy the image? Copy image. Okay, cool. I actually don't know if I can paste it into the console. So I might fall back on using Claude.ai for this.

Paste images is not enabledright now. Okay, let's try Claude.ai then.

So then the question was or the prompt here was

high-performing validated AI model.

Sorry, I like lost all your formatting here.

Okay, so in this image here, if we zoom in, it's supposed to get Matty White and

86 I'm having a hard time reading this.

8687, that's the hard one that it messes up on.

8687 and then 8687 down here. And then in this one, it's just 867.

Yeah, that's a typo.

That's a typo.

By the human. And so I'm hoping it can correct for that. I'm just trying to pull out kind of the average data from all three combined.

Okay, and it said it looks like it said middle mark. So it's misreading the okay, so I don't know too many good tips for image is, but I'll tell you what I have. So one of the things that you said oops, sorry.

One of the things that you said here was

that it works better with zooming in and cropping the image. That's definitely, like, the easiest win that you can have is just giving it, like, a higher quality image, taking out the unnecessary details. That might be hard to do programmatically because it's like you don't know which details are necessary and unnecessary.

But for the same reason that including extraneous information in text form, you probably won't get as good results. If you include extraneous information in image form, the results probably won't be as good either. So the more that you can, like, narrow in on the exact information that you need, the better.

Is the model downsampling large images? I don't know/can't talk about that, but definitely having, like, higher quality, bigger images is better.

I did just read on your website that it says it downsamples to 1,000 pixels by 1,000 pixels.

Okay, great.

That can be found with Google.

Okay. So then any general tips on how to discuss images with Sonnet?

My number one tip for images is to start by having the model describe everything it sees about the image. So I don't know if that will work here. This example is, like, this one is hard enough for me to even read that, like, I kind of doubt that Claude will do well on it regardless of what we say.

But we can give it another shot.

One thing I've noticed when I attempted that where if I asked it to go, like, tube by tube in that image, if it, like, the first tube if it came to the wrong conclusion, it would use that and come to that same wrong conclusion on multiple tubes where, like, it made it so there was some kind of, like, directionality in its thinking where if it got the answer wrong at first, it would project that onto the rest of the image it was analyzing.

Yes, I think that's definitely true. If the model starts off on the wrong path, it probably will just continue going on the wrong path. It's trying really hard to be self-consistent. And it's and this is why self-correction is such a big, like, frontier for LLMs in generalright now.

But as of this month, why use Claude for this task? So I love to use Claude, but why for this specific image text extraction?

I found, like, text tracking OCR models I've played with don't do as well with some of the handwriting. Like, if I zoom in on this image, it actually does perform pretty well from some of these fairly messy handwriting that's even hard for a human.

Okay. Yeah, unfortunately, I don't know how to get better results out of this other than maybe by cropping it better and upsampling.

Okay, cool. Thank you.

Oops.

Here we are. So let's scroll up to where we were before.

Okay, yeah, I still have no screenshots there. How do I enable fragments? What are fragments?

Is fragments like the prefill part?

The beta feature you guys have that artifacts.

Artifacts. Oh, okay. It's just a setting. It's in the bottom left of the you have to enable it. Yeah, you can find it online. Our people will help you.

So I get sorry, can you skip my so I often dump full traceback errors directly into the prompt box. I often dump full traceback errors directly into the prompt box as an API. It seems exceptional at not running into traceback loops.

I don't know, like, if that's intentional. Like, literally, I'll just take the entire traceback, zero context to Claude, and I'll just dump the entire thing in.

Yeah.

And then it'll give me the fix.

Okay, great.

So but is it can you elaborate on how that might have how this for example, like, this only really appeared with, like, very recently. Like, you'd have to explain explicitly a lot.

The models get better, man. They get better every time.

I understand, but this is, like, you know, this isn't prompt. Like, this is prompt engineering we're talking about,right? So I'm just wondering, like, is this a form of prompt engineering, or is just the model being good?

Sounds more like the model being good if you're just dumping it in.

Okay.

I'm going to move to this prompt here.

Okay.

So

to the person who uploaded this, do you and generally, to anyone who's uploaded more examples, if you could just, like, put some stuff in the thread, like, write some stuff about what the issue is that you're having or, like, why it's not working, that would be great.

I want to suggest this is actually a follow-up to, like, what I was doing with the translation. So basically, what I'm trying to get to do is to actually analyze the text. So, like, in this case, you know, there's original English.

Yeah.

There's a bad Japanese translation, and I'm trying to get to score between 1 and 5 how good the translation is. And so what I've been doing is adding a lot of stuff to try to get to do sort of chain of thought to.

Yeah.

You know, see, like because it'll notice errors, but, you know, it just generally does a very bad job at scoring.

Yeah, okay, so this is great. I'm really glad that you asked this because model grading is something that I mean, it would be incredibly useful if it worked. Andright now, it's in a place where it, like, sometimes kind of works and sometimes doesn't.

So let's paste this into the console.

Okay, so we have some English text, and we have some yep, yep, yep.

Then we got the Japanese text. So is this a good translation or a bad translation?

Terrible.

Terrible translation. Okay. Now Claude's actually supposed to be good at Japanese too, so.

Claude is much better. Yeah.

Oh, this is somebody else. But I'm saying if Claude is good at Japanese, it should be good at, like, judging other people's Japanese in theory.

In theory.

In theory. Okay, so now we've got the answer here. So

I guess it's stalled out here for whatever reason. So what are we doing in this prompt? We're scoring between 1 and 5 as below. 1 is many grammatical errors the native would never make, contains multiple grammatical errors, and average quality translation with some errors.

Okay, that looks pretty good. Look for specific clues or indicators.

I don't know why it I think that's just a bug. I think it'll conclude here.

Okay, so it gave it a 3, but we want it to give a 1.

Oh, wait, wait, wait.

Okay. Now, have you found that it's generally too forgiving or too strict or that it's just all over the place?

Well, it's all over the place. Also, it seems to confuse sometimes content versus, you know, so, like, for example, this is from, I think, the HLRF, you know, set. So, you know, the content is fine. But so it might not actually rate it low even though it's a terrible translation because it thinks the content is okay even though if you asked it even to list all the errors, there are a dozen errors in that, you know, single piece of text for the translation.

Gotcha. Yeah, yeah.

So that's what I'm trying to also see if, you know, is there any way to, like, get to separate out the grammatical or the actual translation errors versus.

Yeah. Okay, a few thoughts. So generally, this is, like, a thing where if you ask the model if some text is, like, good or bad, it's sort of if the text is, like, about a nice subject, it's more likely to say that it's good and always.

Like, it's a good translation. Like, it's well-written. It flows very logically. And if it's about, like, a negative subject, it's more likely to, like, criticize it and say that it doesn't flow well.

Yep.

I think you can get at those issues by typing language about it in the prompt. So for instance, here you might say something like.

Is one or five good? You know, you don't specify that.

Hmm?

Is 1 or 5 the best score?

1 is the worst, and 5 should be the best.

That's sort of, like, implicit in this rubric up here.

But it might be good to say it anyways.

The problem with that is before interrupting Claude, how do you get around the fact that you may not have a Japanese tokenizer?

Do you have a Japanese tokenizer in Claude?

The API will tokenize anything.

But how is it trained?

How is it trained?

Yeah, pre-trained.

Off topic for and also, I don't know.

I don't think I actually don't think there is a tokenizer for Claude. Like.

Oh, so.

I mean, unless you say otherwise, there's not.

I mean, if you upload if you upload some text, it will be tokenized.

But it's not pre-trained, so you're not going to get a really good answer for this or Japanese text. I've tested this.

Okay.

Claude speaks the best Japanese of any model available, actually.

We don't need to debate, like, Claude's Japanese skill. That's just, like.

But the tokenizer isn't available, and so that would be interesting.

Okay.

Yeah.

I have a question about programming.

Sorry, can we actually cut the questions off while I type this prompt here? Okay, so it's extra important to so what we're trying to do here is get it to distinguish between the, like, ethical nature of the text and the quality of the translation.

So.

Is it useful to tell it to be, like, extra critical? Say, like, you're grading a, you know, like, I don't know, like, graduate course, you know.

Sorry, I'm a little bit over the place. I need to, like, type this out before I can clear the queue and, like, respond to other questions here. So it's important to distinguish between

the.

What's a good word here? Like,

risqué topics or

yeah. So I don't know. Something like this could help the model not pay so much attention.

The main thing that I would want to do for this prompt is just add a bunch of examples. So for each category, I would add at least one example of that category. So, like, I'd have, like, a really bad translation, and I'd say why it's a 1.

And you can say that in your own words. And I'd have an example of, like, a 2-level translation, a 3-level translation, and so on.

Okay.

And in each case, before you get to the answer, you would have the explanation for why it's good or bad. I can't type all that here, among other reasons, because I don't speak Japanese. But I think that's the most valuable thing that you could do here.

Otherwise, I mean, like, the formatting looks really good. The fact that you're doing the chain of thought in advance looks good. I think

yeah, I think maybe this specific clues are indicators. I think you could go into a little bit more detail here about what is.

Okay, basically, just N-shot every single example or, you know, grade with an example, basically.

Yeah. I mean, it's tedious to write out all these examples, but, A, Claude can help, and then you have the problem of just editing Claude's response versus, like, writing it all out yourself. And, B, it really does lead to better performance versus, like, almost anything else that you could do.

Do you think it's better to use a number scale or ask it to say, like, good, bad, you know?

Yeah. In terms of the scale on these rubrics, I think either a number scale or good, bad is fine. The thing I'd be careful about with a number scale is that I don't think it's very well-calibrated. So if you're telling it, like, choose a number from 1 through 100, it's not going to be, like, an unbiased estimator necessarily.

So I'd probably just limit the granularity to maybe five different classes.

Yeah.

One thing we didn't really talk about at all here, but going back to, like, your questions to the API is.

This is a case where you might be able to utilize, like, log probs, and I'm wondering if that's something you ever use in any of your work or other prompts. Yeah. I think I agree this is could be a case where log probs would be useful if you could get, like, the probability of each grade.

Yeah.

The thing so here's the thing with log probs is you really want the chain of thought beforehand. And the chain of thought, I think, is going to get you more of a win than using the log probs. But log probs there's no in any model, I don't think there's a way where you can just say sample all the way through and then output after you output this, like, closed chain of thought tag, then give me the log probs of whatever comes next.

So if you are going to use the log probs, then you're looking at this, like, multi-turn setup where you first sample all the chain of thought or sample all the, like, pre-computation that it wants to do and then cut it off there and then re-upload that with the chain of thought as a prefill message, and then you could get the log probs.

But for that, you need a model that both has the prefill capacity and has log prob capacity. I'm not sure of what model has both of those characteristics.

Walk me through why it wouldn't just be sufficient to, like, in this case, just ask for I'm asking for a score 1 to 5, only return that, but then, like, look at the log probs of what it returns in that case.

Yeah. So

what you're losing there is whatever intelligence boost you got from having the model do the chain of thought. And my sense is that chain of thought plus have the model say either 1, 2, 3, 4, or 5 is going to be more accurate than, like, the more the additional nuance that you'd get by having it give you the log probs because it's actually doing a lot of its thinking in that chain of thought.

You're, like, leveraging more computation. You're getting more forward passes for all the same reason that chain of thought is, like, usually a good idea.

I see. Are you talking about, like, chain of thought and, like, it's actually out loud writing a chain of thought before the answer?

Exactly.

Okay, okay. Sure, I got you.

Which is what we see in this promptright here,right? Like, we have this analysis section. So if we cut out this whole analysis section, we're really tanking the number of forward passes that the model can do before it gave you the answer.

Okay, cool. That's good to know. Thank you.

What's that?

Okay, I'm being told that I should answer a couple more questions and then get off stage. I was, like before I came here, I was honestly really worried that, like, no one would have questions, and I'd be supplying my own.

But you all have had amazing questions so far and amazing examples, so I really appreciate that. It's made this go on well. Sorry, I'm supposed to say that at the end of this, but I'm giving a pre-thank you.

Now we can do the encore. Yeah.

I just wanted to add that having a numbered list may give more weight to, like, number 1, 2, 3, 4, 5 versus just having an unstructured list. So it may give more weight into your score for the output if you just change it from, like, 1, 2, 3, 4, 5 to, like, little dashes for criteria.

Just as my experience of what I've been seeing.

Okay, cool.

Yeah, so just FYI. I replied back with a prompt that was fixed or that, like, in my own improvement of what I think is a better prompt for this. Yeah.

Okay, awesome.

Nisha, should I do another prompt or should I just answer a couple questions and then head out?

We can do another prompt.

Allright, let's do one last prompt. What's which one should we choose?

Okay, let's do this one. Good old mitigating hallucinations because we haven't really done that. Okay.

Please provide a summary of the text provided as input. Okay, first thing I'll do is just move these instructions down. Now, Matt, did you have an example of the document a document where it hallucinates with this prompt?

Yeah. No, but I can send you one really quick.

Yeah, can you just put that in the thread?

Okay, your summary should be concise while maintaining all important information that would assist in helping someone understand the content. It mentions any dates.

Don't start or end with anything like, "I've generated a summary for you. Here is the summary you asked for." Yeah, so this one we can fix with a prefill.

Okay, allright, we could do something like this.

Now, how about the hallucination part? The best trick that I know for getting around hallucinations in a case like this is to have the model extract relevant quotes first. So what I would say do here is I would say something like.

And now, of course, in this prefill, since we're having relevant quotes here, we wouldn't want to start with summary. That would just be confusing/wrong. So we could say, "Here is the

something like this."

Okay, did you get the doc yet?

One second.

Okay.

And then, of course, I'd put the document here.

Yep.

Okay, I think I should get off stage. So yeah, let me just call it here, and, Matt, we can talk after. Yeah, once again, I really appreciate you all coming out. It's been amazing to have such, like, a great audience engaged.

I've had fun. I learned some things. I hope you all did too. I'm planning to stick around this event for the next for the rest of the afternoon. So I don't know exactly where I'll be, but maybe just DM me if you want to come find me in chat.

I'm always happy to talk prompt engineering. It's, like, my truest passion at this point in the world. So, like, find me, hit me up. We'll talk. And yeah, this has been great. Thank you so much.