# Principles for Prompt Engineering - Karina Nguyen (Claude Instant @ Anthropic) AI Engineer · 2023-10-20 Karina Nguyen, an engineer on Anthropic's Claude, presents principles for effective prompt engineering, treating it as a creative writing process requiring iteration. She advises using XML tags, placing instructions at the end of long prompts (improving accuracy), and decomposing questions to improve faithfulness over chain-of-thought. She covers reducing hallucinations by asking Claude to hedge or quote sources, and using self-consistency and contrastive examples for labeling. Nguyen explains generating evaluation datasets with Claude by splitting documents into multiple-choice questions, and notes that for long-document QA, asking at the end outperforms the beginning. She distinguishes Claude (larger, smarter) from Claude Instant (cheaper, faster, better at math and code), and predicts prompt engineering will remain essential for complex tasks. ## Questions this episode answers ### How should I structure prompts for the Claude API to get the best results? Wrap different parts of your prompt in XML tags like and , as Claude responds strongly to this format. In API calls, always separate turns with newline, newline Human, newline, newline Assistant, and never place 'Human' or 'Assistant' inside XML tags. This explicit structure helps Claude understand the task and produce cleaner, more accurate outputs. [11:28](https://aie.addtry.com/bf1b2ff2-85c2-4f56-ae3a-fcaca9596fea?t=688000) ### What is the decomposition-based prompting method for question answering? Instead of using chain of thought, decompose the main question into atomic, self-contained sub-questions. Each sub-question is answered independently in a separate context, and then the answers are recomposed to form the final answer. This method improves faithfulness because the model's reasoning is less influenced by preceding steps, reducing bias compared to standard chain-of-thought prompting. [21:04](https://aie.addtry.com/bf1b2ff2-85c2-4f56-ae3a-fcaca9596fea?t=1264000) ### What are the best practices for using Claude to answer questions from long documents? For long-document question answering, place the question or instruction at the end of the prompt—Claude pays more attention to later context. Also, ask Claude to first quote relevant passages inside tags before answering; this scratchpad technique improves recall accuracy at the cost of a slight latency increase. [30:44](https://aie.addtry.com/bf1b2ff2-85c2-4f56-ae3a-fcaca9596fea?t=1844000) ### How does Claude Instant differ from Claude 2? Claude 2 is a larger, more capable model. Claude Instant is significantly cheaper and faster, and was fine-tuned specifically to improve performance on math, code, and red-teaming evaluations. This makes Claude Instant much more robust against jailbreaks, while offering comparable reasoning on many other tasks with better cost and latency trade-offs. [52:57](https://aie.addtry.com/bf1b2ff2-85c2-4f56-ae3a-fcaca9596fea?t=3177000) ## Key moments - **[0:00] Introduction** - [0:01] Karina Nguyen worked on reducing hallucinations and self-correction in Anthropic's Claude. - **[0:45] Why Hard** - [0:45] Well-crafted prompts increase the probability of desired and accurate phrases via attention mechanisms, says Karina Nguyen. - [1:51] Karina Nguyen outlines three reasons prompting is hard: knowing what you want but not how to get performance, vague goals, and not knowing what you want. - [2:41] Provide diverse examples and explain simply, like to a five-year-old, when prompting language models, advises Karina Nguyen. - [3:36] Karina Nguyen views prompting language models as an act of creative writing requiring originality and creativity. - **[3:53] Creative Writing** - [4:29] Karina Nguyen describes prompting as forming hypotheses, testing model assumptions, and gaining clarity on strengths and weaknesses. - [5:10] Karina Nguyen's six writing principles for effective prompts: clarity, conciseness, coherence, consistency, direction, and grounding with examples. - **[5:34] Writing Principles** - [5:54] For Claude, Karina Nguyen recommends treating the model like a human and explaining things elaborately, unlike GPT models. - **[8:12] Recommendation System** - [11:28] Claude performs best with XML tags in prompts, and consistent use is crucial for high performance, says Karina Nguyen. - [12:29] Q: Why does Claude prefer XML tags? A: Karina Nguyen says Anthropic fine-tuned Claude on XML formatting, later adding Markdown and JSON per customer needs. - [13:38] Claude can be prompted to output only the content inside XML tags, avoiding extraneous text, a useful extraction trick from Karina Nguyen. - [14:46] Karina Nguyen demonstrates chain-of-thought prompting by having Claude reason in 'thoughts' tags before scoring item relevance. - [18:03] Q: Does scoring with LLMs introduce bias? A: Karina Nguyen says Anthropic's Societal Impacts group is actively researching bias in model scoring distributions. - **[18:46] Long Context** - [19:03] Claude's 100k-token context window can hold an entire book like The Great Gatsby, enabling long-document question answering. - [19:51] Karina Nguyen notes that chain-of-thought prompting can be unfaithful; the model may ignore its own stated reasoning when answering. - [20:45] Anthropic's decomposition method breaks a question into independent sub-questions, answers each separately, then recombines for faithful QA performance. - [22:19] Karina Nguyen illustrates decomposition with 'Could Scooby-Doo fit in a kangaroo pouch?' by asking separate atomic questions about the character and pouch size. - [23:09] Q: How do you prompt Claude to decompose a question? A: Karina Nguyen says they use a meta-prompt instructing the model to break the question into self-contained sub-questions. - [24:10] Karina Nguyen shows a legal Q&A example where Claude decomposes a contract law question into sub-questions like 'What is consideration in contract law?' - **[26:27] Evaluations** - [26:27] Karina Nguyen describes using Claude to generate evaluation datasets by creating multiple-choice questions from document sections and testing recall over long contexts. - [27:54] To avoid pre-training contamination, Karina Nguyen's team used a government document dated after Claude's training cutoff for long-context evaluations. - [29:19] In long-document QA, Karina Nguyen found that placing the question at the end of the prompt significantly outperforms putting it at the beginning. - [30:31] For document Q&A with Claude, placing the instruction at the end of the prompt yields better accuracy, and using a scratchpad thoughts tag helps. - [31:03] Karina Nguyen notes that prompting strategies like placing questions at the end and using thoughts tags can substantially boost Claude Instant's performance. - [32:55] Karina Nguyen hypothesizes that Claude attends more to the end of the prompt, referencing research on language models forgetting in the middle. - **[35:04] Auto-labeling** - [35:28] Karina Nguyen used Claude to auto-label text clusters by aggregating statements from each cluster and asking the model to categorize them. - [37:08] Karina Nguyen recommends self-consistency: generate multiple answers and use majority voting or cross-sample consistency checks to improve reliability. - [38:09] To improve nuance in Claude's categorizations, Karina Nguyen suggests adding contrasting examples, analogies, and clarifying common misconceptions. - **[40:10] API Tips** - [40:27] Karina Nguyen emphasizes that Claude API requires 'Human:' and 'Assistant:' formatting with double newlines, or results will be terrible. - [41:29] To reduce hallucinations, Karina Nguyen advises prompting Claude to hedge and say 'I don't know' when it lacks context. - [41:41] Karina Nguyen's hallucination reduction tip: instruct Claude to quote from documents, but say 'I don't find any relevant quotes' if nothing matches. - [42:04] Karina Nguyen advises that few-shot examples for Claude should be diverse, equally distributed among answer types, and not cause overfitting. - **[44:01] Future** - [44:01] Karina Nguyen predicts prompt engineering will endure, with increased use of synthetic data generation, AI-written evaluations, and reinforcement learning from AI feedback. - [45:40] Prompt engineering enables delightful product features like auto-generating chat titles in Claude.ai, says Karina Nguyen. - **[46:51] Q&A** - [47:24] Karina Nguyen says her prompting experience is mostly with Claude, and she notes strategy differences with GPT, particularly around formatting. - [50:22] Q: Has Anthropic tried function calling for classification? A: Karina Nguyen hasn't extensively used function calling but acknowledges it can limit classification. - [52:51] Karina Nguyen trained Claude Instant, which is cheaper, faster, excels at math and code, and is more robust to red-teaming jailbreaks. - [54:24] Karina Nguyen defines red teaming as testing models against jailbreaks, such as requests for dangerous instructions, and ensuring the model refuses. ## Speakers - **Karina Nguyen** (guest) ## Topics Prompt Engineering ## Mentioned Anthropic (company), OpenAI (company), CLIP (product), Claude (product), GPT (product) ## Transcript ### Introduction **Karina Nguyen** [0:01] My name is Karina. Um, recently I've been working on Claude, which is like a large language model trained by Anthropic. And, um, most recently I was working on, like, reducing hallucinations, how to make Claude self-correct its answers, and many other features that went into a Claude to launch. And so I'm going to talk today about the writing principles for task-tuned prompt engineering, and kind of sort of, like, help you—if you want to use Claude API—help you to guide what's the best practices and tips that I found, and we found, most effective. ### Why Hard **Karina Nguyen** [0:45] So first of all, um, I would like to talk about why prompting is hard. And to understand why prompting is hard, we should understand what prompting is in the first place. So these models estimate the probability of each subsequent word given the preceding words. So, in a way, a well-crafted prompt can increase the probability of generated, desired, and accurate, um, phrases. Due to attention mechanisms in large language models, the models can focus on specific parts of the input text. And so effective prompts ensure that the attention is directed, uh, appropriately for desired outputs. And so it's important to incorporate, like, task-specific keywords and context and examples within the prompt to activate the relevant portions of the model's internal knowledge. And lastly, like, you know, prompting leads to better results because without the need of, like, computational, like, other compute, you just, like, uh, without, like, model retraining. So, uh, you can, like, leverage inference time to test compute for this. And so, like, why prompting is hard. And I think I found, based on, like, my conversations with customers and, like, developers, I think prompting is hard because of, like, three different reasons. First, people know—humans know what they want, but they don't know how to get the best performance from the model. And I think that's what we're going to focus on today. The second reason is that they vaguely know what they want, but they don't know how to explain the best to the model. And so the model gets confused what the human wants from the task. And the third reason is, like, they don't know what they want. So the humans don't know what they want, so it's pretty bad. Um, and it's hard for the model to understand. So basic strategies that you can, like, if you find yourself, like, you kind of vaguely know what the task is, just provide a bunch of examples. And the model will be good at inferring what you're trying to do just based on, like, examples. And the examples may be, um, should be diverse and should en-encapsulate a bunch of, like, edge cases. Try to explain as you would have to explain to a five-year-old or, like, very, you know, in very simple terms. And I think what I found is that, like, as you have to, like, be able to, like, iterate a lot and, like, spend a lot of time just prompting. And in a way, based on my experience, for example, as, like, research engineer, I spend the majority of pairing with people just collaborating on the prompts. Um, in the past, I was thinking, like, how my main experience with the word prompt was only in the creative writing classes. Uh, I graduated from Berkeley, uh, and I took some, like, creative writing classes. And we usually have, like, exercises, like, prompting exercises,right? And so we often forget that prompting language models is actually an act of creative writing. And I see people get annoyed, like, why their prompts just don't work. But in most cases, I think it just means that people lack some kind of, like, originality or creativity to think, uh, of, like, new novel ways how to make it work. ### Creative Writing **Karina Nguyen** [4:15] Um, I recently wrote a blog post about the cultures of writing. And one of the points that I'm making in the blog post is that prompting becomes, like, a form—a new form of writing for any research engineer and, like, scientist who engages in daily. This kind of writing requires forming hypotheses. So you have to ask the model, okay, can the model do this? And you want to test that. Can the model self-correct its responses? Yes or no? Um, and so you start trying, like, forming the hypotheses. Next, you test certain assumptions that you make about the model. And if, uh, as you iterate more, you kind of, like, get new insights. So, like, oh, yeah, the model's pretty good at this particular thing, but it's not super good at another thing to achieve this task. So you gain more clarity about, like, what's—what's the strength and the weaknesses are. Um, I wanted to start with, like, overall broad, like, writing principles. Uh, because ultimately prompting is like writing,right? So the goal is to write prompts that clearly communicate the task objective while providing just enough constraints and guidance to steer the model towards producing high-quality and relevant outputs. ### Writing Principles **Karina Nguyen** [5:34] And so there are, like, four to six kind of, like, writing guidelines that I think I found effective, um, especially working with, like, Claude. Um, so the re maybe to clarify how Claude is different from ChatGPT or GPT models. I think with Claude, you have to almost treat it as, like, another human. So you have to, like, explain things to a five-year-old. Or, like, you have to be elaborate and, like, uh, um, I will share, like, more examples on, like, how to do that. Um, but I think that's, like, a distinguishing, um, feature from, like, GPT models from Claude. Um, so the first principle is clarity. Like, use simple, unambiguous language in your prompts. Avoid confusing syntax or vague phrases that could confuse the model. The second is conciseness. Keep prompts short and focused. Include only key information, uh, that the model needs. Third is coherence. Logically structure the prompt with context at the beginning and clear task at the end. Consistency. Stick to similar formatting. If you use XML tags, use it consistently in the prompt. If you use certain, like, terminology, um, do not, like, kind of, like, put the model off distribution, basically. So make it consistent. Uh, direction. Provide, like, genre, length, style, or any, like, guidelines—guidance to direct the model's response. Ground prompts with examples, sources. Make the model to quote itself. Or, like, if you have, like, a long document in the context, make it to quote from the document to support the argument. Or may or help, help the models to form support the arguments from, like, search results or, like, other supporting, um, contextual, uh, information. Uh, engaging use diverse edge cases examples. Uh, very useful for few-shot prompting. Um, now I'm going to go through certain, like, tasks that I thought would be interesting, um, and see how you can use Claude in those specific tasks. So the first case is obviously a recommendation system. Um, as of last year—last year I made this project in Terrallia. ### Recommendation System **Karina Nguyen** [8:12] Um, I used CLIP to, um, kind of, like, um, so I scraped a bunch of, like, images and, like, uh, clothing items from different brands. Um, and I use it text as, like, an actual search kind of, like, uh, engine. So you can say, like, um, James Bond girl, and you can do, do dress. And in a way, you get, like, results that are, like, dress in the style of James Bond. Or you can do, like, futuristic ethereal outfit. And it's, like, more, like, vibes-based search. Uh, and you can, like, go and look in the shop itself. **Host** [8:59] Can you explain CLIP? **Karina Nguyen** [9:02] Yeah. Oh, yeah. Uh, CLIP is contrastive language-to-image model trained by OpenAI. It's open source, but it's basically, um, they provide embeddings for text and images. So you can, like and the way it works here is that, like, you embed images and you embed, um, text result. And so what you can do is that you can, like, do cosine similarity to find the most similar, um, items based on your database for the user query. I don't know if that's clear. Let me know if you have questions. **Host** [9:37] Is it the same thing as multimodal embeddings? **Karina Nguyen** [9:40] Uh, it's a it's one way to take image and text to become multimodal. There are different things for audio. CLIP is mostly for collecting text and images. Yeah. Um, I think you can read the paper CLIP, yeah, um, if you're interested. So I was c I was thinking, like, okay, how could I use Claude in this project to curate relevant recommendations based on the user's requests? And that's the task. So in a way, you have, like, user's input. Let's say dress in the style of Emma Chamberlain, blazer like in the Gary Gatsby movie, an outfit with a futuristic vibe for the Met Gala. And on the other side, you have, like, image-to-text database, um, with images and their labels. And the labels can be produced either by, like, the original source, uh, or you can use, like, multimodal model to, like, come up with labels based on the images. And so the task with the Claude is to curate, like, based on the labels from the images, decide whether this, like, item relevant. Should I recommend this to the user? Is this accurate? Like, does it matches the, uh, user's per like, can I personalize this? Um, and so we can look at the very simple curation strategy, um, for the prompt. You can just, like, zero-shot it. Like, I need you to decide whether the item is relevant to the user query. Here's the user query. Um, here's the item description. Um, is that item relevant? Or should, should be recommended to the user based on the user's query? Answer yes or no. Please write the answer in answer tags. Let's unpack this. First of all, Claude really likes XML tags. Really loves XML tags. I think, like, this is, like, number one mis not mistake, but, like, um, one thing that people miss. They don't, like, put anything in XML tags. And so they don't have, like, very high, like, good performance. So everything like, yeah, if you love XML tags, then you should, like, put everything in XML tags. And, um, with XML tags, you should, like, be consistent. So what is user query, item? And you can be very descriptive. I can share, like, more examples in the f uh, later on. And here, in a way, you, like the way you interact with Claude is, like, you can see the language here. I need you to decide whether the item is relevant. It almost feels like you talk to a human. Um. **Host** [12:29] Actually, no. **Karina Nguyen** [12:30] Yeah. **Host** [12:31] So, uh, this XML thing recently only came out, like, maybe, like, a month or two ago from official Anthropic advice. **Karina Nguyen** [12:38] Oh, really? **Host** [12:39] Is this something that was on, like, intentionally trained for, or you discovered it after pre-training? **Karina Nguyen** [12:46] I think it was, uh I mean, we tried to XML XML formatting was the first formatting that we, like, kind of, like, fine-tuned on. Like, um, later on, we discovered, you know, customers need, like, Markdown or, like, need Claude needs to, like, use JSONL JSON formatting. So, like, we kind of, like, learned from customers. But originally, it was, like, XML formatting. Yeah. **Host** [13:12] So is that mostly because of the fine-tuning, or is that because you had a training set that had tons of XML stuff in it? **Karina Nguyen** [13:18] Uh, I think it's kind of about both. Yeah. Um, yeah. **Host** [13:23] Do you not need to close your XML tags? **Karina Nguyen** [13:25] Hmm? **Host** [13:26] You don't need to close your XML tags? **Karina Nguyen** [13:28] Close XML tags? Yeah. I don't know. Um. **Host** [13:38] Was it Java? **Karina Nguyen** [13:39] Oh, yeah. Sorry. I had a mistake. Yes. I have. Um, yeah. And so, like, one, one good thing about XML tags is, like, it's really, uh, easy to extract,right, like, the strings inside it. And Claude is pretty good at, like, um, I ca I can, like, say, like, don't put anything in XML tags. So sometimes, like, Claude will, like, say, here's information, blah, blah, blah, blah. But then if you ask, just, like, write the answer in, like, this tag, Claude will not put any additional information, which is one of the most annoying thing with, like, language models. So here's the results, uh, that I put. Um, this is through Claude.AI interface. Um, and so, yeah, you can see, like, is this item relevant? Uh, says no. Is this item relevant? Yes. But I don't think that's, like, 100%, like, um, you know, perfect system. So it's very, like, zero-shot. So basically, you can, like, iterate. And we will try to iterate more on this. Um, strategy number two is that, like, when you put you ask the model to take some time to think whether the item is relevant or not, uh, in thoughts tags based on the criteria above. Um, and you kind of, like, let the model think, um, a little bit more, um, with its reasoning. And then, uh, this is, like, basically chain of thought. You can also add, like, criteria. Uh, so as a part of your critique, consider the following criteria. And so if you want, like, steer the model on, like, does the item match the specific attributes requested by user? Like, help the model to, like, think kind of, kind of, like, think through, like, what means what does it mean to, like, recommend an item to the user? Does the item match the season? Or whether conditions match in, in the user's query? For example, you should not recommend winter coats during summer seasons. So, like, in the criteria, you can, like, give more examples, more elaborate examples. Um, another thing that you iterate on is, like, not just, like, give answer yes or no, but you can, like, based on your critique, score whether the item should be recommended or not, where 1 is least to be recommended and 10 is highly recommended. And put the final score in score tags. And so how does it work? Uh, so, um, here, like, user query James Bond blazer, um, item, which I took from, um, I think it was some brand. Um, and Claude would, like, start, like, thoughts tags. Um, overall, it seems, like, very relevant. And the final score is 9. And here's another example. I want a dress in the style of the Great Gatsby movie. Um, here's the item. Braided cord, cropped waist coast. And the critique is basically, uh, the item is not appropriate for the user's needs based on the context clues in the query. Um, it doesn't, like, you know, match the attributes of the Great Gatsby movie. It, like, tries to, like, have, like, some reasoning. And so the score is 2. And you can be a little bit more elaborate. This is, like, very simple, like, iteration, um, on, on that. Do you guys have any questions? Yeah. **Host** [17:07] So one interesting thing that I saw, I mean, there was the XML tags now here that weren't closed. **Karina Nguyen** [17:12] Yeah. **Host** [17:13] My friend, he is not a native English speaker. His prompts are always in, in very kind of funny English. But he structures them really well, and they work really well despite the English being very incorrect,right? **Karina Nguyen** [17:26] Right. **Host** [17:27] Why? Why does that work? **Karina Nguyen** [17:29] I think it's just the models are, like, pretty good at, like, knowledge transfer between, like, languages. Or, like, can infer very well on, like, the user's intent. Um, yeah. I don't have, like, pretty clear answer. **Host** [17:44] No, no. Probabilistic. So that text with one syntax mistake looks close enough to the text with theright syntax. So the probability of the real answer is close in both cases. **Karina Nguyen** [17:55] Right. **Host** [17:55] Like, yes. I don't know. **Karina Nguyen** [17:57] Yeah. **Guest** [17:57] In this particular example, I'm curious on whether you see any bias with the score. **Karina Nguyen** [18:03] Mm-hmm. **Guest** [18:03] In other words, if you were to look at the distribution of scores. **Karina Nguyen** [18:06] Right. **Guest** [18:06] Would it be a normal distribution? **Karina Nguyen** [18:08] Yeah. This is an interesting question. Like, this is one question that we ask in our research settings. Like, one thing that we are trying to understand. Like, we have a research group, uh, called Societal Impacts. And one thing, uh, that we're, we are trying to understand now is, like, when you summarize, like, news articles and you try to evaluate, like, the bias, what's kind of the distribution? And I feel like this is, like, research active. Like, yeah, I think it depends on the task. Um, really. I did not test on this. Literally, it was yesterday, uh, prompting. Yeah. Cool. Um, the second task, um, so Claude is known for 100k context, uh, size, which is the entire book of the Great Gatsby. You can, like, put into the context, and you can, like, ask the model, uh, summarize the book. ### Long Context **Karina Nguyen** [19:03] Or, like, uh, ask some tasks based on the huge context. And this is, like, basically time test compute, um, thing. And so with long context, um, let me see. The way you can use long context can be in different ways. Like, one way is, like, you put multiple documents and try to summarize. Or, like, uh, retrieve information based on the documents. Another way to use long context is to have a bunch a huge few-shot prompt. And so, as you know, like, chain of thought, um, technique relies on the stated reasoning, faithfully reflecting the model's actual reasoning. And in one of the recent papers, we found that it's not super like, it's not always the case. So it doesn't so basically, what it means is that, like, if you ask the model do a chain of thought, it might not necessarily attend to, you know, chain of thought, uh, to produce the final answer. It might just, like, ignore it. Or, like, uh, not take any account. So it's not we, we call it, like, unfaithful, basically. It's not super faithful. And so we proposed in this paper, um, like, decomposition-based methods can actually achieve, like, strong performance on specifically question-answering tasks, sometimes approaching that of chain of thought performance while improving the faithfulness. Do you guys have any questions? **Host** [20:46] I didn't understand that. Sorry. **Karina Nguyen** [20:48] Okay. Yeah. **Host** [20:50] What is faithfulness here? Is the faithfulness to your prompt or to your? **Karina Nguyen** [20:55] Um, faithfulness is, um, yes, to your prompt. Yeah. **Host** [21:01] What's decomposition? What's decomposition? **Karina Nguyen** [21:04] Yeah. Uh, let me explain what decomposition is. So here's the graph from the paper. Uh, we have, like, three methods. First is a chain of thought method, which is, like, uh, here's the question. Could, could your do fit in a kangaroo pouch? Uh, there are two choices, A yes, B no. Chain of thought prompt saying, like, let's think step by step, gives the reasoning. Um, the human asked the follow-up questions based on the above. What is the single most likely answer choice? Uh, and the model says the correct answer choice is B. Right. The chain of thought decomposition is when you decompose a question when you can ask the model to decompose a question into, like, multiple sub-questions so that each sub-question are kind of independent from each other. Because in chain of thought, like, you have one, two, three, you know, like, they kind of, like, can influence each other,right? Like, in decomposition, you kind of, like, uh, you, you decompose and you, like, put each sub-question in the independent context. So in a way, it kind of, like, reduces the bias. Um, and so in this, um, let's, let's see here is that, like, sub-question one, uh, what type of animal is Scooby-Doo? The answer from the model, Scooby-Doo is a fictional character. Another sub-question for, for the assistant, for Claude, how big is an average kangaroo pouch? And you and what you can see is that, like, each sub-question is kind of, like, self-contained. It's very atomic, self-contained question. Um, and so you have, like, multiple sub-questions like this. And then what you do is you recompose. So, like, you, like, put sub-question, answer, sub-question, answer, sub-question, answer into, like, one context and ask the model based on the above, what is the single most likely answer choice? The correct answer choice is B. Yeah. **Host** [23:09] Um, in, in chain of thought, like, in the system prompt or whatever the user's prompt is, like, we mentioned listing step by step. **Karina Nguyen** [23:16] Mm-hmm. **Host** [23:17] Uh, what, what do you do for the decomposition? Is there, like, a similar, you know, input to the model to make it decompose into multiple questions? **Karina Nguyen** [23:27] Yeah. Uh, I can share the prompt, um, in a few slides, uh, on this. Um, but, uh, yeah. Any other questions? **Host** [23:35] Did we show the graph again? Now I know where it is. **Karina Nguyen** [23:38] The, this graph? **Host** [23:39] Yeah. **Karina Nguyen** [23:49] Um, yeah. Let's look at the prompt. Um, very hard to see, but I'll share the slides. Um, let's, um, I, I'm, I'm gonna give you, like, legal context, like, legal question. Let's say you have a question on, like, a legal question. And you ask us, like, which of the following is the most persuasive argument that a person is liable to the creditor under the terms of the agreement? And here's the context. So that's the question, basically. And so you have, like, choices for the model. So this is, like, multiple-choice question. And before that, you have, like, a huge few-shot prompt. Um, and basically, here, uh, to answer your question, like, it says, I'm going to give you a question. I want you to compose into a series of sub-questions. Each sub-question should be self-contained with all the information necessary. Um, this is really important, blah, blah, blah. Uh, make sure not to decompose more than necessary. Um, be concise, blah, blah, blah. Please put each sub-question in, like, these tags, but include the numbers corresponding to each tag. So, um, and the model says, yes, I understand. Uh, you have a question, uh, multiple-choice answers, and the model provides sub-questions for you. And then what you do is that you try to answer the first sub-question, and you give it to the model. You try to answer the second sub-question, you give it to the model. Third sub-question, you give it to the model. And then later, you say, like, based on everything above, like, you give all the context, um, answer me the question. Um, the correct answer is C. Yeah. And so this is, like, very similar in the legal context. You have sub-questions, like, what is consideration of contract law? Blah, blah, blah. And you have you can have, like, another model to, like, sample here. You can have, like, another model to answer this. It doesn't necessarily should be, like, one model. Um, and then there's, like, another sub-question, and here's the answer. Yeah. Do you guys have any questions? Um, ### Evaluations **Karina Nguyen** [26:27] the second thing that I wanna talk about is how to use Claude to do evaluations. Like, evaluating, like, Claude on, like, long context ability. Let's say you have a lot of, like, documents, and you want to understand how good Claude is answering questions based on the document. Or is it is it able to answer, like, the questions not just, like, from its pre-trained knowledge, but, like, based on the document itself? And so, um, I'm gonna give you an example that we did at Anthropic, um, multiple-choice QA, um, evaluation design. So our goal was to with this experience to evaluate techniques to maximize Claude's chance to correctly recall a specific piece of information from a long document. And so the document that we chose was a government document that contains, like, a bunch of, like, meeting transcripts, different departments. And we also chose the one that was, like, uh, from this year, July 13th, uh, which is, like, way after Claude's, um, training data cut off so that you don't, like, um, you have the document that does not have in the pre-trained knowledge or something. And so what you're trying to do is, like, now you want to use Claude to generate question-answer spares. Um, you, you in a way, like, you create, like, data, dataset-based. You use language models to create, like, datasets. And so the way you do that is that you split the document into sections and use Claude to generate, like, five multiple-choice questions for each section, each with three wrong answers and oneright answer. And if you do that, you then reassemble, like, randomized sets of those sections into, like, long documents that you could pass them to Claude and test its recall, uh, of their contents. This is very meta. Let me know if you have questions. Yeah. **Host** [28:42] Excellent. **Karina Nguyen** [28:45] Um, so here's a prompt to generate multiple-choice questions. Um, I ask we ask, like, please write five factual questions for this, um, some guidelines at the end. Um, and basically, we test different strategies, prompting strategies. Just asking Claude, give Claude two fixed examples of correctly answered general knowledge, and, uh, that are unrelated to the government document. Um, providing two examples and providing five examples of correctly answered questions. And we tested the strategies, uh, on different settings. Like, one is containing the answer positioned at the beginning, the end, or the middle in the input. And we tested with, like, 70K, 95K token documents. You can look at the prompt and more specific how we did this in our blog post. But basically, the result is this. Uh, here we see that, um. **Host** [30:00] Mm-hmm. Uh, what is the metric? **Karina Nguyen** [30:05] Yeah. The metric was, um, to let's see. Like, basically, how many how many times, like, Claude has correctly answered the question. Um,right. And so yeah. Sorry. Uh, basically, what we find is that, like, for document Q&A, asking the question at the end of the prompt performs a lot better than asking at the beginning. You can see it here. Uh, pulling relevant quotes into, like, critique or, like, thoughts tags is helpful. Um, it's, like, it's a small cost to latency, but improves accuracy. Uh, and we tested on, like, both Claude and Claude Instant. Um, and it seems like you can boost way better performance from, like, Claude Instant, um, than Claude 2. Basically, the idea is that, like, if you wanna use long doc Q&A, put the instructions at the end of your prompt. Yeah. That's, like, the result of this. **Host** [31:27] Right. What was the I didn't catch what was the scratchpad in the? **Karina Nguyen** [31:33] Uh, oh, yeah. Like, you just ask the model to, like, put thoughts in, like, thoughts tags before answering the question. So it has, like, more reasoning-based first. Yeah. **Host** [31:42] Can you go to the table again? Sorry. I'm sorry. **Karina Nguyen** [31:44] Yeah. Sorry. I'm, uh. **Host** [31:47] Sorry. I'm sorry. I'm just gonna close that. But, like, the outcome was basically putting it at the end matters more than all the other. **Karina Nguyen** [32:03] Yeah. **Host** [32:03] Optimization strategies. **Karina Nguyen** [32:05] Right. Yeah. Um, yeah. In a way, this is, like, an example to show, like, how to use Claude to generate a dataset that you can, like, evaluate. And, like, you can use it for, like, evaluation, basically. Yeah. **Host** [32:28] So this has to do with putting your instruction at the end of the prompt. **Karina Nguyen** [32:32] Mm-hmm. **Host** [32:33] Uh, first, are there theories on, like, why, why specifically the instruction should be at the end? And are there any things like, do we have any understanding of, like, are there certain things at the beginning of the prompt that still might be weighted? Or is it, like, this sliding scale that, like, the further in the beginning of the prompt, like, the less attention it gets or something? **Karina Nguyen** [32:53] Yeah. I think that's basically the hypothesis. **Host** [32:55] Okay. **Karina Nguyen** [32:55] It's like the, you know, it-it's like the distance. It's like the model attends more to the end of the prompt. **Host** [33:03] Okay. **Karina Nguyen** [33:04] Uh, than the beginning. **Host** [33:05] So like a newer or basic stat? **Karina Nguyen** [33:06] I think there was a paper saying, like, it just forgets in the middle or something. **Host** [33:10] Oh, okay. **Karina Nguyen** [33:11] Um, yeah. I think this is the problem with, like, long context that we're trying to fix or something. Yeah. **Host** [33:17] So to follow that question, you are saying that there was a paper that said that it remembers the beginning and the end and kind of forgets in the middle. **Karina Nguyen** [33:25] Yeah. **Host** [33:25] But what you're saying is for Claude 2, it seems to do best if you give it at the end. **Karina Nguyen** [33:30] Yeah. **Host** [33:30] So that paper doesn't apply to Claude. **Karina Nguyen** [33:33] Um, I did not read that paper, like. **Host** [33:38] No, I'm just curious. **Karina Nguyen** [33:39] Yeah. **Host** [33:39] But what you're saying is you're finding it at least for Claude. **Karina Nguyen** [33:42] Right. **Host** [33:43] The end part gets more attention. **Karina Nguyen** [33:45] Yeah. For, like, a specific task is, like, long context, uh, Q&A for, like, long documents. Yeah. Um, yeah. We have not tested on other tasks, to my knowledge. **Host** [33:59] Um, so the prompts you showed were using regular pros and then the XML tags. **Karina Nguyen** [34:06] Mm-hmm. **Host** [34:07] Um, I think that's also what's in the Anthropic docs. Have you guys ever done experiments on, like, that kind of format versus markdown versus everything is in XML? Do you have any thoughts on that? **Karina Nguyen** [34:20] Yeah. So, um, in general, I think I think it's because markdown was kind of like there's not that much of, like I don't know. It's, like, best in XML tags. Like, I'm, I'm, like, thinking v I've, like, tried Claude to, like, you know, use, like, JSONL or, like, uh, use markdown. But sometimes it's, like, you know, it's not as good as, like, XML. With XML, it's almost 100% accuracy. Yeah. Let's see. Yeah. Um, let's go to another task, um, which is, like, you can use language models to, like, auto-label basically anything. Um, so one of the examples that we did last year, um, we asked Claude to categorize the labels for the clusters. ### Auto-labeling **Karina Nguyen** [35:28] And so, um, this was for the paper, but the approach was very simple. We have a bunch of, like, you know, text, and we embed them in UMAP. Um, and we do, like, KNN clustering. And for each cluster sorry, KMENS clustering. And for each cluster, select, like for each cluster, aggregate all the, you know, little, like, statements or claims. And we ask the model to come up with the category for this cluster. So that's the approach. And you can look at the other labels. Uh, labels here are not super good because we used Claude 1.3 at that time. Claude 2 is supposed to be, like, way better at this. Um, but this is, like, you know, cached. It was, like, last year. Um, where's my slides? And so one thing that you can do with this kind of task, we call it self-consistency. You can generate end samples for the question. Um, so let's say you have a question, like, how do you label this cluster? And you generate independently end times. And you can ask just, like, come up with, like, one category. Um, well, this method is mostly useful for, like, quantitative, like, if you have, like, a math question and you sample, like, different like, sample multiple times and come up with the answer. Um, like, the most common answer is, uh, the one that you select for the final answer. And this is called the majority of vote. Another technique that you can use is, like, um, have, like, two generated samples and ask another model to evaluate whether those samples are consistent or not. And if the samples are consistent, well, you gain more confidence that this is correct,right? And if it's not consistent, you just, like, deselect. Um, another thing that you wanna do with Claude is, uh, if you if Claude is kind of, like, misses the nuance, especially for, like, categorizing a lot of labels and you have, like, a lot of, like, categorizations, um, you can add contrasting conceptual distinctions in your instruction. And you can do it in multiple ways. One way to do it is, like, you provide bad example. Let's say, like, here's a very bad category, and you should never come up with it because this is, like, too narrow or, like, too general. And this is not what I want. Uh, like, give, like, contrastive, like, examples. Uh, vary the contexts. Use examples in different contexts and settings. Um, not just, like just, like, have, like, more diversity. Like, diversity is, like, um, and, uh, uh, the more diverse, like, future prompt examples, the, the better. Use analogies and metaphors. Um, if the concept is, like, too hard to understand for the model, try to, like, decompose and, like, bring analogy. Um, point out, like, common misconceptions. Um, especially for, like, categorizing, like, let's say, what is false presupposition,right? Like, uh, point out the common misconception and, like, clarify, like, why this is, like, incorrect. Like, provide examples that explicitly show why common misconception is wrong. Uh, yes. Do you guys have any questions? Yeah. **Host** [39:32] So for the, the latency, the goal is you rank your means to get most. And then do you wanna sort of get into the group? Or is that the, the only action? Just to clarify. **Karina Nguyen** [39:44] Um, I cannot super hear, but, um. **Host** [39:46] Yeah. If so the goal here is you're, you're running, uh, a clustering algorithm to get group. **Karina Nguyen** [39:51] Mm-hmm. **Host** [39:51] And then the, the task of the LLM is to put a name on each group? **Karina Nguyen** [39:56] Yeah. **Host** [39:56] Okay. Okay. **Karina Nguyen** [39:57] Like, come up with a category. Like, um, yeah. Come up with a category or, like, classify, uh, like, label that cluster, basically. Um, so here's, like, ba very basic, like, tips and strategies with Claude API. Um, number one is formatting. Um, like, human assistant is, like, what Claude loves. And if you misses you miss it, like, you'll get, like, very, very terrible results. ### API Tips **Karina Nguyen** [40:27] Uh, newline, newline human, newline, newline assistant. Um, yeah. Uh, you can also put words in Claude's mouth to, like, kind of, like, say, like, do you understand it? And you can, like, put in the, um, Claude's mouth, "Yes, I understand it," in a way to, like, you know, put, put the model in, into this mode. Have Claude repeat instructions back. Um, you can say, like, "Do you understand the instructions?" Um, and you can put, like, "Assistant, yes, I understand instructions." Blah, blah, blah. Uh, to reduce hallucinations, like, let Claude hedge and, like, say, like, "I don't know," or, like, uh, "I don't have enough information or, like, context to answer the question." Um, here's another thing. Um, if you have, like, generate direct quotes. If you have, like, a document or, like, um, a long document in the context, um, make Claude to say, "Find appropriate quotes," but also say, like, um, if there are no quotes in this document that seems relevant to this question, please just say, "I don't find any relevant quotes," so that it doesn't make up or fabricate new quotes. Uh, how to give good examples. Um, are the examples similar to the ones you need to classify? Are the examples diverse enough for Claude not to overfit to, to the specifics? Equally distributed among answer types. Don't always choose option A, but, like, you, you kind of, like, have the diversity. Um, yeah. I get a lot of questions. **Host** [42:31] Oh. **Karina Nguyen** [42:31] Yeah. **Host** [42:31] When you first got into formatting because the, the ad copy was wrong, I can't see exactly what's wrong with formatting in that context. **Karina Nguyen** [42:40] Oh, formatting in a way oh, here? **Host** [42:42] Yeah. **Karina Nguyen** [42:44] Um, I think they didn't put, like, newline, newline. Pretty sure. Oh, the h or here. Sorry. Here, you put, like, human assistant inside the XML tags. You only need you only have to use human assistant as, like, tokens to, like, sample, but you should never put it in, like, like, inside the context itself. Either use, like, user and, like, other, like, um, you know, um, words, like user AI or something, or, like, H or A. But you should never use human assistant. Human assistant is, like, very special, special words. **Host** [43:30] If, if that didn't have the XML tags, would it would it be okay? **Karina Nguyen** [43:34] Um, it would be okay, but you would, like, make you should, like, have human and then assistant in between and then human assistant and then another human and assistant, basically. Uh, yeah. Formatting is that, like, human assistant, human assistant. You should never have, like, human human assistant assistant or something. Uh, that's bad. Um, yeah. I think we have, like, more extensive, uh, explanations in the API docs. If you can look at it. Yeah. I get a lot of questions, like, what the future of prompt engineering is. Um, and I think the answers are pretty clear. ### Future **Karina Nguyen** [44:17] Like, prompting will stay. We'll just ask, like, more complicated, nuanced questions or, like, tasks for the model. Um, prompt engineering is a we will, like we are moving towards the world where we'll have, like, more and more synthetic data generation. And so I'm pretty optimistic about, like, mod using models to, like, generate, like, diverse sets of, like, data sets. Um, you can also use language models to write, like, evaluations. Um, so you use prompting to do that. Um, reinforcement learning from AI feedback, um, is an alternative to, like, reinforcement human from human feedback, which is, like, a little more scalable. But basically, you ask the model to revise its own, uh, responses in the process. So you give the model you, you, like, ask the model to, like, self-reflect or, like, self-revise. Um, and so you use prompting in that process to do this. And especially, like, prompt engineering will become, like, a, a standard part of, like, product development. I feel like, um, things that we did in Claude products, such as, like, auto-generating titles. Like, these things was never, like, done before, like, l before, like, large language models. And so you can, like, create delightful mini UX experiences, uh, such as, like, that using just prompting or something. And you can, like, have personalization. Uh, maybe you can embed all the users' conversations and, like, suggest, like, new topics for the conversation. Um, and you can use models to do that. Um, and the most, like, interesting thing is, like, uh, finding most optimal prompts for specific tasks. Maybe you want to, like, minimize the number of tokens to get the highest accuracy for the task. Um, yeah. Uh, here are some resources. Uh, we just, uh, launched Anthropic Cookbook with, like, certain, like, demos on research uh, on retrieval and search. Um, we have prompt design guide in API, uh, book. Um, you can also read out the papers that we publish. Uh, oftentimes, we have, like, appendix with, like, all the prompting that we do. Yeah. Thank you so much. And, uh, if you have any questions, let me know. **Host** [46:51] Yeah. We have five minutes for questions. Charles is coming up. We also have water. Thanks to Sean, um, for bringing in some water. So, uh, yeah. Hydrate the people. But five minutes for questions for anything about Anthropic or prompt engineering. ### Q&A **Karina Nguyen** [47:09] Yeah. **Guest** [47:09] So have you tried these with different things besides Claude? Like, other, have you had similar kind of results? 'Cause you're talking about Claude in this particular way. Like, maybe, like, with OpenAI or other? **Karina Nguyen** [47:24] Yeah. I think, uh, most, um, experience with Claude because I use it, like, every day. Um, less experience with GPT. Uh, I did not look carefully, to be honest, at their, like, API docs. Um, but it seems like the strategy is a little bit different. Yeah. They don't have, like, formatting as we are, let's say. Yeah. **Host** [47:50] Uh, great, great presentation. And, and, uh, well, it was very interesting. **Karina Nguyen** [47:53] Thanks. **Host** [47:54] Uh, have you seen this prompt reader that came out recently? Is that, uh, where you basically use an element to improve the prompt, uh, considering the outcome? Uh, do you see that in, in the sight of future, say, optimal tasks? Is that what you mean? Or do you see other directions for, uh, optimizing prompts? **Karina Nguyen** [48:13] Yeah. I think, uh, that's, that's actually one of the directions too, like, um, rem don't remember how what was that paper called? Like, LLMs as, like, optimizers, I think,right? Um, but yeah. I guess, like, um, in a way, there are, like, certain tasks that the models are, like, not good at currently. Like, for example, like, self-correction. Like, the models are not really good at, like, self-correcting their, like, answers. And, like, can you find, like, a prompt that was, like, pretty good at it? Or, like, um, other tasks that you want. Yeah. **Host** [48:51] So I'm curious about. **Karina Nguyen** [48:52] Yeah. **Host** [48:53] What techniques your team is using for actually, like, evaluating the quality of the responses. That's something that my team has kind of struggled. Like, how do we actually automate this? **Karina Nguyen** [49:02] Yeah. I think, um, depends on the task. Sometimes we just, like, have to look manually, qualitatively, uh, at outputs. Um, sometimes you, let's say, um, you wanna evaluate, you know, how much does the model refuses and if it refuses in a relevant context or not. And so, uh, you use, you know, generated answers, and you categorize refusals in different categories and use the model to categorize that. And so you just, like, see the rate. Um, yeah. I can think of that example. Yeah. It depends on the task. Some tasks are, like, you know, um, for, like, hallucinations, you actually have to, like, look yourself or something. Yeah. Yeah. **Host** [49:54] So, uh, you mentioned, uh, something about clustering, uh, different kinds of questions that we need to ask different kinds of bodies. And, uh, we're going to label. Uh, so have you tried using OpenAI functions, which has, uh, some enumeration, enumerated values that are predefined classes or intents, user intents? And have you seen if that improves the accuracy of the classification? Or have you tried anything else? **Karina Nguyen** [50:20] Uh, with OpenAI? **Host** [50:22] Or, or for any kind of function calls. Uh, so the OpenAI function call is a model, uh, which any kind of, uh, model will take structured data where you can, uh, identify the, uh, all the functions after. **Karina Nguyen** [50:35] Yeah. Yeah. I think, uh, um, I won't say too much about this, but I, I actually have not, like, extensively used function calling from OpenAI, like, other models. **Host** [50:46] Yeah. It limits, uh, the classification if you do. **Karina Nguyen** [50:49] Right. **Host** [50:49] Things like that. **Karina Nguyen** [50:50] Yeah. Cool. Yeah. Yes. **Host** [50:55] So you mentioned something about, uh, using an LLM to generate titles for a customer. How do you actually try to evaluate if the titles are relevant. **Karina Nguyen** [51:05] Mm-hmm. **Host** [51:05] And factually consistent. **Karina Nguyen** [51:07] Mm-hmm. **Host** [51:07] Beyond human? **Karina Nguyen** [51:09] Yeah. I think, uh, one interest well, actually, this is an interesting question. Like, I worked on the auto-generating titles for Claude.ai. And, um, one thing that I asked Claude is to be, like, an editor, like, have an editorial taste. And what we did is actually we took previous titles, and we put in the context to generate a new title. And so in a way, it's, like, a little bit more consistent to, um, what the style of the user is. Yeah. **Host** [51:42] Would you be able to share the recognization in terms of catching defects if they do happen? **Karina Nguyen** [51:47] Yeah. **Host** [51:47] When it comes to. **Karina Nguyen** [51:47] Uh, yeah. I'm not sure if I can share that. Oh, yeah. **Host** [51:52] Okay. Was it reliable enough for use in production? **Karina Nguyen** [51:54] Uh, yeah. We, we use it in production. I can, like, show you, like, Claude.ai interface. And, uh, one thing that we changed recently is that, like, if you have, like, pretty, like, short, like, um, you know, sometimes you don't have you don't need, like, LLM to come up with a title. You just take if, if the prompt is, like, very short, you just, like, um, use the, like, the first, like, words. Uh, but here, yeah. Like, I don't know. Um, let's see. Hey is introduction, but then, like, recommend some books. Um, yeah. I don't know. Uh, yeah. **Host** [52:41] Uh, I have a quick question. Did you cover the difference between Claude and Claude Instant? **Karina Nguyen** [52:45] Um, no, but I can tell. Uh, so Claude. **Host** [52:51] For those that don't know, she trained Claude Instant. And I'm like, you might as well cover the thing that you did. **Karina Nguyen** [52:57] Yeah. So let's look at the is there some docs on this? **Host** [53:03] I think you guys announced the 1.2. **Karina Nguyen** [53:06] When did we announce? Uh, August 9th. August 9th. Um, basically, Claude 2 is a larger model. Uh, is a little bit smarter is, like, smarter than Claude Instant. Claude Instant is way cheaper and way faster. But Claude Instant is better than Claude Instant 1, um, in, uh, like, more, like, reasoning-based tasks. So it's way better at math. Uh, it's way better at code. Um, other benchmarks are, like, pretty similar. But I think we specifically trained Claude Instant to be good at, like, math and code. Um, and it's, uh, way better at, like, red teaming, um, like, automated red teaming evaluation. So it's more robust to, like, jailbreaks. Um, yeah. I really like this model. You guys should use it. Yeah. **Host** [54:04] Yeah. **Karina Nguyen** [54:05] Yes. **Host** [54:06] So you expo exposed my ignorance. So when you talk about training, like, when you trained this client, was that ta fine-tuning, or was that something different? **Karina Nguyen** [54:13] Uh, yeah. Fine-tuning. Yes. **Host** [54:18] Last question. **Guest** [54:20] Can you say more, I guess, red teaming? Like, what is red teaming? **Karina Nguyen** [54:24] Yeah. Red teaming is an interesting concept. It's, um, basically, you like, the models are pretty, like, uh, vulnerable to, like, certain, like, jailbreaks. Um, so sometimes, let's say, like, a very simple example. Like, can the model give you instructions how to build a bomb? And so we, we consider it as a jailbreak. And so the goal is to, like, uh, in that cases, like, the model should, like, refuse or do not, like, provide any additional information in case of, like, um, unsafe, like, prompts or something like this. And so this is, like, the internal evaluation that we have. Um, you can read in the model card that we ha we launched in Claude 2 how we specifically do that. Uh, but it's basically the amount of, like, um, how robust the model is to, like, those jailbreaks. Yeah. Cool. **Host** [55:21] Great. Thank you very much. **Karina Nguyen** [55:23] Yeah. Thank you. **Host** [55:29] Yeah. Uh. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.