# Pragmatic AI with TypeChat: Daniel Rosenwasser AI Engineer · 2023-11-14 Daniel Rosenwasser, TypeScript program manager, introduces TypeChat, an experimental library that uses TypeScript type definitions to guide and validate unstructured LLM output into structured JSON for traditional apps. He demonstrates a coffee shop ordering system where types define the schema, showing how the library can handle ambiguous inputs like 'a purple gorilla' by including unknown text for recovery. TypeChat also generates programs as JSON using a fake language to safely script multi-step operations, avoiding sandboxing issues from real code. A Python prototype extends the same approach, with examples like CSV data manipulation via a class-based API. The library aims to make AI tools accessible to all engineers by leveraging types they already use. ## Questions this episode answers ### How does TypeChat use TypeScript types to get structured data from large language models? TypeChat is an experimental npm library that uses TypeScript type definitions both to guide an LLM in generating structured JSON and to validate the output using the TypeScript compiler. You define the expected schema, send it with user intent to an LLM, and TypeChat returns well-typed data. If validation fails, errors are fed back to the model for repair, ensuring reliable, schema-conformant results. [4:08](https://aie.addtry.com/b8c1971d-e9bf-444e-bed0-53f2d4d1ae7b?t=248000) ### How does TypeChat handle user input that doesn't match the expected schema, like a 'purple gorilla' in a coffee order? In the coffee shop demo, when the user orders 'a medium purple gorilla named Bonsai,' TypeChat's schema includes an 'unknownText' field. The LLM places the unrecognizable part there, and the app responds with 'I didn't understand: a medium purple gorilla named Bonsai.' This pattern lets you anticipate failures and recover gracefully, rather than getting junk data. [8:44](https://aie.addtry.com/b8c1971d-e9bf-444e-bed0-53f2d4d1ae7b?t=524000) ### Can TypeChat generate and execute multi-step programs, like a calculator that does 'add 1 to 41 and then divide by 7'? Yes, TypeChat includes a program translator for multi-step tasks. You define a fake JSON-based language representing a sequence of steps, and TypeScript validates that only allowed operations are used and inputs match expected types. In the math example, it evaluates 'Add 1 to 41 then divide by 7' by generating a safe program and interpreting it, effectively making a language model good at math. [14:29](https://aie.addtry.com/b8c1971d-e9bf-444e-bed0-53f2d4d1ae7b?t=869000) ## Key moments - **[0:00] Intro** - [0:35] Traditional apps require structured data, unlike chat-based AI interfaces that handle free-flowing natural language. - **[1:09] Problem** - **[2:55] JSON Output** - [2:55] "Parsing natural language is extremely hard, if not a fool's errand, for most people." - **[4:08] Types Guide** - [6:20] Types are all you need — they guide LLMs to desired output and validate the JSON results. - **[6:31] TypeChat** - **[7:08] Coffee Demo** - [8:06] TypeChat instantly translates "One latte with foam, please" into well-typed JSON for a coffee shop schema. - [8:44] TypeChat captures unrecognizable input like "a medium purple gorilla named Bonsai" in an unknown-text field for graceful recovery. - **[11:17] Commands** - [11:37] Chaining multiple commands requires variables, argument reuse, and program-like generation beyond simple command lists. - **[13:34] Fake Language** - [13:34] TypeChat generates a fake JSON-based language to produce safely sandboxed programs from LLMs, avoiding the risks of full code generation. - **[14:29] Math Demo** - [14:44] TypeChat makes an LLM good at math by generating a safe program for "Add 1 to 41 and then divide by 7" and evaluating step-by-step. - **[16:16] Python & CSV** - [16:16] Early Python support for TypeChat shows promise, matching the TypeScript approach with similar type-guided translation. - **[17:52] Outro** ## Speakers - **Daniel Rosenwasser** (guest) ## Topics Prompt Engineering ## Mentioned Anthropic (company), Cohere (company), Microsoft (company), OpenAI (company), Llama (product), NPM (product), Python (product), TypeChat (product), TypeScript (product) ## Transcript ### Intro **Daniel Rosenwasser** [0:15] Good afternoon, my AI engineering friends. How are we all feeling today? Great. Woohoo! **Daniel Rosenwasser** [0:20] There we go. We got some energy, even post-lunch. Allright, you heard: I'm Daniel Rosenwasser. I'm the Program Manager on TypeScript, as well as a new little experimental library I'm here to talk about today called TypeChat. Now, this is an AI engineering conference. Everybody here has used something like ChatGPT,right? We've used it for this continuous flow of information. We've been able to prototype things with it, just get useful answers just by having this adorable little chat interface,right? But that's this one end of the spectrum. And on the other end of the spectrum, we have our traditional apps, these apps that are looking for this more precise sort of data to work with. So the question is: how do we make all of the new AI tools, all these language models that are so powerful, accessible to every engineer out there? ### Problem **Daniel Rosenwasser** [1:09] And so, just to start things off, what if we had this cute, you know, this little appright here. You have some basic user input at the very top, followed by these items, and each of these items has a venue name and a description. So this just helps me figure out what I need to do on a rainy day in Seattle, because this is every day in Seattle for me. A lot of weather apps at this conference. But the problem that you may find with trying to bridge together these language models and these traditional apps is that you find that you need to sort of massage the data. You need to sort of, like, really, really, really pamper the models to give you what you're looking for. And even after all that's said and done, by default, these apps will give you natural language, which is great for people, but it's not great for code. So if we just prototype this in, you know, something like a ChatView, maybe you'd actually use the Playground to do this, you'd find yourself saying certain things to pamper, like, "Keep it short," and "Do this," and "Put everything on a different line," and "Do whatever." You might find that you're starting to glom onto the patterns of what the language model gives you because you've seen it in a certain way,right? And you've noticed, "Oh, well, it gives me this format." Each of these things is on its own line. Each of the lines has a leading number. They're always separating the venue name by the description, by a colon. So I'll just do some basic parsing: split by newline, remove the trailing, the leading numbers, and then split on the colon. That is a disaster waiting to happen, because you can't rely on the language model to always do this. And you can't know whether or not you're going to have something in the middle of that input that is going to just sort of wreck your parsing strategy,right? Parsing natural language is extremely hard, if not a fool's errand, for most people. The thing that many people at this conference and elsewhere have discovered is you can say, "Pretty pretty please, give me some JSON." And it works pretty well,right? ### JSON Output **Daniel Rosenwasser** [3:13] You know, just, "Here's an example of what I'm expecting. Please respond with the answer." And voilà, it comesright back. But there's two issues with this. One is, just doing that on its own is not enough to guarantee that your app is actually going to get the data it's looking for, because maybe there's an extra property that doesn't seem to align. Maybe there's not enough data in the actual response. So you need to do some level of validation. But not just that. You can't comprehensively describe all of the things that you want, practically. In this case, I have really, really simple schema, or a really, really simple, like, example. All the objects are uniform. They all have the same properties. End of story,right? But what if something is optional? What if something is required but needs to be null in some cases? What if this could be a string or a number but never something else? I don't know. So you will not be able to get that far for more complex examples, because you end up with this combinatorial explosion. So what we found is that you can use types. Types are this great description to actually guide the model. ### Types Guide **Daniel Rosenwasser** [4:26] Here, I'm just using type definitions in TypeScript. These are just plain interfaces. All I want is a thing with a list, and the list has these objects, and the objects have these two properties that are both strings on them. And the beauty of these type definitions is that the types can guide the model,right? So you can actually use these types to tell a model, "Hey, here's some user input. Here's a user intent." Now use this with the types that I'm actually going to use in my application. Throw it through your cool AI service, whatever that is. That may be OpenAI, Cohere, Anthropic. Maybe it's a local model. Maybe it's Llama code. I don't know. But the point is, what we found is that if you use a language model that is sufficiently trained on both human prose, natural language, and code, this actually bridges the two worlds together. But like I said, the guidance is not it's only half of the problem,right? You need to be able to actually validate what you're getting. And that's the key insight, is that the types can also validate the results. And so what we found is, in our experience, you know, we're using TypeScript. TypeScript's great for JSON because it's a superset of JavaScript, which is a superset of JSON, which means that you can actually construct a miniature little program that, underneath the hood, the TypeScript compiler is using to do that validation. And if that all goes well, then great, you have well-typed data from your language model. And if it doesn't go well, well, underneath the covers, what we actually end up with is an error message,right? Because it's actually using the TypeScript compiler under the hood. That error message can be used to perform a repair when you are reaching out to a language model to say, "No, no, no, no, no. That's not what I wanted. Try again." And so the key insight is, types are all you need. Types can actually guide and validate, and it becomes a very powerful model, because whoops. What's that? **Daniel Rosenwasser** [6:27] Well, yes, actually. That's the key insight that we have with TypeChat. It's a library on npmright now. It's a TypeScript library at the moment. And basically, we've bundled this all together and made it easy to just guide a language model, perform these queries, and actually, like, make sure that you're actually getting well-typed data from the language models. ### TypeChat **Daniel Rosenwasser** [6:51] And so you can actually use much more complex examples as well. You might say, like, "I have a coffee shop, and the coffee shop has this schema, these types. You define them like this." And basically, you can use that to combine that with a user intent and input, and you get well-typed output. ### Coffee Demo **Daniel Rosenwasser** [7:08] And I'll actually demo thatright now. What I have here is my, you know, the TypeChat repository cloned, npm installed, everything set up, and we have an examples directory. And I think if you're just curious to get started with TypeChat, the examples directory gets you started. We have a table. If you look at the README, we have a table of all of our examples. They kind of increase in complexity and difficulty. And the first one is like a sentiment thing where we say if something is positive, negative, or neutral. But that's so basic, it's like our hello world. I actually want to go back to that coffee shop example that I showed you just now. So we have this coffee shop schema, and this is just a bunch of types,right? You probably have something similar in your preferred language as well. And what I can do here is I'm just going to run our entry point. And from the command prompt, I actually have a little prompt, and I can actually just make orders here. So I can say, "One latte with foam, please." Ta-da! Right? Yeah. So, you know, it's this is the key thing, is that it's actually so simple, and it actually just works very well in a surprising way. Now, that's I could just tell you about that, and I could walk off, and that's not really good enough, I know. What happens if I say, "One latte and a medium purple gorilla named Bonsai"? So what actually happened here is, technically, when we ran this prompt, this thing succeeded. But even though we got a successful result, we were able to do this sort of recovery here. We actually, in our app, are able to say, "I didn't understand the following: a medium purple gorilla named Bonsai." And that actually showed up in the JSON. And the reason that it did is because we have this thing called unknown text. So we've started to see these patterns in that, instead of doing this sort of prompt engineering, you're doing schema engineering. You're able to sort of thread through these results into your app, because if you actually, you know, remove this stuff and let me show you what this actually looks like. If you look at the coffee shop example, this is under 40 lines of code,right? The magic here actually comes from we create a model, we infer it based on your environment settings, and then the actual magic is that we have this JSON translator. You give us the contents of your types, you select the type that you're expecting, and then every single time you need to translate a user intent, you just run this translate function. Now I'm getting type errors because I removed a type, and it's telling me, like, "This will never happen." Whoops. Not that. So if I rerun this thing and I say, "One cappuccino cappuccino" I can't spell anything today "and a purple gorilla named Bonsai." I want to be precise here. So I got a bagel with butter, because I asked for a Bonsai. And the thing is that the language what's going to happen is that the language model really doesn't want to disappoint you. It really wants to make sure you're getting what you want. So this is the thing, is you can actually define a schema that is rich enough to encounter, you know, anticipate failure, gives you a chance to recover, show that to the user, say, "I got this and this and this and that. It wasn't so clear on that." And that's kind of the beauty of this approach. It's very simple, and it's really just about defining types, which you're going to use in your application anyway. Now, there's this other thing that we started encountering when we showed this off to teams internally. ### Commands **Daniel Rosenwasser** [11:17] People said, "Well, that's all cool. You're turning coffee into code." I do, too. "How do I actually do something more rich, like commands? What if I want to actually script my application in some way?" Well, this approach that I just showed you actually works for very simple stuff as well,right? You can imagine something where you say, "Schedule an appointment for me," and that turns into a specific command for a calendar app. In fact, in our examples, we actually have that. What if you want to string together multiple things? Hey, that's just a list of commands,right? Kind of. What's the problem with this is, if I want these to kind of thread through to each other, this is a simple example. So it's just going, "Input," you know, "Run the command, get the output, go to the input," et cetera, et cetera, et cetera, et cetera. What if you have something that expects multiple arguments? What if you want to reuse a result? Sure seems like you need variables and other things like that here. So we asked ourselves, "Is there a thing here where you can imagine you can just generate code and just take the same approach where types are all you need?" So what if you could just define, "Here's all the methods that I want you to be able to call. Come back with some code that only calls those methods, and then generates a program like this." The problem is that you really want to have some sort of sandboxing and safety constraints in place,right? And so you might start saying, "I need availability. I can't just endlessly loop here. So I'm not going to allow loops. I'm not going to allow lambdas and whatever." And the problem is that even if you decide, "I'm going to pick a subset of a language, like JavaScript or Python or whatever you have," the language models have seen so much of that code that they're going to draw outside the lines. And then you'll hit this failure case, and then you just won't get a result. You won't get a bad result. You just won't get a result that conforms to what you're expecting. And then you still have to worry about sandboxing. And then there's all these questions about synchronous versus asynchronous APIs and all this other stuff, too, that language models don't tend to understand, because I guess most people don't either. So what we actually have been trying is we generate a fake language. We have the language models generate a fake language still based on the types, but it's in the form of JSON, actually. And so you have things like refs, and refs are just, you know, references to prior results. ### Fake Language **Daniel Rosenwasser** [13:50] And if you're familiar with, like, you know, if you're a compiler, this may look like SSA. It might look like an AST, whatever. But we use that to construct a fake TypeScript program in memory as well, and use that to make sure that not just are you calling all the only the methods that are available to you, that you can only do certain actions, but also that the inputs from prior steps matches up with the types that you're defining from your API. And so that kind of comes back to types are all you need. We have another really simple example for we have a math schema. This is basically a calculator in sheep's clothing. So if you go back and we run this here, we have another prompt. That's an abacus. That's the closest thing to a calculator I could get. ### Math Demo **Daniel Rosenwasser** [14:44] If we could say something like, "Add 1 to 41 and then divide by 7." Now, basically, what happened here is we made a language model good at math. So we've also solved a whole other set of problems. Right? Yeah. More seriously, though, so at each of these steps, we're actually performing not having the language model call a method, perform an operation. And if you actually look at the code here, math main, this is all under 50 lines of code. We are able to do the same sort of translation. We have a separate thing called a program translator. And in that program translator, when you are successfully able to validate your results, you know, you say, "If this thing is a success" or not a success, just jump out. Otherwise, do some stuff with it. We have this evaluate function, and this evaluate function takes a callback, and that callback is just sort of like this instruction interpreter. And so you can do this with objects, you can do this with a function, with a switch case, or whatever. But the point is that this actually allows you to do some richer tasks. Now, there are other approaches for many of these things, and they overlap with what TypeChat does. But the cool thing is that TypeChat is able to actually give you this level of validation for both JSON and programs. And it's something that we're also experimenting with with other languages, too. So, for example, people at this conference have been saying, "Yeah, I, you know, TypeScript is very cool." And I agree with them, because I work on TypeScript. But how would I work make this work with Python? And so we have been experimenting with this, and we've been getting fairly good results. ### Python & CSV **Daniel Rosenwasser** [16:32] I'm able to do something like the coffee shop with a very similar approach using types. I'm able to do something similar with the calculator app, just defining methods on a class with comments and all this other stuff that helps the model do a little bit better, and it works really well. We can even do more complex examples, too. Like, we have this CSV example. Maybe I want to be able to well, I'm not going to get into the, oh, pip env. The demos demo gods are going to kill me here. That. Brutal. Okay. I can just create a program that does this now. I have this entire API that grabs columns and is able to perform certain operations and then do joins. I do filtering and joining and all this other stuff as well, because it just sort of does this selection based on Booleans. So read a CSV, find all the values that equal NA, and then drop the rows. And so this becomes this sort of powerful approach. And this is just a prototype of the Python stuff that we've been working on as well. It's not prime time, and if you want to talk to me about it, I'm definitely game. ### Outro **Daniel Rosenwasser** [17:52] So what I want from you all is to try TypeChat out. Reach out. What I'm here at this conference for is to learn about what you're all trying to build, trying to help bridge the gap as well between what we're all learning on the cutting edge and making that more accessible to everyday engineers who have been at this more precise end of the spectrum. Bring the power of these language models that are so rich to the traditional apps. Thank you very much. Come see me at the Microsoft booth. I'll be hanging out for a little bit. And thank you. --- This library is powered by PodHood (https://podhood.com), the podcast website platform.