Building a Chess Coach — Anant Dole and Asbjorn Steinskog, Take Take Take

Afternoon, everyone. So, our next talk will be something a little bit different: we're going to dive into the world of chess. Quick show of hands: who has heard of Magnus Carlsen? Okay, fantastic. No introduction needed, but widely considered the best chess player in the world.

He also founded a company called Take Take Take. This is where myself, Anant, and my colleague Asbjørn currently work at. And we're going to talk to you today about how we built our AI chess coach that now you can use and is in production.

So first up, quick agenda. We'll quickly discuss a bit more about Take Take Take: what it is we actually built, what it is we actually launched. I'll split it with them going into a quick history of chess and AI, a lot of links there.

We'll briefly touch on why LLMs are actually bad at chess, and how we managed to solve this problem. We're then going to deep dive into actually understanding our game review and sort of closing the loop with our autonomous agent, and you'll get a demo.

And then finally, some latency versus quality trade-offs, as this is a consumer-focused AI application. And then lastly, some learnings. So first up: what is Take Take Take? In its simplest form today, it's currently an iOS and Android application.

You can go on and play your friends, and you can post about your games. What's relevant for our particular talk is that after you play a game, you get presented with our game review. And this is powered by our AI pipeline.

So for example, just showing you how it works: in this particular position, it's leading to a checkmate. The last move that White has played has moved the knight from this yellow square over here on f3, captured the pawn on e5.

It is a brilliant move, so it automatically gets the brilliant sort of notation. And the commentary below is actually generated by our system, and we're using an LLM, and the pipeline we'll get into in a second. But what's quite interesting about it is we're able to give you the nuance of why it is a tactic, what detectors from a positional and tactical sense have fired, what are the threats you're trying to do, and actually explain sort of the why behind the move.

So that's the system we're going to be talking about today.

Finally, on the last sort of step of our application, we've started revealing insights about your play. And this could be things like how accurate you played in a particular game phase, maybe your current rating, or your current depth in a particular opening.

And these insights form the next layer of analysis that we present to the coach, who then gives them back to you as opportunities for learning and improving. We hope by using this, you'll be able to improve and become better at the game.

Allright. So first, a brief history of chess and AI, since they've been intertwined for so long. Just to give you a little bit of backstory: in 1949, Claude Shannon, the OG Claude, wrote the paper "Programming a Computer to Play Chess."

And here he envisioned that, or he proposed that there are two types of chess engines: type A and type B. Type A were these brute-force engines that searched through all possible moves and figured out the best move. While type B were those who we know from 2017 and onward that can selectively pick out the best moves.

Back then, he assumed that we would need type B computers to play chess because computers were so weak back then. You couldn't search through the whole tree of moves. But computers quickly became better, and people just started scaling these type A computers.

They got better and better until they, in 1997, Deep Blue versus Kasparov, the first time a chess engine beat the best chess player at the time. So people didn't really bother about these type B computers for a while, these intuitive engines, until DeepMind, shout out to DeepMind, released first AlphaGo.

Because Go is a much more complex game than chess, so you can't solve this with these type A computers. You would need this intuitive approach, neural network approach, to actually selectively figure out which lines to calculate. But after that, they released AlphaZero, who could play not only Go, but also chess and Shogi.

And some years later, LLMs came, and people started playing chess against the LLMs and quickly turned out that they can't really play chess. Sometimes they make someright moves and they can't, to an extent, play a nice opening, but they quickly start to hallucinate.

So let's see if we can show the yeah, there's a video of.

Rock went for the poisoned pawn line with queen b6 early on and lost pretty badly. Not necessarily because of the opening, but because it doesn't really know how to play chess.

That was Magnus Carlsen commentating a LLM chess tournament from our office in Oslo.

That was a tournament organized by Kaggle when they launched their GameArena, which was a benchmark for benchmarking LLMs when playing different types of games. One of them was chess, and now they've started to add more games. They also added Werewolf recently, where you can watch LLMs try to deceive each other in social deduction games, which I can recommend watching.

But yeah, we see that LLMs often hallucinate because obviously they're trained on language. They can't calculate. Like high reasoning models can, to an extent, calculate through the reasoning steps where they can actually play out moves, but they quickly fall apart.

But there's nothing inherently wrong about the architecture of the transformer architectures to play chess. DeepMind has trained a transformer to, instead of predicting the next token, they predict the evaluation based on a chess position where they've trained it on millions of chess positions to Stockfish evaluations pair.

And that has actually led the transformer to play at a grandmaster level strength. But these aren't trained on language, so these can't explain chess. So how do we bridge the gap between these old chess computers that can understand and play really good chess between the LLMs that can explain chess?

So we're going to go through our pipeline of how our game review explains chess in our app. When you play a game, the first thing we do is we run Stockfish through the whole game. Stockfish is the leading chess engine now.

That's like a classical chess engine that calculates the best move. So what Stockfish says is considered to be the solution in a chess position. We then extract a lot of context in the position because we want to explain not only the best move.

We want to explain the threats, the plans, the tactics that could arise in the position, what you should have played. A lot of these nuances that are useful if you want to learn how to become better at chess.

So we have a lot of detectors that try to figure out all of this: the forks, pins, skewers, positional structural themes. Doubled pawns, for example, is a disadvantage, so we need to be aware of all of those kind of things.

And there's also a new novel chess engine called Maya, which is behind a research project by the University of Toronto, where instead of building a chess engine that is trained to play the best, they've trained a chess engine.

It's a neural network that predicts the moves that humans would play in certain positions. So given a rating, for example, an online rating of 1,500, it outputs the probability distribution over all the moves in the position. And by doing this, we could actually say that a move is really the best move.

We know that because of Stockfish. But we also know it's really hard to find that move because the probability of playing it at certain levels are so low. And all of this information, we feed that to the LLM.

And

for now, the LLM's job is only to translate this information into English. Because we really don't want it to try to figure out too much on its own because it could quickly lead to hallucination. It still does, but we want everything to be grounded in the information that we give it.

And that could result in a comment like this if you play chess or know about chess: this is a game that I played. My opponent played f5 here, which is a bad move. So by using Stockfish, you could see that you get like a bad move indicator.

But that's not so useful to just know it's a bad move. So we are running our detectors to figure out that, OK, f5 is threatening to trap my queen. You can also see it draws a line with bishop g5.

But it can also say, while it threatens to trap the queen, I can just capture the pawn in the middle because that defends this square so that my queen can get out of the situation.

So that's how we get to that situation. Now I'm going to explain a bit on how we improve our game review using agents. We have closed loop from user feedback to the pull request, essentially, with humans in the loop.

But what happens when users download the commentary in your app? Because you can download it if you think it's bad. It posts it to Slack, but it also sends it to a Claude Code channel. Channel is a new feature in the research preview that is essentially an MCP server that can inject events into a running Claude Code session.

So kind of like OpenClaw, if you use that. So you have this continuously running channel, and you can inject events to it. And then Claude Code starts working on the commentary. It gets all the information. It runs a commentary triage skill that we created that outlines its process, how it should go about to investigate what's wrong in the position.

It has some scripts to actually run the generation, so it can modify, for example, the prompt. It could change some of the detectors, create some new detectors. And then it can generate the commentary again, given this new information, and verify its own work.

And then it will also ask questions back to Slack so that I could be on the bus, and I could get a message from Claude who is working on this problem, where it will ask me if this seemsright, and I can guide it.

And if it looksright, I'll just tell it to submit the PR, and I open GitHub on my mobile, walk through, and merge it. I'm going to show how this works

by so we have a running Claude Code channel here. Here is the Slack channel where the commentary appears. This is just me having tested a bunch of time. I'm going to open up the app on my phone, go to Comment, and report it as bad.

Now we see it posts a comment. We can see the position, the commentary that was generated. Now, I haven't really looked at the commentary, so it could be it's probably good. But we can also see that it injects it to the Claude channel, who invokes the commentary triage skill, and starts working.

Now, this is now running on high effort, so this could take a while. So I'm thinking we should just go to the next slide, and then we could get back to it to see if something is happening.

Fantastic. So we'll come back to that in a few seconds. So as we built this for end users, we had to really kind of consider this trade-off between latency versus quality. So typically, when you finish a chess game, you want to get the analysis and the results pretty quick.

You want to cycle through the moves kind of one by one. So we really couldn't show you a coach's thinking screen indefinitely while reasoning tokens are kind of running in the background, as an example. So we had to get this done, which felt almost instant.

In an AI world, that's a few seconds at best. So we were aiming for sub-3 seconds to generate our coach sort of feedback. How do we do this? We use Gemini 3 Flash. Time to first token is typically about a second.

End-to-end latency, on average, is about 3 seconds, which kind of meets our criteria. We have experimented with other reasoning models, and we'll get into that on the next slide. The analysis is not incorrect, so the quality is definitely good.

But the challenge is it's unpredictable as to how long it's going to take to finish. So we have a new set of features kind of planned for a more chat-with-your-coach type experience, where we can kind of expect the user to be more patient and wait for a response, rather than in this sort of phase where it needs to be more instantaneous.

The last thing about quality is Asbjørn and I are both good chess players. So we ultimately kind of have the final say as when we look at a position to actually use how we would calculate and how we would play and compare it to the LLM's response.

This allows us to actually evaluate whether it's doing theright thing or not. So if we talk about evals in more detail, like I said, Gemini Flash is kind of our benchmark. But we have multiple chess scenarios. Currently, we have 16 different scenarios that we created.

These are around themes like tactical patterns, blunders, and sort of limiting hallucination. So as an example, there might be a knight fork on the particular chess position, and we're trying to assert that can the LLM actually understand and mention this when we run it through with our sort of context engine?

And how do we do this? We extract scenarios from real games. We use LLM as a judge, very powerful sort of technique to test. We then run the model in OpenRouter. OpenRouters come in handy because new models are being released so fast, so frequently.

We just want to be able to quickly swap in and swap out maybe a new version of, say, Gemini. We want to check out the latest GPT-5 model or one of the Claude models. So we'll then compare and see sort of the quality, ultimately relying on our own skill to detect whether this is good or bad.

And as a sort of final point on this, we ran our three models, Gemini, Claude, and GPT-5. And typically, Gemini Flash is about 75%. It still doesn't pass all the scenarios we've set it. So we're always kind of seeing if a new model will actually exceed some of the tricky cases we've set up.

Claude, on more thinking, gets us to about just under 60%, but the latency is much longer. GPT-5 Mini, given a smaller sort of model, lower latency, or rather slower latency, but lower accuracy as well. So we kind of continuously run through these to update.

Last thing on sort of our learnings and how this can sort of apply to your world sitting in front of us. Number one, really important to separate that sort of data pipeline from the language generation. LLMs can do a lot of different things, but if you need high latency or quick latency, it's a good sort of technique.

Really try to close the loop with autonomous agents. Kind of the flow that Asbjørn showed is now very common and very powerful and really allows you to iterate quickly. Always try to build a very clear sort of context extraction model.

This, unfortunately, in the beginning, is a very slow, sort of painful process. It's a large, ultimately, JSON file that you keep sort of starting big, and you start to prune step by step, and you see how quality improves over time.

Automated evals really do help. And I hope in your domains, you also have a set of SMEs that you can help rely on to evaluate the output. And sometimes that's not necessarily the person actually building it. It could be someone else who is a domain expert.

So remember to sort of partner if needed. The last thing, just on a fun sort of note before we go back to the output of the sort of coding agent, we do have some chess sets on the third floor at the entrance you may have seen.

We're going to host a chess simul today in the afternoon, around 3:45 p.m. A chess simul, for those who aren't familiar, is when one person, in this case me or Asbjørn, plays multiple people at the same time. So we have four chess sets.

We will play four people at the same time. We'll have slightly more time for us because we have to walk around to play multiple boards. If you happen to play and you happen to win, you will get one of the wooden chess boards at the end of the event.

They're very nice, high-quality chess sets. If no one wins, we will still determine who the two best players are, and we will still give you a set.

And if everyone wins, we need to buy more boards.

Hopefully, not everyone wins. There's a QR code if you want to sign up or just stop by at 3:45. You're welcome to do that. And then, yeah.

Yeah, close it off.

Let's go back to see what has been happening here. Oh, you see it's still thinking. It has actually added a comment. Looking into this now, investing into the position. Quick question. What specifically feels wrong about the commentary? Yeah, it got me there.

There's nothing wrong. You're absolutelyright. Nothing wrong. So yeah, now it's going to close this off because, yeah, it worked well.

Fantastic. Well, thank you so much, and happy to take any questions.