How to Improve Your Agents: Academic Lit Review

Hello everybody. Today I'm going to talk about AI agents beyond ChatGPT. This talk is tailored for a more research-oriented audience. My name is Joe, I'm from Columbia University, and I'm also the founder of Arklex AI. So many people are talking about agents.

What are they? Bill Gates is very bullish on it, talking about the biggest revolution in computing. Andrew Ng is talking about this is a massive AI progress. Sam Altman from OpenAI is talking about 2025 as the year of agent.

We also hear a lot of negative voice about, oh, these are just some wrapper of large language model, they really can't plan, then we can't really have agents. And we're also talking about, oh, Auto-GPT is not great, they can never solve practical solutions.

But before we hear all these kind of questions, can we talk about what exactly is AI agents? So let's go back to the base. Also in Norvik. So agents is not new. It's really coming down to the large language model have powered agents that become more powerful these days.

First, we have the step we call perception. So like humans, agent, AI agent needs to understand the world, the environment, through sensing informations from text, image, audio, video, touch, and so on. And then once we get the information through these sensors, we're going to have this go through this reasoning process.

Sometimes people are also kind of in their monologue. To understand how do we process this information, how do we complete tasks, break down tasks into individual steps, and utilize these inputs of the environment to help us to better think what are different tools, what are different actions to take.

So this inner planning process, sometimes we also call them chain of thoughts reasoning, that because most of the time are powered by large language model. And then we also have some meta reasoning steps that we can perform, kind of reflect.

Basically, whenever you execute a certain action, you can ask yourself, so far, did I make theright choice? If not, can I go back? Finally, we talk about what considers to be actions of AI agents. Anything that you perform that may be talking to human, maybe moving from point A to point B, are also talking about, are all actions.

So in general, we're really talking about agents are interacting with the environment through actuations of actions. So in general, this is like the total process of agents.

Though it is difficult to understand how easy it is for us to deploy agents, we're just going to use a nice analog of different levels of autonomy in self-driving cars. So you can think about the very first base was a chatbot.

It started with 2017, that you can just retrieve information and that's it. And level two is more about we call agent assist. You have a customer service agent, they're using large language model to generate suggested responses, and people still has to be the person to approve the sending message.

Level three is what we have been talking moreright now, it's called agent as a service. So we use large language model to automate the AI workflows that we have, and then usually using it as a service, for example, meeting bookings, writing a job descriptions.

And level four is really coming down to one person is not only doing one task at a time, it can delegate the AI to do multiple tasks, and these tasks have intersections about sharing components, sharing knowledges and resources as well.

So this we call autonomous agents that can behave and perform multiple tasks together. And then level five is one level above is what we really think about the Jarvis and Iron Man. So we trust the agents 100%, we delegate the agents with all our security measures, like our keys, and then agents would perform on behalf of us.

So self-driving car is having this similar five levels,right? But self-driving car, you can consider it as one example of agents, because it's agent but doing the perception, doing the actions of reasoning about planning and trajectory of executing in terms of driving.

But self-driving is very high risk, which means that you have to make sure that there is no errors happening. Otherwise we are talking about lives and death, live and death. So, but AI agents, what we talk about is you always like it can separate them into low risk tasks to high risk tasks.

So when you talk about low risk tasks, maybe there are some back office tasks, like for example, filing reimbursements, which you can have human supervision and over time, trusted over time to be able to automate it. And then customer facing tasks are usually considered to be more high risk tasks.

And over time we can see people going from back office into front office over time to deploy these agents.

Today we're going to talk a little bit more about how do we improve large language models to make it to be able to reason better, to reflect better. And then we're going to talk about how do we elicit better behaviors from these existing large language models that optimize towards AI agent tasks.

Finally, how do we learn from the examples or traces we search over time to feed this information back to the large language model so that we can have a large language model that is optimized towards AI agent tasks.

So let's start with the first work, which is a NARCO best outstanding paper on self-improvements. So let's start with an example reasoning task, which is a mathematical reasoning task. So here is an simple example. There are usually two ways that we can actually solve it through using large language model.

One is on the left side, we call it few-shot prompting. We generate examples of the same mathematical, similar mathematical problems and its corresponding answers, and use them as a context, or sometimes we call it prompts, to a large language model, for example, GPT-4, and then get theright answer.

And the other methods we call chain of thoughts, which is instead of giving an example, we give it an instruction saying, let's think step by step. So the models actually are following this step by step chain of thought process.

Sometimes we also call it reasoning over the tokens and be able to reach the correct answer.

More recently, there is a prompting method that combines the two methods together that actually improved the outcome. So instead of asking the model to just simply think step by step, we actually give some information about these steps in the prompting in order to improve its performance.

So for example, we give the same question, we ask the models to solve it step by step, and then we take the question and the initial answer and then prompt the large language model to generate feedback. So based on the question and answer, what do you think about, do you think the answer is correct or not?

And then we have these generated things like in step two, the part blah blah is incorrect. This is because blah blah. So what we call this process is self-reflection or self-improvement. And then we combine the reflection or feedback together with the original question answer and then prompt the models again.

And then finally the model is able to update the answer and the internal processes to get the final feedback correct. So what we call this process self-refine or self-improvement. So we do find that this kind of prompt feedback could improve, and you can do it multiple times if you couldn't reach the correct answer.

And then you can stop it when you feel 100% that you are reaching the correct answer. But in reality, we do find this interesting self-improvement process could be having a problem with smaller large language model. So here previously we were all talking about large language model that is beyond 7 billion of 13 billion.

But if you're using a smaller model that is cost efficient like a Llama, 7 billion, we do find that the feedback that it generates sometimes contains what we call noise. And these noise can propagate down to the correction steps as well.

So this is what we usually call the blind is leading the blind. So we actually, instead of improving the results, are getting even less results over time. And then people may say, oh, these feedbacks generated by these smaller models are not great.

Can we change it into a bigger model? So you can think about these kind of feedbacks also nowadays people call them verifiers. One of the interesting things that we saw is because these large language models demonstration or internal logics are not compatible with the smaller models.

For example, the steps are completely different. If we feed this kind of internal logic back to the smaller model, there is incompatibility. So the feedback is not useful at all. So you can really think about this in human learning as well.

If you find your kid is making a mistake on using pronouns, you can't just say that you should use a second person pronoun instead of a first person pronoun in this situation. Your kid would never understand you. You should simply just say, at this point, you should, I want it, instead of say, you want it.

So we have to tone it down in terms of the feedback to cater for these smaller models internal logic. So this is where we come from in terms of how do we help the smaller models to also acquire the self-improvement process by distilling information from large language models.

So first we're going to reformulate the problem in terms of we wanted to retrain the model with attempts of solving the problem as an input, and then we're going to generate feedback and updates. And then we wanted to consider not using smaller models to generate the feedback step.

Instead, we're actually using large language model to edit the smaller model's feedback so that it's more tailored towards the smaller model. You can also use Python scripts for this particular mass reasoning task, because which gives you more correct feedback.

So now you can think about our proposed method tripath. We have the same question, we use the small model to prompt it to get the answer, and then we prompt the smaller model to generate the self-reflection. And then we ask the large model to edit this feedback and then use this corrected feedback as input to further generate the updated answer.

And this process of correction and getting the feedback could be iterated multiple times until the problem is solved correctly. Because it's mathematical problems, we have the ground truths in some of the benchmarks. So we can actually utilize this like you're just, for every problem we'd be able to generate traces like that until we reach the correct answer.

And then we can filter the trajectories of these kind of trial and error for a balanced set, and then use it to train our existing smaller models to be able to do self-improvement with the guidance of larger model or other toolings like Python scripts.

We find like if we using a weighted, self-weighted supervised training policy, and specifically we're talking about we have to train on policy data. We have to generate these feedback in real time, because your models over time is going to change.

So the model has to be updated in real time. We're going to evaluate this on a big bench hard mathematical problems for arithmetic, word sorting, data understanding, logical deductions, and so on. And then we through all these kind of different tasks, we collected, we do find like after three iterations of using our tripath, so when we say three iterations, like you provide feedback, you edit the answer, you provide feedback, and you edit the answer, you have these kind of a loop by three times, you're able to reach 48%, which is even better than using some supervised train data to do it.

And then the question is really coming down to, we improve the performance, which means that these kind of policy self-supervision is useful. But if the model fine-tuned with the data that we talk about self-improvement, it really learns the self-improvement in itself.

So we do find in this all the four tasks, we looked at the first time correction, which is the pink one, and then if the second time later ones it was correct after self-improvement, the dark pink, it actually obviously happening all these kind of different tasks.

And the range is pretty high in terms of 7 to 18%. So this question is really coming down to, this is definitely we are getting extra answers correctly, but where does these kind of delta really come from? Is it can it just obtain from fine-tuning on goat answers instead of giving them the internal logic and policies?

So we do find that if you give own policy data, we do much better. And then you just use simple SFT data, you just like the improvement is cutting in half. So the takeaway is very simple that we can actually improve these models reflection or self-learning abilities without explicitly human and supervision data.

We could use these synthetic ways to generate data on policy to help the models to improve. And of course we have the limitation that is that what we call about the verifiers or the editors, which are large language model.

If these large language model cannot edit it correctly, then we always have the ceiling effect that smaller models wouldn't be able to distill these information from these large language model. So we're still hoping there will be a big language model is pushing the boundaries of all the reasoning tasks.

And then we can actually use these kind of syncing in by step or syncing in scale, test time scale, in order to be able to use smaller models to catch up to large models ability. Now we talked about a meta reasoning type of we call it reflection.

Now we go into can we actually elicit better results or planning results from large language model itself. We all know in large language model area, there are three factors that would impact the training results of a large language model in the pre-training process.

The number of compute, the data size, and the parameter size. And we definitely see it's like linear related with how the performance look like. So this is what we call like the scaling law. We still believe that currently we're not reaching towards the scaling law, but we are close.

We're relying on more GPUs, but unfortunately data size is just, we have one internet as Ilia talked about in Europe. How do we generate more data? How do we get better data is a million dollar question. And then parameters wise, we're already reaching like billion parameters.

If we increase it even more without the support of abounded diversified data, the models are harder to improve as well. But pre-training itself, you can do it with if you're not Google, if you're not OpenAI. For smaller companies or academia, what kind of things could we do?

So there is a couple of new work related to GPT-4.01 is getting us a new direction, which is called test time compute scaling. So instead of pre-training a large language model, you take the model as it is, but give it more steps or budgets in terms of inference.

So specifically like what we talked about, as it to sync by step by step, as it to do reflection, and so on. You can actually reach better results in the end of the task,right? So given the budget, we see it's like a linearly increased in terms of task assess.

So can we actually further elicit better behaviors with existing pre-trained model? And today is what we're going to talk about. How do we elicit stronger model behavior through chain of thoughts like processing that is even more on the tree search side that is more complex, but it gives you better results.

And finally, how do we further improve the base large language model by using the data we generated from these synthetic processes? So let's start with a very concrete task. I have been working on dialogues for many years, and then this is one of the examples.

But many dialogue tasks, you can think about it as a sequential decision making. You are at a certain point talking to another person. You need to make a decision given the conversational context. What should I say next? So this is processes you also have to look ahead,right?

In terms of, oh, this is my plan. How do I strategize my conversational strategies to do some of the tasks? For example, we have the donation persuasion task. The boy on the left is trying to persuade the girl on theright to donate to a charity called Save the Children.

So the story with the conversation saying, how are you doing? I'm good. And then we talk about some of the strategies people use for persuasion. We call it increase. Great. Have you ever donated to charities? Right? And then given the user's response, I'm in theright place at theright time that I would do it.

This boy needs to think about and ahead of time, what kind of strategies to apply at this point in order to convert this persuasion. So let's take a step back and think about this sequential decision making in a more toy problem, for example, like chess.

It's really coming down to, you have to think about what's your next move. And you can think about what's your move in a more efficient way. You can stimulate, if you do this, what your opponent is going to do,right?

So you can have these kind of what we usually call planned out or like simulate and evaluate process. And we do find that in chess games, grandmasters usually have many steps ahead of time in order to see what is your best practice at this particular moment.

So this is very similar to AlphaGo and all these kind of algorithms that we're talking about that uses what we call tree search. So the idea is very simple. You propose a certain move, and then you simulate what are the value or like changes after you make this move.

And then you evaluate in terms of what are the outcomes of this particular move. And then you do this multiple times until you can obtain a stronger move by simulation.

So in our EMLP 2020-13 work, we basically brought the similar ideas of chess into a conversational setting, which has a very clear goal. We don't really need any training data. We hopefully could be able to train this model with simulation.

So we start with a multi-color tree search. We call it zero training. How do we design this multi-color tree search model? Is we're going to prompt large language models that's ChatGPT. So first, we're going to have the search potential, which is promising actions.

You basically prompt a large language model to act as the policy. And then you simulate action outcomes. So basically, if you say these persuasion strategy, what would be the outcome? We also do prompt a large language model. Evaluate, then finally we're going to do evaluate action quality.

So once you perform this action, what are the quality of the actions after interacting with the environment? We also prompt a large language model. Finally, we basically update each action quality over time. You will be able to have this process.

And then when we talk about self-play,right, you also have to simulate what your opponent's behavior. So here we used another large language model to simulate the user. Again, we also use prompting large language model. So basically, given the conversational history, I want the models to generate the simulation.

So we want to see what are the prompts here particularly. We have, for example, prompting of negative reactions. That is one of the process. And then we want to see, given the prompt, what are the evaluation of the success?

For example, is it a 0.2, 0.3, or 0.5? This is all very important in terms of how to evaluate the performance of a particular action. Traditionally, for these kind of MCTs with zero training, you would have a closed loop of MCTs.

Basically, is given the history, what are your actions? But in reality, especially in our conversational task, we have to account for a lot of variance within humans' response. People are going to try different things. So you can consider this process of stochastic process.

Instead of using closed loop MCTs, we're using open loop MCTs. So basically, given the conversational history, we're going to stochastically randomly sample one of the one possible simulated task. So basically, you can simulate different conversational strategies to persuade your opponent.

And then the idea of this particular task is that we have a very objective way of evaluating it, which is whether the person donated to the charity or not. But in itself, if you can't actually run with real humans, then it's really hard to have a good policy because it's very subjective.

It's very hard to train. So here we particularly wanted to have GDP zero, which is our proposed method, to be able to compare it to without training, be able to generate more stronger competitive results. So we performed evaluations.

First is we asked another large language model, for example, ChatGPT, to look at the conversations and look at the general outputs. One is using our planning algorithm. One is not using our planning algorithm. Which one is more likely to be persuasive, can lead to better outcomes of the task.

And then on the other side, we have to run the human studies. So basically, we release the model that is actually basically performing at the task of a persuader. We asked the mechanical turkers online to interact with this chatbot to see whether they wanted to donate or not.

So the models that we deploy with our planning algorithm actually definitely gets better donations. And also we find that people find it's more convincing, more natural, more coherent as well because the process is optimized for the goal. We also find through the analysis, we can learn that these models can self-discover a lot of task information.

For example, we do find that they wouldn't do the task like, would you like to donate in the very early on of the conversation? Because that usually, according to the literature, is also not good. We wanted to pave the way into the big ask.

And we also do find it learns how to diversify its strategies as well. It's not using one strategy only in the entire conversation. It learns to use different persuasion strategies, emotional appeal, logical appeal to diversify its potential of convincing various different people.

So in general, this work really talks about for dialogue tasks, you can use multi-color trees to simulate what are the possible behaviors and then use these kind of policies to drive in real world these decision makings. Although this is not as clear as the chess board that has very detailed and limited action space.

But if you use these kind of quantifiable search spaces within dialogue tasks, we can still gain better performances than not planning. So of course, this is just a conversation task. Can we extend it into even larger AI agent space that not only perform actions of talking, but it performs actions of other tool use and manipulations?

Can we actually transfer the policy that we learned into other tasks? So this comes to our recent work in iClear with the name of Exact. So for large language models to be able to do various AI agent tasks, we got to teach it how to perceive the world.

And then we're going to start with a visual large language model. We call it a visual language large language model because this is at least we want to have to be able to process image as well as text.

So the traditional visual large language model, or we call it like VLMs, are trained with a task such as a visual question answer. Like what is he doing? He's performing a skateboard trick. But in reality, for AI agent tasks, we want the computer to do things, which for example, the input is a image of your computer screenshots.

And then the actions ways that can help me clear my shopping cart. And then you have to perform actions like click button of the shopping cart, click delete, and so on. So the entire training process or the entire usage of the current AI agent that we need for a large language model that is multimodal is very different from the previous VQA task.

So we really wanted to see if there is a way that we can adapt the traditional VQA visual language model into a more action-based visual language model,right? If you look at humans can do these kind of benchmark visual web arena like a clean my shopping carts, find me a green pair of shoes, and things like that.

88%. Well, if you just use a simple GPT-4V without anything in terms of plannings or whatever smart things, you get 16%. So the absence of these kind of agent environment interacting with data and training is very lacking in these agentic tasks.

So here we wanted to see if we don't have the budget to retrain GPT-4O or do anything with it, is there a better way that we can actually elicit better performance through test time compute? So we introduced a new algorithm called RMCTS, which is short for Riffler Multi-Color Tree Search.

Very like multi-color tree search, it is a search algorithm that can explore the vast action space on the fly and also improves decision making by incrementally constructing a search tree. Unlike simple multi-color tree search, we extended to incorporate contrasted reflection.

So we basically allow agents to learn from past interactions and dynamically improve their search efficiency. And also we are using multi-agent debate to get reliable state evaluation instead of using single prompting. So here we have two modifications that we did.

So the idea is that on top of performing multi-color tree search and improve its decision using a task, we can equip the entire system with a memory module such that it can improve its behavior across different tasks. Practically, the idea is that after completing a task, we use reflective, use contrastive reflection to help the model internalize the success or the error it has made and save this experience to a vector database.

So we basically cache the knowledge that we just learned. Then during any future task, the agent will also retrieve reflection similar to the current task, all the current computer state. To enhance its decision, we're basically making this as a new environment.

Finally, the RMCTS, we also introduced the message to improve the success or the value estimate used during the specifically instead of directly prompting the language model to evaluate the progress. We find this evaluation process can be much more robust if we use in a form of a debate.

So basically we're asking the models what if it's a good action, why it is a good action. If it's a bad action, why it is a bad action. So that you can have these kind of more holistic overview to counterbalance biases from one model's prompt.

So basically through these two modifications of caching information and using more robust verifier, we're able to create this multi-color tree search with a reflection. During each task, the RMCTS builds a search tree on the fly by interaction with the by interacting with the environment.

And then RMCTS use multi-agent debate to provide a more reliable state estimate and better guide the search process. Then after each task, RMCTS is now originally performs this contrastive self-reflection at the end of the task in order to improve its future execution.

So we basically cache the information that we learned through reflection. To evaluate our method, we use two benchmark, Visual Web Arena and OS World, which are the most popular AI agent benchmark. So the Visual Web Arena is a benchmark that evaluates AI agents performance on browsing tasks.

So you have a browser that you actually do various things like doing Redis or doing shoppings and so on. While the OS World is a benchmark that consists of computer tasks such as navigating the file system, using apps such as VS Code, Excel, and so on in a Linux computer environment.

So we first wanted to look at the Visual Web Arena. We find that our search algorithm outperforms other existing search algorithms such as Breadth First Search, Depth First Search, ASTARs, and then Vanilla MCTS. We also compared our method to non-search methods like a simple React or as well as like other search methods from various different ways.

We find that augmenting LLMs or the visual large language models with search algorithm can basically improve the performance without additional human supervision. So through this kind of caching, through better verifications of the states, we're able to get better performance.

We then basically also compared another task, which is OS World. We are actually number one in the Visual Web

Arena leaderboard. We also have the best non-trained methods in the OS World, which means that we only do test time instead of fine-tuning original models.

So given what we see, like autonomous method can actually improve agents without human supervision, can we actually consider transfer these additional knowledge that we obtain through the search into the training process of the base large language model? And we basically also invented the algorithm exploratory learning.

So compared to imitation learning, which is a direct transfer train the model using the best action found in the tree, we propose to use exploratory learning, which treats the tree search process as single trajectory. So basically the models will learn how to actually linearize the search tree traversal to motivate the model to learn how to explore, backtrack, and evaluate.

So through this process of learning, teaching the model to do the exploration instead of giving it the final good answer, we actually make the models to be able to improve its decision processes by itself. So if we look at the on theright, if we let the model to do exploratory learning instead of imitation learning, which is the darker color, through test time compute, basically if you give it

a constrained budget, it can improve the performance over time.

Here we basically provide an example trajectory after exploratory learning. For example, like this, given the task such as find the most relevant recent coffee maker with a touchscreen, then comment with great item. So this process, the agent basically learns how to perform an action, release it, did not lead to desired state, then go back and try other actions.

So this evaluate and backtracking process happen through the process. So the model actually learned this process, which we find is very surprising and great. So in summary, the reflective MCTS models that we institute just simply relies on test time compute scaling, is able to help the models to get better performance.

And through exploratory learning, by training on the examples that we serve, we're able to actually further improve the large language model. So there are many more things we can try that don't rely on human supervised information, which we can provide in an academic or smaller company setting.

So for example, how do we get better RL methods to reduce reliance on the search tree? How do we do model predictive control methods to reduce this really expensive environment setup and interaction? So some of the so far, we basically talked about how do we elicit better large language model stability through test time compute, through retraining from refined or backtracked exploratory traces.

So some of the recent things and ongoing work that we have done is more focused on how do we improve controllabilities and also at the same time the autonomous exploration together for the agent orchestration layer. So we just released our Arklex open source agent framework that has more features and more control compared to many other existing frameworks such as LangChain.

We have various interesting components that we have, for example, continuous learning, task decomposition, which could give you developers more flexibility in selecting different things. What other things that we have been doing, we think, is combining not only the machine learning expertise, but combining the system expertise, the HCI expertise, and also security expertise together to form a group to learn how do we advance the system in a more deeper and practical way.

So currently, all the benchmarks I talked about is one single agent performing one single task that human assigns to you,right? What if like you're this is super inefficient to run this like in a separate instance? What if I have one human wants the model to perform multiple tasks together on the same computer?

Then we will encounter all these kind of system level problems about scheduling. How do we interact with database to avoid side effects? How do we have better securities in terms of knowing when to do human handover, when to actually request human supervision?

And then what's even more is if we have think about the entire process as a community, you will have multiple humans interacting with multiple different agents and assigned different works. The multi-users, multi-agents planning is even more complicated, can even lead to a lot of these kind of adversal settings that we want to quantify.

So nowadays we're working together to establish more realistic benchmarks that have system integrations and also algorithm that I will build on top to not only consider task completion, but also consider efficiencies, consider securities, and so on, which you want to provide for the basis for all these kind of applications in the future.

Okay, thank you so much for listening. If you're interested to know more about our framework that integrates all the research that we just talked about, please join us at the Arklex AI.