If Your AI Scores 100%, Your Eval Is Wrong - Ramakrishnan Lokanathan, eslams.com

I think why all of us are here primarily is because evals are currently very broken, right? And like that is what we're trying to explore and understand how to go about it. More often than not, I think evals are broken because we overcomplicate it and we don't really understand, right? We hear things like the eval is more important than the model and the eval is more important than the hardness or whatnot, but what even is an eval, right? It's a question that we all sort of come to and like this is something that I consistently hit when I speak with people building high-stake applications to even understand to the first principle level of what an eval is right? And what is an eval is just a abbreviation of the term evaluation, right? And how we understand evaluation is like we know exams, which are examinations and this is a concept that we understand in human beings, right? And if you think about it, as people we write a variety of exams through our life, like we write the SATs, the GMAT, the bar exam and like regardless of what we do. That is the same sort of idea you can apply as a parallel into evals and then you can think about evals as those examinations for AI systems, right? You have the Sweet Bench, the humanities law exam and all of this.

In multiple ways, this is trying to evaluate and try to understand how a model or an agent will be working in something, right? At the end of the day, examinations and evals are just tasks, standards, and evidences to sort of evaluate and understand how something is to perform on something as you expect, right? And I think that brings us to a very key distinction here as well. An exam must answer a bounded question and let's take the example of the SATs. Um it's a college entrance exam and it tells you how good a student is on language and math, but it's just one signal of a college readiness, right? I come from India and even today in India your college admissions are done with a single examination. However, let's say you look at the US exam, SAT is one signal of college readiness, right? And I think that's extremely key. You should not index on one eval or one test and it should just be one signal of your readiness to the platform rather than trying to sort of over index on your model being really good at humanities class exam or at W bench, right? So, no eval or no test is a measure of someone's complete abilities and that should always be clear going into that.

So, at the end of the day, what we try to answer with evals is is our performance matching the promise that we're making to the consumer or the user at any given point in time, right? And that's the job of any eval to make sure that we go ahead and ensure that the promise of the product of the agent that we're providing is meeting its performance to the consumer. Before we move ahead, I wanted to sort of give a small brief on who I am. I'm Ramakrishnan. I come from the city called Cochin in in South of India. I started coding very early on and started building machine learning classifiers in high school and whatnot. And I started my career with an Indian EdTech unicorn which gave me the opportunity in '21 to get a demo of a GPT-3 based product on how to better our content flow which changed my mind being someone who really loved machine learning from an early age. And I was like, this is going to really change the world, right? And my claim to fame or like what I was right in doing was in September 2022, we decided that there is a new wave of applications that are going to be possible that are going to be built on the GPT-3 API, right? And about 2 months after we went full-time, OpenAI released released their research preview for GPT-3.5 called ChatGPT and like that literally changed life for us and it validated us and killed us as well, right? So, went through multiple pivots and then we eventually landed on a pet care AI assistant and we monetized it using e-commerce and recommendation and engines on top of it and we got acquired in India and that was the end of that

journey, right? Uh after that, I've been super interested in understanding how these systems evolve and because we were super early on building on GPT-3, evals were critical for us to build. So, that took me to being a red teamer at for the Nematron 3 models. I work at the Lawrence Berkeley Lab right now on a AI safety research and also with the London Business School data science and AI initiative. Broadly to explore how to sort of look at the new technology that we are working on and then how we see and explore, right? So, I am not aligned or like, you know, with anybody here, so I'm not going to sponsor a vendor. It's just to understand how do you think about eval and like build these high-stake applications, right? And the consolidation of the conversations that I've had over time with multiple implementations that we have done with millions of users using our AI product, sorry. So, this is something I strongly believe. So, evals are not going to be relevant in a decade's time. Because if we are to believe that model capability is to keep improving, we will not have to have as much eval as we do. However, in the midterm, like production cannot wait. The problem is if you don't have the right amount of eval structure, you get reports like the MIT GenAI divide where 95% of organizations are not able to get any returns because you are not delivering the promise in production that you're trying to make with the LLM, right? And that is what we have to solve with evals today and in 10 years time, evals is just going to be a validation of this.

But today, it is to make sure that in production you are delivering the promise that you're going ahead for, right? So, the first thing that you have to think about as you build evals for high-stake applications is to define the most ambitious success, right? Before designing any test. Like, assume in in of AGI, what is your AI agent going to do? Or what is the best outcome you can get out of it, right? And setting this goal really gives you the scope of what you can go ahead and design further, right? And you start with the job. And simple the first question that you ask is what must happen. And what must never happen? And when should it stop? And you can extend it into other WH questions and first principles to go ahead and think about what could be the most ambitious outcome that you can get out of it, right? And aim at the job you need and not the score you can get. If you have an eval that is scoring a 100 of a 100, your eval is wrong, right? And you're not ambitious enough and you're not testing enough, right? If you look at any model eval today, people are at like 85, 86 max. Nobody gets to 100. So, you have to make sure that the evaluation and your ambition is beyond what the model ability today is for you to make sure that you're putting it to the stress test and not making it easy, right?

And always test the whole workflow, right? And not just the final answer. And this is the original picture, and my request would be only change the background, right? And I am just looking at the output. And the output has changed the background to a cream background, but it has two handles now, right? So, it has to be very clear that you have to always test the work. And this is a very simplified example of this. Let's say you have a 100 steps in your workflow. Each step has to be evaluated for what goes into that step and what comes out of that step and what is the expected outcome. You can't look at the start point and the end point and be like, "Okay, if you look at the start point and the end point, the background has changed." But is that the objective that you're trying to achieve, right? So, always for look at the whole workflow when you're approaching things, right? And the third thing, and I'm I'm going to talk a little bit about something I've been working on in this space as well, is one score is not the system, right? And the idea is to build a holistic evaluation system.

When you started off with models in 2022 and 2023, you were measuring for coding ability or let's say you were measuring for writing ability, etc. But today you have agentic persistent systems that have to do multiple things, right? So you have to make sure that you're designing a holistic evaluation system at all points in time. And the first element of that is that automated checks and eval being part of your CI pipeline, right? And your eval questions have to be asked to your agent every time they are going through a production deployment process, right? So that being part of the automated checks makes sure that it's not like one eval score you are taking at the time you're deciding the model. Continuously human review and make sure that the results make sense. Don't rely on your agents to be the reviewer of your eval at any point always, right? You can fast-track your eval process with it, but make sure you're doing human review. And the third part is adversarial test, right? Where can you break the workflow and like put it into challenging situations like where you can have adversarial testing? And one thing I've been recently working on and would love to talk about is um the idea of how you can choose the test for the failures it can reveal, right?

And the idea of a test that can fight back. Today you have a task on SW bench, the agent or the model is asked to go ahead and perform one task and give an output. But there is no continuous thinking or evaluation, right? And getting one right answer more and more is not the same as making a good sequence of decisions and you need agentic action and long-chain reasoning and all of these things. For the longest time the best measure of agentic action and long-chain reasoning in human beings have been board games like chess and go. And even when you go ahead to um like like evaluation of like a model, AlphaGo is a very important step to AI development, but it never very specifically trained model for playing Go. Today's general-purpose LLMs we don't test on Go or chess, and that is what I'm trying to solve with an open-source repo called E-Slams, which is electronic slams, and we make AI play games, right? So, you can just install the package. It's a Python PyPi package, and you can make your agent and your model play games and see how it works. And how can you put it to test, right? You have You're replacing an agent with an improved model. You might want your old agent to play with the new agent on on a chess game of chess. And let's say I don't I'm not saying the winner of that match is a better model, but how they enter that match and play that match will tell you a lot more about how to sort of go ahead with that model.

Because playing chess, you are changing your strategy, you have to get out of blunders, and move that you make can become liabilities for you in the past. So, how is your agent reasoning through that is an additional paradigm to the holistic evaluation you can have. And the idea here is do a coding evaluation, do a writing evaluation, do humanities last exam, and try to have as many paradigms as possible, and that's what I'm trying to build with E-Slams, right? So, we support about 50 games and 90 different like providers. We have chess and Go, which looks at planning invisible state. We have poker, which is decision-based on hidden information. And we have cooperation games based on shared goals and cooperation. And we also have trials for arcade games, right? And all of this tells you a lot about models and how they think. And on our platform, we have preview on eslams.com of how you can use the repo. And this is an example of Fable 5.1 playing tic-tac-toe against GPT-6 Astra, right? And you can look at how Fable and Astra are looking at each other's turn, and how they go ahead and play this. And this match, interestingly, ends in a tie. And most cases, a model wins, but Fable and Astra ended up knocking their heads to each other, right? And let's say you are deploying an agent in production. You can You want to go ahead and play against these AI models and see how they are. You can come on Islam.com and like play against these AI models on any different game that you want to. And you can all do all of this using the open source repo and make it play with your own agents and like evaluate your

systems right? And we are going to host the Grand Slams of AI on in February and the idea is to put an Olympics of for AI models to go ahead and compete and know who will stand out of all of this, right? So, please do check out Islam to know where we're going with it and this could be one paradigm of model evaluation you can add into your stack. So, now coming back, the main key part after all of this, right? Like you keep evolving, you keep adding paradigms, is to make sure that you keep testing the evaluation workflow, right? And the most important part of that is keeping what works in the flow. And my best example of this is Google Assistant moving to Gemini. I used to use Google Assistant for all of my reminders. And one day it moved to Gemini and it stopped giving me reminders and it said it's a language model that cannot set up reminders, right? So, it was excellent in the old Google Assistant model they had and before they moved it to Gemini and it stopped working and this is an example of Google not doing eval on what used to work, right? When they went ahead and launched the Gemini product.

And that's extremely important to go ahead and make sure that you keep testing like the regression cases, right? And known task and past failures because a model that is extremely better than the old one could very well be incapable of something the old one did very well, for example, setting a reminder, right? And um fresh challenge cases at all points goes back to what I was saying in terms of setting the most ambitious target. The idea is that you have to keep pushing your model. If you feel that your model your is getting to an 85 or a 90 on your eval test, it's time that you better your eval and like push the frontier, right? And make sure that your model is challenged and has cases where it is not able to perform to expectations. And at all points, right? And I think this has been true in software throughout history is that you have to turn real-world failures into future tests, right? Your customer and your user is the most important evaluator of your agent and your model.

So, every failure should have a proper workflow for it to enter into your system and make sure that like be that, right? So, one is to keep observing and there are multiple ways of making sure that you can observe. There are multiple observability tools that give you automatic data, give option for your users to report it to you with your simple thumbs up or a thumbs down, and collect as much data where your customers are telling you where there is a failure in the pipeline right? Classic QA on software, go ahead and reproduce that issue, and try to understand where is the model failing? Is it a tool call? Is it Is it And what part of your eval suit failed in going ahead and doing that, right? And review it, add it to the pipeline, and add that to the eval pipeline. This becomes extremely important to keep this feedback loop going so that you go ahead and access new frontiers of things that you are missing out in every point in time. And I think the two ways to go ahead and get to eval that is as holistic as possible is one to consistently human review a process, and then ensure that your customer feedback is part of your eval pipeline, right? Because these are the last 5% edge cases that really come out which a model or an existing eval framework might not catch.

So, to conclude, test the job and not just the answer, right? And that's extremely key to the whole process. And I'll just wrap up with all the things that we've discussed is one is always define your most ambitious success. Think about what life would be in AGI, right? And think about life in a model that has no inabilities and can absolutely perform like the human that you're trying to sort of emulate and try to define in this process right? Second is continuously test the workflow at every part of the workflow. Don't assume things in the workflow or like assume for an input and output to be the outcome of anything, right? And when you test the workflow, I think something I slightly missed on here is to make sure that your your tests are designed for what that element of the workflow does. For example, let's say in that in your workflow there's a specific part that is doing tool calls consistently to get the information, your eval has to be designed for tool calling and not for any different process separately, right?

Because the objective in that workflow process is to know what is your tool call ability and the tool tool call accuracy, right? At that at that point. So, every time make sure that when you're testing the workflow, you design the test to that specific part of the workflow and measure what is the success in that element of the workflow as well. And continuously gather evidence holistically. In an agentic world, what evidence for agentic action eval is something that keeps evolving. Uh and games is something I'm exploring to see how it works out, but there are so many different frontiers for you to go ahead and look at, right? Something super interesting that I came across while here I'm here is the parallel search model index, right? And how models are good at fetching search from parallels like AI search index. And that becomes very important for you to evaluate when you are designing a model that needs real-time data, right? So, make sure that you as a company or a person building these products is consistently looking for more evidence and more arenas and more paradigms for you to evaluate your models and that is one sure shot way to making sure that you cover most edge cases.

The fourth one is to evolve the test as time flow and making sure that you cover all aspects from the past and the future into the test and the simplest thing is if your test is scoring high is it 85 90 that is the time you change your test and you have to make it harder. Right? The simplest thing is if you think everyone is scoring like in the 99th percentile for the SATs, it is time that the test have to change in the internet of itself, right? And always learn from reality. Reality is going to be your best teacher and the idea of eval that I started off with is to make sure that we deliver the promise that we are making to the consumer and the user in production and that is most defined in the customer feedback and the reality that really happens, right? So, consistently think about how you can incorporate real user feedback and real world test and how you can have human review and human testing into the workflow as you go ahead and get started. I think with these five thought processes and the idea that you go ahead and test the job and not just the answer, you will have a good mental framework to go ahead and build eval for high stake applications. Thank you for coming. I'm happy to catch up for questions afterwards.