Hill-Climbing Skills: Improve Agents Without Changing the Model — Shubhankar Srivastava, Browserbase

OK. Greetings. Thank you to OK. Greetings. Thank you to everyone who came. I everyone who came. I everyone who came. I know this is the last know this is the last know this is the last workshop today. I workshop today. I workshop today. I appreciate you all appreciate you all appreciate you all coming here. I am Shub. I'm coming here. I am Shub. I'm coming here. I am Shub. I'm here to talk here to talk here to talk about hill climbing skills about hill climbing skills about hill climbing skills , specifically how , specifically how , specifically how you can improve you can improve you can improve your agents without your agents without your agents without getting too deep into the getting too deep into the getting too deep into the models or weighting models or weighting models or weighting factors themselves. I factors themselves. I factors themselves. I also have a slide here. I'll also have a slide here. I'll also have a slide here. I'll just show the QR code just show the QR code just show the QR code in case in case in case any of you want to any of you want to any of you want to ask questions, ask questions, ask questions, and we can and we can and we can go over them at go over them at go over them at the end. A little about the end. A little about the end. A little about me. I am Shub. I work on the me. I am Shub. I work on the sales engineering team at BrowserStack. I sales engineering team at BrowserStack. I mainly work with mainly work with mainly work with our customers who are our customers who are our customers who are implementing implementing implementing browser agents into browser agents into browser agents into production. I don't know, production. I don't know, production. I don't know, maybe we'll do a show of maybe we'll do a show of hands poll. Who among you knows what hands poll. Who among you knows what browser browser browser agents are? Someone? Okay, agents are? Someone? Okay, agents are? Someone? Okay, some people know. Perfectly. Have some people know. Perfectly. Have some people know. Perfectly. Have any of you any of you any of you built browser built browser built browser agents before? agents before? agents before? Has anyone used these Has anyone used these Has anyone used these tools?

tools? tools? Raise your hands. Okay, there Raise your hands. Okay, there Raise your hands. Okay, there are a few. Perfectly. are a few. Perfectly. are a few. Perfectly. Good. Maybe I'll Good. Maybe I'll Good. Maybe I'll explain a little bit about what explain a little bit about what explain a little bit about what browser browser browser agents are before agents are before agents are before diving into the diving into the diving into the concept of climbing a concept of climbing a concept of climbing a skill hill skill hill skill hill so it's clear so it's clear so it's clear what we're doing. what we're doing. At the end of the talk, At the end of the talk, we will also look at we will also look at we will also look at how you can how you can how you can apply this to apply this to apply this to your industry or field your industry or field . OK. So, a . OK. So, a . OK. So, a brief introduction to what brief introduction to what browser browser browser agents are. If you have an agents are. If you have an agents are. If you have an agent that is supposed to agent that is supposed to agent that is supposed to control the browser, control the browser, control the browser, you actually call you actually call you actually call it a browser it a browser it a browser agent, right? The browser agent, right? The browser agent, right? The browser becomes a tool becomes a tool becomes a tool for your agent. And for your agent. And for your agent. And how do you manage it? You how do you manage it? You how do you manage it? You control it control it control it using a CLI, MCP, or using a CLI, MCP, or using a CLI, MCP, or some other method where some other method where some other method where you can give a command to you can give a command to you can give a command to the browser, like: " the browser, like: " Hey, click this Hey, click this Hey, click this red button." " red button." " Click the Click the Click the update button." "Take update button." "Take update button." "Take a screenshot or snapshot of the a screenshot or snapshot of the a screenshot or snapshot of the accessibility tree of accessibility tree of accessibility tree of this site, okay?" this site, okay?" So, these are So, these are low-level low-level low-level tools that tools that tools that your agent can your agent can your agent can access to control access to control access to control the browser. BrowserBase is a the browser. BrowserBase is a the browser. BrowserBase is a platform for platform for platform for creating and running creating and running creating and running such browser such browser such browser agents. We provide agents. We provide agents. We provide the infrastructure, SDK, the infrastructure, SDK, the infrastructure, SDK, skills, and CLI skills, and CLI skills, and CLI needed to needed to needed to build these build these build these browser agents. And browser agents. And browser agents. And we're going to we're going to we're going to use all of use all of use all of these tools to these tools to these tools to see what see what see what going up a hill looks like.

going up a hill looks like. OK. I want to talk a little bit OK. I want to talk a little bit about the about the about the biggest problem biggest problem biggest problem agents face today. agents face today. agents face today. Essentially, it's reliability, Essentially, it's reliability, Essentially, it's reliability, right? People are right? People are right? People are annoyed by annoyed by annoyed by browser agents browser agents browser agents freezing, freezing, freezing, encountering encountering encountering anti-bot systems or anti-bot systems or anti-bot systems or CAPTCHAs, or simply CAPTCHAs, or simply CAPTCHAs, or simply forgetting or being too forgetting or being too forgetting or being too lazy to complete lazy to complete lazy to complete tasks. We tasks. We tasks. We view view view reliability primarily reliability primarily reliability primarily as an engineering as an engineering as an engineering problem. I think the problem. I think the problem. I think the latest models like latest models like latest models like Opus and Fable, when they Opus and Fable, when they Opus and Fable, when they came out for those three came out for those three came out for those three days, were really really days, were really really days, were really really smart, weren't smart, weren't smart, weren't they? But we see that a they? But we see that a they? But we see that a significant part of the significant part of the significant part of the work to work to work to ensure ensure ensure reliability lies in reliability lies in reliability lies in the complexities around the complexities around the complexities around them. These are the tools them. These are the tools them. These are the tools you provide them. These are the you provide them. These are the you provide them. These are the skills you skills you skills you give them. And that's why give them. And that's why give them. And that's why we're going to we're going to we're going to look at how we can look at how we can look at how we can improve the reliability of improve the reliability of improve the reliability of these browser these browser these browser agents. I wanted to agents. I wanted to agents. I wanted to give you some give you some give you some examples of where these examples of where these examples of where these agents agents agents are doing poorly right now, are doing poorly right now, are doing poorly right now, right? If I right? If I right? If I zoom in a little bit, you'll see zoom in a little bit, you'll see zoom in a little bit, you'll see that this browser that this browser that this browser agent is trying agent is trying agent is trying to go to Google Flights and to go to Google Flights and to go to Google Flights and click on...it's click on...it's click on...it's trying trying trying to drag the to drag the to drag the time filter, but it can't time filter, but it can't time filter, but it can't figure out how to figure out how to figure out how to do it.

do it. do it. Dragging is a Dragging is a Dragging is a complicated operation for complicated operation for complicated operation for such agents, and he such agents, and he such agents, and he becomes completely becomes completely becomes completely confused. He doesn't confused. He doesn't confused. He doesn't know what HTML know what HTML element it is or what element it is or what element it is or what JavaScript it needs. JavaScript it needs. JavaScript it needs. It can't It can't It can't do it, freezes and is do it, freezes and is do it, freezes and is unable to book unable to book unable to book my plane tickets. my plane tickets. my plane tickets. Another example Another example Another example I wanted to show is the I wanted to show is the I wanted to show is the browser agent for browser agent for browser agent for OpenTable. Let's say OpenTable. Let's say OpenTable. Let's say I have an OpenClaw or I have an OpenClaw or I have an OpenClaw or Hermes agent and I need to Hermes agent and I need to Hermes agent and I need to book a table book a table book a table for the evening, and I say, " for the evening, and I say, " Hey, go on OpenTable, Hey, go on OpenTable, Hey, go on OpenTable, find me something"—and find me something"—and find me something"—and look at this look at this look at this antibot, right? And there it antibot, right? And there it antibot, right? And there it just says "access just says "access just says "access denied." I don't denied." I don't denied." I don't know what's going on here, know what's going on here, know what's going on here, but can my but can my but can my browser agent browser agent browser agent get around these issues? get around these issues? get around these issues? Can they be more Can they be more Can they be more reliable and reliable and reliable and can they can they can they book a restaurant book a restaurant book a restaurant for my date— for my date— that's the main question that's the main question that's the main question I wanted to I wanted to I wanted to ask here. This is where ask here. This is where ask here. This is where we look at different we look at different we look at different ideas for improving the ideas for improving the ideas for improving the reliability of these reliability of these reliability of these agents, and one of them is agents, and one of them is ... actually, ... actually, ... actually, let's get into the let's get into the let's get into the root cause, okay root cause, okay ? Why is this ? Why is this ? Why is this happening? We have happening? We have happening? We have models like Opus, these models like Opus, these models like Opus, these 55s, they are very good 55s, they are very good 55s, they are very good for general purposes.

for general purposes. for general purposes. You may You may You may have used Codex have used Codex have used Codex computer use, it's very computer use, it's very computer use, it's very good. But when you good. But when you good. But when you want to work with want to work with want to work with long, complex long, complex long, complex websites like websites like websites like OpenTable or Google Flights, it becomes OpenTable or Google Flights, it becomes OpenTable or Google Flights, it becomes very difficult to run very difficult to run very difficult to run them in a productive them in a productive them in a productive environment, and the agent environment, and the agent environment, and the agent just doesn't know about these just doesn't know about these site-specific tricks or site-specific tricks or JavaScript that needs to be JavaScript that needs to be JavaScript that needs to be executed in executed in executed in the browser, right? And the browser, right? And the browser, right? And most importantly, most importantly, most importantly, every launch every launch every launch starts from scratch. You starts from scratch. You don't really have don't really have any built-in any built-in any built-in learning as you run the programs. I learning as you run the programs. I learning as you run the programs. I think Dwarkesh think Dwarkesh think Dwarkesh talked about this a few talked about this a few talked about this a few days ago. He days ago. He days ago. He asked: why is asked: why is asked: why is progress in progress in progress in the use of the use of the use of computers still computers still computers still so slow? so slow? so slow? Using Using Using a computer seems like a a computer seems like a a computer seems like a verifiable verifiable verifiable problem, so why has problem, so why has problem, so why has n't it been solved yet? In n't it been solved yet? In n't it been solved yet? In essence, no learning essence, no learning takes place at all. Each takes place at all. Each agent launch agent launch agent launch starts from scratch; starts from scratch; starts from scratch; it does not learn from its it does not learn from its it does not learn from its own work own work own work traces. These browser traces. These browser traces. These browser agents—I call them agents—I call them agents—I call them geniuses who never geniuses who never geniuses who never learn. Some learn. Some learn. Some might say, “Why do might say, “Why do might say, “Why do n’t you train n’t you train n’t you train the model on something like the model on something like the model on something like OpenTable or OpenTable or OpenTable or restaurant websites?” restaurant websites?” restaurant websites?” But that's not very But that's not very But that's not very scalable, right?

scalable, right? scalable, right? If you plan If you plan If you plan to create agents, to create agents, to create agents, you can't you can't you can't retrain them for retrain them for retrain them for each new each new each new task. You task. You task. You need a mechanism where need a mechanism where need a mechanism where it can learn from it can learn from it can learn from its results and its results and its results and improve over improve over improve over time. That's when we time. That's when we time. That's when we started thinking about it started thinking about it started thinking about it . It . It . It was about was about was about four to six four to six four to six months ago when months ago when months ago when Kartik proposed Kartik proposed Kartik proposed the idea of ​​auto-research the idea of ​​auto-research . Essentially, it's a cycle, right? . Essentially, it's a cycle, right? This cycle takes a This cycle takes a testable task, testable task, testable task, works on it, works on it, works on it, tries again and tries again and tries again and again, examines its own again, examines its own again, examines its own actions, and tries actions, and tries actions, and tries to improve. And we to improve. And we to improve. And we asked ourselves: how asked ourselves: how asked ourselves: how can we apply can we apply can we apply this to browser this to browser this to browser agents? How can we agents? How can we agents? How can we apply this to apply this to apply this to complex complex complex workflows, to sites workflows, to sites workflows, to sites that are non- that are non- that are non- deterministic, deterministic, deterministic, change daily, and change daily, and change daily, and have new captchas every time have new captchas every time have new captchas every time ? How can we ? How can we ? How can we learn from this? This is where learn from this? This is where learn from this? This is where I want to go into a I want to go into a I want to go into a demonstration and demonstration and demonstration and show this in show this in show this in real time. I'll real time. I'll real time. I'll start with...this start with...this start with...this query. So, I want to query. So, I want to query. So, I want to do something do something do something like this: go to Google like this: go to Google like this: go to Google Flights and find business Flights and find business class from San Francisco class from San Francisco class from San Francisco to New York this to New York this to New York this Saturday. I need Saturday. I need Saturday. I need one carry-on bag and one carry-on bag and one carry-on bag and this flight is in this flight is in this flight is in the morning. I hope I the morning. I hope I ...Okay. That's right.

...Okay. That's right. I'll just plug it in and I'll just plug it in and run it, okay? So, run it, okay? So, run it, okay? So, let's start a session in let's start a session in let's start a session in the browser. What you the browser. What you the browser. What you see in the center is a see in the center is a see in the center is a remote browser remote browser remote browser that has just that has just that has just launched and launched and launched and switched to Google Flights. switched to Google Flights. If I deploy my If I deploy my agent, look, it agent, look, it agent, look, it performs all these actions. These are the performs all these actions. These are the performs all these actions. These are the reasoning traces reasoning traces reasoning traces you see in real you see in real you see in real time, and it time, and it time, and it tries to tries to tries to apply a apply a apply a business-class filter. So, business-class filter. So, business-class filter. So, it will do its it will do its it will do its thing, we'll thing, we'll thing, we'll come back to that, come back to that, come back to that, but I wanted to but I wanted to but I wanted to talk about what's talk about what's talk about what's actually actually actually going on here and the going on here and the going on here and the architecture architecture architecture behind it, or the behind it, or the behind it, or the research cycle. research cycle. So we've run a So we've run a task for Google Flights, task for Google Flights, task for Google Flights, and the architecture here is what and the architecture here is what we call an auto-review loop. we call an auto-review loop. We took We took We took auto-research and auto-research and auto-research and review, that's why we review, that's why we review, that's why we called it called it called it auto-review. This auto-review. This auto-review. This loop essentially loop essentially loop essentially consists of two consists of two consists of two agents based on the Claude SDK agents based on the Claude SDK . You can call . You can call . You can call one the one the performing agent and the other the performing agent and the other the teaching agent.

teaching agent. teaching agent. The task of the executing agent is The task of the executing agent is to directly engage in review . It has access to the . It has access to the . It has access to the browser CLI we browser CLI we browser CLI we talked about. He has talked about. He has talked about. He has access to the browser, access to the browser, access to the browser, presses buttons, presses buttons, presses buttons, looks at looks at looks at screenshots, and screenshots, and screenshots, and tries to get tries to get tries to get information for me. information for me. information for me. And the teacher's task is And the teacher's task is to analyze the to analyze the to analyze the performer's actions and learn performer's actions and learn performer's actions and learn from what went from what went from what went wrong and what was wrong and what was wrong and what was done right. done right. They continue to They continue to work together until they work together until they work together until they find an find an find an optimal solution optimal solution optimal solution that converges. Isn't that right that converges. Isn't that right that converges. Isn't that right ? We don't tell it ? We don't tell it how many cycles to how many cycles to how many cycles to perform. We do perform. We do perform. We do not specify what not specify what not specify what success rate or success rate or success rate or success rate success rate success rate we require. We don't we require. We don't we require. We don't say, "Hey, we need to say, "Hey, we need to say, "Hey, we need to do this in such and such a do this in such and such a do this in such and such a number of tokens number of tokens number of tokens or seconds." We just or seconds." We just or seconds." We just give a general give a general give a general request and accept the request and accept the request and accept the bitter lesson, saying, " bitter lesson, saying, " Okay, let Okay, let Okay, let the performer and the teacher the performer and the teacher the performer and the teacher work on it." work on it." The key point The key point is that is that is that there is a strategy.md file, there is a strategy.md file, there is a strategy.md file, which is an artifact that is which is an artifact that is which is an artifact that is saved between saved between saved between different runs.

different runs. After each After each iteration, the teacher iteration, the teacher iteration, the teacher reflects on what reflects on what reflects on what happened, looks at happened, looks at happened, looks at the screenshots, the logs, and the screenshots, the logs, and the screenshots, the logs, and then thinks: "Oh, you didn't then thinks: "Oh, you didn't then thinks: "Oh, you didn't press that button press that button ." "You ." "You ." "You should press should press should press another button instead." He another button instead." He another button instead." He writes this into the writes this into the writes this into the strategy.md file, this artifact strategy.md file, this artifact strategy.md file, this artifact goes to the goes to the goes to the next iteration, and next iteration, and next iteration, and each time things each time things each time things get better. get better. get better. Maybe we'll go back Maybe we'll go back Maybe we'll go back to our to our to our demonstration and demonstration and demonstration and see what see what see what happened. So, happened. So, happened. So, iteration number one, iteration number one, iteration number one, this is where the failure occurred, this is where the failure occurred, this is where the failure occurred, and the conclusion from this and the conclusion from this and the conclusion from this iteration was: iteration was: iteration was: skip the entire skip the entire form calendar widget, because you can form calendar widget, because you can just use the just use the just use the query parameter, and you query parameter, and you query parameter, and you don't have to don't have to don't have to click anywhere, right? click anywhere, right? The agent was The agent was smart enough in this smart enough in this smart enough in this case to think, case to think, "Oh, I'm wasting "Oh, I'm wasting "Oh, I'm wasting so many tokens and so many tokens and so many tokens and so much time on clicks so much time on clicks , when I could just , when I could just , when I could just use the use the use the Google Flights query parameter." Google Flights query parameter." Google Flights query parameter." Of course, it's Of course, it's Of course, it's obvious now, but as a obvious now, but as a obvious now, but as a human, I probably wouldn't have human, I probably wouldn't have human, I probably wouldn't have guessed it and guessed it and guessed it and written written written such a such a such a complex complex complex query parameter on my own, right? Here is the query parameter on my own, right? Here is the query parameter on my own, right? Here is the conclusion from the conclusion from the conclusion from the first iteration.

first iteration. first iteration. It is currently working and It is currently working and It is currently working and performing tasks performing tasks performing tasks within the second within the second within the second iteration. He'll iteration. He'll iteration. He'll keep going until he's keep going until he's keep going until he's exhausted all the exhausted all the exhausted all the tricks, all the JavaScript tricks, all the JavaScript hacks he could hacks he could hacks he could use, and then he'll use, and then he'll use, and then he'll think, "Okay, this is a think, "Okay, this is a think, "Okay, this is a skill I can skill I can skill I can use from here." use from here." I think the key thing I think the key thing I also wanted to I also wanted to I also wanted to say is—after say is—after say is—after this this this convergence cycle is complete, I convergence cycle is complete, I convergence cycle is complete, I want to get a want to get a want to get a skill.md file. This skill.md file I skill.md file. This skill.md file I skill.md file. This skill.md file I will pass on to will pass on to will pass on to future agents, and it future agents, and it future agents, and it will contain a collection of will contain a collection of will contain a collection of all the hacks, tricks, all the hacks, tricks, all the hacks, tricks, or takeaways from or takeaways from or takeaways from previous runs. previous runs. We'll see what We'll see what this skill might look like this skill might look like this skill might look like . It might . It might . It might look like this: “Hey, the look like this: “Hey, the look like this: “Hey, the pre- pre- pre- filled URL is filled URL is filled URL is required. Don't required. Don't required. Don't waste clicks and waste clicks and waste clicks and tokens pressing tokens pressing tokens pressing these buttons." There these buttons." There these buttons." There is a bug with reopening the calendar is a bug with reopening the calendar . Don't do this.

If you want to If you want to move, move, move, drag the drag the drag the timestamp, timestamp, timestamp, use JavaScript use JavaScript injection. Don't injection. Don't injection. Don't try to do try to do try to do this with the mouse, okay? this with the mouse, okay? These are the techniques that are These are the techniques that are saved for saved for saved for every future every future every future launch, launch, launch, providing me with providing me with providing me with significant savings in significant savings in significant savings in terms of performance terms of performance , reliability, and , reliability, and , reliability, and cost. Now let cost. Now let cost. Now let 's go back to 's go back to 's go back to our launch and our launch and our launch and see if it see if it see if it went through. So, the second went through. So, the second went through. So, the second iteration was iteration was iteration was successful. And, as you can see, the successful. And, as you can see, the successful. And, as you can see, the cost savings cost savings cost savings are about 4 are about 4 are about 4 times. This graph times. This graph times. This graph is created "on the fly is created "on the fly ". It started at ". It started at ". It started at $12 and the second one $12 and the second one $12 and the second one was $3. was $3. Please note that my Please note that my agents here are Opus 48, agents here are Opus 48, agents here are Opus 48, both for the performer and the both for the performer and the both for the performer and the teacher. Now the teacher. Now the teacher. Now the third iteration has begun third iteration has begun third iteration has begun . The number of . The number of . The number of steps has decreased. steps has decreased. steps has decreased. The delay has also become The delay has also become The delay has also become less. But in the third less. But in the third less. But in the third iteration, iteration, iteration, an error occurred. So, he an error occurred. So, he an error occurred. So, he made some made some made some overcorrection, and the overcorrection, and the overcorrection, and the fourth iteration fourth iteration fourth iteration should probably should probably should probably learn from that.

learn from that. learn from that. We'll see if We'll see if We'll see if he does it. While he's he does it. While he's he does it. While he's busy with his busy with his busy with his work, I wanted to work, I wanted to work, I wanted to show some preliminary show some preliminary show some preliminary runs I did runs I did runs I did in preparation for in preparation for in preparation for this demonstration. this demonstration. this demonstration. For example, I went to For example, I went to For example, I went to AutoBrowse and said, "Hey, AutoBrowse and said, "Hey, AutoBrowse and said, "Hey, go to the AI go to the AI go to the AI engineers website and tell me engineers website and tell me engineers website and tell me what talks and what talks and what talks and workshops are on workshops are on hill climbing?" And he hill climbing?" And he did it in a few did it in a few did it in a few iterations. And you can iterations. And you can iterations. And you can see a replay of see a replay of see a replay of each one. These are each one. These are each one. These are repetitions coming repetitions coming repetitions coming from the browser. Very from the browser. Very from the browser. Very useful, important for, useful, important for, useful, important for, uh, the uh, the uh, the observability perspective. And, observability perspective. And, observability perspective. And, as you can see, uh, um, it's, it's as you can see, uh, um, it's, it's as you can see, uh, um, it's, it's gotten better in terms of gotten better in terms of gotten better in terms of value, 7.4 value, 7.4 value, 7.4 times. Although it times. Although it times. Although it was able to go through all the was able to go through all the was able to go through all the iterations, I still iterations, I still iterations, I still got a got a got a cost improvement. Now I cost improvement. Now I cost improvement. Now I wanted to actually wanted to actually wanted to actually see what see what see what the conclusions were from this the conclusions were from this the conclusions were from this skill, right? Uh skill, right? Uh , how do you, like, , how do you, like, , how do you, like, store all these store all these store all these findings? It says findings? It says : go directly to the : go directly to the : go directly to the schedule page. schedule page. schedule page. If there are If there are If there are opening errors, don't opening errors, don't opening errors, don't try to try to try to retry retry retry blindly. Run this blindly. Run this . Uh, enter a . Uh, enter a . Uh, enter a keyword in the session search field keyword in the session search field keyword in the session search field . Wait a . Wait a . Wait a minute. So, take these minute. So, take these minute. So, take these breaks because sometimes breaks because sometimes breaks because sometimes websites websites websites take a long time to load take a long time to load . So you can . So you can . So you can use these use these use these techniques. Uh, and then techniques. Uh, and then techniques. Uh, and then read the read the read the results. Uh, results. Uh, results. Uh, you know, there's a you know, there's a lot of lot of learning that goes into these skills. Maybe I'd learning that goes into these skills. Maybe I'd learning that goes into these skills. Maybe I'd like to show like to show like to show a few interesting ones. I a few interesting ones. I a few interesting ones. I want to show this want to show this want to show this OpenTable because we OpenTable because we OpenTable because we started with this started with this started with this blockade, like " blockade, like " access denied" access denied" access denied" when we started this when we started this when we started this demo. Uh, and

demo. Uh, and demo. Uh, and if we move on to the if we move on to the if we move on to the uh screenshot. This was the uh screenshot. This was the uh screenshot. This was the same " same " access denied" error access denied" error access denied" error we got, right? we got, right? we got, right? In iteration number one In iteration number one . Let's see . Let's see . Let's see how he learned from how he learned from how he learned from this. Uh, he this. Uh, he this. Uh, he realized that realized that realized that blocking is blocking is blocking is environmentally friendly because environmentally friendly because environmentally friendly because the website doesn't want the website doesn't want the website doesn't want this agent to do anything. this agent to do anything. Uh, in iteration number Uh, in iteration number two, he figured out that two, he figured out that two, he figured out that if I use a if I use a if I use a query parameter and query parameter and query parameter and enter the latitude, enter the latitude, enter the latitude, longitude, and longitude, and longitude, and metro ID, metro ID, metro ID, I automatically I automatically I automatically get the get the get the interface that interface that interface that works. Do not works. Do not works. Do not open {slash} r. This is open {slash} r. This is open {slash} r. This is blocked, and don't blocked, and don't blocked, and don't reload {slash reload {slash } s. These are the places where the } s. These are the places where the locks are triggered. If you locks are triggered. If you don't do that, but don't do that, but don't do that, but instead instead instead follow this follow this follow this technique, your agent technique, your agent technique, your agent will work, right? will work, right? will work, right? Uh, and as you can see, uh, Uh, and as you can see, uh, Uh, and as you can see, uh, the price changed from $6.22 the price changed from $6.22 the price changed from $6.22 to $0.51. The to $0.51. The to $0.51. The time spent was reduced from 189 time spent was reduced from 189 time spent was reduced from 189 seconds to 27 seconds. Uh seconds to 27 seconds. Uh , and the skill he , and the skill he , and the skill he invented was right invented was right invented was right here. Uh, create a here. Uh, create a search URL. search URL. search URL. perform the perform the perform the date calculation like this. Uh, these are date calculation like this. Uh, these are date calculation like this. Uh, these are famous subway foams for famous subway foams for famous subway foams for San Francisco. Uh, and, San Francisco. Uh, and, San Francisco. Uh, and, and, and follow these and, and follow these and, and follow these steps, okay? Never steps, okay? Never steps, okay? Never call {slash} R call {slash} R call {slash} R because it will return " because it will return " access denied".

access denied". access denied". This is the knowledge that This is the knowledge that This is the knowledge that a person could never a person could never a person could never acquire, but acquire, but acquire, but you just accept you just accept you just accept this bitter lesson and this bitter lesson and this bitter lesson and let the agents let the agents let the agents do the work, uh do the work, uh , and they will find these , and they will find these , and they will find these uh, tricks for you. I uh, tricks for you. I uh, tricks for you. I want to, uh, also want to, uh, also want to, uh, also show, uh, a few show, uh, a few show, uh, a few examples of what kind of examples of what kind of examples of what kind of improvements we're improvements we're improvements we're seeing. Uh, the reason seeing. Uh, the reason seeing. Uh, the reason this issue is so close to my heart is this issue is so close to my heart is because I'm because I'm implementing implementing implementing these browser these browser these browser agents into the work agents into the work agents into the work environment from a sales perspective. Uh, environment from a sales perspective. Uh, environment from a sales perspective. Uh, a sale is a sale is a sale is only considered complete when only considered complete when only considered complete when the browser the browser the browser agent is running successfully, agent is running successfully, agent is running successfully, so reliability so reliability so reliability is the is the is the most important thing to me. And this is the most important thing to me. And this is the most important thing to me. And this is the key key key tool that I tool that I tool that I offer to my offer to my offer to my clients, saying: " clients, saying: " Just run Just run Just run your agents, your agents, your agents, let them let them let them do this do this do this auto-search, and your auto-search, and your auto-search, and your agents will be agents will be agents will be successful in their work." successful in their work." successful in their work." Uh, and these are exactly the Uh, and these are exactly the Uh, and these are exactly the performance indicators performance indicators , uh, that people are looking for , uh, that people are looking for . Up to 16 times cheaper, 12 . Up to 16 times cheaper, 12 . Up to 16 times cheaper, 12 times cheaper, 3 times times cheaper, 3 times times cheaper, 3 times cheaper, uh, on cheaper, uh, on cheaper, uh, on different tasks. different tasks. different tasks. Of course, every Of course, every Of course, every task is different. The task is different. The task is different. The bottlenecks in bottlenecks in bottlenecks in each task are each task are each task are different, so, different, so, different, so, uh, you can see uh, you can see uh, you can see differences there, but, differences there, but, differences there, but, uh, the idea is that you uh, the idea is that you uh, the idea is that you can gradually can gradually can gradually improve improve improve using these, uh, using these, uh, using these, uh, methods. I wanted to methods. I wanted to methods. I wanted to go back to our go back to our go back to our live demo live demo live demo and see if there were and see if there were and see if there were any, uh, improvements any, uh, improvements . So, we have a ready-made . So, we have a ready-made . So, we have a ready-made skill, uh, and it is skill, uh, and it is skill, uh, and it is applied from applied from applied from start to finish without start to finish without start to finish without additional training additional training . The results . The results . The results are displayed are displayed are displayed correctly: correctly: correctly: one-way ticket, business class, one-way ticket, business class, one-way ticket, business class, from San Francisco to from San Francisco to from San Francisco to New York, July 4th, with New York, July 4th, with New York, July 4th, with carry-

carry- carry- on included, uh, and here's on included, uh, and here's on included, uh, and here's this flight at 8:00 AM for this flight at 8:00 AM for this flight at 8:00 AM for $649, right? So he was able to $649, right? So he was able to $649, right? So he was able to get all of that, and I get all of that, and I get all of that, and I can look through the can look through the can look through the entire recording to entire recording to entire recording to make sure it's make sure it's make sure it's accurate. Uh, he was able to accurate. Uh, he was able to accurate. Uh, he was able to sort of figure out this sort of figure out this sort of figure out this whole whole whole skill thing. skill thing. skill thing. A quick way is to A quick way is to URL encode the entire request, this is strictly URL encode the entire request, this is strictly recommended. Again, recommended. Again, recommended. Again, I know I'm showing these I know I'm showing these I know I'm showing these things, but when I things, but when I things, but when I first started first started first started looking at these looking at these looking at these results, it was a results, it was a results, it was a moment for me, as they moment for me, as they moment for me, as they say, a mythic say, a mythic say, a mythic moment for moment for moment for browser agents, browser agents, browser agents, you know, just the crazy you know, just the crazy you know, just the crazy reverse engineering of reverse engineering of reverse engineering of everything around. I everything around. I everything around. I show it to some of show it to some of show it to some of my clients and my clients and my clients and they are simply amazed. they are simply amazed. Previously, they had to Previously, they had to write scripts for write scripts for write scripts for each of their each of their each of their websites. They websites. They websites. They said, "Hey, I have to said, "Hey, I have to said, "Hey, I have to spend spend spend so many so many so many engineering hours engineering hours engineering hours just to just to just to reverse engineer these reverse engineer these reverse engineer these websites." How can I websites." How can I websites." How can I scale this to scale this to scale this to hundreds or thousands of hundreds or thousands of hundreds or thousands of sites? And now our sites? And now our sites? And now our customers have actually customers have actually customers have actually built entire built entire built entire pipelines of such pipelines of such pipelines of such skills that skills that skills that are created on the fly are created on the fly are created on the fly and pull and pull and pull requests that are generated requests that are generated requests that are generated automatically. If a automatically. If a automatically. If a failure occurs in the failure occurs in the failure occurs in the working environment, the working environment, the working environment, the agent intervenes and agent intervenes and agent intervenes and begins to independently begins to independently begins to independently correct this correct this correct this skill. There are many skill. There are many skill. There are many engineering solutions engineering solutions engineering solutions that we can that we can that we can implement to implement to implement to effectively effectively effectively use this use this use this technique. I also technique. I also technique. I also wanted to show what we wanted to show what we wanted to show what we did with it, did with it, did with it, you know? We went you know? We went you know? We went further and created this further and created this further and created this product called product called product called browse.sh. We thought: browse.sh. We thought: browse.sh. We thought: if we can if we can if we can do it for one do it for one do it for one site, we can do it for site, we can do it for site, we can do it for five, why not

five, why not five, why not do it for 500 do it for 500 do it for 500 sites and make it sites and make it sites and make it free for everyone. free for everyone. free for everyone. If you remember the 1990s If you remember the 1990s or the time before or the time before or the time before the internet, you had the the internet, you had the the internet, you had the Yellow Pages. There Yellow Pages. There Yellow Pages. There was information on was information on was information on how to find people, how to find people, how to find people, businesses, etc. This is the businesses, etc. This is the Yellow Pages for Yellow Pages for Yellow Pages for browser agents, browser agents, browser agents, right? Here are the details on how right? Here are the details on how right? Here are the details on how to complete any to complete any to complete any task on task on any website. I would like to any website. I would like to demonstrate this to demonstrate this to you all. Uh-huh. You you all. Uh-huh. You you all. Uh-huh. You can install can install can install browse CLI and browse CLI and browse CLI and use all use all use all these skills absolutely these skills absolutely these skills absolutely free. We have taken free. We have taken free. We have taken the cost of tokens upon the cost of tokens upon the cost of tokens upon ourselves and are making it ourselves and are making it ourselves and are making it available to you. available to you. So if I want to, So if I want to, say, find say, find say, find products on Amazon, I see this products on Amazon, I see this products on Amazon, I see this whole skill here. whole skill here. whole skill here. When should you When should you When should you use this use this use this skill? How do skill? How do skill? How do you check prices, you check prices, you check prices, availability, and all that stuff availability, and all that stuff availability, and all that stuff , right? Here's how , right? Here's how , right? Here's how to use it. There's to use it. There's to use it. There's a lot of detail here about a lot of detail here about a lot of detail here about Amazon, screenshots, and I Amazon, screenshots, and I Amazon, screenshots, and I just pass this just pass this just pass this browse skills CLI to my open browse skills CLI to my open browse skills CLI to my open claw, and it knows when claw, and it knows when claw, and it knows when to plug in the to plug in the to plug in the alltrails.com skill, when to weather.gov, and alltrails.com skill, when to weather.gov, and alltrails.com skill, when to weather.gov, and it makes my life a lot easier it makes my life a lot easier it makes my life a lot easier . We . We . We actually run actually run actually run the same the same the same automated automated automated research cycle behind the research cycle behind the research cycle behind the scenes and make scenes and make scenes and make it available to you.

it available to you. In fact, if you In fact, if you want, you can want, you can want, you can go here and say, go here and say, go here and say, "Hey, make me "Hey, make me "Hey, make me a skill to find a skill to find a skill to find events in San Francisco events in San Francisco events in San Francisco for this week." for this week." Do this, Do this, generate a skill, and generate a skill, and generate a skill, and it will do it for it will do it for it will do it for you. You will be sent a you. You will be sent a you. You will be sent a letter and you can letter and you can letter and you can use your use your use your skill on any skill on any skill on any agent you agent you agent you wish. Yes? So, wish. Yes? So, wish. Yes? So, we did this for we did this for we did this for browser agents, and browser agents, and browser agents, and I wanted to talk I wanted to talk I wanted to talk about how this could about how this could about how this could potentially be potentially be potentially be applied in other applied in other applied in other areas. I think areas. I think code writing agents are where I code writing agents are where I see frustration see frustration see frustration when the agent has to when the agent has to when the agent has to relearn the conventions of the relearn the conventions of the relearn the conventions of the repository every time. He doesn't repository every time. He doesn't repository every time. He doesn't apply apply apply linting rules correctly, and linting rules correctly, and linting rules correctly, and I have to I have to remind him of this every time. And maybe you can you can you can use your use your use your Claude MD and MD agents Claude MD and MD agents Claude MD and MD agents to improve it, but the to improve it, but the to improve it, but the main idea is main idea is main idea is that the agent shouldn't have to that the agent shouldn't have to retrain your retrain your task from scratch, task from scratch, task from scratch, right? I've seen it in right? I've seen it in right? I've seen it in research research research agents. We agents. We agents. We run a bunch of run a bunch of deep learning agents deep learning agents for our internal for our internal for our internal workflows, and workflows, and workflows, and every time they every time they every time they run into run into run into difficulties and difficulties and difficulties and try to try to try to retrain themselves, "Oh, I retrain themselves, "Oh, I retrain themselves, "Oh, I took this information from took this information from took this information from Wikipedia, but I didn't Wikipedia, but I didn't Wikipedia, but I didn't take it from this take it from this take it from this other site." And it's other site." And it's other site." And it's like: why not? We like: why not? We like: why not? We always always always use this use this use this other site. You should other site. You should other site. You should learn from your learn from your learn from your previous actions and previous actions and previous actions and extract extract extract information.

information. information. Internal Operations: Internal Operations: Internal Operations: You have all these dashboards, You have all these dashboards, You have all these dashboards, databases, and databases, and databases, and tools, but your tools, but your tools, but your agents don’t know which agents don’t know which agents don’t know which ones are working and which ones are ones are working and which ones are ones are working and which ones are prioritized. prioritized. prioritized. Ideally, you should Ideally, you should Ideally, you should apply this apply this auto-research technique, and auto-research technique, and for this to work, for this to work, for this to work, two conditions are necessary: ​​A—you two conditions are necessary: ​​A—you two conditions are necessary: ​​A—you must have a must have a must have a reproducible reproducible reproducible environment to environment to environment to run these cycles, and run these cycles, and run these cycles, and B—you must have B—you must have B—you must have some kind of verified some kind of verified some kind of verified result, some result, some result, some quantitative metric quantitative metric quantitative metric to run these to run these to run these cycles. Yes, I see a cycles. Yes, I see a cycles. Yes, I see a question there. Yes. Yes.

question there. Yes. Yes. Yes, that's right. So I Yes, that's right. So I think the question is think the question is think the question is how do you know that the how do you know that the how do you know that the auto-study loop is auto-study loop is auto-study loop is accurate, whether it matches accurate, whether it matches accurate, whether it matches , and when it says it , and when it says it , and when it says it matches—is matches—is matches—is it really so, when it it really so, when it it really so, when it says it's good—is it says it's good—is it says it's good—is it really good, and when it really good, and when it really good, and when it says it's bad—is it says it's bad—is it says it's bad—is it really bad. How can really bad. How can really bad. How can I be sure of I be sure of I be sure of what the agent says and what the agent says and what the agent says and what the cycle does? I think this is where what the cycle does? I think this is where what the cycle does? I think this is where I want to I want to I want to talk about the talk about the talk about the verified verified verified result. For result. For result. For Karpaty, this was Karpaty, this was Karpaty, this was pretty easy because pretty easy because pretty easy because he was looking at the he was looking at the he was looking at the logarithmic loss of logarithmic loss of logarithmic loss of the experiment when the experiment when the experiment when training neural training neural training neural networks, right? So it's networks, right? So it's networks, right? So it's very easy to see very easy to see very easy to see whether your logarithmic losses have decreased whether your logarithmic losses have decreased or not. But in many or not. But in many real-world scenarios, real-world scenarios, real-world scenarios, this is not the case. For this is not the case. For this is not the case. For browser agents, browser agents, browser agents, as I explained, did as I explained, did as I explained, did you book a you book a you book a plane ticket—that plane ticket—that plane ticket—that might be clear, might be clear, might be clear, but did you find the but did you find the but did you find the best best best flight options for me— flight options for me— that might be a bit that might be a bit that might be a bit blurry and not always blurry and not always blurry and not always optimal. So, if optimal. So, if optimal. So, if I understand correctly, I understand correctly, I understand correctly, you're saying that one you're saying that one you're saying that one mistake in the loop can mistake in the loop can mistake in the loop can ruin the entire ruin the entire auto-research process, and how do auto-research process, and how do you protect against that you protect against that you protect against that ? I have

? I have ? I have actually seen actually seen actually seen cases where some cases where some cases where some iterations have iterations have iterations have failed. For example, failed. For example, failed. For example, if we look at the if we look at the if we look at the previous slide, there previous slide, there previous slide, there are failures are failures are failures after there were after there were after there were successes, right? So, it is not successes, right? So, it is not successes, right? So, it is not always the case that always the case that always the case that after success, all after success, all after success, all subsequent iterations subsequent iterations subsequent iterations will also be successful. will also be successful. will also be successful. But I believe that it is the But I believe that it is the But I believe that it is the design of your loop that design of your loop that design of your loop that allows you to make allows you to make allows you to make it it error-resistant, so that the system error-resistant, so that the system does not get stuck on does not get stuck on does not get stuck on them and still them and still them and still corrects corrects corrects itself to some extent after any itself to some extent after any itself to some extent after any failures in previous failures in previous failures in previous attempts. I guess we're attempts. I guess we're attempts. I guess we're trying trying trying to prevent what they to prevent what they to prevent what they call call call explosive loss scenarios? explosive loss scenarios? I think you can I think you can do it. Actually, do it. Actually, do it. Actually, Karpaty talks Karpaty talks Karpaty talks about this as one about this as one about this as one way to way to way to automate automate automate research. You research. You research. You can actually can actually can actually automatically automatically automatically research different research different research different ways to ways to ways to automate the automate the automate the research itself. Is the research itself. Is the research itself. Is the "teacher-performer" cycle "teacher-performer" cycle "teacher-performer" cycle the best? Should there the best? Should there the best? Should there be a multi-agent be a multi-agent be a multi-agent architecture for architecture for architecture for this? Or is there this? Or is there this? Or is there some other approach some other approach some other approach that might work that might work that might work better for better for better for research research research tasks in neural tasks in neural tasks in neural networks compared to networks compared to networks compared to browser agents browser agents ? You can ? You can ? You can use use use different loop designs different loop designs different loop designs to prevent to prevent to prevent such situations.

such situations. Going back to my Going back to my point, as long as you have a point, as long as you have a point, as long as you have a verified verified verified outcome or some outcome or some outcome or some quantitative metric, you quantitative metric, you quantitative metric, you can design a can design a can design a good cycle. good cycle. The ability to The ability to reproduce reproduce reproduce the environment, in my the environment, in my the environment, in my opinion, is sometimes not opinion, is sometimes not opinion, is sometimes not discussed discussed discussed enough, but it is enough, but it is enough, but it is extremely important. extremely important. I think it was in I think it was in a conversation with Dwarkesh, a conversation with Dwarkesh, a conversation with Dwarkesh, he said, "It's important he said, "It's important he said, "It's important to have a verified to have a verified to have a verified result, but you result, but you result, but you should also have environments should also have environments should also have environments or or or use cases that use cases that use cases that can be reproduced many times can be reproduced many times can be reproduced many times , , , right?" If your agent right?" If your agent right?" If your agent can't " can't " push push push through" failures, it will have a through" failures, it will have a through" failures, it will have a hard time getting hard time getting hard time getting through multiple through multiple through multiple iterations and correcting iterations and correcting iterations and correcting itself. So, do you have any itself. So, do you have any itself. So, do you have any examples of coding examples of coding examples of coding agents? This is a good agents? This is a good agents? This is a good example because example because example because you can start with a Docker you can start with a Docker image or a specific image or a specific image or a specific EC2 instance, and you EC2 instance, and you EC2 instance, and you can replicate it can replicate it can replicate it from different parts of the stack from different parts of the stack . But perhaps . But perhaps . But perhaps intellectual intellectual intellectual work is not so easy work is not so easy work is not so easy to reproduce.

to reproduce. to reproduce. There are probably many different There are probably many different There are probably many different actions involving humans actions involving humans actions involving humans that make it very difficult that make it very difficult that make it very difficult to reproduce from a to reproduce from a to reproduce from a specific point in specific point in specific point in time. So investing time. So investing time. So investing in your environment in your environment in your environment can also be can also be can also be extremely extremely extremely beneficial in such beneficial in such beneficial in such cases. Perfectly. I cases. Perfectly. I cases. Perfectly. I just wanted to just wanted to just wanted to share the URLs of the share the URLs of the auto-review loop I was auto-review loop I was demonstrating demonstrating demonstrating today. If you today. If you today. If you scan the QR code, you scan the QR code, you scan the QR code, you can get a can get a can get a link to GitHub. And link to GitHub. And link to GitHub. And also the second QR code also the second QR code also the second QR code here—this is the skill itself. You here—this is the skill itself. You here—this is the skill itself. You can just type can just type can just type `/auto browse` in your Cloud `/auto browse` in your Cloud `/auto browse` in your Cloud Code or Codex and say, " Code or Codex and say, " Hey, `/auto browse`, Hey, `/auto browse`, Hey, `/auto browse`, use opus 48 for use opus 48 for use opus 48 for teacher and performer teacher and performer teacher and performer and create and create and create a skill for me to do that a skill for me to do that ." Uh...I'm reviewing ." Uh...I'm reviewing ." Uh...I'm reviewing one of these questions. By one of these questions. By one of these questions. By the way, if you have the way, if you have the way, if you have ideas for what you want to ideas for what you want to ideas for what you want to test the agent with, test the agent with, test the agent with, write to Slido and we write to Slido and we write to Slido and we will try to do it live will try to do it live will try to do it live . But . But . But let's consider let's consider let's consider this question. Is there a this question. Is there a this question. Is there a way to optimize way to optimize way to optimize convergence? I think this is the same convergence? I think this is the same convergence? I think this is the same question question question we were just we were just we were just talking about. Uh-huh. The high talking about. Uh-huh. The high talking about. Uh-huh. The high cost is interesting, cost is interesting, cost is interesting, because you would think because you would think because you would think that longer iterations that longer iterations that longer iterations would become would become would become more expensive over time. I ran a more expensive over time. I ran a more expensive over time. I ran a bunch of tests where bunch of tests where bunch of tests where Sonnet could be used Sonnet could be used Sonnet could be used for this task, but with for this task, but with for this task, but with longer iterations, longer iterations, longer iterations, compared to Opus for

compared to Opus for compared to Opus for shorter ones. I think shorter ones. I think shorter ones. I think you can achieve the same you can achieve the same you can achieve the same result result result using Sonnet with using Sonnet with using Sonnet with more more more iterations, which would still iterations, which would still iterations, which would still be cheaper. be cheaper. be cheaper. Besides, these costs Besides, these costs Besides, these costs are amortized, are amortized, are amortized, right? One right? One right? One auto-review cycle that auto-review cycle that auto-review cycle that we usually we usually we usually run costs run costs run costs somewhere around $30-40, or a somewhere around $30-40, or a somewhere around $30-40, or a maximum of $100 if maximum of $100 if maximum of $100 if the task is very complex. the task is very complex. the task is very complex. So these costs So these costs So these costs are amortized, and if are amortized, and if are amortized, and if it's acceptable for you to it's acceptable for you to it's acceptable for you to implement in implement in implement in production, just production, just production, just go for it. There are many go for it. There are many go for it. There are many interesting things you interesting things you interesting things you can do by can do by can do by evaluating the cycles evaluating the cycles evaluating the cycles you generate. Yes, I you generate. Yes, I you generate. Yes, I will share will share will share the slides as well. I can the slides as well. I can the slides as well. I can do this directly on do this directly on do this directly on GitHub. Uh-huh. Can you GitHub. Uh-huh. Can you GitHub. Uh-huh. Can you tell us how you tell us how you tell us how you came up with the idea of came up with the idea of came up with the idea of auto-research in auto-research in auto-research in auto-review? auto-review? Well, the process was pretty Well, the process was pretty simple. I took the GitHub simple. I took the GitHub repository that repository that repository that Karpatiy posted. And Karpatiy posted. And Karpatiy posted. And then, given then, given then, given the context of my the context of my the context of my clients, who were having clients, who were having clients, who were having some some reliability issues. The way reliability issues. The way I solved this I solved this I solved this for them was by for them was by for them was by visiting their visiting their visiting their office. I sat with them, office. I sat with them, office. I sat with them, working very closely.

working very closely. working very closely. And I tried to figure it out simply And I tried to figure it out simply And I tried to figure it out simply by brute force by brute force by brute force and reverse and reverse and reverse engineering engineering engineering . That's . That's . That's what I do. So, I am an what I do. So, I am an agent teacher. agent teacher. agent teacher. Can you Can you Can you automate my automate my automate my participation in the process participation in the process participation in the process by creating a cycle by creating a cycle by creating a cycle borrowed from borrowed from borrowed from Karpaty and Karpaty and Karpaty and building the same thing? building the same thing? building the same thing? Of course, there Of course, there Of course, there were many iterations after that. There were many iterations after that. There were many iterations after that. There was also a lot of was also a lot of was also a lot of engineering work engineering work engineering work to simplify it. But to simplify it. But to simplify it. But yeah, that's exactly what the yeah, that's exactly what the yeah, that's exactly what the process of turning process of turning process of turning this into was. Can I this into was. Can I this into was. Can I use this use this use this through a proxy? through a proxy? through a proxy? Can I be Can I be Can I be logged in to the site? logged in to the site? logged in to the site? Yes. Again, this Yes. Again, this Yes. Again, this brings us back to the brings us back to the brings us back to the environment you environment you environment you are designing. Browse base, are designing. Browse base, are designing. Browse base, the infrastructure on the infrastructure on the infrastructure on which all this works, which all this works, which all this works, allows you to allows you to allows you to log in to sites, log in to sites, log in to sites, save cookies, and save cookies, and save cookies, and use them on use them on use them on subsequent launches subsequent launches subsequent launches so you don't have to log in every time so you don't have to log in every time . This greatly simplifies the . This greatly simplifies the . This greatly simplifies the reproduction of real-world reproduction of real-world reproduction of real-world workflows. And workflows. And workflows. And yes, you can also yes, you can also yes, you can also use a use a use a proxy, if you have proxy, if you have proxy, if you have certain IP addresses, certain IP addresses, certain IP addresses, such as such as such as residential ones, you residential ones, you residential ones, you can connect them.

can connect them. Success Success depends largely on the quality of depends largely on the quality of depends largely on the quality of your infrastructure your infrastructure your infrastructure and environment, and environment, and environment, especially in the face of especially in the face of especially in the face of countermeasures like captchas countermeasures like captchas countermeasures like captchas and anti-bots. Therefore, and anti-bots. Therefore, and anti-bots. Therefore, creating the creating the creating the highest quality highest quality highest quality infrastructure is infrastructure is infrastructure is certainly certainly certainly important. important. Surprisingly. Are there any Surprisingly. Are there any requests for something we requests for something we requests for something we could run could run could run live? Yes, a live? Yes, a live? Yes, a question. question. question. One question. Do you One question. Do you One question. Do you force the teacher force the teacher force the teacher to analyze the entire to analyze the entire to analyze the entire trajectory of the trajectory of the trajectory of the keyword once or at keyword once or at keyword once or at each step? How each step? How each step? How exactly is this a exactly is this a exactly is this a great question. great question. great question. Actually, I wanted to Actually, I wanted to Actually, I wanted to show the code as well. If you show the code as well. If you show the code as well. If you look on GitHub, you look on GitHub, you look on GitHub, you can find the file can find the file can find the file evaluate.mjs. In essence, it evaluate.mjs. In essence, it evaluate.mjs. In essence, it describes the entire cycle that is describes the entire cycle that is describes the entire cycle that is happening here. But happening here. But happening here. But to answer your to answer your to answer your question, the teacher and the question, the teacher and the question, the teacher and the performer work performer work performer work one after the other in one after the other in one after the other in each cycle. So, each cycle. So, each cycle. So, the teacher finds the teacher finds the teacher finds conclusions at each conclusions at each conclusions at each iteration. Maybe iteration. Maybe iteration. Maybe if I show a demo if I show a demo if I show a demo here. Let's here. Let's here. Let's go back to what go back to what go back to what we just did.

we just did. we just did. The first iteration had The first iteration had The first iteration had these conclusions. And then the these conclusions. And then the these conclusions. And then the second iteration had second iteration had second iteration had some different findings. some different findings. some different findings. Yes? You should add a Yes? You should add a Yes? You should add a clear ISO date in this clear ISO date in this clear ISO date in this format. The third format. The third format. The third iteration would have even iteration would have even iteration would have even different conclusions. different conclusions. different conclusions. Resolving the date still Resolving the date still Resolving the date still opens a blank opens a blank opens a blank page and spends page and spends page and spends three steps reading three steps reading three steps reading today's date today's date today's date or having it calculated by or having it calculated by or having it calculated by the executor. So, he's the executor. So, he's the executor. So, he's trying to trying to trying to figure out the date and figure out the date and figure out the date and time. So at each time. So at each time. So at each iteration, the teacher iteration, the teacher iteration, the teacher learns something new learns something new learns something new and contributes it to strategy.md. and contributes it to strategy.md. and contributes it to strategy.md. Maybe I'll Maybe I'll Maybe I'll demonstrate this in demonstrate this in demonstrate this in practice practice practice by looking at by looking at by looking at the trace for the trace for the trace for one of these runs one of these runs . Uh...So, here are all . Uh...So, here are all . Uh...So, here are all my launches. my launches. For example, this was an For example, this was an example of an example of an example of an OpenTable reservation. And it was the OpenTable reservation. And it was the OpenTable reservation. And it was the strategy.md file that the teacher strategy.md file that the teacher strategy.md file that the teacher added to every time. added to every time. So, step zero, you So, step zero, you should do this, and so should do this, and so should do this, and so on. It was a on. It was a on. It was a task. Here is the query task. Here is the query task. Here is the query I gave him.

I gave him. I gave him. Find me Find me Find me a reservation for a reservation for a reservation for Bixby Bix restaurant. And these Bixby Bix restaurant. And these Bixby Bix restaurant. And these were the tracings were the tracings were the tracings he studied on. he studied on. he studied on. He took screenshots. He took screenshots. He took screenshots. He took these kinds of He took these kinds of He took these kinds of tracings. At each tracings. At each tracings. At each step, it performs a step, it performs a step, it performs a browser opening. browser opening. browser opening. This is the result we This is the result we This is the result we get. Then step get. Then step get. Then step number two, number two, number two, browser snapshot. The snapshot browser snapshot. The snapshot browser snapshot. The snapshot essentially captures the essentially captures the essentially captures the accessibility tree. And here is accessibility tree. And here is accessibility tree. And here is this picture. So, he this picture. So, he this picture. So, he reads all of this and analyzes at reads all of this and analyzes at reads all of this and analyzes at every step every step every step what was what was what was done right and done right and done right and what wasn't. Yes, this is what wasn't. Yes, this is what wasn't. Yes, this is actually machine actually machine actually machine code. I won't code. I won't code. I won't show all of it show all of it show all of it . But yes, this is the . But yes, this is the . But yes, this is the summary he also summary he also summary he also writes on each writes on each writes on each iteration. So, there are iteration. So, there are iteration. So, there are a lot of engineering a lot of engineering a lot of engineering tweaks that tweaks that tweaks that can be made to can be made to can be made to adapt this to adapt this to adapt this to your field. your field. The easiest way The easiest way to get to know it is to get to know it is we also posted it we also posted it we also posted it on skills.sh. In Auto Browse. And you on skills.sh. In Auto Browse. And you on skills.sh. In Auto Browse. And you can do something can do something can do something similar. Excellent similar. Excellent similar. Excellent question. Very good

question. Very good question. Very good question. I guess this question. I guess this question. I guess this brings us back to brings us back to brings us back to how to increase the how to increase the how to increase the reliability of your reliability of your agents? Reliability is an engineering is an engineering is an engineering problem. You can problem. You can problem. You can design your design your design your system to system to system to use one use one use one skill for each skill for each skill for each task. Or you could task. Or you could task. Or you could say, "Hey, one say, "Hey, one say, "Hey, one skill for every skill for every skill for every task you'll task you'll ever encounter." ever encounter." ever encounter." I think it I think it I think it depends entirely on how depends entirely on how depends entirely on how you design you design you design the infrastructure. It the infrastructure. It the infrastructure. It completely depends completely depends completely depends on your on your on your use case and you use case and you use case and you evaluate evaluate evaluate all of these all of these all of these strategies to strategies to strategies to see which one see which one see which one works for you. For works for you. For works for you. For us it was like that. Yes. us it was like that. Yes. Yes. Yes, and this is where infrastructure comes into play again. If infrastructure comes into play again. If the infrastructure has the infrastructure has the infrastructure has the intelligence to the intelligence to the intelligence to pull in skills from pull in skills from pull in skills from OpenTable, Resy, and Google OpenTable, Resy, and Google OpenTable, Resy, and Google Reservations, maybe it Reservations, maybe it Reservations, maybe it can combine all of them can combine all of them can combine all of them to figure out the to figure out the to figure out the best way best way best way to book me a to book me a to book me a date night. That's date night. That's date night. That's your decision, is your decision, is your decision, is n't it?

n't it? n't it? The infrastructure The infrastructure The infrastructure must be able must be able must be able to determine which to determine which to determine which tools tools tools to use to to use to to use to get the job done. Here's get the job done. Here's get the job done. Here's how I would how I would how I would approach it. Determine approach it. Determine approach it. Determine which which which use cases are use cases are use cases are most common most common most common among your customers. among your customers. among your customers. For these cases, what For these cases, what For these cases, what skill files, skill files, skill files, tips, and tips, and tips, and tools are tools are tools are best? Maybe best? Maybe best? Maybe you want you want you want to use to use to use specific specific specific tools for tools for tools for specific tasks, specific tasks, specific tasks, right? Maybe you right? Maybe you right? Maybe you want to want to want to use use use universal universal universal tools for all tools for all tools for all tasks. It's the same tasks. It's the same tasks. It's the same argument. Should you argument. Should you argument. Should you use one use one use one skill for everyone, skill for everyone, skill for everyone, or one skill for or one skill for or one skill for everyone? Evaluate all these everyone? Evaluate all these everyone? Evaluate all these strategies and strategies and strategies and let the let the let the infrastructure infrastructure infrastructure handle it. handle it. Yes. Yes. Yes. I Yes. Yes. Yes. I mean, it's a very mean, it's a very mean, it's a very similar concept, similar concept, similar concept, right? So, DSPY. Yes. I right? So, DSPY. Yes. I right? So, DSPY. Yes. I think they stumbled think they stumbled think they stumbled upon something really cool upon something really cool . They were doing this . They were doing this . They were doing this 3-4 years ago when they 3-4 years ago when they 3-4 years ago when they started building DSPY, started building DSPY, started building DSPY, Jepa and everything else. I Jepa and everything else. I Jepa and everything else. I think we're also think we're also think we're also doing a lot of doing a lot of doing a lot of research in this research in this research in this direction. Browser direction. Browser direction. Browser versions work with versions work with versions work with Microsoft. We released an Microsoft. We released an Microsoft. We released an article on how to article on how to article on how to create the right create the right create the right verifiers for verifiers for verifiers for your agents. I think your agents. I think your agents. I think conducting assessments is conducting assessments is conducting assessments is easy until easy until easy until the question arises: "How do the question arises: "How do the question arises: "How do I assess the I assess the I assess the assessment itself?" True? Do we assessment itself?" True? Do we assessment itself?" True? Do we need to involve need to involve need to involve so many people to so many people to so many people to verify all these verify all these verify all these results, or is it possible results, or is it possible results, or is it possible to create a high-quality to create a high-quality to create a high-quality verifier? And all of this verifier? And all of this verifier? And all of this uses uses uses the lessons learned from these the lessons learned from these the lessons learned from these articles. I can also

articles. I can also articles. I can also share all of this share all of this share all of this on GitHub. Now on GitHub. Now on GitHub. Now I'll see if there are any I'll see if there are any I'll see if there are any new questions here. Where and new questions here. Where and new questions here. Where and how are these skills how are these skills how are these skills stored and stored and stored and retrieved on a retrieved on a retrieved on a large scale? large scale? Everything in this repository is currently Everything in this repository is currently stored in my stored in my stored in my cloud environment. cloud environment. cloud environment. So, each of these So, each of these So, each of these files is a separate files is a separate files is a separate skill. This was my skill. This was my skill. This was my search query for search query for search query for recipes. I wanted to recipes. I wanted to recipes. I wanted to cook cook cook dinner yesterday, so I asked, dinner yesterday, so I asked, dinner yesterday, so I asked, "Hey, give me "Hey, give me "Hey, give me some recipes." So, some recipes." So, some recipes." So, all of this is now all of this is now all of this is now in my in my in my local local local skills folder. But if you skills folder. But if you skills folder. But if you want to implement this want to implement this want to implement this in production, of course in production, of course , you can , you can , you can integrate this into your integrate this into your integrate this into your agent cycle. Or for agent cycle. Or for agent cycle. Or for brows.sh where we have brows.sh where we have brows.sh where we have public public public skills, we skills, we skills, we use use use Vercel blob storage to Vercel blob storage to Vercel blob storage to store them and store them and store them and call them on request. call them on request. I hope this I hope this helps. Websites helps. Websites helps. Websites change often. What is change often. What is change often. What is your strategy for your strategy for your strategy for monitoring and monitoring and monitoring and updating your updating your updating your website skills?

website skills? website skills? Yes, that's a great Yes, that's a great Yes, that's a great question. This is exactly what question. This is exactly what question. This is exactly what I mentioned earlier. I mentioned earlier. I mentioned earlier. How do you keep How do you keep How do you keep your skills up to date your skills up to date when your website is when your website is constantly changing? constantly changing? Our Current Theory Our Current Theory is that it is not is that it is not is that it is not worth wasting worth wasting worth wasting tokens on daily tokens on daily tokens on daily automatic automatic automatic research. It's better research. It's better research. It's better to run to run to run auto-research only auto-research only auto-research only when there's a when there's a when there's a failure in the production failure in the production failure in the production environment, right? environment, right? We build systems We build systems where any where any where any error triggers an error triggers an error triggers an auto-research loop auto-research loop to understand to understand to understand the cause, fix it, the cause, fix it, the cause, fix it, and create a and create a and create a merge request for an engineer. merge request for an engineer. The skill The skill is automatically updated. So yes, is automatically updated. So yes, is automatically updated. So yes, we take into account the we take into account the we take into account the failure cases, but it's a failure cases, but it's a failure cases, but it's a matter of compromise. matter of compromise. You can run You can run an auto-research or an auto-research or an auto-research or test run test run test run every 2–3 hours to every 2–3 hours to every 2–3 hours to check if check if the skill is working perfectly. So, there are different the skill is working perfectly. So, there are different ways to set up ways to set up ways to set up cron jobs to cron jobs to cron jobs to test the accuracy of test the accuracy of test the accuracy of these skills. This is exactly what these skills. This is exactly what these skills. This is exactly what I am currently I am currently I am currently actively working actively working actively working with the client to with the client to with the client to implement. And it raises implement. And it raises implement. And it raises a lot of questions, does a lot of questions, does a lot of questions, does n't it? People don't n't it? People don't n't it? People don't always need 100% always need 100% accuracy. Maybe accuracy. Maybe accuracy. Maybe they want to reduce they want to reduce they want to reduce costs by 10%. Therefore, this is a costs by 10%. Therefore, this is a costs by 10%. Therefore, this is a question that often question that often question that often arises. What are arises. What are arises. What are your teacher's criteria in your teacher's criteria in your teacher's criteria in this example, besides this example, besides this example, besides achieving the goal?

achieving the goal? achieving the goal? It appears to be performing It appears to be performing It appears to be performing some some some step count optimizations. What step count optimizations. What step count optimizations. What other criteria do you other criteria do you other criteria do you use? So, again use? So, again use? So, again , it comes down to the , it comes down to the , it comes down to the design of the design of the auto-research loop. There's a auto-research loop. There's a way to tell way to tell way to tell your your your autoresponder, "Hey, autoresponder, "Hey, autoresponder, "Hey, I want to focus I want to focus I want to focus only on success." " only on success." " only on success." " Once you Once you Once you succeed, I will be succeed, I will be succeed, I will be satisfied." "Just satisfied." "Just satisfied." "Just give it to me." give it to me." give it to me." This is usually not what This is usually not what This is usually not what everyone wants. People everyone wants. People everyone wants. People want to minimize want to minimize want to minimize latency, perhaps latency, perhaps latency, perhaps because this is a because this is a because this is a real-time launch where real-time launch where real-time launch where how quickly you how quickly you how quickly you complete a task is important. complete a task is important. complete a task is important. So, if this is an So, if this is an So, if this is an optimization metric optimization metric , you focus , you focus , you focus on getting on getting on getting it done as quickly as possible. I've it done as quickly as possible. I've it done as quickly as possible. I've sometimes seen these sometimes seen these auto-exploration loops be auto-exploration loops be so smart that so smart that so smart that they understand that they understand that they understand that browsers aren't needed at all browsers aren't needed at all browsers aren't needed at all , they just , they just , they just reverse-engineer the API reverse-engineer the API reverse-engineer the API and use and use and use it. And that's the fastest it. And that's the fastest it. And that's the fastest result you result you result you can achieve, can achieve, can achieve, right? So sometimes I've seen right? So sometimes I've seen right? So sometimes I've seen situations where situations where situations where solutions were found in solutions were found in solutions were found in 500 milliseconds. Sometimes 500 milliseconds. Sometimes 500 milliseconds. Sometimes it depends on the it depends on the it depends on the number of steps, number of steps, number of steps, right? You want to right? You want to right? You want to minimize the minimize the minimize the context loop and context loop and context loop and the number of steps the number of steps the number of steps required to required to required to complete a task. In complete a task. In complete a task. In my case, I my case, I my case, I think what I did was think what I did was allow you allow you allow you to stop or to stop or to stop or end the process end the process end the process whenever you whenever you whenever you want. I'm not want. I'm not want. I'm not too concerned about too concerned about too concerned about the costs. I the costs. I the costs. I care most about reliability.

care most about reliability. care most about reliability. So continue as long as So continue as long as So continue as long as you think it's you think it's you think it's appropriate, and then it's the appropriate, and then it's the teacher agent's job to complete the teacher agent's job to complete the process when they process when they process when they want to. But yes, you want to. But yes, you want to. But yes, you can try can try can try different strategies. If different strategies. If different strategies. If the website supports Web the website supports Web the website supports Web MCP, will it be MCP, will it be MCP, will it be used? Web used? Web used? Web MCP is really cool. This is MCP is really cool. This is MCP is really cool. This is basically a concept, I don't basically a concept, I don't basically a concept, I don't know if everyone knows about it know if everyone knows about it know if everyone knows about it , but it's the idea , but it's the idea , but it's the idea of ​​making your website of ​​making your website of ​​making your website programmable. This programmable. This programmable. This came out in a came out in a came out in a recent release of recent release of recent release of Google Chrome, where you tell Google Chrome, where you tell Google Chrome, where you tell your site that if your site that if your site that if someone wants someone wants someone wants to book a seat, to book a seat, to book a seat, they can do XYZ on the they can do XYZ on the they can do XYZ on the site. This is similar to the API, site. This is similar to the API, site. This is similar to the API, but works in the browser but works in the browser . Uh, right now I don't think . Uh, right now I don't think . Uh, right now I don't think our research our research our research cycle supports that, cycle supports that, cycle supports that, but I don't think it would be but I don't think it would be but I don't think it would be too difficult too difficult too difficult to do. In fact, we to do. In fact, we to do. In fact, we recently added recently added recently added Web MCP support to our Web MCP support to our Web MCP support to our SDKs that we released. SDKs that we released. SDKs that we released. We also created We also created We also created tools for tools for tools for site owners to site owners to implement Web MCP at scale.

implement Web MCP at scale. But yes, it's an interesting But yes, it's an interesting But yes, it's an interesting point. We haven't point. We haven't point. We haven't seen significant seen significant seen significant adoption of Web MCP yet. Uh, adoption of Web MCP yet. Uh, adoption of Web MCP yet. Uh, truth for a pause. This is truth for a pause. This is truth for a pause. This is interesting. I think as long as interesting. I think as long as interesting. I think as long as the work is done— the work is done— for example, for example, for example, the cheapest tickets are found on Google Flights the cheapest tickets are found on Google Flights the cheapest tickets are found on Google Flights , , , a reservation is made on OpenTable a reservation is made on OpenTable —the system evaluates —the system evaluates —the system evaluates itself. This is usually a itself. This is usually a itself. This is usually a binary indicator, and binary indicator, and binary indicator, and yes. Again, I yes. Again, I yes. Again, I fully accept the " fully accept the " bitter lesson." I bitter lesson." I bitter lesson." I try not to put too try not to put too try not to put too many restrictions on these many restrictions on these many restrictions on these assessments. Have assessments. Have assessments. Have you considered a " you considered a " verifier as a verifier as a verifier as a service" for service" for web browsing tasks, web browsing tasks, including something that including something that including something that allows you to simulate allows you to simulate allows you to simulate certain side effects, certain side effects, certain side effects, such as purchases? Uh such as purchases? Uh , I might , I might , I might have to ask have to ask ... for clarification on ... for clarification on ... for clarification on that. Rather, if you're that. Rather, if you're that. Rather, if you're here, what do you here, what do you here, what do you mean by " mean by " verifier as a verifier as a verifier as a service"? Yes. Uh-huh.

service"? Yes. Uh-huh. So, uh, you So, uh, you mean that...Yeah, if I mean that...Yeah, if I mean that...Yeah, if I understand correctly, understand correctly, understand correctly, you have different you have different you have different environments that you can environments that you can environments that you can use for use for reinforcement learning (RL), and reinforcement learning (RL), and you want to simulate you want to simulate you want to simulate what happens if I what happens if I what happens if I copy Amazon, if I copy Amazon, if I copy Amazon, if I click "buy," what click "buy," what click "buy," what happens next? happens next? Err. No, because we're Err. No, because we're Err. No, because we're trying to trying to trying to rely more on rely more on rely more on real websites. real websites. Regarding browsers, we Regarding browsers, we have the privilege of have the privilege of have the privilege of accessing most of the accessing most of the accessing most of the internet internet internet through accessing through accessing through accessing various sites, various sites, various sites, solving captchas, solving captchas, solving captchas, etc. So we etc. So we etc. So we try to try to try to emulate the emulate the emulate the real world as much as possible, but we real world as much as possible, but we real world as much as possible, but we see a lot of these see a lot of these see a lot of these RL environments where people RL environments where people RL environments where people simulate websites simulate websites simulate websites to train and to train and to train and build browser build browser build browser agents that agents that agents that do very well with, do very well with, do very well with, for example, Salesforce for example, Salesforce for example, Salesforce or SAP. We try to or SAP. We try to or SAP. We try to do this on real do this on real do this on real portals as much as portals as much as portals as much as possible. Is it easy for possible. Is it easy for possible. Is it easy for models to bypass hCaptcha, models to bypass hCaptcha, models to bypass hCaptcha, or does it require a lot of or does it require a lot of or does it require a lot of tokens? What is the point of tokens? What is the point of tokens? What is the point of hCaptcha for hCaptcha for hCaptcha for website owners? hCaptcha, if website owners? hCaptcha, if website owners? hCaptcha, if I understand correctly, I understand correctly, I understand correctly, is a special type of is a special type of is a special type of captcha. It's something like captcha. It's something like captcha. It's something like Google reCAPTCHA or something.

Google reCAPTCHA or something. Yes, I would probably Yes, I would probably talk about that talk about that talk about that separately. I don't think separately. I don't think separately. I don't think we can we can we can discuss which discuss which discuss which captchas we can captchas we can captchas we can solve and which we can't. solve and which we can't. solve and which we can't. But this is not a task But this is not a task But this is not a task that requires a lot of that requires a lot of that requires a lot of tokens, since it is tokens, since it is tokens, since it is done at the done at the done at the infrastructure level, infrastructure level, infrastructure level, right? Uh-huh. We right? Uh-huh. We right? Uh-huh. We don't actually don't actually don't actually need to need to need to use AI use AI use AI to solve to solve to solve captchas. This is handled captchas. This is handled captchas. This is handled at the Chrome binary level at the Chrome binary level at the Chrome binary level , and we do , and we do , and we do it out of the box. How it out of the box. How it out of the box. How does the teacher evaluate does the teacher evaluate does the teacher evaluate the performer and when is the performer and when is the performer and when is a skill considered a skill considered a skill considered learned? Yes. So, learned? Yes. So, learned? Yes. So, he evaluates the performer, he evaluates the performer, he evaluates the performer, essentially, by reading these essentially, by reading these essentially, by reading these traces. A separate folder is created for traces. A separate folder is created for traces. A separate folder is created for each iteration each iteration . There are these JSON files. He reads them. files. He reads them. files. He reads them. It has a It has a It has a cloud agent SDK loop and cloud agent SDK loop and cloud agent SDK loop and essentially determines what essentially determines what essentially determines what conclusions can be conclusions can be conclusions can be drawn from that loop drawn from that loop . Actually, I think I have a . Actually, I think I have a . Actually, I think I have a slide that slide that slide that talks about this talks about this talks about this at the end. Yes, it's a kind of at the end. Yes, it's a kind of at the end. Yes, it's a kind of hint hint hint that says your that says your browser automation agent can browser automation agent can use these use these use these tools, and the tools, and the tools, and the strategy.md file is added strategy.md file is added strategy.md file is added during each iteration.

during each iteration. And this is how the teacher And this is how the teacher evaluates the evaluates the evaluates the performer's work. So, yes, performer's work. So, yes, performer's work. So, yes, it reads the trace, it reads the trace, it reads the trace, has a system prompt, a has a system prompt, a has a system prompt, a previous strategy, previous strategy, previous strategy, and develops a better one. and develops a better one. Surprisingly. Uh-huh. I hope we I hope we answered all your answered all your answered all your questions. Uh-huh. Are there questions. Uh-huh. Are there questions. Uh-huh. Are there any other questions on the any other questions on the any other questions on the live broadcast that we live broadcast that we live broadcast that we could discuss? could discuss? could discuss? Yes. Yes, it can Yes. Yes, it can Yes. Yes, it can work with Stage Hand SDK, it work with Stage Hand SDK, it work with Stage Hand SDK, it can work in can work in can work in your cloud code. your cloud code. your cloud code. This is essentially markdown, right? This is essentially markdown, right? This is essentially markdown, right? I took a skill and I took a skill and I took a skill and said to my said to my said to my cloud code, " cloud code, " cloud code, " Can you find events Can you find events Can you find events in San Francisco for this in San Francisco for this in San Francisco for this Friday, July 3rd?" And Friday, July 3rd?" And Friday, July 3rd?" And he went and did it. he went and did it. he went and did it. So he doesn't So he doesn't So he doesn't have to have to have to use use use browser-base, he doesn't browser-base, he doesn't browser-base, he doesn't have to have to have to use Stage use Stage use Stage Hand. This is, in essence, a Hand. This is, in essence, a Hand. This is, in essence, a skill. This is a skill. This is a skill. This is a portable artifact portable artifact that can be that can be that can be used in used in used in any agent or any agent or any agent or SDK. Yes. Yes, great SDK. Yes. Yes, great SDK. Yes. Yes, great question. I think question. I think question. I think if I if I if I understand correctly how you understand correctly how you

understand correctly how you protect against protect against protect against prompt injection, prompt injection, prompt injection, since websites since websites since websites can trick my can trick my can trick my auto-explore loop auto-explore loop into learning into learning into learning something bad, which something bad, which something bad, which will lead to will lead to will lead to malicious behavior malicious behavior malicious behavior in the work in the work in the work environment. There are several environment. There are several environment. There are several ways we ways we ways we do this. One of them is at do this. One of them is at do this. One of them is at the level of the the level of the the level of the infrastructure itself. We infrastructure itself. We infrastructure itself. We have a bunch of have a bunch of have a bunch of protections around protections around protections around our our our browsers themselves. We have a browsers themselves. We have a browser-based safe mode where browser-based safe mode where you can you can you can control what control what control what URLs it can URLs it can URLs it can go to. You can go to. You can go to. You can control what actions control what actions control what actions it can perform it can perform it can perform on different websites. on different websites. on different websites. So, at the So, at the So, at the infrastructure level, at the infrastructure level, at the infrastructure level, at the network network network firewall level, we firewall level, we firewall level, we control everything. control everything. control everything. Now you might Now you might Now you might say that it's possible say that it's possible say that it's possible that amazon.com will be that amazon.com will be that amazon.com will be hacked someday. How do you hacked someday. How do you hacked someday. How do you prevent prevent prevent malicious activity on malicious activity on malicious activity on these portals? I think these portals? I think these portals? I think we also try to we also try to we also try to run the created run the created run the created skills through skills through skills through some kind of some kind of security review agent security review agent security review agent before publishing before publishing before publishing them to browser.sh. This is one them to browser.sh. This is one them to browser.sh. This is one way to do way to do way to do it. We encourage it. We encourage it. We encourage our customers to have a our customers to have a our customers to have a good level of good level of good level of protection against prompt protection against prompt injections. If they injections. If they injections. If they provide it to provide it to provide it to consumers, consumers, consumers, for example, some of our for example, some of our for example, some of our customers want to customers want to customers want to automate something automate something automate something on Amazon. What if someone on Amazon. What if someone on Amazon. What if someone places places places an ad on Amazon that an ad on Amazon that an ad on Amazon that contains hidden contains hidden contains hidden instructions? And how do you instructions? And how do you instructions? And how do you prevent such

prevent such prevent such things? This is where we things? This is where we things? This is where we work with clients work with clients work with clients to build the to build the to build the right barriers before right barriers before right barriers before the instruction even the instruction even the instruction even enters the system. enters the system. So, yes, I would highlight So, yes, I would highlight three levels of protection here three levels of protection here . The first is to control . The first is to control . The first is to control the request even at the stage of the request even at the stage of the request even at the stage of its receipt in its receipt in its receipt in our system. At the our system. At the our system. At the skill level, we skill level, we skill level, we guarantee that our guarantee that our guarantee that our tool will not be tool will not be tool will not be fooled. And at the fooled. And at the fooled. And at the infrastructure level, we infrastructure level, we infrastructure level, we have network have network have network restrictions and restrictions and restrictions and action restrictions. You action restrictions. You action restrictions. You could even could even could even do something similar do something similar do something similar to what to what to what Codex or Cloud Code do. Codex or Cloud Code do. Codex or Cloud Code do. For example, "always For example, "always For example, "always allow", " allow", " allow once", allow once", allow once", or "allow or "allow any action". This is what the any action". This is what the any action". This is what the browser's browser's browser's safe mode offers out of safe mode offers out of safe mode offers out of the box. I know the box. I know the box. I know the browser is a the browser is a the browser is a crazy crazy crazy threat vector, if you threat vector, if you threat vector, if you think about it. He has think about it. He has think about it. He has access to everything on the access to everything on the access to everything on the internet, and we internet, and we internet, and we take that take that take that extremely seriously extremely seriously extremely seriously from a prompt from a prompt injection perspective. Perfectly. injection perspective. Perfectly. injection perspective. Perfectly. I'll wait another 10 seconds to see I'll wait another 10 seconds to see I'll wait another 10 seconds to see if there are any more if there are any more if there are any more questions. But it questions. But it questions. But it was fun. I was fun. I was fun. I really liked it.

really liked it. really liked it. I also really I also really I also really liked the liked the liked the questions. And yes, I'll questions. And yes, I'll questions. And yes, I'll put the QR code back on put the QR code back on put the QR code back on the screen if any of the screen if any of the screen if any of you want to you want to you want to scan it. Yes. Yes. scan it. Yes. Yes. Yes. Yes, that's right. Yes , I think we , I think we , I think we talked about this talked about this talked about this a while ago. You a while ago. You a while ago. You could do as could do as could do as I often I often I often did using did using did using Anthropic managed agents Anthropic managed agents . I was running cron . I was running cron jobs on jobs on jobs on managed agents that managed agents that managed agents that said, "Use said, "Use said, "Use this skill every day at 9:00 this skill every day at 9:00 this skill every day at 9:00 AM to perform AM to perform AM to perform this task." And this task." And this task." And if everything is successful, then if everything is successful, then if everything is successful, then good. If good. If good. If an error occurs, send me a an error occurs, send me a an error occurs, send me a notification that notification that notification that will trigger the will trigger the will trigger the recovery cycle. This is how recovery cycle. This is how recovery cycle. This is how I can implement I can implement I can implement checks and balances checks and balances checks and balances for for for website changes, website changes, website changes, anti-bot updates, etc.

anti-bot updates, etc. Surprisingly. I'll be here for a while Surprisingly. I'll be here for a while . I will also . I will also . I will also be at the Browserbase booth be at the Browserbase booth be at the Browserbase booth all days of the conference. all days of the conference. So if you're interested and So if you're interested and want to want to want to chat chat chat more, I'd be happy to. more, I'd be happy to. more, I'd be happy to. Thank you. Thank you.