Build with Perception Agents — Emile Baizel & Shruti Arora, Amazon AGI Lab

Hello everyone, thank you for Hello everyone, thank you for coming to our coming to our coming to our workshop on creating workshop on creating workshop on creating perception agents. perception agents. perception agents. My name is Emile My name is Emile My name is Emile Basel. I lead a Basel. I lead a Basel. I lead a team of team of team of solution architects at Amazon AGI Lab. solution architects at Amazon AGI Lab. solution architects at Amazon AGI Lab. Our team is Our team is Our team is focused on two focused on two focused on two things. The first is things. The first is things. The first is working with our working with our working with our very first customers very first customers who want to who want to who want to use our use our use our early models and early models and early models and products for products for production implementation. The second is closing the feedback loop is closing the feedback loop from customers, from customers, creating assessments for creating assessments for creating assessments for our research our research our research team to train team to train team to train models, and models, and models, and providing providing providing product and product and product and engineering feedback to these engineering feedback to these engineering feedback to these teams. With me teams. With me teams. With me today is one of our today is one of our today is one of our top top top solution architects, solution architects, solution architects, Shruti Arora, who Shruti Arora, who Shruti Arora, who will be leading this will be leading this will be leading this workshop. A little about what workshop. A little about what workshop. A little about what we will talk about today we will talk about today we will talk about today . I will provide a . I will provide a . I will provide a brief overview of Amazon AGI brief overview of Amazon AGI brief overview of Amazon AGI Lab and talk about the Lab and talk about the Lab and talk about the first product first product first product we released we released we released last year—Amazon Nova last year—Amazon Nova last year—Amazon Nova Act. Then we will move on Act. Then we will move on Act. Then we will move on to to to perception agents, which is the perception agents, which is the perception agents, which is the main topic of main topic of main topic of today's today's today's seminar, and after that seminar, and after that seminar, and after that we will start the we will start the we will start the practical part. practical part.

So, a little about So, a little about the laboratory. We are a the laboratory. We are a cutting-edge cutting-edge cutting-edge research and research and research and product team product team product team that combines that combines that combines the speed of a startup with the speed of a startup with the speed of a startup with the scale and the scale and the scale and resources of Amazon. We are resources of Amazon. We are resources of Amazon. We are focused on focused on focused on creating intelligent agents creating intelligent agents that can that can perceive, think, and perceive, think, and perceive, think, and act in different act in different act in different environments to environments to environments to perform real-world perform real-world perform real-world tasks. Our offices tasks. Our offices tasks. Our offices are located in San are located in San Francisco and Boston. Francisco and Boston. Francisco and Boston. We will have We will have We will have several more several more several more performances throughout the week. We have a performances throughout the week. We have a performances throughout the week. We have a booth at the show, booth at the show, booth at the show, so stop by to so stop by to so stop by to get souvenirs, get souvenirs, get souvenirs, meet the meet the meet the team, and team, and team, and listen to performances. I listen to performances. I listen to performances. I want to share want to share want to share our approach and how our approach and how our approach and how we perceive we perceive we perceive agents. In three agents. In three agents. In three aspects. The first is aspects. The first is aspects. The first is cooperation. We cooperation. We cooperation. We strive to create strive to create strive to create agents as agents as agents as full full full partners, not partners, not partners, not just as a "black just as a "black just as a "black box." When we box." When we box." When we say “partners,” we say “partners,” we say “partners,” we want the agent want the agent want the agent to understand our goal and to understand our goal and to understand our goal and objectives and be able to objectives and be able to objectives and be able to share their share their share their thoughts on ways thoughts on ways thoughts on ways to achieve that goal. to achieve that goal.

And all this is based on And all this is based on real client real client real client cases. We work very closely cases. We work very closely cases. We work very closely with a with a with a large number of large number of large number of clients to clients to clients to substantiate all the substantiate all the substantiate all the work we work we work we do with agents. do with agents. do with agents. The second aspect is The second aspect is The second aspect is perception. Agents perception. Agents perception. Agents must understand must understand must understand their environment. their environment. their environment. Since it is a Since it is a Since it is a perception-based agent, perception-based agent, perception-based agent, its environment its environment its environment spans various spans various spans various devices. Last devices. Last devices. Last year we started with year we started with year we started with browsers, but you browsers, but you browsers, but you can imagine can imagine any visual any visual any visual screen as an environment for screen as an environment for our agent to do intelligent work. our agent to do intelligent work. So, mobile So, mobile So, mobile devices, as well as devices, as well as computer use—that's computer use—that's another aspect. And the third another aspect. And the third is reliability. We is reliability. We is reliability. We have a great testimonial from have a great testimonial from have a great testimonial from a client: "If my a client: "If my a client: "If my agent works 50% of the time, agent works 50% of the time, agent works 50% of the time, he doesn't work at all." he doesn't work at all." he doesn't work at all." This is because you This is because you This is because you will either not will either not will either not run it at all, or run it at all, or run it at all, or if you do, if you do, if you do, you will spend a lot of time you will spend a lot of time you will spend a lot of time debugging, debugging, debugging, negating all negating all negating all the benefits of the benefits of the benefits of using this using this using this agent. Therefore, we agent. Therefore, we agent. Therefore, we spend a lot of spend a lot of spend a lot of time building and time building and time building and training our training our training our models in models in models in simulated simulated reinforcement learning environments. And reinforcement learning environments. And again, a lot of again, a lot of again, a lot of it is based on it is based on it is based on real-world real-world real-world use cases that we use cases that we use cases that we get not only from get not only from get not only from external customers, but external customers, but external customers, but also from many of also from many of Amazon's internal teams doing the Amazon's internal teams doing the smart smart smart work. Steeply. So, with work. Steeply. So, with work. Steeply. So, with that in mind, I want that in mind, I want that in mind, I want to talk about the to talk about the to talk about the first product first product first product we launched we launched we launched last year. Amazon Nova last year. Amazon Nova last year. Amazon Nova Act is a premium Act is a premium Act is a premium AWS service that AWS service that AWS service that enables developers enables developers enables developers to build and to build and to build and run robust run robust run robust browser-based

browser-based browser-based workflows at scale. workflows at scale. Using Using natural language, you natural language, you natural language, you can control the can control the can control the browser: browser: browser: fill in data, fill in data, fill in data, navigate, navigate, navigate, search for information, search for information, search for information, fill out forms, and fill out forms, and fill out forms, and involve a human if involve a human if involve a human if you need to approve an you need to approve an you need to approve an expense report, expense report, expense report, if you encounter if you encounter if you encounter difficulties, or in other difficulties, or in other difficulties, or in other similar situations. similar situations. There are four main use cases for use cases for which customers which customers which customers have already started using have already started using have already started using this tool. this tool. this tool. The first is The first is The first is filling out forms. So filling out forms. So , if you need to , if you need to , if you need to do this at scale do this at scale do this at scale across many different across many different across many different sites and you don't want sites and you don't want sites and you don't want to write a separate to write a separate to write a separate prompt or prompt or prompt or deterministic deterministic deterministic script for each one script for each one script for each one , you can , you can , you can use Nova Act. use Nova Act. Data search and extraction if you want to Data search and extraction if you want to conduct conduct conduct research, collect research, collect research, collect data from multiple data from multiple data from multiple sites, make sites, make sites, make purchases, and conduct purchases, and conduct purchases, and conduct testing. Now testing. Now testing. Now a little more detail about what a little more detail about what a little more detail about what we launched. we launched. we launched. Let me Let me Let me briefly walk through briefly walk through briefly walk through all of this. So, this is a all of this. So, this is a all of this. So, this is a first-class first-class first-class AWS service. If you go into the AWS service. If you go into the AWS service. If you go into the AWS console and type AWS console and type AWS console and type Nova Act, you will Nova Act, you will Nova Act, you will see it.

see it. You can You can view the entire view the entire view the entire history of actions directly in the console. This is a very history of actions directly in the console. This is a very history of actions directly in the console. This is a very convenient way convenient way convenient way to see everything to see everything to see everything your agent has done: your agent has done: your agent has done: a frame around the a frame around the a frame around the elements it is elements it is elements it is analyzing and analyzing and analyzing and trying to influence trying to influence . And behind the scenes, . And behind the scenes, . And behind the scenes, our our our special special special computer computer computer usage model works, usage model works, usage model works, performing all the performing all the performing all the necessary calculations necessary calculations . This year, we . This year, we . This year, we launched the MCP server, launched the MCP server, launched the MCP server, as well as the corresponding as well as the corresponding as well as the corresponding skill that you will skill that you will skill that you will use in the use in the use in the workshop, so that your workshop, so that your workshop, so that your agents can easily agents can easily agents can easily work with Nova Act. work with Nova Act. work with Nova Act. Additionally, we have Additionally, we have Additionally, we have support for support for support for human participation and human participation and human participation and the use of the use of the use of tools. And tools. And tools. And we also have a gaming we also have a gaming we also have a gaming environment. So, environment. So, environment. So, if you go to the if you go to the if you go to the Nova Nova Nova Act homepage, you can Act homepage, you can Act homepage, you can see Nova Act in action see Nova Act in action see Nova Act in action without writing any code. without writing any code. Looking to the future, Looking to the future, the topic of today's the topic of today's the topic of today's seminar is seminar is seminar is perception agents. Why do we perception agents. Why do we perception agents. Why do we think there is a think there is a think there is a need for need for need for perceptual agents? For those perceptual agents? For those perceptual agents? For those of you who are not of you who are not of you who are not familiar with the term " familiar with the term " computer usage," it's computer usage," it's essentially a way for an essentially a way for an essentially a way for an agent to perform agent to perform agent to perform actions on a actions on a actions on a computer screen, right? computer screen, right? computer screen, right? For example, you see a For example, you see a For example, you see a blue button in the right blue button in the right blue button in the right corner and you tell your corner and you tell your corner and you tell your agent, "Hey, click agent, "Hey, click agent, "Hey, click this blue button." This is this blue button." This is this blue button." This is very important and very very important and very very important and very powerful. This is critically powerful. This is critically powerful. This is critically important for important for important for automating automating automating workflows. But workflows. But workflows. But we believe that there is we believe that there is we believe that there is another side—the agent another side—the agent another side—the agent should understand what exactly should understand what exactly should understand what exactly he sees on the screen, he sees on the screen, he sees on the screen, and not just blindly and not just blindly and not just blindly click on click on click on elements. We want the elements. We want the elements. We want the agent to be able agent to be able agent to be able to understand the screen just like a to understand the screen just like a human does, right?

human does, right? If I show someone an If I show someone an If I show someone an email client from the email client from the email client from the late 90s, like late 90s, like late 90s, like Yahoo or Hotmail, that they've Yahoo or Hotmail, that they've Yahoo or Hotmail, that they've never seen, never seen, never seen, they'll likely they'll likely they'll likely figure out how to figure out how to figure out how to send an send an send an email email email without even without even without even using the using the using the service before. service before. Because we have the Because we have the collective understanding collective understanding gained from gained from gained from thousands of hours of working with thousands of hours of working with thousands of hours of working with these screens, we these screens, we these screens, we intuitively understand intuitively understand how how how to use them, and that's what to use them, and that's what to use them, and that's what we strive for we strive for we strive for with our agents. with our agents. Second, these agents don't Second, these agents don't Second, these agents don't need APIs to need APIs to need APIs to work, right? work, right? work, right? If you think about If you think about If you think about intelligent intelligent intelligent work, many people work, many people work, many people use use use application screens that don't application screens that don't application screens that don't have APIs, which is why have APIs, which is why have APIs, which is why this work is still this work is still this work is still done by humans. We done by humans. We done by humans. We want these agents to be able to want these agents to be able to want these agents to be able to do the do the do the same work across same work across same work across different sites. We different sites. We different sites. We want the models want the models want the models to see the screen the same way to see the screen the same way we do, because it makes we do, because it makes collaboration much easier, right? collaboration much easier, right? So, instead of So, instead of So, instead of writing long writing long writing long texts about what the texts about what the texts about what the agent should do on the agent should do on the agent should do on the screen and describing screen and describing screen and describing that screen, you can that screen, you can that screen, you can simply point, simply point, simply point, click, and click, and click, and describe the desired describe the desired describe the desired changes, and that's what we'll do in changes, and that's what we'll do in changes, and that's what we'll do in today's today's today's workshop. You workshop. You workshop. You just click on just click on just click on a few elements and a few elements and a few elements and say, "Hey, I want say, "Hey, I want say, "Hey, I want to change the background to to change the background to to change the background to red." And that's it.

red." And that's it. red." And that's it. After that, you simply After that, you simply After that, you simply give the agent the give the agent the give the agent the task to do it. task to do it. To start, we To start, we have opened the source have opened the source have opened the source code of two basic code of two basic code of two basic tools for tools for tools for perception agents, perception agents, perception agents, which you will which you will use in detail in use in detail in today's workshop. today's workshop. The first is a visual The first is a visual annotation. It annotation. It annotation. It allows you to allows you to point, click, point, click, and simply describe and simply describe and simply describe the changes you want to the changes you want to the changes you want to make on any web page. When you're make on any web page. When you're make on any web page. When you're done, you done, you done, you click "export" click "export" and the system creates a JSON and the system creates a JSON file with all your file with all your file with all your changes and a request changes and a request changes and a request that you can that you can that you can pass to your pass to your pass to your coder agent along coder agent along coder agent along with that file to with that file to with that file to say, "Hey, say, "Hey, say, "Hey, make these changes." make these changes." make these changes." It is important to note that It is important to note that It is important to note that this has nothing to do with this has nothing to do with inference. This is a purely inference. This is a purely deterministic deterministic deterministic process. It just process. It just process. It just outputs JSON, not outputs JSON, not outputs JSON, not tied to any tied to any tied to any model or anything model or anything model or anything like that. The second like that. The second like that. The second part, the second part, the second part, the second basic tool basic tool basic tool that we launched, is that we launched, is that we launched, is visual visual visual verification. If you verification. If you verification. If you think about your think about your think about your website, you have certain CSS website, you have certain CSS styles and designs styles and designs styles and designs that your site must that your site must that your site must adhere to. You adhere to. You adhere to. You have different have different have different workflows that workflows that workflows that your your your site needs to support. Maybe the site needs to support. Maybe the checkout process, login, checkout process, login, or something else.

or something else. or something else. You can describe them You can describe them You can describe them in Gherkin files. So you in Gherkin files. So you in Gherkin files. So you can can can use use use visual visual visual verification—this verification—this verification—this skill—to run skill—to run skill—to run your your coder agent against these coder agent against these coder agent against these design artifacts design artifacts design artifacts and Gherkin files to and Gherkin files to and Gherkin files to make sure your make sure your make sure your site complies with them. site complies with them. site complies with them. Again, this is what Again, this is what Again, this is what you will you will you will use in the use in the use in the workshop today. And once again workshop today. And once again workshop today. And once again , all of this is in the , all of this is in the , all of this is in the public domain, public domain, public domain, available available available today. And yes, that's exactly what today. And yes, that's exactly what today. And yes, that's exactly what we'll be doing in the we'll be doing in the we'll be doing in the workshop. I think workshop. I think workshop. I think I've said I've said I've said enough for now, and I'll enough for now, and I'll enough for now, and I'll hand over to Shruti, hand over to Shruti, hand over to Shruti, who will start our who will start our who will start our workshop. Can workshop. Can you just you just click on it? click on it? click on it? Thank you. Thank you. OK. Can you OK. Can you hear me well? hear me well? hear me well? Yes. Yes. Yes. Good. Good. Good. Welcome. Thanks to Emil Welcome. Thanks to Emil Welcome. Thanks to Emil for this great for this great for this great introduction to our introduction to our introduction to our Amazon AGI Labs and the Amazon AGI Labs and the Amazon AGI Labs and the products we products we products we launched earlier launched earlier launched earlier this May. this May.

So, as he said, So, as he said, we'll have a hands-on we'll have a hands-on we'll have a hands-on workshop workshop workshop where we'll work with what where we'll work with what where we'll work with what we released: we released: we released: the interface, the the interface, the the interface, the visual visual visual annotator, and the annotator, and the annotator, and the visual visual visual validation. I have a validation. I have a validation. I have a short demonstration short demonstration , but before we , but before we , but before we get to it, I get to it, I get to it, I want you all to want you all to want you all to try out the try out the try out the role. Just, just role. Just, just role. Just, just bear with me. bear with me. Imagine you are a Imagine you are a front-end engineer and you front-end engineer and you front-end engineer and you are building a web application are building a web application . You have a team of . You have a team of . You have a team of designers and you designers and you designers and you draw inspiration from draw inspiration from draw inspiration from them. They have them. They have them. They have Figma files that you Figma files that you Figma files that you work from. You are also work from. You are also work from. You are also collaborating with an collaborating with an collaborating with an adjacent team adjacent team adjacent team that wants to place that wants to place that wants to place some of their some of their some of their information on the information on the information on the web page you are web page you are web page you are building. So if you building. So if you building. So if you think about it, you take a think about it, you take a think about it, you take a design and probably design and probably design and probably spend spend spend four to five four to five four to five weeks weeks weeks implementing it, implementing it, implementing it, writing the code, and writing the code, and writing the code, and building that site.

building that site. Now, in the third or Now, in the third or fourth week of fourth week of fourth week of development, you suddenly development, you suddenly development, you suddenly think, "Oh, maybe it's think, "Oh, maybe it's think, "Oh, maybe it's worth going back and worth going back and worth going back and checking if everything checking if everything checking if everything looks good, if looks good, if looks good, if everything is okay with the rest of the everything is okay with the rest of the everything is okay with the rest of the team." You guys are team." You guys are team." You guys are always always always collaborating, collaborating, collaborating, right? So you right? So you right? So you check if what check if what check if what you have done is exactly what you have done is exactly what you have done is exactly what you want to you want to you want to release. You release. You release. You submit your code, submit your code, submit your code, share a link share a link share a link to GitHub or GitLab, or wherever to GitHub or GitLab, or wherever to GitHub or GitLab, or wherever you have your code, you have your code, you have your code, they run they run they run a web server locally, a web server locally, a web server locally, open the site, and open the site, and open the site, and say, “Oh, something’s say, “Oh, something’s say, “Oh, something’s wrong here. You know, wrong here. You know, wrong here. You know, the headline probably the headline probably the headline probably shouldn't be shouldn't be shouldn't be red. "I would like to red. "I would like to red. "I would like to take it all back." take it all back." And they provide this And they provide this feedback. feedback. feedback. They will probably write a They will probably write a They will probably write a message in Slack message in Slack message in Slack or in a shared or in a shared or in a shared document, for example. document, for example.

And when you get And when you get these reviews about your these reviews about your these reviews about your site, you go through site, you go through site, you go through them one by one. them one by one. them one by one. You disagree with something You disagree with something You disagree with something , , , agree with something, agree with something, agree with something, find a compromise find a compromise find a compromise and decide what and decide what and decide what changes should be made. changes should be made. changes should be made. You have to You have to You have to go through each of go through each of go through each of them, one by one, and them, one by one, and them, one by one, and start making changes. start making changes. Now, with coding agents, what agents, what agents, what used to take used to take used to take a week can a week can a week can be reduced to be reduced to be reduced to a few days. Now a few days. Now a few days. Now imagine if you could imagine if you could imagine if you could pass their pass their pass their feedback directly to the coding feedback directly to the coding agent without your agent without your agent without your supervision, and when the changes supervision, and when the changes supervision, and when the changes were ready, just were ready, just were ready, just step in and say, step in and say, step in and say, “Oh, it looks “Oh, it looks “Oh, it looks good now.” All changes good now.” All changes good now.” All changes have been made. "How does this have been made. "How does this have been made. "How does this cut cut cut time? Maybe a time? Maybe a time? Maybe a few hours. You few hours. You few hours. You just collect just collect just collect feedback, send feedback, send feedback, send it to your agent, it to your agent, it to your agent, go get a coffee, go get a coffee, go get a coffee, come back—and all come back—and all come back—and all the changes are already merged. the changes are already merged. This is exactly what This is exactly what Emil mentioned. That Emil mentioned. That Emil mentioned. That we have a we have a we have a Chrome extension that we call the Chrome extension that we call the Chrome extension that we call the visual visual visual annotator. There are three annotator. There are three annotator. There are three ways to add ways to add ways to add annotations. As you can see, annotations. As you can see, annotations. As you can see, you can either draw on the you can either draw on the you can either draw on the page, or you can select page, or you can select page, or you can select an element, or you can an element, or you can an element, or you can put a mark put a mark put a mark like in Google Maps and like in Google Maps and like in Google Maps and add your add your add your comments. When you have comments. When you have the annotations ready, you can the annotations ready, you can upload them and upload them and upload them and send them to the team send them to the team send them to the team that needs to make the changes.

that needs to make the changes. Or, even if you're Or, even if you're making those changes yourself making those changes yourself , we provide an , we provide an , we provide an AI-friendly query that you AI-friendly query that you AI-friendly query that you can send to can send to can send to your agent, and it will your agent, and it will your agent, and it will automatically make them, automatically make them, automatically make them, you just have to you just have to you just have to give it an instruction give it an instruction give it an instruction to change the codebase. to change the codebase. So that was the So that was the visual annotator. visual annotator. The next, um... Oh. The next, um... Oh. The next, um... The next, um... primitive that we primitive that we launched—is the launched—is the launched—is the visual visual visual inspection skill. And when I inspection skill. And when I inspection skill. And when I say "skill," it's say "skill," it's say "skill," it's really a skill that you really a skill that you really a skill that you can install in can install in can install in Claude or Keynote. And you'll Claude or Keynote. And you'll Claude or Keynote. And you'll see these see these see these checks, as I checks, as I checks, as I mentioned, are the design files mentioned, are the design files that you're going to be that you're going to be doing your doing your doing your visual checks against. visual checks against. For example, do the design For example, do the design colors look good colors look good colors look good ? Is the ? Is the font size right? Is the font size right? Is the user journey right? But user journey right? But user journey right? But one question that one question that one question that comes to mind is how comes to mind is how comes to mind is how do you create these do you create these do you create these design specs design specs design specs and these paths? How, how, how are and these paths? How, how, how are and these paths? How, how, how are you going to you going to you going to build that? Don't build that? Don't build that? Don't worry, we have a worry, we have a worry, we have a solution for that too.

solution for that too. When you run When you run the check for the first time, the the check for the first time, the the check for the first time, the UI UI Verification tool will do a Verification tool will do a Verification tool will do a deep surface deep surface deep surface scan of your scan of your scan of your website to website to website to check the links check the links check the links and the content; based on and the content; based on and the content; based on that, it will create that, it will create that, it will create user journeys in user journeys in user journeys in Gherkin syntax that the UI Verification tool Gherkin syntax that the UI Verification tool will will will use in the use in the use in the next next next check. So check. So check. So let me let me let me run run run a demo. You a demo. You a demo. You can see that you're can see that you're can see that you're just invoking just invoking just invoking the tool. It'll the tool. It'll the tool. It'll look at the design look at the design documents, documents, documents, the specs, and the the specs, and the the specs, and the user journeys. It'll user journeys. It'll walk through them. It'll open a browser, it'll It'll open a browser, it'll walk through the walk through the walk through the design design design user journeys, and then it'll user journeys, and then it'll user journeys, and then it'll generate a report. So generate a report. So generate a report. So those are two tools those are two tools those are two tools that you'll be that you'll be that you'll be working with in this working with in this working with in this workshop. I'll go workshop. I'll go workshop. I'll go into how into how into how our our our workshop is organized at this workshop is organized at this workshop is organized at this point. First, as I point. First, as I point. First, as I mentioned, these are two mentioned, these are two mentioned, these are two hands-on activities.

hands-on activities. hands-on activities. The second thing we're The second thing we're The second thing we're trying to do trying to do is show that these are is show that these are is show that these are just two basic just two basic just two basic elements. Perception elements. Perception is not just these two is not just these two is not just these two basic elements. How basic elements. How basic elements. How would you combine these two would you combine these two would you combine these two elements for a elements for a elements for a specific specific specific use case? The use case? The use case? The workshop workshop workshop has a has a has a scenario. We're scenario. We're scenario. We're trying to make trying to make trying to make it fun, so it fun, so it fun, so try it yourself, it's try it yourself, it's try it yourself, it's like a game day like a game day . You'll see some . You'll see some . You'll see some clues to clues to clues to solve the problem: solve the problem: solve the problem: what it is and what it is and what it is and what the ultimate goal is. what the ultimate goal is. what the ultimate goal is. So, it's all laid out in the So, it's all laid out in the So, it's all laid out in the workshop materials. workshop materials. workshop materials. This is the link to GitHub. This is the link to GitHub. This is the link to GitHub. We'll share it with We'll share it with We'll share it with you in a little bit. you in a little bit. you in a little bit. So, how are you So, how are you So, how are you going to combine going to combine going to combine these two things? Next up these two things? Next up is our is our is our prize sponsors. At the end, we're prize sponsors. At the end, we're prize sponsors. At the end, we're going to have some going to have some going to have some contests. Thank you to contests. Thank you to contests. Thank you to Team B for Team B for Team B for sponsoring the prizes sponsoring the prizes sponsoring the prizes for our event.

for our event. for our event. We'll have a way for you to We'll have a way for you to use other use other interesting methods to interesting methods to interesting methods to add annotations add annotations add annotations or feedback on or feedback on or feedback on the web page and its the web page and its the web page and its design. Team B design. Team B design. Team B will show you how to will show you how to will show you how to do this, not do this, not do this, not just by just by just by printing feedback printing feedback . Finally, before that, . Finally, before that, . Finally, before that, after the after the after the workshop, we have workshop, we have workshop, we have three different three different three different bonus task options. bonus task options. bonus task options. You will have enough You will have enough You will have enough time to try time to try time to try these tools in these tools in these tools in your own case, your own case, your own case, maybe combine it maybe combine it maybe combine it with B and see how it with B and see how it with B and see how it works with works with any other type of any other type of any other type of feedback from your feedback from your feedback from your team. Or, if you team. Or, if you team. Or, if you think it's think it's think it's really cool. I want to really cool. I want to really cool. I want to create something that I create something that I create something that I can show my can show my can show my team on Monday, team on Monday, team on Monday, next Monday next Monday . So go ahead and if you're . So go ahead and if you're . So go ahead and if you're brave enough, brave enough, go on stage, go on stage, go on stage, show your show your show your case, and you'll get the case, and you'll get the case, and you'll get the B device. Finally, B device. Finally, B device. Finally, we'll do we'll do we'll do a little fun a little fun a little fun exercise: once you've exercise: once you've exercise: once you've mastered these two mastered these two mastered these two primitives, you can primitives, you can primitives, you can think of tons of think of tons of think of tons of ways to use them in your ways to use them in your ways to use them in your daily life daily life daily life . You'll . You'll . You'll submit your submit your submit your ideas in the format of, "Oh, it would be

ideas in the format of, "Oh, it would be ideas in the format of, "Oh, it would be cool cool cool to use this like this, to use this like this, to use this like this, " and we'll have " and we'll have " and we'll have a vote. The a vote. The a vote. The three best ideas three best ideas three best ideas will get device B, will get device B, will get device B, and one more—the and one more—the and one more—the audience award. audience award. So let's move on to the So let's move on to the workshop. Here's workshop. Here's workshop. Here's the link to the workshop. the link to the workshop. You can type it You can type it into a search, it's into a search, it's into a search, it's available on GitHub, available on GitHub, available on GitHub, so just type " so just type " so just type " perception agents workshop" and you'll perception agents workshop" and you'll perception agents workshop" and you'll see the link see the link see the link first. For the workshop, first. For the workshop, first. For the workshop, you'll also you'll also you'll also need a Novact API key need a Novact API key need a Novact API key . The . The . The workshop materials have the first three workshop materials have the first three workshop materials have the first three letters hidden. letters hidden. They're displayed here They're displayed here on the screen. You can on the screen. You can on the screen. You can just copy those just copy those just copy those three letters—ACE—and your three letters—ACE—and your three letters—ACE—and your Novact API key is ready to Novact API key is ready to Novact API key is ready to go. So, go. So, go. So, let me let me stop this. In stop this. In the workshop, the readme file the workshop, the readme file the workshop, the readme file will take you will take you will take you directly to the directly to the directly to the workshop.md file. Here's workshop.md file. Here's workshop.md file. Here's the script I was the script I was the script I was talking about. I'll read talking about. I'll read talking about. I'll read it right from here. You're a it right from here. You're a it right from here. You're a front-end developer front-end developer front-end developer getting ready to getting ready to getting ready to launch a landing page for a launch a landing page for a launch a landing page for a podcast. Yesterday you

podcast. Yesterday you podcast. Yesterday you showed the page to showed the page to showed the page to a few colleagues, and a few colleagues, and a few colleagues, and they gave feedback. The they gave feedback. The hero title hero title seems too big. The seems too big. The seems too big. The accent color doesn't accent color doesn't accent color doesn't stand out like it should stand out like it should stand out like it should . Some of the buttons are . Some of the buttons are . Some of the buttons are too big, and there are too big, and there are too big, and there are a few little things that a few little things that a few little things that look a little look a little look a little inappropriate. You agree inappropriate. You agree inappropriate. You agree with these comments, with these comments, with these comments, so today you're here so today you're here so today you're here with renewed with renewed with renewed vigor and are ready vigor and are ready vigor and are ready to make changes. So what do you to make changes. So what do you to make changes. So what do you have at your have at your have at your disposal? The disposal? The Novact Novact Chrome extension, the UI verification skill, the Chrome extension, the UI verification skill, the Chrome extension, the UI verification skill, the Novact SDK, and the Novact MCP server Novact SDK, and the Novact MCP server . These are tools that . These are tools that . These are tools that you can you can you can use to use to use to implement the solution implement the solution implement the solution and essentially and essentially and essentially build this entire build this entire build this entire cycle yourself. You just cycle yourself. You just cycle yourself. You just create annotations, create annotations, create annotations, click a button in the click a button in the click a button in the Chrome extension, and all Chrome extension, and all Chrome extension, and all the changes are applied the changes are applied the changes are applied automatically. After automatically. After automatically. After that, the that, the UI verification runs, and you UI verification runs, and you get a report. Here's the get a report. Here's the get a report. Here's the task, but there are task, but there are task, but there are a few prerequisites. You'll a few prerequisites. You'll a few prerequisites. You'll need macOS or need macOS or need macOS or Linux, because the Linux, because the Novact MCP server runs Novact MCP server runs Novact MCP server runs on those on those on those operating systems operating systems . You should have . You should have . You should have Node.js installed.

Node.js installed. Node.js installed. If not, you can If not, you can If not, you can install it. NPM, install it. NPM, install it. NPM, Python, and the Chrome browser Python, and the Chrome browser Python, and the Chrome browser for testing, for testing, for testing, because it's a because it's a because it's a Chrome extension, and an Chrome extension, and an Chrome extension, and an AI agent for AI agent for AI agent for coding. So if coding. So if coding. So if you have Claude, that's you have Claude, that's you have Claude, that's great. great. great. We also have integration with We also have integration with We also have integration with Kiro. So if you Kiro. So if you Kiro. So if you don't have any AI don't have any AI assistant for assistant for assistant for coding, you can coding, you can coding, you can install Kiro and install Kiro and install Kiro and start start start experimenting. And experimenting. And experimenting. And finally, we have UV, or finally, we have UV, or finally, we have UV, or rather UVX. This is actually rather UVX. This is actually rather UVX. This is actually for installation. for installation. for installation. If you don't want to If you don't want to If you don't want to install UV or install UV or install UV or UVX, please UVX, please UVX, please raise your hand and we'll raise your hand and we'll raise your hand and we'll help you help you help you install everything via install everything via install everything via pip. Um. So that's it. pip. Um. So that's it. The workshop The workshop is designed to is designed to take about 5 to 7 minutes to set up. Then you can Then you can start learning start learning start learning these skills. This will give you a these skills. This will give you a these skills. This will give you a good understanding of good understanding of good understanding of how to use how to use how to use Chrome extensions and the UI Chrome extensions and the UI Chrome extensions and the UI testing skill testing skill testing skill . That's what it's all about . That's what it's all about . That's what it's all about .

. You have about 15 You have about 15 minutes to complete the exercises if you minutes to complete the exercises if you minutes to complete the exercises if you want to go that want to go that want to go that route, which is to do it route, which is to do it route, which is to do it step by step. I'm going to step by step. I'm going to scroll down quickly now. Okay. scroll down quickly now. Okay. So choose your So choose your So choose your path. This workshop path. This workshop path. This workshop is designed on two is designed on two is designed on two levels. Remember, I levels. Remember, I levels. Remember, I said I want you to have said I want you to have said I want you to have fun fun . So, two levels: if . So, two levels: if . So, two levels: if you want to challenge you want to challenge you want to challenge yourself, choose the yourself, choose the yourself, choose the middle path. If middle path. If middle path. If you feel like you just you feel like you just you feel like you just want to follow want to follow want to follow the instructions and go through the the instructions and go through the the instructions and go through the workshop. You'll workshop. You'll workshop. You'll happy with happy with happy with the result. So, the result. So, the result. So, yes, we have an easy yes, we have an easy yes, we have an easy mode where you mode where you mode where you get clear get clear get clear instructions on what to do instructions on what to do , and at the end of the , and at the end of the , and at the end of the hands-on part, hands-on part, hands-on part, you connect these two you connect these two you connect these two primitives. primitives. primitives. We're giving you 15 We're giving you 15 We're giving you 15 minutes for the easy way, and minutes for the easy way, and 30 to 35 minutes for the medium way. 30 to 35 minutes for the medium way. After the After the allotted 15 allotted 15 allotted 15 minutes, I'll come up and minutes, I'll come up and minutes, I'll come up and show you exactly where you show you exactly where you show you exactly where you need to add the code need to add the code need to add the code to connect these two to connect these two to connect these two primitives and see primitives and see primitives and see how it all comes to life in action how it all comes to life in action . And we also have a bonus . And we also have a bonus . And we also have a bonus task. If task. If task. If you finish early, you finish early, you finish early, you can start the you can start the you can start the bonus task. But bonus task. But bonus task. But before that, before that, Giovanni from Team B will join us Giovanni from Team B will join us to give a to give a to give a presentation. After.

presentation. After. presentation. After. Yes. Sure, no Yes. Sure, no Yes. Sure, no problem. problem. problem. I noticed that while I was I noticed that while I was I noticed that while I was installing these... installing these... Umm. Umm. What are these What are these Nova Act UI Checker skills? It Nova Act UI Checker skills? It said that there are some said that there are some said that there are some default clients default clients default clients that are that are that are always supported always supported , including Codex and Cloud , including Codex and Cloud , including Codex and Cloud Code. But there are also these Code. But there are also these Code. But there are also these additional skills below, and additional skills below, and additional skills below, and one of them is one of them is one of them is Cloud Code as well. I wonder if are Cloud Code as well. I wonder if are Cloud Code as well. I wonder if are they two different things, or are they two different things, or are they two different things, or are they the same thing? The they the same thing? The they the same thing? The fact that there are two Cloud Codes fact that there are two Cloud Codes fact that there are two Cloud Codes Yes, there are two skills. Yes, there are two skills. Yes, there are two skills. One is the Nova Act skill One is the Nova Act skill . One is the . One is the Nova Act UI validation skill. So you Nova Act UI validation skill. So you probably see both probably see both . Yeah. So I'm going to give you . Yeah. So I'm going to give you . Yeah. So I'm going to give you time so I can open the time so I can open the time so I can open the slides. I'm going to ask slides. I'm going to ask slides. I'm going to ask Emil to come over here. Emil to come over here. Emil to come over here. The slides are on the GitHub The slides are on the GitHub repository so repository so repository so you have the API you have the API key handy in case you key handy in case you key handy in case you need to change it.

need to change it. need to change it. If you have If you have If you have a question, feel free to a question, feel free to a question, feel free to raise your hand. We have a raise your hand. We have a great team behind our work. There great team behind our work. There they are, you see, they are, you see, they are, you see, waving. So waving. So waving. So reach out to reach out to any of us and we'll any of us and we'll any of us and we'll come over and help. Good come over and help. Good luck. Okay, luck. Okay, I'm just going to try to I'm just going to try to I'm just going to try to walk you through the solution so that walk you through the solution so that walk you through the solution so that Okay. Okay. Okay. Okay. So here's the step-by-step So here's the step-by-step guide. You see guide. You see we have a we have a we have a Chrome extension. Chrome extension. Chrome extension. We have your We have your We have your agent running, your AI agent. agent running, your AI agent. If you want If you want your agent to your agent to automatically automatically apply changes from the apply changes from the apply changes from the extension, you extension, you extension, you need a bridge because there's need a bridge because there's need a bridge because there's no bridge between them. How no bridge between them. How no bridge between them. How can an extension can an extension can an extension send a request to send a request to send a request to the agent for the agent for the agent for programming? To do programming? To do programming? To do this, we created what we this, we created what we this, we created what we call an agent call an agent call an agent bridge, which bridge, which bridge, which automatically takes automatically takes automatically takes the annotations and passes the annotations and passes the annotations and passes them to the agent. That's when them to the agent. That's when them to the agent. That's when it will start it will start it will start applying those applying those applying those changes and running the changes and running the UI check skill. So

UI check skill. So actually, actually, actually, we don't have an we don't have an apply changes button in the extension in its apply changes button in the extension in its apply changes button in the extension in its native mode, but native mode, but native mode, but for this workshop, we've for this workshop, we've for this workshop, we've added this added this apply changes button to the apply changes button to the apply changes button to the Chrome extension. I'll Chrome extension. I'll Chrome extension. I'll show you that in a little bit show you that in a little bit show you that in a little bit . So to get all of . So to get all of . So to get all of this working, you this working, you this working, you need to make need to make need to make changes to two files. The changes to two files. The first file is the first file is the manifest.json file in the manifest.json file in the manifest.json file in the Chrome extension, where you add Chrome extension, where you add Chrome extension, where you add this line for the this line for the this line for the host permissions to host permissions to host permissions to start accepting start accepting start accepting requests from the requests from the requests from the localhost. And then you localhost. And then you localhost. And then you add this snippet of add this snippet of add this snippet of code. It's a huge code. It's a huge code. It's a huge chunk of code. Um, and you chunk of code. Um, and you chunk of code. Um, and you can, what this can, what this can, what this part does in the part does in the part does in the extension is it extension is it extension is it creates an creates an creates an apply changes button, apply changes button, apply changes button, binds it to binds it to binds it to the extension, the extension, the extension, makes sure that the makes sure that the makes sure that the UI check UI check UI check is run is run is run using the Nova Act SDK using the Nova Act SDK using the Nova Act SDK instead of the skill, and starts instead of the skill, and starts instead of the skill, and starts generating a report generating a report generating a report that you can click on that you can click on that you can click on and view.

and view. Um, so this snippet of Um, so this snippet of code goes into this file, code goes into this file, code goes into this file, which is, um, let's see which is, um, let's see , content.js in the extension, , content.js in the extension, , content.js in the extension, and it needs to and it needs to and it needs to be added be added be added about 261 lines in. Um, between about 261 lines in. Um, between about 261 lines in. Um, between this append child export button, this append child export button, this append child export button, right before it, and I right before it, and I right before it, and I think I have a hash, um think I have a hash, um , there's an " , there's an " add code here" comment in add code here" comment in add code here" comment in that file, and that's where that file, and that's where that file, and that's where you're going to put this you're going to put this you're going to put this code. So once code. So once code. So once you run this now you run this now you run this now you'll see this you'll see this you'll see this "apply changes" button "apply changes" button "apply changes" button here in the extension. Um, here in the extension. Um, here in the extension. Um, I think, um, Emil I think, um, Emil I think, um, Emil was testing this, so he was testing this, so he was testing this, so he changed it to something changed it to something changed it to something like like like World Cup, why World Cup, why World Cup, why not? Um, so I'm not? Um, so I'm not? Um, so I'm going to pick going to pick going to pick one of those and one of those and one of those and refresh the extension again.

refresh the extension again. Let's see. Annotate the current Annotate the current page. Um, I'm page. Um, I'm page. Um, I'm going to try changing the going to try changing the going to try changing the publish date. Change it publish date. Change it to today, to today, June 29th. Okay. And June 29th. Okay. And June 29th. Okay. And save it. And this save it. And this save it. And this extension. extension. Apply changes. Apply changes. So in the background So in the background So in the background it will open it will open it will open your your your programming agent. It's programming agent. It's programming agent. It's been tested on been tested on been tested on Cloud Code and Kiro, as I Cloud Code and Kiro, as I Cloud Code and Kiro, as I mentioned. So if you're mentioned. So if you're mentioned. So if you're trying Cursor or Codex, the trying Cursor or Codex, the trying Cursor or Codex, the agent bridge is something that agent bridge is something that agent bridge is something that you're going to have to you're going to have to you're going to have to edit so that it edit so that it edit so that it can accept these can accept these can accept these requests from the requests from the requests from the apply changes button. Um apply changes button. Um , so it's going to , so it's going to start applying start applying start applying the changes soon, and you're also the changes soon, and you're also the changes soon, and you're also going to see some going to see some going to see some logs that are going to logs that are going to logs that are going to show up in the show up in the show up in the repository. Um, that's the repository. Um, that's the repository. Um, that's the key, the CLI logs, and key, the CLI logs, and key, the CLI logs, and also when also when the validation runs, Nova Act has the validation runs, Nova Act has these logs that give these logs that give these logs that give you an idea of ​​what you an idea of ​​what you an idea of ​​what exactly the AI ​​model was thinking exactly the AI ​​model was thinking exactly the AI ​​model was thinking and what part of and what part of and what part of the screen it the screen it the screen it was trying to influence was trying to influence was trying to influence in making its in making its in making its decision. And you can decision. And you can decision. And you can see that the button see that the button see that the button has changed to " has changed to " run validation run validation ." So, the validation

." So, the validation ." So, the validation phase is now running phase is now running phase is now running . And . And . And it's also going it's also going it's also going to produce a report soon that to produce a report soon that to produce a report soon that you can you can you can look at. So that's look at. So that's look at. So that's the point where you the point where you the point where you just apply just apply just apply the changes, go to lunch the changes, go to lunch the changes, go to lunch or a break, and then or a break, and then or a break, and then come back and come back and come back and look at the look at the look at the validation report. I'll give it validation report. I'll give it validation report. I'll give it a minute, because it a minute, because it a minute, because it has to go through all the has to go through all the has to go through all the design flows, design flows, design flows, all the user all the user all the user flows, and then flows, and then flows, and then produce the report. So it produce the report. So it produce the report. So it takes a little while. takes a little while. Let's see if I can Let's see if I can I'm going to open I'm going to open I'm going to open the repository. Oh. the repository. Oh. the repository. Oh. Damn. The Damn. The

Wi-Fi seems to be really Wi-Fi seems to be really slow, so But, I slow, so But, I ... I think for now I ... I think for now I ... I think for now I can invite can invite can invite Giovanni up on stage Giovanni up on stage Giovanni up on stage to show you a to show you a to show you a cool way to cool way to cool way to annotate annotate annotate web pages, instead of web pages, instead of web pages, instead of just just just typing something typing something typing something directly into the directly into the directly into the page you're page you're page you're working on. working on. Hi. Hi. Oh, okay. Oh, okay. Oh, okay. Okay. Okay. Okay. Can we Can we Can we Uh, I'm done. Can Uh, I'm done. Can Uh, I'm done. Can I see the slides? I see the slides? I see the slides? Slides, yes. Slides, yes. Hi. Hi everyone, I'm Hi. Hi everyone, I'm Giovanni. Imagine Giovanni. Imagine Giovanni. Imagine you're in a design meeting and you you're in a design meeting and you you're in a design meeting and you 're discussing changes 're discussing changes 're discussing changes that you want to make to a that you want to make to a that you want to make to a website. And then your website. And then your website. And then your perceptual agent perceptual agent perceptual agent can automatically can automatically can automatically change the website. That's change the website. That's change the website. That's what you can what you can what you can do right now do right now do right now with this with this with this workshop. And why, how?

workshop. And why, how? workshop. And why, how? Using Using Using B wearables. B wearables. B wearables. So B is a device So B is a device So B is a device that we have on Amazon. And you that we have on Amazon. And you that we have on Amazon. And you can buy it can buy it can buy it on b.computer. It's a on b.computer. It's a on b.computer. It's a wearable, your wearable, your wearable, your AI assistant. It has AI assistant. It has AI assistant. It has a microphone. So you a microphone. So you a microphone. So you can press a can press a can press a button and it starts button and it starts button and it starts listening to you. listening to you. No don't worry, there's a No don't worry, there's a light light light that turns on when that turns on when that turns on when it's listening, and all your it's listening, and all your it's listening, and all your conversations are stored conversations are stored conversations are stored in your private in your private in your private cloud. After that, the AI cloud. After that, the AI agent running on B agent running on B agent running on B can give you can give you can give you suggestions for suggestions for any conversation that's going on any conversation that's going on any conversation that's going on . So, I . So, I . So, I integrated this integrated this integrated this B agent with the B agent with the B agent with the perception agent that we perception agent that we perception agent that we used for used for used for the workshop. Taking the workshop. Taking the workshop. Taking input from the AI input from the AI conversation, which is the conversation, which is the conversation, which is the design discussion. design discussion. Uh, I can't see the video. I'm Uh, I can't see the video. I'm going to... here it is. Yes.

going to... here it is. Yes. going to... here it is. Yes. Thank you. You can Thank you. You can Thank you. You can achieve this in three achieve this in three achieve this in three easy steps. easy steps. First, of course, you First, of course, you First, of course, you need to have B. I have a need to have B. I have a need to have B. I have a device that you device that you device that you can use right can use right can use right here and now, or we here and now, or we here and now, or we can set up a can set up a can set up a test account to test account to test account to download so you download so you download so you can record can record can record conversations however you want conversations however you want . Then you need to . Then you need to . Then you need to install the B CLI. It's a install the B CLI. It's a install the B CLI. It's a command command command line interface that will allow you line interface that will allow you line interface that will allow you to get to get to get your conversation data, and your conversation data, and your conversation data, and the instructions for the instructions for the instructions for installing the B CLI installing the B CLI installing the B CLI via the NPM command are in via the NPM command are in via the NPM command are in the workshop. Then you the workshop. Then you the workshop. Then you start the start the start the workshop server, start a workshop server, start a workshop server, start a bridge that listens to bridge that listens to bridge that listens to B's conversation, and B's conversation, and B's conversation, and applies it to the applies it to the applies it to the perceiving agent. perceiving agent. Then you just Then you just activate your activate your activate your device B. device B. device B. The LED lights up The LED lights up . You say something . You say something . You say something like, "Oh, I want to like, "Oh, I want to like, "Oh, I want to change the change the change the title color to title color to title color to red." I want to red." I want to red." I want to change the emoji, change the emoji, change the emoji, instead of the brain emoji, instead of the brain emoji, instead of the brain emoji, I want the robot emoji I want the robot emoji . And on the right, you'll . And on the right, you'll . And on the right, you'll see your see your see your conversation, conversation, conversation, the transcript of it, and the transcript of it, and the transcript of it, and even the even the even the highlights. Here you highlights. Here you highlights. Here you see the API response, the see the API response, the see the API response, the API response API response API response coming from B. You can coming from B. You can coming from B. You can open the panel and open the panel and open the panel and inspect it. And then inspect it. And then inspect it. And then you can just you can just you can just click " click " apply." And so here's apply." And so here's apply." And so here's what the typical what the typical what the typical workflow for a Perception workflow for a Perception workflow for a Perception Agent workshop would be.

Agent workshop would be. Agent workshop would be. So it So it So it would change the color, the would change the color, the would change the color, the color that you color that you color that you just said, and just said, and just said, and change the emoji. And change the emoji. And change the emoji. And after it applies after it applies after it applies the changes to the website, it would the changes to the website, it would the changes to the website, it would automatically automatically automatically run a run a run a no act validation stage for you. no act validation stage for you. So we have a test So we have a test account that account that account that you can you can you can use with a use with a use with a preloaded preloaded conversation, a preloaded design design conversation that just conversation that just conversation that just says, "Oh, we're says, "Oh, we're says, "Oh, we're changing the color, we're changing the color, we're changing the color, we're changing the emoji." changing the emoji." changing the emoji." Something like that. Something like that. Something like that. And after you install And after you install And after you install the CLI and log in the CLI and log in the CLI and log in with account B with account B with account B using a using a using a test device with a test device with a test device with a test test test account, you just account, you just account, you just click the button at the click the button at the click the button at the bottom, the button at the bottom, the button at the bottom, the button at the bottom. Uh, bottom. Uh, bottom. Uh, this is the this is the this is the validation page that you validation page that you validation page that you can check. can check. can check. But if you want to But if you want to But if you want to test test test device B, we can device B, we can ... you can ... you can ... you can use mine, and I'll use mine, and I'll use mine, and I'll show you everything if you show you everything if you show you everything if you want. We can want. We can want. We can actually actually actually talk about the talk about the talk about the design, and you'll see design, and you'll see design, and you'll see that it actually that it actually that it actually applies this after applies this after applies this after two or three minutes two or three minutes two or three minutes of talking.

of talking. of talking. My Wi-Fi is a little My Wi-Fi is a little My Wi-Fi is a little slow, so I didn't slow, so I didn't slow, so I didn't do a do a do a live demo. live demo. live demo. Uh, yeah, and Uh, yeah, and Uh, yeah, and there's a test report at the end there's a test report at the end there's a test report at the end that you can that you can that you can look at. The look at. The B devices are also B devices are also our our our contest prizes. So, yeah. You contest prizes. So, yeah. You contest prizes. So, yeah. You get one if get one if get one if you win the contest. And you win the contest. And you win the contest. And we're also at the booth we're also at the booth we're also at the booth today and today and today and all week. So, yeah all week. So, yeah . Let me know if you . Let me know if you . Let me know if you want to integrate B, want to integrate B, want to integrate B, and I'm here for you. and I'm here for you. So, device B, if So, device B, if you want to come up you want to come up you want to come up on stage, we have 10 on stage, we have 10 on stage, we have 10 minutes for you to minutes for you to minutes for you to explore these two explore these two explore these two primitives and primitives and primitives and think about an idea of ​​how think about an idea of ​​how think about an idea of ​​how you might want to you might want to you might want to use this. And use this. And use this. And if you want to come out, if you want to come out, if you want to come out, you'll get one of you'll get one of you'll get one of these devices. Uh, these devices. Uh, these devices. Uh, so, yeah. If you just so, yeah. If you just so, yeah. If you just get creative, get creative, get creative, I think. Hey everyone, this is I think. Hey everyone, this is I think. Hey everyone, this is Jed, who's Jed, who's Jed, who's made something really made something really made something really cool, and he'd like to cool, and he'd like to cool, and he'd like to show it to everyone. show it to everyone. show it to everyone. Jed, come up on Jed, come up on Jed, come up on stage. Yeah. Jed. Oh.

stage. Yeah. Jed. Oh. stage. Yeah. Jed. Oh. What am I doing? Where am I going? What am I doing? Where am I going? What am I doing? Where am I going? Oh, I called Jed, Oh, I called Jed, Oh, I called Jed, but yeah. but yeah. but yeah. Oh, I, I... my name is Oh, I, I... my name is Oh, I, I... my name is Jeff, so I thought... Jeff, so I thought... Jeff, so I thought... Sorry. After that, Sorry. After that, Sorry. After that, Jeffrey's going to show you Jeffrey's going to show you Jeffrey's going to show you something interesting. something interesting. something interesting. Oh, okay. Okay, that Oh, okay. Okay, that Oh, okay. Okay, that was just funny, was just funny, was just funny, but it's actually a really but it's actually a really but it's actually a really interesting little interesting little interesting little process—the way these process—the way these process—the way these animators go animators go animators go through it... Oh, you through it... Oh, you through it... Oh, you don't see it. don't see it. Change it to a podcast Change it to a podcast about ducks. That's it. about ducks. That's it. So, just asked to So, just asked to turn it into a turn it into a turn it into a retro NES style. Throw in retro NES style. Throw in retro NES style. Throw in some some some duck jokes and go through the duck jokes and go through the duck jokes and go through the validation phase. Probably the most validation phase. Probably the most validation phase. Probably the most interesting thing for me interesting thing for me interesting thing for me here is watching here is watching here is watching this wonderful AI this wonderful AI this wonderful AI say, "I made this with a say, "I made this with a say, "I made this with a duck, you know, I duck, you know, I duck, you know, I made a duck icon made a duck icon ." But a lot of the time, as everyone ." But a lot of the time, as everyone ." But a lot of the time, as everyone here probably knows, here probably knows, here probably knows, you don't see it you don't see it you don't see it happening, but happening, but happening, but it went through the it went through the it went through the whole whole whole validation process and validation process and validation process and came back with a report saying, came back with a report saying, came back with a report saying, "Yeah, it's done." I "Yeah, it's done." I "Yeah, it's done." I did it. I really did it. I really did it. I really did it. So that was did it. So that was did it. So that was cool. Oh, yeah. I just cool. Oh, yeah. I just cool. Oh, yeah. I just asked him

asked him asked him to tell some to tell some to tell some jokes. Which he's jokes. Which he's jokes. Which he's pretty pretty pretty bad at. Here you go bad at. Here you go bad at. Here you go . . Oh, because you came up with a Oh, because you came up with a cool demo. Do cool demo. Do cool demo. Do you need to come out? you need to come out? As promised, you're As promised, you're going to win this. going to win this. going to win this. Oh, thanks. Oh, thanks. Oh, thanks. Device B. Next up, we have Device B. Next up, we have Device B. Next up, we have Jeffrey who wants to Jeffrey who wants to Jeffrey who wants to come out and present what come out and present what come out and present what he did. he did.

Yeah, I'm still building a Yeah, I'm still building a testbed to testbed to testbed to actually show it, actually show it, actually show it, because the default because the default because the default B account has an example B account has an example B account has an example conversation that's not conversation that's not conversation that's not relevant. relevant. So this is just trying to So this is just trying to bring B to life. The idea was to bring B to life. The idea was to access the B access the B CLI and pull up a CLI and pull up a CLI and pull up a summary of the conversation where summary of the conversation where summary of the conversation where they were discussing they were discussing they were discussing backlog features that backlog features that backlog features that need to be added to the need to be added to the need to be added to the repository that repository that repository that you're working on, you're working on, you're working on, you know? You're you know? You're you know? You're working on a working on a working on a specific specific specific GitHub repository GitHub repository GitHub repository that's cloned, and that's cloned, and that's cloned, and then your conversation then your conversation then your conversation will be used in will be used in will be used in the context of all the the context of all the the context of all the code that's there. He code that's there. He code that's there. He will create a backlog will create a backlog will create a backlog based on this based on this based on this summary and summary and summary and use Atlassian MCP use Atlassian MCP use Atlassian MCP to bring it into Jira. to bring it into Jira. So that's the idea here. And So that's the idea here. And the main thing that I'm the main thing that I'm the main thing that I'm building in the background right now building in the background right now building in the background right now is a is a is a testbed for inserting a testbed for inserting a testbed for inserting a new conversation that new conversation that new conversation that matches this matches this matches this repository. So it's...

repository. So it's... Pretty simple and... yeah. Pretty simple and... yeah. Anyway. Anyway. Anyway. That's awesome. Thank you very much. That's awesome. Thank you very much. That's awesome. Thank you very much. Thank you. Please. Thank you. Please. Thank you. Please. So here you go, you win So here you go, you win So here you go, you win your new device. your new device. your new device. I know you wanted to I know you wanted to I know you wanted to go and buy it, go and buy it, go and buy it, but you won it. but you won it. but you won it. Thank you very much. Thank you very much. Oh, do you have any questions? Ryan, who's Ryan, who's next? next? Hi everyone. Good to Hi everyone. Good to be here. Let me be here. Let me be here. Let me take a look. I'm starting take a look. I'm starting take a look. I'm starting the broadcast. I'm the broadcast. I'm the broadcast. I'm late for this late for this late for this session because the previous one session because the previous one session because the previous one ended at 3:00. So ended at 3:00. So ended at 3:00. So I'm going to I'm going to I'm going to Giovanni. Let me Giovanni. Let me Giovanni. Let me change the key here.

change the key here. Oh, perfect. That's good. Oh, perfect. That's good. I'm handing this over to Giovanni I'm handing this over to Giovanni . I'm wearing a . I'm wearing a . I'm wearing a bee hat right now. I'm bee hat right now. I'm bee hat right now. I'm at at Wharton School of Business. Wharton School of Business. Wharton School of Business. My group mascot is a My group mascot is a bee. I'm one of bee. I'm one of bee. I'm one of three people who don't three people who don't three people who don't care that the mascot care that the mascot is a bee. So you can is a bee. So you can is a bee. So you can imagine my imagine my imagine my excitement when I excitement when I excitement when I got to the got to the got to the conference and conference and conference and saw that we were saw that we were saw that we were presenting a presenting a presenting a "bee" device. "bee" device. "bee" device. Of course, being late Of course, being late Of course, being late and a and a and a Wharton graduate, I asked if I Wharton graduate, I asked if I Wharton graduate, I asked if I could get a could get a could get a free free free device. Giovanni device. Giovanni device. Giovanni said, "Have an idea said, "Have an idea said, "Have an idea you want to you want to you want to share?" "I'm currently share?" "I'm currently share?" "I'm currently working at an working at an AI training startup. I'm not a AI training startup. I'm not a big fan of big fan of big fan of Linear. Sorry if Linear. Sorry if Linear. Sorry if any of you work at any of you work at any of you work at Linear. I just Linear. I just Linear. I just needed a better needed a better needed a better way to understand what way to understand what way to understand what to do during the day, to do during the day, to do during the day, from capturing an idea to from capturing an idea to from capturing an idea to discussing it with my discussing it with my discussing it with my co-founder and co-founder and co-founder and updating the updating the updating the project repository. project repository. project repository. This is a natural This is a natural This is a natural use case for a use case for a use case for a bee. bee. bee. We have an internal We have an internal We have an internal tool tool tool called Sidekick. called Sidekick.

called Sidekick. I emphasize I emphasize internal because there's internal because there's internal because there's no design in it. In no design in it. In no design in it. In defense of my defense of my defense of my co-founder: he co-founder: he co-founder: he asked Claude to make asked Claude to make asked Claude to make it beautiful, without it beautiful, without it beautiful, without bugs, and this is the bugs, and this is the bugs, and this is the best result. best result. So here's how it works: So here's how it works: we have a shared inbox we have a shared inbox we have a shared inbox . You can . You can . You can ignore "Go Touch Crash SF ignore "Go Touch Crash SF ignore "Go Touch Crash SF ." I see all these ." I see all these ." I see all these billboards, and it billboards, and it billboards, and it bothers me. We bothers me. We bothers me. We send emails send emails send emails together to together to together to [email protected]. It's [email protected]. It's [email protected]. It's all our different all our different all our different inputs, thoughts for inputs, thoughts for inputs, thoughts for the week, newsletters. I don't the week, newsletters. I don't the week, newsletters. I don't know if anyone here is know if anyone here is know if anyone here is subscribed to AI subscribed to AI newsletters. We see something newsletters. We see something newsletters. We see something interesting—we send it to the interesting—we send it to the interesting—we send it to the inbox. We believe in the inbox. We believe in the inbox. We believe in the method GTD ("Getting Things Done" method GTD ("Getting Things Done" ) is a ) is a ) is a task management system task management system task management system developed by David developed by David developed by David Allen in 2010. Allen in 2010. If you have Wi-Fi, you If you have Wi-Fi, you can see it. can see it. Sorry, it just did Sorry, it just did n't work on n't work on n't work on the screen. One second.

the screen. One second. So the to-do tree shows all the projects, shows all the projects, and under each project and under each project and under each project there's a strategy, a there's a strategy, a there's a strategy, a completion description, tasks, completion description, tasks, completion description, tasks, next steps, and next steps, and next steps, and links. We links. We links. We created the MCP so you could created the MCP so you could created the MCP so you could sit down and sit down and sit down and talk to Claude and talk to Claude and talk to Claude and strategize: strategize: strategize: "Hey, here's what we're "Hey, here's what we're "Hey, here's what we're thinking about this thinking about this thinking about this project. What do you project. What do you project. What do you think we should think we should think we should do?" All of this is to do?" All of this is to do?" All of this is to address address address the question that arises the question that arises the question that arises after the meeting: “What should after the meeting: “What should after the meeting: “What should I do now that I do now that I do now that aligns with aligns with aligns with my overall my overall my overall strategy?” So, how strategy?” So, how strategy?” So, how we see the we see the we see the Bee device working Bee device working , because Giovanni , because Giovanni , because Giovanni told me there's an told me there's an told me there's an MCP: we capture our MCP: we capture our MCP: we capture our meetings or ideas, meetings or ideas, meetings or ideas, send them to the inbox send them to the inbox send them to the inbox , and then , and then , and then we use that we use that we use that to update our to update our to update our strategy. Hmm, strategy. Hmm, strategy. Hmm, I wish it would I wish it would I wish it would upload, because, upload, because, upload, because, let me see let me see . Oh, here's why. My . Oh, here's why. My . Oh, here's why. My phone. See, phone. See, phone. See, it worked. Hmm. Yeah. it worked. Hmm. Yeah. it worked. Hmm. Yeah. You know, I've had 0 You know, I've had 0 You know, I've had 0 out of 11 successful live out of 11 successful live out of 11 successful live demos right now. But this is demos right now. But this is demos right now. But this is my concept for the my concept for the my concept for the Bee device. I'm Bee device. I'm Bee device. I'm looking forward to looking forward to using it and using it and telling LinkedIn how telling LinkedIn how telling LinkedIn how much I love bees.

much I love bees. much I love bees. Go bees! Buzz- Go bees! Buzz- buzz. Thank you. buzz. Thank you. How would you integrate How would you integrate perception agents into perception agents into perception agents into your... your... Oh, I actually have Oh, I actually have an answer for that too. an answer for that too. an answer for that too. Okay. Okay. So, as I wrote to So, as I wrote to myself in Sidekick to myself in Sidekick to myself in Sidekick to discuss later with discuss later with discuss later with my co-founder my co-founder , we've developed a lot of , we've developed a lot of , we've developed a lot of products like products like products like Sidekick, different Sidekick, different Sidekick, different tools for tools for tools for training. Our whole training. Our whole training. Our whole mission is to make sure mission is to make sure mission is to make sure that everyone knows how that everyone knows how that everyone knows how to adapt to the to adapt to the to adapt to the future changes in the future changes in the future changes in the job market. But job market. But job market. But the main obstacle the main obstacle the main obstacle right now is the right now is the right now is the product design, to create product design, to create product design, to create something where something where something where users users users understand exactly what to do. understand exactly what to do. So we would use the So we would use the annotation feature of the annotation feature of the annotation feature of the perception agent and perception agent and perception agent and then send then send then send that description through Sidekick that description through Sidekick that description through Sidekick or our own or our own or our own agent to make agent to make agent to make improvements and close the improvements and close the improvements and close the development cycle. development cycle. development cycle. Oh. Oh. Oh. So, yeah, if AGI So, yeah, if AGI So, yeah, if AGI wants to sponsor wants to sponsor wants to sponsor Ozzie, I'll give you my Ozzie, I'll give you my Ozzie, I'll give you my card and we'll card and we'll card and we'll talk later.

talk later. talk later. So, thank you. So, thank you. So, thank you. Great. Thank you very much for Great. Thank you very much for Great. Thank you very much for coming and coming and coming and showing, showing, showing, here's the here's the here's the Bee device as promised. Bee device as promised. Bee device as promised. Thank you. I appreciate it. Thank you. I appreciate it. So, to wrap up, So, to wrap up, Emil, I'm just Emil, I'm just Emil, I'm just going to... We're going to... We're going to... We're going to put the survey up going to put the survey up going to put the survey up on the board on the screen. It's on the board on the screen. It's on the board on the screen. It's something that we... Yeah. You something that we... Yeah. You something that we... Yeah. You want to... Oh, okay. want to... Oh, okay. Okay, there's also Jonathan Okay, there's also Jonathan who wants to who wants to who wants to come on. come on. Um, so I have Um, so I have more of a concept because I have more of a concept because I have more of a concept because I have n't had time to n't had time to n't had time to build everything. But I'll build everything. But I'll build everything. But I'll just just just talk a little bit about it. talk a little bit about it.

Okay. So I Okay. So I was looking at the proxy for was looking at the proxy for was looking at the proxy for B and trying B and trying B and trying to figure out how we to figure out how we to figure out how we get the information get the information get the information from the CLI. Oh, it's not working. from the CLI. Oh, it's not working. from the CLI. Oh, it's not working. It's not It's not It's not being displayed. Oh, being displayed. Oh, being displayed. Oh, there it is now. Okay. there it is now. Okay. So I was trying So I was trying to figure out how to figure out how the proxy actually works and how we the proxy actually works and how we get the information get the information get the information from B, and is it possible, other than from B, and is it possible, other than from B, and is it possible, other than transcription, transcription, transcription, to pull data from to pull data from to pull data from different sources? Is it different sources? Is it different sources? Is it possible to get it from an IoT possible to get it from an IoT device for some device for some device for some reason? And what reason? And what reason? And what I started to come to is that I started to come to is that I started to come to is that at the company I at the company I at the company I work for—it's a financial work for—it's a financial work for—it's a financial company—we have company—we have company—we have a lot of non-technical a lot of non-technical a lot of non-technical people. And we people. And we people. And we need to not only need to not only need to not only get our get our get our criteria, but we also need to criteria, but we also need to capture a lot of UI/UX that we have to capture, and capture a lot of UI/UX that we have to capture, and we don't always we don't always we don't always get it in the get it in the get it in the finest detail finest detail finest detail for the for the for the acceptance criteria. So I acceptance criteria. So I acceptance criteria. So I was pushing the team was pushing the team was pushing the team to use Jam MCP, to use Jam MCP, to use Jam MCP, which allows us to do which allows us to do which allows us to do recording, capture all recording, capture all recording, capture all the console logs, the console logs, the console logs, network requests, and network requests, and network requests, and then pass that then pass that then pass that information. Also information. Also , I built an AI , I built an AI harness so that we can harness so that we can harness so that we can push that push that push that information there. So information there. So the architecture the architecture the architecture looks something like looks something like looks something like this: similar to the CLI, you this: similar to the CLI, you this: similar to the CLI, you can take Jam MCP, can take Jam MCP, can take Jam MCP, push that push that push that information into a proxy, information into a proxy, information into a proxy, have LLM call have LLM call have LLM call Nova, and then Nova, and then Nova, and then implement the UI. So

implement the UI. So implement the UI. So if you say something if you say something if you say something like, "Oh, I'd like, "Oh, I'd like, "Oh, I'd like this like this like this investor schedule to be like this investor schedule to be like this investor schedule to be like this , or this , or this , or this metric should be a metric should be a metric should be a research level and research level and research level and be in a different be in a different be in a different location," we could location," we could location," we could pull that pull that pull that information from MCP, information from MCP, information from MCP, send it to Nova, and send it to Nova, and send it to Nova, and make changes to the AI- make changes to the AI- binding that's running binding that's running binding that's running locally. So you locally. So you locally. So you can can can use use use the proxy in the same way the proxy in the same way , but through a different , but through a different , but through a different device. I did device. I did device. I did some initial some initial some initial testing. It seems to testing. It seems to testing. It seems to work even through work even through work even through mocking, but over Wi-Fi I could mocking, but over Wi-Fi I could mocking, but over Wi-Fi I could n't get n't get n't get Jam MCP to run. But that's going to be my Jam MCP to run. But that's going to be my Jam MCP to run. But that's going to be my job when job when job when I get home. I get home. I get home. So thank you. So thank you. Thank you very much, hold on. So now we have another So now we have another 15 minutes, and then 15 minutes, and then 15 minutes, and then we'll wrap up the workshop.

we'll wrap up the workshop. We'll put a QR code on the We'll put a QR code on the screen. If you have a screen. If you have a screen. If you have a great idea but you don't great idea but you don't great idea but you don't want to come up and want to come up and want to come up and talk about it, talk about it, talk about it, you can send it you can send it you can send it here, and we We'll here, and we We'll here, and we We'll vote on these vote on these vote on these ideas and the top three ideas and the top three ideas and the top three will get a TB device. will get a TB device. So if you have an idea, So if you have an idea, please add please add please add it to this it to this it to this poll. Give poll. Give poll. Give me a second. Here you go. Do me a second. Here you go. Do me a second. Here you go. Do you see this? Yes. We you see this? Yes. We you see this? Yes. We want you to want you to want you to scan the QR code and scan the QR code and scan the QR code and enter an innovative enter an innovative enter an innovative idea using idea using idea using what we've learned. what we've learned. what we've learned. How would you use How would you use How would you use this this this perception system? Everything you've perception system? Everything you've perception system? Everything you've seen today, everything you've seen today, everything you've seen today, everything you've worked with, what worked with, what worked with, what use case do use case do you think would be you think would be you think would be useful and make useful and make useful and make your daily your daily your daily life easier? We've seen life easier? We've seen life easier? We've seen quite a few ideas. And if you quite a few ideas. And if you quite a few ideas. And if you think you have think you have think you have one but are shy, there one but are shy, there 's a chance to win another 's a chance to win another device.

device. So what do you say? 5 So what do you say? 5 minutes and then we're minutes and then we're minutes and then we're 5 minutes and then we're 5 minutes and then we're 5 minutes and then we're like. like. Oh. Let me Let me fix that. fix that. Okay, I've stopped Okay, I've stopped the poll. So, all the the poll. So, all the the poll. So, all the ideas are great. I've ideas are great. I've ideas are great. I've been looking at them and been looking at them and been looking at them and discussing with Emil, discussing with Emil, discussing with Emil, how difficult it is to make a how difficult it is to make a how difficult it is to make a choice. Isn't this choice. Isn't this choice. Isn't this Emil? So we have an Emil? So we have an idea with Pixel Judge. Can idea with Pixel Judge. Can someone raise hand, someone raise hand, someone raise hand, who suggested this who suggested this who suggested this idea? Nobody? Pixel Judge? What is idea? Nobody? Pixel Judge? What is idea? Nobody? Pixel Judge? What is it? Okay.

it? Okay. Next idea— agent-based end-to-end agent-based end-to-end agent-based end-to-end product testing. product testing. Okay. Come Okay. Come here. Do you want to have Yes. here. Do you want to have Yes. here. Do you want to have Yes. Great. Thank you. Reframe—a Great. Thank you. Reframe—a canvas for thinking, canvas for thinking, canvas for thinking, brainstorming, and brainstorming, and brainstorming, and planning. Is that you? planning. Is that you? planning. Is that you? Come here. We Come here. We Come here. We decided to give away decided to give away decided to give away three devices. And one three devices. And one three devices. And one went went went unclaimed. So unclaimed. So unclaimed. So we actually chose we actually chose we actually chose four winners. four winners.

The last one—creating a The last one—creating a canvas app canvas app canvas app where a perception agent and a where a perception agent and a where a perception agent and a human work on the human work on the human work on the design of the interface design of the interface design of the interface in real time. in real time. Come here. Thank you Come here. Thank you everyone. Big applause everyone. Big applause everyone. Big applause for all our for all our for all our winners. Okay, I winners. Okay, I winners. Okay, I want to end the session with a want to end the session with a want to end the session with a main conclusion. main conclusion. The most important thing in this The most important thing in this workshop, as you have already workshop, as you have already workshop, as you have already understood, is that understood, is that understood, is that the way the way the way human and agent collaborate is human and agent collaborate is human and agent collaborate is changing. We changing. We changing. We need tools need tools that will make that will make that will make the visualization of the visualization of the visualization of human and human and human and agent interactions transparent, and that is the agent interactions transparent, and that is the agent interactions transparent, and that is the next step in next step in next step in building building building perception agents. So you perception agents. So you perception agents. So you can get started can get started can get started with Nova Act, if you still haven't with Nova Act, if you still haven't with Nova Act, if you still haven't checked out the checked out the checked out the GitHub repository.

GitHub repository. GitHub repository. It's available, you It's available, you It's available, you can scan can scan can scan this QR code. This is the SDK, and this QR code. This is the SDK, and this QR code. This is the SDK, and we also have a we also have a we also have a blog post from our blog post from our blog post from our team team team explaining our explaining our explaining our vision for vision for vision for perceptual agents in general. perceptual agents in general. Thank you all for that, and Thank you all for that, and we've also put up a QR we've also put up a QR code for feedback. code for feedback. If you enjoyed If you enjoyed the session, Emil, me, or the session, Emil, me, or the session, Emil, me, or Giovanni, who Giovanni, who Giovanni, who spoke here, please spoke here, please spoke here, please leave us a review. leave us a review. We appreciate all of you for We appreciate all of you for coming. coming. coming. Thank you very much. Thank you very much.