Teaching Agents to Search with NVIDIA Data Designer — Dhruv Nathawani, NVIDIA

This seminar is This seminar is about something about something about something more general and more general and more general and powerful. It's about powerful. It's about powerful. It's about how we build how we build how we build data pipelines using a data pipelines using a tool we tool we created, Nemo Data Designer, created, Nemo Data Designer, created, Nemo Data Designer, to be able to to be able to to be able to create or create or create or add any add any add any functionality to a model or functionality to a model or functionality to a model or agent. So, let's begin. agent. So, let's begin. agent. So, let's begin. By the way, I have a By the way, I have a By the way, I have a repository on GitHub repository on GitHub repository on GitHub where I store several where I store several where I store several notebooks. So if notebooks. So if notebooks. So if you're interested, you can you're interested, you can you're interested, you can scan the QR code scan the QR code scan the QR code or go to this or go to this or go to this repository on GitHub. And the repository on GitHub. And the repository on GitHub. And the same QR code will be same QR code will be same QR code will be on all other slides on all other slides . So if you . So if you . So if you miss it, there will be miss it, there will be miss it, there will be more opportunities more opportunities more opportunities to get it. So, what is to get it. So, what is to get it. So, what is this seminar about and this seminar about and this seminar about and what will we learn here? what will we learn here? what will we learn here? The first thing we're The first thing we're The first thing we're going to learn going to learn is how to actually is how to actually is how to actually take the target take the target take the target agent behavior agent behavior agent behavior you want and you want and you want and turn it into a turn it into a turn it into a data problem? data problem? data problem? Right? So, Right? So, Right? So, essentially, can we essentially, can we essentially, can we generate data for generate data for generate data for this target this target this target agent behavior? And agent behavior? And agent behavior? And then can we then can we then can we train the model to train the model to train the model to learn to perform learn to perform learn to perform this task? And so we this task? And so we this task? And so we created a tool created a tool created a tool called Data Designer called Data Designer called Data Designer to create to create to create synthetic synthetic synthetic data pipelines that data pipelines that data pipelines that does just that. And does just that. And does just that. And today we today we today we will look at a will look at a will look at a use case–this is use case–this is use case–this is search. By the way, this is a search. By the way, this is a search. By the way, this is a very reusable very reusable very reusable framework. So, if framework. So, if framework. So, if you want to create you want to create you want to create data for something data for something data for something other than search, other than search, other than search, this is how you this is how you this is how you do it. There are two do it. There are two do it. There are two ways to participate ways to participate ways to participate in today's in today's in today's seminar. Of course, seminar. Of course, seminar. Of course, you can follow the you can follow the you can follow the live broadcast, live broadcast, live broadcast, try running try running try running notebooks. But notebooks. But notebooks. But since we only have since we only have since we only have an hour or less, you an hour or less, you an hour or less, you might as well might as well might as well listen now and

listen now and listen now and then start the then start the then start the notebooks later. So notebooks later. So notebooks later. So make sure you make sure you make sure you get the get the get the GitHub link. And if you GitHub link. And if you GitHub link. And if you want the slides, they are want the slides, they are want the slides, they are also available in the also available in the also available in the GitHub repository. You will need GitHub repository. You will need GitHub repository. You will need four things if you are four things if you are four things if you are going to going to going to follow up now or follow up now or follow up now or after the seminar. after the seminar. after the seminar. Of course, you Of course, you Of course, you need a machine with need a machine with GPU support , i.e. a cloud , i.e. a cloud , i.e. a cloud virtual machine virtual machine virtual machine or workstation. or workstation. or workstation. You will also You will also You will also need a need a need a Tavily search API key. Tavily is a Tavily search API key. Tavily is a Tavily search API key. Tavily is a company that company that company that searches for searches for searches for agents. If you agents. If you agents. If you register on register on register on their website, their website, their website, they will give you 1000 they will give you 1000 they will give you 1000 credits for free credits for free . And you need Data . And you need Data . And you need Data Designer, our Designer, our Designer, our data creation tool. data creation tool. data creation tool. It is completely It is completely It is completely open open open source. Anyone can source. Anyone can source. Anyone can download it, download it, download it, use it, use it, use it, do whatever they want with it do whatever they want with it do whatever they want with it . And all . And all . And all the notebooks I provided are the notebooks I provided are the notebooks I provided are in the GitHub repository. in the GitHub repository. So, here's the agenda So, here's the agenda for today. for today. for today. First, we'll First, we'll First, we'll spend a few spend a few spend a few minutes minutes minutes talking about what talking about what talking about what Data Designer is and why do Data Designer is and why do Data Designer is and why do we actually we actually we actually need such a need such a need such a framework? And then we'll framework? And then we'll framework? And then we'll talk about the talk about the talk about the basics. There are three basics. There are three basics. There are three notebooks. I'll show notebooks. I'll show notebooks. I'll show you the code. The first you the code. The first you the code. The first notebook is dedicated to notebook is dedicated to notebook is dedicated to the basics of Data Designer. How do the basics of Data Designer. How do the basics of Data Designer. How do you actually create you actually create you actually create the data you need? And the the data you need? And the the data you need? And the second notebook second notebook second notebook is dedicated to the is dedicated to the Model Context Protocol or MCP. How do Model Context Protocol or MCP. How do you actually create you actually create you actually create agent data? And the agent data? And the agent data? And the third notebook, which third notebook, which third notebook, which is, um, an extremely is, um, an extremely is, um, an extremely interesting interesting interesting use case for search.

use case for search. use case for search. How do you actually train How do you actually train How do you actually train a model to perform a model to perform a model to perform a search? And then we'll a search? And then we'll a search? And then we'll leave some time at leave some time at leave some time at the end for questions. the end for questions. So, I want to start with a So, I want to start with a quote from Andriy quote from Andriy quote from Andriy Karpaty, who Karpaty, who Karpaty, who said: “Data from said: “Data from said: “Data from the Internet is not the the Internet is not the the Internet is not the data you data you data you need for your need for your need for your transformer.” So transformer.” So transformer.” So that's, um, a very, very valid that's, um, a very, very valid that's, um, a very, very valid point, because data from point, because data from point, because data from the Internet can the Internet can the Internet can be, you know, be, you know, be, you know, high-quality high-quality high-quality sources like sources like sources like Wikipedia, or it Wikipedia, or it Wikipedia, or it can be, you know, can be, you know, can be, you know, someone's opinion on Twitter, X, someone's opinion on Twitter, X, someone's opinion on Twitter, X, or some other or some other or some other platform, right? platform, right? platform, right? So how do we actually So how do we actually So how do we actually know, ahem, what data is know, ahem, what data is know, ahem, what data is useful for training useful for training useful for training AI models and what data isn't? AI models and what data isn't? So, um, we need to So, um, we need to process this data from process this data from process this data from the internet, curate the internet, curate the internet, curate it, and then, um, it, and then, um, it, and then, um, create create create training data from it. So, training data from it. So, training data from it. So, this is the traditional this is the traditional this is the traditional approach. We propose approach. We propose that we can that we can that we can actually synthetically actually synthetically actually synthetically generate data, ahem, generate data, ahem, generate data, ahem, either from scratch or from an either from scratch or from an either from scratch or from an initial initial initial dataset. And that's exactly what dataset. And that's exactly what dataset. And that's exactly what Data Designer is for. Data Designer is for. Data Designer is for. So, Data Designer is a So, Data Designer is a So, Data Designer is a framework for framework for framework for creating creating creating synthetic data. And synthetic data. And synthetic data. And what do I mean by, what do I mean by, what do I mean by, um, initial um, initial um, initial data? So, you can use raw data? So, you can use raw data? So, you can use raw data, such as Wikipedia data, data, such as Wikipedia data, as a as a starting point, and then starting point, and then starting point, and then try to create try to create try to create synthetic training synthetic training synthetic training data from Wikipedia. So data from Wikipedia. So data from Wikipedia. So your synthetic data is your synthetic data is your synthetic data is not just a not just a not just a model.

model. It is based on It is based on real distribution real distribution real distribution and real data. and real data. So, Data Designer is a So, Data Designer is a completely open completely open completely open framework. We have a framework. We have a framework. We have a large team at NVIDIA large team at NVIDIA large team at NVIDIA working on this. Um working on this. Um , and um, you can , and um, you can , and um, you can just install just install just install it, um, by running it, um, by running it, um, by running pip install Data Designer. And I pip install Data Designer. And I pip install Data Designer. And I think that, um, what we're think that, um, what we're think that, um, what we're really proud of is that really proud of is that really proud of is that Data Designer has been Data Designer has been Data Designer has been used to used to used to create 6.5 create 6.5 create 6.5 trillion tokens of trillion tokens of trillion tokens of training data, and training data, and training data, and that number is growing that number is growing that number is growing every day. Um, and every day. Um, and every day. Um, and yes, we also yes, we also yes, we also track track track GitHub stars. So, if you GitHub stars. So, if you GitHub stars. So, if you like our like our like our repository, repository, repository, give us a star. give us a star. So, what is synthetic data generation, synthetic data generation, right? Um, well, one right? Um, well, one right? Um, well, one way to generate way to generate way to generate synthetic data synthetic data is to take a loop, is to take a loop, is to take a loop, run it a run it a run it a million times, and million times, and million times, and then if you want to then if you want to then if you want to create a create a create a product review dataset, product review dataset, product review dataset, just keep just keep just keep asking LLM, "Give asking LLM, "Give asking LLM, "Give me the product, give me the product, give me the product, give me the review, and then me the review, and then me the review, and then give me the give me the give me the customer profile." This is one customer profile." This is one customer profile." This is one way to create a way to create a way to create a dataset, but you will dataset, but you will dataset, but you will run into a bunch of run into a bunch of run into a bunch of problems if you do problems if you do problems if you do this. So, what's this. So, what's this. So, what's missing? Um, missing? Um, missing? Um, diversity, diversity, diversity, diversity and more diversity and more diversity and more diversity.

diversity. diversity. That's the most important thing, is That's the most important thing, is That's the most important thing, is n't it? If you n't it? If you n't it? If you keep asking for LLM keep asking for LLM keep asking for LLM after 10, 12, maybe 20 after 10, 12, maybe 20 after 10, 12, maybe 20 attempts, it will start attempts, it will start attempts, it will start to repeat itself. He won't to repeat itself. He won't to repeat itself. He won't give you what you give you what you give you what you need. You are also need. You are also need. You are also missing missing missing correlations between all correlations between all correlations between all the features you are the features you are the features you are looking for in your looking for in your looking for in your dataset. You dataset. You dataset. You don't have don't have don't have multimodality. multimodality. You know, you You know, you don't really have control over don't really have control over don't really have control over the creation of this the creation of this the creation of this dataset, and there's no dataset, and there's no dataset, and there's no validation, validation, validation, verification, etc. And verification, etc. And verification, etc. And if I create a if I create a if I create a dataset, I can't dataset, I can't dataset, I can't share it with you share it with you , because how do you actually , because how do you actually , because how do you actually take the process that I take the process that I take the process that I used to used to used to create my create my create my dataset to dataset to dataset to create something create something create something like this? That's why we like this? That's why we like this? That's why we created Data Designer. created Data Designer. So, how does it work? In So, how does it work? In Data Designer, we have many Data Designer, we have many Data Designer, we have many different columns or different columns or different columns or functions. These columns functions. These columns functions. These columns are essentially are essentially are essentially an abstraction of the Data Designer. an abstraction of the Data Designer. In Data Designer, we In Data Designer, we generate data generate data generate data column by column. column by column. column by column. So, first you have So, first you have So, first you have your starting your starting your starting column, which is column, which is column, which is completely completely completely optional. So, optional. So, optional. So, I gave the example of I gave the example of I gave the example of Wikipedia, which can Wikipedia, which can Wikipedia, which can be an initial be an initial be an initial data set. We data set. We data set. We can can can use use any data set as the any data set as the any data set as the starting one. That is, starting one. That is, starting one. That is, any CSV, JSONL, any CSV, JSONL, any CSV, JSONL, Parquet file. So you can Parquet file. So you can Parquet file. So you can use use use your company data as a your company data as a your company data as a starting point. And then starting point. And then starting point. And then you can try you can try you can try adding adding adding sample columns to add sample columns to add sample columns to add variety.

variety. variety. Let's say you're Let's say you're Let's say you're trying to create a trying to create a trying to create a dataset for dataset for dataset for encoding. Well, a very encoding. Well, a very encoding. Well, a very simple simple simple sample column would be: sample column would be: sample column would be: easy, medium, and easy, medium, and easy, medium, and hard hard hard coding questions. So this is a coding questions. So this is a coding questions. So this is a sample column sample column sample column that you can add that you can add that corresponds to that corresponds to that corresponds to complexity. You complexity. You complexity. You can also add can also add can also add subsamples. So, for an subsamples. So, for an subsamples. So, for an easy task, you easy task, you easy task, you can match can match can match it to a specific type of it to a specific type of it to a specific type of task. For an task. For an task. For an average task, average task, average task, you can map you can map you can map it to a specific it to a specific it to a specific task type, etc. So, we'll task type, etc. So, we'll task type, etc. So, we'll look at all of this. I look at all of this. I look at all of this. I know this seems know this seems know this seems a bit abstract, a bit abstract, a bit abstract, but we'll look into all of this but we'll look into all of this but we'll look into all of this once once once we look at the code. And we look at the code. And we look at the code. And then we have columns then we have columns then we have columns for text generation, for text generation, for text generation, for for for structured data generation structured data generation , for code generation. , for code generation. , for code generation. We also have Data We also have Data We also have Data Designer, so you Designer, so you Designer, so you can generate can generate can generate images as well. And you images as well. And you images as well. And you can also can also can also check your check your check your generations. So the whole generations. So the whole generations. So the whole idea is idea is idea is that we create a DAG or a that we create a DAG or a that we create a DAG or a graph that describes graph that describes graph that describes what your what your what your dataset should be like, and then dataset should be like, and then dataset should be like, and then that DAG, when you're that DAG, when you're that DAG, when you're ready to execute it, ready to execute it, ready to execute it, you can just you can just you can just hit one click and hit one click and hit one click and get a dataset out of it get a dataset out of it get a dataset out of it . So, . So, . So, today we'll today we'll today we'll look at three different look at three different look at three different notebooks. Um, I know notebooks. Um, I know notebooks. Um, I know some of you some of you some of you are thinking, "Well, in 2026, are thinking, "Well, in 2026, are thinking, "Well, in 2026, he's going to be showing us he's going to be showing us he's going to be showing us code, right? I code, right? I code, right? I mean, now we have mean, now we have mean, now we have Codex and Cloud Code and all these Codex and Cloud Code and all these Codex and Cloud Code and all these amazing amazing amazing tools. So, I tools. So, I tools. So, I think if you think if you think if you understand code, it's going to be understand code, it's going to be understand code, it's going to be a lot easier a lot easier a lot easier to use Codex to use Codex to use Codex or Cloud Code to or Cloud Code to or Cloud Code to write your own, write your own, uh, uh, uh, synthetic synthetic synthetic data workflows. Uh, so we're going to go through data workflows. Uh, so we're going to go through each each notebook briefly now. And by the way, this is notebook briefly now. And by the way, this is notebook briefly now. And by the way, this is our repository. So our repository. So , this is our

, this is our , this is our data designer repository. It's data designer repository. It's data designer repository. It's completely open completely open completely open source. Uh, and this is the source. Uh, and this is the source. Uh, and this is the GitHub repository that, GitHub repository that, uh, I created for uh, I created for uh, I created for this workshop, where this workshop, where this workshop, where we have notebooks. Um, we have notebooks. Um, we have notebooks. Um, so you can so you can so you can try notebook 1 2 try notebook 1 2 try notebook 1 2 3. And then we also 3. And then we also 3. And then we also have, uh, slides here if have, uh, slides here if have, uh, slides here if you want to follow along. you want to follow along. you want to follow along. Or if you want to Or if you want to Or if you want to access access access the slides after the workshop. the slides after the workshop. the slides after the workshop. So, So, So, let's go to the first let's go to the first let's go to the first notebook. So, this is our notebook. So, this is our notebook. So, this is our introduction to the introduction to the introduction to the data designer notebook. Does data designer notebook. Does data designer notebook. Does everyone see okay, everyone see okay, everyone see okay, should I increase should I increase should I increase it? Is that good? Okay it? Is that good? Okay . So first of all, we . So first of all, we . So first of all, we set up our set up our set up our environment. Um, there's a environment. Um, there's a environment. Um, there's a bunch of, um, prompts here bunch of, um, prompts here bunch of, um, prompts here that we import, and that we import, and that we import, and then we just, you then we just, you then we just, you know, import the know, import the know, import the data designer. That's it data designer. That's it data designer. That's it , literally. Uh, and , literally. Uh, and , literally. Uh, and once you once you once you do that, you'll see do that, you'll see do that, you'll see that in the data designer, that in the data designer, that in the data designer, there's this concept of there's this concept of model providers. So right now model providers. So right now we're using v l l we're using v l l we're using v l l m. So this is local m. So this is local m. So this is local inference. So I'm inference. So I'm inference. So I'm running this on an running this on an Nvidia GPU cluster. And Nvidia GPU cluster. And then we have access then we have access then we have access to different types of to different types of to different types of models, right? So on models, right? So on models, right? So on my local my local my local cluster I have access cluster I have access cluster I have access to the GPT OS, which is to the GPT OS, which is to the GPT OS, which is 120 billion. You 120 billion. You 120 billion. You can can can use use any model or any model or any open model any open model any open model to generate to generate to generate synthetic data.

synthetic data. synthetic data. Some closed models Some closed models Some closed models prevent you from prevent you from generating data from them generating data from them and using and using and using them for training them for training them for training models because of licensing issues. But open models because of licensing issues. But open models because of licensing issues. But open models are usually models are usually models are usually fair game. So I'm fair game. So I'm fair game. So I'm using this using this using this model, and the goal here is can model, and the goal here is can I actually I actually take a set of Wikipedia topic data take a set of Wikipedia topic data take a set of Wikipedia topic data for this for this for this notebook. So, as you notebook. So, as you notebook. So, as you know, different, know, different, know, different, for example, Wikipedia for example, Wikipedia for example, Wikipedia has so many different has so many different has so many different pages and many pages and many pages and many different topics. So we different topics. So we different topics. So we just made a list of just made a list of just made a list of those, put it in a those, put it in a those, put it in a CSV, and then we just CSV, and then we just CSV, and then we just call our call our configuration builder with the configuration builder with the initial initial initial dataset. And so, basically, dataset. And so, basically, dataset. And so, basically, now we're trying to now we're trying to now we're trying to create a create a create a data design workflow data design workflow data design workflow but with but with but with Wikipedia topics. So Wikipedia topics. So Wikipedia topics. So once you do that, once you do that, once you do that, you can add you can add you can add question difficulty. You question difficulty. You question difficulty. You can specify can specify can specify easy, medium, easy, medium, easy, medium, hard, whatever you hard, whatever you hard, whatever you want your want your want your dataset to be. And then there are dataset to be. And then there are dataset to be. And then there are weights that you can weights that you can weights that you can specify. So you specify. So you specify. So you can have more can have more can have more medium questions than medium questions than medium questions than you know easy or you know easy or you know easy or hard. And then there are hard. And then there are hard. And then there are all kinds of all kinds of all kinds of selection columns that you selection columns that you selection columns that you can add on can add on can add on question difficulty.

question difficulty. For example, for For example, for easy, you can easy, you can easy, you can do a straight lookup do a straight lookup do a straight lookup or a simple or a simple or a simple factor definition, factor definition, factor definition, right? But hard right? But hard right? But hard can be if you can be if you can be if you want something more want something more want something more detailed or detailed or detailed or obscure. And obscure. And obscure. And then once we then once we then once we get to the columns get to the columns get to the columns generated by LLM, you generated by LLM, you generated by LLM, you just call this just call this just call this add add add column function and then column function and then column function and then you have your query you have your query you have your query which will essentially which will essentially which will essentially look something like look something like look something like this. Just this. Just this. Just trying to find it. trying to find it. Yeah. So you're an expert Yeah. So you're an expert at writing at writing at writing quiz questions. Hmm, so quiz questions. Hmm, so here's the topic, the difficulty here's the topic, the difficulty , the question type. So , the question type. So , the question type. So every time you every time you every time you add these add these add these topic templates, it will do what topic templates, it will do what topic templates, it will do what the topics that you the topics that you the topics that you created in created in created in the previous one, or in the the previous one, or in the the previous one, or in the original query original query original query that you had from your that you had from your that you had from your Wikipedia, it Wikipedia, it Wikipedia, it will insert them at will insert them at will insert them at runtime into this runtime into this runtime into this query. So basically query. So basically query. So basically every query that every query that every query that LLM sees is now LLM sees is now LLM sees is now unique and unique and unique and diverse, so diverse, so diverse, so the model will give you the model will give you the model will give you diverse data diverse data diverse data when you actually when you actually when you actually hit hit hit generate. Hmm, and generate. Hmm, and generate. Hmm, and then, hmm, you then, hmm, you then, hmm, you can also have something can also have something can also have something like a like a like a structured structured structured column. So a column. So a column. So a structured structured structured column is when column is when column is when instead of trying to instead of trying to instead of trying to generate text generate text generate text data, you can data, you can data, you can try to try to try to generate it in generate it in generate it in JSON or Markdown or JSON or Markdown or JSON or Markdown or whatever your whatever your whatever your own format is that own format is that own format is that you're interested in. We you're interested in. We you're interested in. We know that with LLM we know that with LLM we know that with LLM we have to be polite to them have to be polite to them have to be polite to them , we have to , we have to , we have to talk to them, but

talk to them, but talk to them, but sometimes they might not sometimes they might not sometimes they might not give you the give you the give you the structured structured structured format that you want. So we just format that you want. So we just format that you want. So we just delete those records. delete those records. So the way data creation So the way data creation works is this: in the works is this: in the works is this: in the data designer, there's a data designer, there's a data designer, there's a preview mode preview mode preview mode . Basically, . Basically, . Basically, when you click the when you click the when you click the preview button preview button preview button , the system , the system , the system tries to create tries to create tries to create three records for you, and three records for you, and three records for you, and then you can then you can then you can look at those look at those look at those records. Let's records. Let's records. Let's look at some look at some look at some records so that it all makes records so that it all makes records so that it all makes more sense. more sense. more sense. The topic here is the World Wide The topic here is the World Wide The topic here is the World Wide Web, and the question, Web, and the question, Web, and the question, or I guess some or I guess some World Wide Web consortium , the question is, , the question is, , the question is, which which which W3C recommendation W3C recommendation W3C recommendation is identified by the is identified by the is identified by the same three-letter same three-letter same three-letter abbreviation that abbreviation that abbreviation that serves as the serves as the serves as the airport code for airport code for airport code for some city in some city in some city in Norway. So, I have Norway. So, I have Norway. So, I have no way of knowing no way of knowing no way of knowing the answer to that the answer to that the answer to that question, right? question, right? question, right? That's why these questions are That's why these questions are That's why these questions are essentially complex, because essentially complex, because essentially complex, because you want to call a you want to call a you want to call a model for a search, model for a search, model for a search, right? But in this right? But in this right? But in this first notebook, we're first notebook, we're first notebook, we're just trying to just trying to just trying to create create create question-answer pairs. question-answer pairs. question-answer pairs. So, uh, this is a very So, uh, this is a very So, uh, this is a very simple dataset.

simple dataset. simple dataset. So, I think this is So, I think this is So, I think this is the answer that I the answer that I the answer that I got. model. And got. model. And got. model. And then you can also then you can also then you can also add LLM as an evaluator add LLM as an evaluator add LLM as an evaluator to evaluate how to evaluate how to evaluate how your model is doing. your model is doing. your model is doing. So this is the question that the So this is the question that the So this is the question that the model got. This is model got. This is model got. This is the answer that it the answer that it the answer that it got. And then, um, got. And then, um, got. And then, um, this is the quality of the question. this is the quality of the question. this is the quality of the question. So we can So we can So we can ask another ask another ask another model to evaluate what model to evaluate what model to evaluate what it thinks about it thinks about it thinks about the question that the the question that the the question that the previous model created. Uh, previous model created. Uh, previous model created. Uh, and so on. So, and so on. So, and so on. So, ultimately, you get ultimately, you get ultimately, you get this dataset. And this dataset. And this dataset. And by the way, we have a by the way, we have a by the way, we have a feature where feature where feature where you can just you can just you can just feed your feed your feed your datasets directly datasets directly datasets directly into Hugging Face. Uh, it's very into Hugging Face. Uh, it's very into Hugging Face. Uh, it's very easy to easy to easy to use. And, um, use. And, um, use. And, um, you can also you can also you can also visualize all visualize all visualize all your questions here, as your questions here, as your questions here, as you can see. And then you you can see. And then you you can see. And then you can get can get can get some statistics about some statistics about some statistics about your dataset. your dataset. your dataset. Obviously, we Obviously, we Obviously, we only created three records here. only created three records here. only created three records here. So, you know, these So, you know, these So, you know, these statistics are not very statistics are not very statistics are not very meaningful. But if you're meaningful. But if you're meaningful. But if you're creating hundreds or creating hundreds or creating hundreds or thousands of records, you thousands of records, you thousands of records, you can do that. And can do that. And can do that. And then we have something then we have something then we have something called batch called batch called batch mode. So this is data mode. So this is data mode. So this is data designer.create. And here, designer.create. And here, designer.create. And here, instead of 100, I can instead of 100, I can instead of 100, I can just, you know, just, you know, just, you know, put as many put as many put as many records as I records as I records as I need. And then I need. And then I need. And then I can create all these can create all these can create all these records. And as I records. And as I records. And as I mentioned earlier, we mentioned earlier, we mentioned earlier, we generate data generate data generate data column by column.

column by column. column by column. So in batch So in batch So in batch mode, first you mode, first you mode, first you generate the questions. And generate the questions. And generate the questions. And then you generate then you generate then you generate the answers, right? the answers, right? the answers, right? So this is obviously a So this is obviously a So this is obviously a very simple example. very simple example. very simple example. And then you have LLM as the And then you have LLM as the And then you have LLM as the judge. Um, and here's your judge. Um, and here's your judge. Um, and here's your final final final dataset. This is what it dataset. This is what it dataset. This is what it looks like. So that was the looks like. So that was the looks like. So that was the first notebook. I first notebook. I first notebook. I think it's pretty think it's pretty think it's pretty simple if simple if simple if you look at it you look at it you look at it and try it out. Um, and try it out. Um, and try it out. Um, we have a second notebook we have a second notebook we have a second notebook that's more that's more that's more focused on MCP. And focused on MCP. And focused on MCP. And then we're going to then we're going to then we're going to move on to a third move on to a third move on to a third notebook, which I notebook, which I notebook, which I think is going to be interesting think is going to be interesting think is going to be interesting for everyone because that's where for everyone because that's where for everyone because that's where we're going to talk about we're going to talk about we're going to talk about search. Excuse me? Yes. Oh, search. Excuse me? Yes. Oh, search. Excuse me? Yes. Oh, yes. All the yes. All the yes. All the GitHub links are the same. I'm GitHub links are the same. I'm GitHub links are the same. I'm just putting them on just putting them on just putting them on every slide. So every slide. So every slide. So if you missed if you missed if you missed the first one, you can the first one, you can the first one, you can find it. But I'll find it. But I'll find it. But I'll pause for a minute pause for a minute pause for a minute if you want to if you want to if you want to review. So review. So review. So let's talk let's talk let's talk about MCP. MCP is a standard about MCP. MCP is a standard that allows that allows an agent to connect an agent to connect to tools, to tools, to tools, right? So right? So right? So the tool here is the tool here is the tool here is Tavily, which is nothing more Tavily, which is nothing more Tavily, which is nothing more than a search tool than a search tool than a search tool for agents. There are for agents. There are for agents. There are many other many other many other search tools out there.

search tools out there. search tools out there. That's essentially what we That's essentially what we That's essentially what we need, what we're need, what we're need, what we're trying to teach the trying to teach the trying to teach the model, is how to model, is how to model, is how to do a search, do a search, do a search, right? So Tavily is a right? So Tavily is a right? So Tavily is a good tool good tool good tool for the model to for the model to for the model to learn how to do learn how to do learn how to do that search. that search. Going back to the Going back to the second notebook, so second notebook, so second notebook, so here we're just going to here we're just going to here we're just going to go over some go over some go over some simple concepts from the simple concepts from the simple concepts from the first notebook and first notebook and first notebook and add in how to create add in how to create add in how to create agent data. So agent data. So agent data. So again, you have again, you have again, you have the import, you have the the import, you have the the import, you have the model configurations, model configurations, model configurations, and then the same and then the same and then the same set of initial set of initial set of initial data, the same data, the same data, the same sample columns, so the sample columns, so the sample columns, so the question difficulty, the question difficulty, the question difficulty, the question type, and so on. And question type, and so on. And question type, and so on. And then you actually then you actually then you actually create the question, and create the question, and create the question, and then at the end you then at the end you then at the end you get something get something get something like a configuration, like a configuration, like a configuration, right? I don't know why it's right? I don't know why it's right? I don't know why it's printed in, um, printed in, um, printed in, um, light color, but light color, but light color, but yeah, that's essentially your yeah, that's essentially your yeah, that's essentially your graph, um, that you graph, um, that you graph, um, that you create. And then create. And then create. And then you have the answer. So it's the same thing you have the answer. So it's the same thing you have the answer. So it's the same thing . Only . Only . Only now when you now when you now when you actually create actually create actually create the answer, you add the answer, you add the answer, you add here's the here's the here's the tool alias. And so tool alias. And so tool alias. And so the goal is that the goal is that the goal is that instead of instead of instead of just trying to just trying to just trying to answer the answer the answer the question, the model question, the model question, the model will now use the will now use the Tavily search tool to try Tavily search tool to try to find the answer to find the answer to find the answer online, right? So online, right? So online, right? So this is what we're building, this is what we're building, this is what we're building, we're building on the we're building on the we're building on the first notebook where we first notebook where we first notebook where we just said, " just said, " Okay, please Okay, please Okay, please generate the question, and generate the question, and generate the question, and then please then please then please generate the answer generate the answer ." Now please ." Now please ." Now please generate the question, and generate the question, and generate the question, and then, "Hey, you have

then, "Hey, you have then, "Hey, you have this tool. this tool. this tool. Can you Can you Can you use this use this use this tool to tool to tool to try to find try to find try to find the answer, instead of the answer, instead of the answer, instead of trying trying trying to generate to generate to generate the answer." So that's the answer." So that's the answer." So that's the difference between the first notebook and the the difference between the first notebook and the the difference between the first notebook and the second notebook. And second notebook. And second notebook. And again, you have a again, you have a again, you have a preview mode preview mode preview mode where you where you where you can generate can generate can generate a few records and a few records and a few records and then you can review then you can review then you can review your records. And by the way your records. And by the way , data creation is , data creation is , data creation is very iterative. It's very iterative. It's very iterative. It's not like it's the not like it's the not like it's the same thing. So same thing. So same thing. So when you create when you create when you create some data, you some data, you some data, you review it. If review it. If review it. If you're not happy, then you're not happy, then you're not happy, then you probably change you probably change you probably change your workflow, your workflow, your workflow, change your change your change your prompts, and then, prompts, and then, prompts, and then, you know, iterate them you know, iterate them . And then when you're . And then when you're . And then when you're happy, you can happy, you can happy, you can send them to send them to send them to batch execution. batch execution. batch execution. So let's So let's So let's see what the see what the see what the second second second notebook produced. So notebook produced. So notebook produced. So let's say the question let's say the question let's say the question is: To which is: To which is: To which German city did German city did German city did Telefunken move its Telefunken move its Telefunken move its headquarters in headquarters in headquarters in 1949? Does anyone know 1949? Does anyone know 1949? Does anyone know the answer? No, really? the answer? No, really? the answer? No, really? I mean, I don't I mean, I don't I mean, I don't know the answer. So know the answer. So we want to create we want to create complex questions that complex questions that complex questions that involve involve involve reasoning and calling out reasoning and calling out reasoning and calling out tools from the tools from the tools from the model. And when the model model. And when the model model. And when the model tries tries tries to answer that, as to answer that, as to answer that, as you can see, it you can see, it you can see, it uses Tavily.

uses Tavily. uses Tavily. The first search The first search The first search it does is " it does is " it does is " Telefunken moved its Telefunken moved its Telefunken moved its headquarters in headquarters in headquarters in 1949 to which company 1949 to which company ?" ". So basically it ?" ". So basically it ?" ". So basically it uses one uses one uses one call to the call to the call to the search engine. And then, um, search engine. And then, um, search engine. And then, um, it gets it gets it gets the response. So this is the response. So this is the response. So this is the query from the model, and here are the query from the model, and here are the query from the model, and here are the results. These are all the the results. These are all the URLs. And then for URLs. And then for URLs. And then for each URL it each URL it each URL it will give you will give you will give you the title. And then the title. And then the title. And then it will also it will also it will also give you the content again. give you the content again. give you the content again. So this is what Tavily gives us So this is what Tavily gives us So this is what Tavily gives us , right? If you , right? If you , right? If you make a query, or this is what it make a query, or this is what it make a query, or this is what it gives the agent gives the agent gives the agent when the agent makes a when the agent makes a when the agent makes a query. And then you query. And then you query. And then you get the response. get the response. get the response. And finally, at the end, you And finally, at the end, you And finally, at the end, you can build your can build your can build your dataset in dataset in dataset in batch mode, like batch mode, like batch mode, like we did last we did last we did last time. And if you time. And if you time. And if you think it think it think it just flew just flew just flew through all of these first through all of these first through all of these first two notebooks, that's because the whole point of two notebooks, that's because the whole point of this is this is this is in the third in the third in the third notebook, right? notebook, right? notebook, right? So, uh, that's what So, uh, that's what So, uh, that's what I want us to I want us to focus on today. So focus on today. So let's talk let's talk let's talk about search. We just about search. We just about search. We just went through a couple of went through a couple of went through a couple of notebooks that notebooks that notebooks that show us how show us how show us how to use the to use the to use the data designer, and data designer, and data designer, and that's good, but what that's good, but what that's good, but what problem are we problem are we problem are we actually actually actually trying to solve trying to solve ? So how do we I ? So how do we I ? So how do we I mentioned earlier, I'm a mentioned earlier, I'm a mentioned earlier, I'm a research research research scientist at NVIDIA, and scientist at NVIDIA, and scientist at NVIDIA, and we're also part of the we're also part of the we're also part of the NeMo Tron team. So, NeMo Tron team. So, NeMo Tron team. So, NeMo Tron is a collection of NeMo Tron is a collection of open open source, underlying models that we've source, underlying models that we've source, underlying models that we've built. We just built. We just built. We just released our released our released our latest model, Ultra, latest model, Ultra, latest model, Ultra, but we also have two but we also have two but we also have two other models. So other models. So other models. So we have models in three we have models in three we have models in three different sizes. And different sizes. And different sizes. And so, as part of this so, as part of this so, as part of this work, we had to work, we had to work, we had to train NeMo Tron train NeMo Tron train NeMo Tron to do search, and to do search, and to do search, and that's really important, that's really important, that's really important, right? Because it might right? Because it might right? Because it might seem like a long time ago, but seem like a long time ago, but seem like a long time ago, but maybe two or three

maybe two or three maybe two or three years ago, I'm years ago, I'm years ago, I'm sure you all sure you all sure you all remember remember remember looking at, I don't know looking at, I don't know , Chat GPT, Claude, or Gemini, and , Chat GPT, Claude, or Gemini, and , Chat GPT, Claude, or Gemini, and then you look at the web then you look at the web interface, and it interface, and it interface, and it says, " says, " says, " Knowledge collection end date is Knowledge collection end date is December 2023." So December 2023." So December 2023." So the reason it says that is the reason it says that is because, well, you because, well, you have to start have to start have to start training the model at a training the model at a training the model at a certain point. So certain point. So certain point. So you have to you have to you have to stop collecting and stop collecting and stop collecting and generating data at a generating data at a generating data at a certain point, certain point, certain point, right? So let's say right? So let's say right? So let's say they train the model they train the model they train the model in January 2024, until in January 2024, until in January 2024, until December 2023, they December 2023, they December 2023, they need to essentially need to essentially need to essentially freeze the data. So freeze the data. So freeze the data. So anything the model knows anything the model knows anything the model knows after December 2023 after December 2023 after December 2023 is not part of its is not part of its is not part of its training data. So training data. So if you need to if you need to if you need to ask a question ask a question ask a question like, "What's like, "What's like, "What's the weather like today?" ", the weather like today?" ", the weather like today?" ", the model can't possibly the model can't possibly the model can't possibly answer that answer that answer that question from its question from its question from its training data, training data, training data, right? Uh, question? right? Uh, question? Yeah. So, yeah, Yeah. So, yeah, definitely definitely definitely scan the QR code. scan the QR code. scan the QR code. So the idea is So the idea is So the idea is , how do , how do , how do you actually train a model on you actually train a model on you actually train a model on current events?

current events? current events? because learning about current because learning about current because learning about current events seems to be just as events seems to be just as events seems to be just as important, if not more important, if not more important, if not more important, than important, than important, than learning about, uh, things from learning about, uh, things from learning about, uh, things from the past, right? the past, right? So, uh, that's why So, uh, that's why search exists, and that's search exists, and that's search exists, and that's why models need to why models need to why models need to learn how to do learn how to do learn how to do search. Uh. So, search. Uh. So, search. Uh. So, going back to going back to going back to the slide, uh, teaching the the slide, uh, teaching the the slide, uh, teaching the Nematron model Nematron model Nematron model to use to use to use search tools search tools . That was our . That was our . That was our motivation. And, uh, that's why motivation. And, uh, that's why motivation. And, uh, that's why the model needs the model needs the model needs access to access to access to real- real- real- time information to be able to time information to be able to time information to be able to do useful things. do useful things. do useful things. So, how did we So, how did we So, how did we actually do that? actually do that? actually do that? So, before I So, before I So, before I get to how we get to how we get to how we did that, we did that, we did that, we need something to need something to need something to measure measure measure progress against, right? progress against, right? progress against, right? So, uh, in the So, uh, in the So, uh, in the baseline model community, when baseline model community, when baseline model community, when you need to you need to you need to measure progress on measure progress on measure progress on something, you something, you something, you use a use a use a benchmark, which is a benchmark, which is a benchmark, which is a standardized standardized standardized test set with no test set with no test set with no answers, right? answers, right? answers, right? It's like we don't have It's like we don't have It's like we don't have the answers. So the answers. So the answers. So one of the benchmarks one of the benchmarks one of the benchmarks for search is BrowseComp, for search is BrowseComp, uh, which is a very complex uh, which is a very complex uh, which is a very complex benchmark. And, uh, the benchmark. And, uh, the World Cup is on right now World Cup is on right now . So, I don't . So, I don't . So, I don't know how many know how many know how many soccer fans there are here, soccer fans there are here, soccer fans there are here, but I'm a big but I'm a big but I'm a big soccer fan. But soccer fan. But soccer fan. But we don't call it we don't call it we don't call it soccer. But, um, between soccer. But, um, between soccer. But, um, between 1990 and 1994, 1990 and 1994, 1990 and 1994, teams that played in a teams that played in a teams that played in a soccer match with a soccer match with a soccer match with a Brazilian referee Brazilian referee Brazilian referee got four yellow got four yellow got four yellow cards, two from cards, two from cards, two from each team, each team, each team, three in total, blah blah three in total, blah blah blah. I'm not going to read blah. I'm not going to read blah. I'm not going to read the rest of the question, but it's a the rest of the question, but it's a the rest of the question, but it's a pretty complex pretty complex pretty complex question, isn't it?

question, isn't it? question, isn't it? None of us know None of us know None of us know the answer to that the answer to that the answer to that question, or maybe question, or maybe question, or maybe there's someone who, um, is really there's someone who, um, is really there's someone who, um, is really good at it good at it good at it and and and has the answer in their has the answer in their has the answer in their head. What we used to head. What we used to head. What we used to do before, do before, do before, like three or four like three or four like three or four years ago, is we years ago, is we years ago, is we tried to break tried to break tried to break the problem down into pieces, the problem down into pieces, the problem down into pieces, if we would have a if we would have a if we would have a complex, um, query complex, um, query complex, um, query like this, and then like this, and then like this, and then we would think about what we would think about what we would think about what queries we actually queries we actually queries we actually needed to ask. needed to ask. needed to ask. We would search on Google or We would search on Google or We would search on Google or somewhere, somewhere, somewhere, access additional access additional access additional information, and then we would information, and then we would information, and then we would think about think about think about what our what our what our original goal was, original goal was, original goal was, what information we what information we what information we accessed, and accessed, and accessed, and then we would go then we would go then we would go from there and finally from there and finally from there and finally get to the get to the get to the answer, right? answer, right? answer, right? So the answer is So the answer is Ireland versus Ireland versus Ireland versus Romania. I didn't Romania. I didn't Romania. I didn't know that. So how did we know that. So how did we know that. So how did we actually create a actually create a actually create a dataset that dataset that dataset that allows us to train a allows us to train a allows us to train a model to go model to go model to go from 0% to 30% on this from 0% to 30% on this from 0% to 30% on this BrowseCom benchmark? BrowseCom benchmark? So that's what we're So that's what we're going to learn about today. going to learn about today. going to learn about today. So to do that, So to do that, So to do that, we did something, uh, we did something, uh, we did something, uh, we used the seed we used the seed we used the seed dataset dataset dataset that I described earlier.

that I described earlier. that I described earlier. But our seed But our seed But our seed dataset is not dataset is not dataset is not just Wikipedia. It's just Wikipedia. It's just Wikipedia. It's something that is essentially a something that is essentially a something that is essentially a graph of graph of graph of Wikipedia topics. So it's Wikipedia topics. So it's Wikipedia topics. So it's called Wikidata called Wikidata . And we took a few . And we took a few . And we took a few steps along that steps along that steps along that graph, and that's our graph, and that's our graph, and that's our seed seed seed dataset. And I know dataset. And I know dataset. And I know the diagram on the right the diagram on the right the diagram on the right looks like a tree. looks like a tree. looks like a tree. It's not a tree. It's It's not a tree. It's It's not a tree. It's a graph. So if it a graph. So if it a graph. So if it looks like a tree, looks like a tree, looks like a tree, it's my fault. Uh, it it's my fault. Uh, it it's my fault. Uh, it should be, it should be should be, it should be should be, it should be a graph. Um, and basically, a graph. Um, and basically, a graph. Um, and basically, uh, you can see, uh, you can see, uh, you can see, uh, on the right, the first uh, on the right, the first uh, on the right, the first topic is Nvidia. And then Nvidia topic is Nvidia. And then Nvidia topic is Nvidia. And then Nvidia is related to Jensen is related to Jensen , who is our , who is our CEO. And then CEO. And then it's related to it's related to Oregon State University, because Wikidata Oregon State University, because Wikidata says that's where he says that's where he says that's where he went to college. And went to college. And went to college. And then this college then this college then this college is located in is located in is located in Benton County, and Benton County, and Benton County, and Benton County is Benton County is actually named actually named after a man after a man after a man named Thomas Hart named Thomas Hart named Thomas Hart Benton. So if you Benton. So if you Benton. So if you take, uh, if you take, uh, if you take, uh, if you randomly pick randomly pick randomly pick an object in Wikidata, and an object in Wikidata, and an object in Wikidata, and then you do five to then you do five to eight passes on eight passes on eight passes on it, and I'll explain why it, and I'll explain why , five to eight, you , five to eight, you , five to eight, you get, uh, a get, uh, a get, uh, a pass like this pass like this pass like this green, you know, green, you know, green, you know, set of objects with the set of objects with the set of objects with the same arrows same arrows same arrows between them. So this is, by the way between them. So this is, by the way between them. So this is, by the way essentially our essentially our essentially our initial data set for initial data set for initial data set for creating, uh, creating, uh, creating, uh, search puzzles.

search puzzles. search puzzles. So one search So one search So one search puzzle that I could puzzle that I could puzzle that I could create from this, and create from this, and create from this, and I'll have to I'll have to I'll have to read this from my read this from my read this from my speaker notes, speaker notes, speaker notes, is, uh, what company is, uh, what company is, uh, what company was founded by was founded by was founded by someone who someone who someone who went to college went to college went to college where the district was named where the district was named where the district was named after Thomas Hart after Thomas Hart after Thomas Hart Benton? So there's no way LLM Benton? So there's no way LLM Benton? So there's no way LLM can know can know can know the answer to this the answer to this the answer to this question in the training question in the training question in the training data, right? Because data, right? Because data, right? Because if we give the agent if we give the agent if we give the agent the opportunity the opportunity the opportunity to cheat, it's to cheat, it's to cheat, it's going to cheat. It's going to cheat. It's going to cheat. It's just going to give us just going to give us just going to give us the answer. It's not the answer. It's not the answer. It's not going to use going to use going to use tool calls tool calls tool calls or a tool, or a tool, or a tool, excuse me, like excuse me, like excuse me, like search that we want to search that we want to search that we want to use, use, use, right? So we want to right? So we want to right? So we want to make the questions make the questions make the questions as complex as possible as complex as possible as complex as possible and probably and probably and probably puzzle-like so that we can puzzle-like so that we can puzzle-like so that we can trigger trigger trigger search behavior from the model. So that's search behavior from the model. So that's search behavior from the model. So that's our initial our initial our initial value. So I value. So I value. So I talked about the first talked about the first talked about the first step, which is to step, which is to step, which is to create random create random create random walks with the initial walks with the initial walks with the initial value. And why value. And why value. And why four to eight four to eight four to eight or five to or five to or five to eight transitions? eight transitions? eight transitions? Sometimes it's just, Sometimes it's just, Sometimes it's just, you know, if you you know, if you you know, if you do these transitions, do these transitions, do these transitions, it's just completely it's just completely it's just completely off off off topic. Like, he topic. Like, he topic. Like, he says, um, I don't know, says, um, I don't know, says, um, I don't know, like, the like, the like, the person's name, and then he says, person's name, and then he says, person's name, and then he says, well, the person. Well, well, the person. Well, well, the person. Well, of course, that's true, of course, that's true, of course, that's true, but, you know, it's but, you know, it's but, you know, it's not really not really not really related. Basically, related. Basically, related. Basically, your transitions your transitions your transitions are going to be off are going to be off are going to be off topic. So, that's why topic. So, that's why topic. So, that's why we want to, as a we want to, as a we want to, as a heuristic, heuristic, heuristic, stay between stay between stay between four and eight.

four and eight. four and eight. That's what That's what That's what worked well for me. worked well for me. worked well for me. And then you And then you And then you turn those turn those turn those seed values seed values seed values into search puzzles, into search puzzles, into search puzzles, and then, um, you and then, um, you and then, um, you do the search using a do the search using a search tool, search tool, which is Tavily. So, that's which is Tavily. So, that's which is Tavily. So, that's how you create how you create how you create a dataset with a a dataset with a a dataset with a query, a query, and a query, a query, and a query, a query, and a search answer, search answer, search answer, right? And then we right? And then we right? And then we evaluate our data, and evaluate our data, and evaluate our data, and then something that's not then something that's not then something that's not mentioned here is that mentioned here is that mentioned here is that we train our we train our we train our models, right? models, right? models, right? So once you So once you So once you have the dataset, you have the dataset, you have the dataset, you train the model on train the model on train the model on it, and it, and it, and then you measure then you measure then you measure the performance against a the performance against a the performance against a standardized standardized standardized benchmark. So, benchmark. So, benchmark. So, let's look let's look let's look at the notebook and at the notebook and at the notebook and let's see how we let's see how we let's see how we can do that. Are there can do that. Are there can do that. Are there any questions at this point? any questions at this point? Yeah. So, uh. So the reason why it So the reason why it was important to do this, was important to do this, was important to do this, I think, is that these are not hard I think, is that these are not hard I think, is that these are not hard questions. These are questions questions. These are questions questions. These are questions that the model can't that the model can't that the model can't easily find easily find easily find answers to answers to answers to online, can't online, can't online, can't easily find easily find easily find answers to answers to answers to online. So if online. So if online. So if you take this whole you take this whole you take this whole block of text that block of text that block of text that we had in the we had in the we had in the browse.com benchmark and put browse.com benchmark and put browse.com benchmark and put it online, you're it online, you're it online, you're not going to get an not going to get an not going to get an answer. I answer. I answer. I mean, you're going to mean, you're going to mean, you're going to get an answer get an answer get an answer from the from the from the AI ​​search in AI ​​search in AI ​​search in Gemini right now, but that's what Gemini right now, but that's what Gemini right now, but that's what we're essentially we're essentially we're essentially trying to teach trying to teach trying to teach the model. Yeah. Okay.

the model. Yeah. Okay. So this is a So this is a tutorial for tutorial for tutorial for end-to-end end-to-end end-to-end agent search. We've agent search. We've agent search. We've talked about talked about talked about how we do this and how how we do this and how how we do this and how we create the data. We've we create the data. We've we create the data. We've talked about talked about talked about how we import all how we import all how we import all these prompts, and I these prompts, and I these prompts, and I wrote a function to wrote a function to wrote a function to use Tavily with MCP. use Tavily with MCP. use Tavily with MCP. You can You can You can look at the code look at the code look at the code later. Um, and here's where later. Um, and here's where later. Um, and here's where we have our initial we have our initial we have our initial entities. Here's the example entities. Here's the example that I showed you. that I showed you. that I showed you. This is just one This is just one This is just one initial entity. initial entity. initial entity. So, my seeding algorithm So, my seeding algorithm works in a way works in a way that you can that you can that you can run it a run it a run it a million times and you'll million times and you'll million times and you'll get a million get a million get a million different traversals of different traversals of different traversals of Wikidata, right? Wikidata, right? Wikidata, right? These are just the themes that These are just the themes that These are just the themes that we want to we want to we want to base base base our search our search our search puzzle on. So another puzzle on. So another puzzle on. So another theme—we theme—we theme—we like modern like modern like modern art, and I'm not really art, and I'm not really art, and I'm not really into into into art, but I don't art, but I don't art, but I don't understand most of understand most of understand most of these words, but basically these words, but basically these words, but basically this is another test this is another test this is another test case that you case that you case that you can see. So can see. So the way we build the way we build the way we build this dataset is this dataset is this dataset is this: you're a this: you're a this: you're a search expert who search expert who search expert who uses web uses web search to accurately search to accurately search to accurately answer answer answer questions, providing questions, providing questions, providing valid JSON with valid JSON with valid JSON with final final URLs to URLs to URLs to validate the validate the validate the answer and a answer and a answer and a short short short justification. Here are justification. Here are justification. Here are the tools that the tools that the tools that you have. You have one you have. You have one you have. You have one tool— tool— tool— Tavily search. And here are some Tavily search. And here are some Tavily search. And here are some guidelines for guidelines for guidelines for using that using that using that tool. We tool. We tool. We say, "Okay, say, "Okay, say, "Okay, please don't

please don't please don't use use use more than, you know, more than, you know, more than, you know, call the call the call the tool a hundred times tool a hundred times tool a hundred times and don't spend, and don't spend, and don't spend, you know, a thousand you know, a thousand you know, a thousand dollars or something dollars or something ." I guess these are all kinds of ." I guess these are all kinds of ." I guess these are all kinds of instructions that you instructions that you instructions that you can put in there. can put in there. can put in there. But basically, joke. But basically, joke. So what you want So what you want is that your agent is that your agent is that your agent doesn't go crazy and doesn't go crazy and doesn't go crazy and try to spend try to spend try to spend too much money too much money too much money answering a very answering a very answering a very simple question or simple question or simple question or something. That's why something. That's why something. That's why we have all these we have all these we have all these instructions. And we instructions. And we instructions. And we also have an example. It's, also have an example. It's, also have an example. It's, um, kind of like um, kind of like um, kind of like contextual contextual contextual learning, right? You learning, right? You learning, right? You add an example to the add an example to the add an example to the query, and the model can query, and the model can query, and the model can essentially, um, learn essentially, um, learn essentially, um, learn from that example how to from that example how to from that example how to actually invoke the actually invoke the actually invoke the search tool. search tool. Here's your query that you Here's your query that you want to create. So want to create. So , we collect the , we collect the , we collect the Wiki transitive data. Wiki transitive data. We can insert this We can insert this transitive data into transitive data into transitive data into the query here, um, this is our, the query here, um, this is our, the query here, um, this is our, um, well, you know, um, well, you know, um, well, you know, seed entity. And seed entity. And seed entity. And then we ask then we ask then we ask the model to formulate the model to formulate the model to formulate a question a question a question using using any or all of the seed any or all of the seed any or all of the seed entities from this entities from this entities from this transitive transitive . And then we ask . And then we ask . And then we ask the model to generate the model to generate the model to generate an answer an answer an answer using, um, using, um, using, um, this agent decision diagram this agent decision diagram this agent decision diagram that I that I that I showed you earlier. So showed you earlier. So showed you earlier. So in the second notebook.

in the second notebook. in the second notebook. So once you So once you So once you do that, you'll have a do that, you'll have a preview dataset. Here's our preview dataset. Indian preview dataset. Indian medical medical medical research council is the research council is the research council is the initial entity initial entity initial entity for the initial for the initial for the initial element. And obviously element. And obviously element. And obviously it's located in a it's located in a it's located in a city called city called city called Bhubaneswar, which is Bhubaneswar, which is Bhubaneswar, which is the capital of some the capital of some the capital of some state in India. So that's state in India. So that's state in India. So that's our initial our initial our initial path. And here's the question path. And here's the question path. And here's the question that came out of the that came out of the simulation. This simulation. This agent decision string agent decision string agent decision string isn't isn't isn't shown here, but shown here, but shown here, but if you look at if you look at if you look at it, we can see what it, we can see what it, we can see what it looks like. So it looks like. So it looks like. So it's done a bunch of it's done a bunch of tool calls, it's tool calls, it's thought about the question, and thought about the question, and thought about the question, and then it tries to then it tries to then it tries to find the answer find the answer find the answer online. So that's how you online. So that's how you online. So that's how you create a create a create a search tool called a " search tool called a " search tool called a " dataset." And I guess dataset." And I guess I'll pause here I'll pause here I'll pause here for questions. And then for questions. And then for questions. And then we can move on to the we can move on to the we can move on to the next few next few next few slides. Any questions?

slides. Any questions? slides. Any questions? If anything was "Yes." If anything was "Yes." Yes. Absolutely. Yes. Yes. Absolutely. Yes. So we moved the So we moved the Bird benchmark from 26% to 41%. So you Bird benchmark from 26% to 41%. So you asked, you know, asked, you know, asked, you know, absolutely the right absolutely the right absolutely the right question. And we also question. And we also question. And we also have a blog post about that have a blog post about that . You can read . You can read . You can read about that. So, for all the about that. So, for all the about that. So, for all the datasets that we've datasets that we've datasets that we've created using the created using the data designer, we have there are data designer, we have there are blog posts. And you can blog posts. And you can blog posts. And you can see right here. see right here. see right here. So I was also So I was also So I was also involved in this work. involved in this work. involved in this work. Yes, it's about Yes, it's about Yes, it's about SQL here. And I can also SQL here. And I can also SQL here. And I can also share the link with you share the link with you share the link with you later. later. later. Hmm, but basically, the Hmm, but basically, the Hmm, but basically, the data designer can be data designer can be data designer can be used to used to used to create any create any create any dataset for dataset for dataset for training AI models.

training AI models. training AI models. Maybe other Maybe other Maybe other datasets too, if you're datasets too, if you're datasets too, if you're looking for test looking for test looking for test datasets that are not datasets that are not datasets that are not related to training related to training related to training models, etc. Hmm, and models, etc. Hmm, and models, etc. Hmm, and we have a bunch of we have a bunch of data tools that data tools that we've built in our we've built in our we've built in our team, in addition to the team, in addition to the team, in addition to the data designer, data designer, data designer, which I'll talk about which I'll talk about which I'll talk about at the end. So the SOTA at the end. So the SOTA at the end. So the SOTA on Bird is 70 on Bird is 70 on Bird is 70 because it's agent-based. because it's agent-based. because it's agent-based. Our, um, current Our, um, current Our, um, current next next next dataset is not agent-based. And dataset is not agent-based. And , um, also the SOTA on , um, also the SOTA on , um, also the SOTA on benchmarks is very benchmarks is very benchmarks is very model-dependent. model-dependent. We're talking about We're talking about porting, um, Nemotron porting, um, Nemotron porting, um, Nemotron Nano, which is our Nano, which is our Nano, which is our little, um, LLM or little, um, LLM or little, um, LLM or SLM. I don't know if you can SLM. I don't know if you can SLM. I don't know if you can call it SLM because call it SLM because call it SLM because it's MOE, but it's LLM. So, um it's MOE, but it's LLM. So, um , for the model this , for the model this , for the model this size is pretty size is pretty size is pretty decent decent decent performance on Bird. performance on Bird. performance on Bird. Yeah. Question? Yeah.

Yeah. Question? Yeah. Yeah. Question? Yeah. No, it's a deterministic No, it's a deterministic No, it's a deterministic algorithm to algorithm to algorithm to create the initial create the initial create the initial data. And then the data. And then the data. And then the tool calls tool calls , and the agent part , and the agent part , and the agent part is to create the is to create the is to create the actual search, actual search, actual search, like doing the like doing the like doing the actual search actual search actual search when you have a question, when you have a question, when you have a question, and creating and creating and creating the trajectory. We the trajectory. We the trajectory. We call it a call it a call it a trajectory. So trajectory. So trajectory. So creating the creating the creating the search trajectory. It's search trajectory. It's search trajectory. It's deterministic. It's deterministic. It's deterministic. It's just what I just what I just what I came up with to came up with to came up with to create the initial create the initial create the initial data. Yeah. You can data. Yeah. You can data. Yeah. You can have any have any have any initial data. initial data. initial data. Maybe you, you know, Maybe you, you know, Maybe you, you know, can think of a can think of a can think of a better way better way better way to create the initial to create the initial to create the initial data than this. Oh, data than this. Oh, data than this. Oh, great question. So great question. So great question. So we could do that. we could do that. we could do that. We created We created We created datasets that can be datasets that can be datasets that can be used in the used in the used in the pre-training and pre-training and pre-training and post-training post-training post-training process. My main process. My main process. My main focus is on the focus is on the focus is on the post-training post-training post-training process. Um, or, process. Um, or, process. Um, or, you know, creating you know, creating you know, creating datasets for the datasets for the datasets for the post-training post-training post-training process. So this process. So this process. So this particular, um, I particular, um, I particular, um, I think the project was SFT, think the project was SFT, think the project was SFT, um, in the um, in the um, in the post-training post-training post-training process. Yeah. There's no process. Yeah. There's no process. Yeah. There's no reason other than reason other than reason other than the fact that, um, I the fact that, um, I the fact that, um, I think someone at Nvidia think someone at Nvidia think someone at Nvidia made a deal with Tavily to made a deal with Tavily to made a deal with Tavily to get credits. You get credits. You get credits. You can can can use Exa. Exa use Exa. Exa is another is another search agent provider, um , I think. Yeah, that's also , I think. Yeah, that's also . Yeah. Basically. So the "

. Yeah. Basically. So the " golden" answers golden" answers golden" answers are generated by Data Designer are generated by Data Designer are generated by Data Designer using using using the model that you chose. And the model that you chose. And the model that you chose. And so, in the first so, in the first so, in the first notebook... Yeah, and notebook... Yeah, and notebook... Yeah, and sorry, I think, you sorry, I think, you sorry, I think, you know, I was a little know, I was a little know, I was a little rushed because we rushed because we rushed because we only have an hour for this only have an hour for this only have an hour for this seminar. The previous seminar. The previous seminar. The previous version of this version of this version of this seminar was 2 seminar was 2 seminar was 2 hours. So the model hours. So the model hours. So the model that I used is that I used is that I used is GPT-OSS-120B for GPT-OSS-120B for GPT-OSS-120B for data generation. But data generation. But data generation. But Data Designer is not just a Data Designer is not just a Data Designer is not just a set of LLMs that you can set of LLMs that you can set of LLMs that you can use to use to use to create data. It create data. It create data. It also has a bunch of also has a bunch of also has a bunch of LLMs and non-LLMs LLMs and non-LLMs LLMs and non-LLMs that work together that work together that work together to help you to help you to help you create create create all kinds of data. all kinds of data. Say that again. Is that what you Is that what you mean? Yeah. Yeah, yeah. mean? Yeah. Yeah, yeah. Yeah. Yeah. Yeah, you Yeah. Yeah. Yeah, you distill, but in distill, but in distill, but in some cases you some cases you some cases you can even go can even go can even go beyond distillation framework.

beyond distillation framework. beyond distillation framework. And that's what I'm trying to And that's what I'm trying to And that's what I'm trying to do, because we do, because we do, because we built this built this built this tool that tool that tool that allows you to create a allows you to create a allows you to create a variety of data. variety of data. variety of data. For example, in a completely For example, in a completely For example, in a completely different use case that different use case that I've been working on, GPT-OSS I've been working on, GPT-OSS gets a certain gets a certain gets a certain percentage in this percentage in this percentage in this benchmark called benchmark called benchmark called StructEVAL, which is a StructEVAL, which is a StructEVAL, which is a way of measuring how well way of measuring how well way of measuring how well models perform in models perform in models perform in JSON, markdown, XML, JSON, markdown, XML, JSON, markdown, XML, YAML, and CSV. And NeMo-Tron YAML, and CSV. And NeMo-Tron YAML, and CSV. And NeMo-Tron is actually a is actually a is actually a distillation of GPT-OSS, distillation of GPT-OSS, distillation of GPT-OSS, but it outperforms but it outperforms but it outperforms GPT-OSS. GPT-OSS. GPT-OSS. So it's a distillation, So it's a distillation, So it's a distillation, but it can also but it can also but it can also be more powerful be more powerful be more powerful because of all the because of all the because of all the things that we've things that we've things that we've built into Data Designer. built into Data Designer. built into Data Designer. Yes. You can Yes. You can Yes. You can use Data use Data use Data Designer for RL. By the way, we're Designer for RL. By the way, we're Designer for RL. By the way, we're doing this for search right now doing this for search right now doing this for search right now , or the , or the , or the NVIDIA team is doing this for NVIDIA team is doing this for NVIDIA team is doing this for search. So instead of search. So instead of search. So instead of trying to trying to trying to create a golden create a golden create a golden solution like he solution like he solution like he described it, or, you know, described it, or, you know, described it, or, you know, do search do search do search puzzles like it's written puzzles like it's written puzzles like it's written on my slide, you on my slide, you on my slide, you just don't do that just don't do that just don't do that step, and then you step, and then you step, and then you essentially get a essentially get a essentially get a dataset full of dataset full of dataset full of clues, and then clues, and then clues, and then we have a tool we have a tool we have a tool called NeMo Gym, which is essentially called NeMo Gym, which is essentially called NeMo Gym, which is essentially is an is an is an RL reward function. You know, RL reward function. You know, RL reward function. You know, I think it's a I think it's a I think it's a tool that tool that tool that essentially trains your essentially trains your essentially trains your models to do RL, models to do RL, models to do RL, but the thing is, in NeMo but the thing is, in NeMo but the thing is, in NeMo Gym, you write something Gym, you write something Gym, you write something like a function that like a function that like a function that essentially gives you a essentially gives you a essentially gives you a reward, and then reward, and then reward, and then you can you can you can use our use our RL training framework to do it in RL training framework to do it in real real real time with these

time with these time with these tools, and then tools, and then tools, and then provide the reward provide the reward provide the reward in RL. That's a great idea. in RL. That's a great idea. in RL. That's a great idea. We're thinking about We're thinking about We're thinking about how to do that now, but how to do that now, but how to do that now, but building something building something building something universal is the universal is the universal is the challenge, right challenge, right ? We don't want to ? We don't want to ? We don't want to build something build something build something specific to specific to specific to one one one use case. Yeah. use case. Yeah. Yeah. Is that relevant Yeah. Is that relevant and accurate for ETL? I have and accurate for ETL? I have and accurate for ETL? I have n't tried it myself, n't tried it myself, n't tried it myself, but I think it is, because it but I think it is, because it but I think it is, because it should be, because we've should be, because we've should be, because we've built a built a built a diverse set of diverse set of diverse set of datasets using the datasets using the data designer. And I think data designer. And I think we showed we showed we showed about 6.5 trillion about 6.5 trillion about 6.5 trillion tokens early on. And if that's tokens early on. And if that's tokens early on. And if that's not reliable, you should not reliable, you should not reliable, you should talk to us and talk to us and talk to us and see how we see how we see how we can help you.

can help you. I can't hear you. Sorry. Sorry, I can't hear you. Oh, I don't know about that. I'll... I'll come back to you at I'll come back to you at I'll come back to you at the end and we can the end and we can talk more. Yeah, talk more. Yeah, sorry about that. Okay. sorry about that. Okay. sorry about that. Okay. We have about 15 We have about 15 We have about 15 minutes left. So let minutes left. So let minutes left. So let me move on to the me move on to the me move on to the next questions, and at next questions, and at next questions, and at the end we can the end we can the end we can ask a few more ask a few more ask a few more questions about something questions about something questions about something really interesting, which is really interesting, which is really interesting, which is evaluating your evaluating your evaluating your synthetic data, synthetic data, synthetic data, right? And that's not easy right? And that's not easy right? And that's not easy to do. Because without to do. Because without to do. Because without that evaluation step, the that evaluation step, the that evaluation step, the synthetic data is not synthetic data is not synthetic data is not as useful. So as useful. So as useful. So when I started, when I started, when I started, just to give you just to give you just to give you an idea of ​​how an idea of ​​how an idea of ​​how SDG works, as we SDG works, as we SDG works, as we call it, call it, call it, synthetic data generation, I synthetic data generation, I synthetic data generation, I started with 50,000 seeds. And I started with 50,000 seeds. And I started with 50,000 seeds. And I ended up with ended up with ended up with only 7,000 only 7,000 only 7,000 high-quality high-quality high-quality end-to-end SFT records end-to-end SFT records . And then I used . And then I used . And then I used that to train the that to train the that to train the model, and we model, and we model, and we moved the moved the moved the benchmark benchmark benchmark from zero to 30, right? So that's from zero to 30, right? So that's from zero to 30, right? So that's great. But from 50,000 great. But from 50,000 great. But from 50,000 to 7,000, that's lots of, to 7,000, that's lots of, to 7,000, that's lots of, you know, you know, you know, sales funnels and sales funnels and sales funnels and filtering for filtering for filtering for high quality. So what high quality. So what high quality. So what I'm trying to I'm trying to I'm trying to focus on is focus on is focus on is that data quality is the that data quality is the that data quality is the most important most important most important factor when factor when factor when building data. And, um building data. And, um , what do I mean by , what do I mean by , what do I mean by quality? It can quality? It can quality? It can be be be task-specific, but it task-specific, but it task-specific, but it can also be general.

can also be general. can also be general. Diversity is Diversity is Diversity is one way of, um, one way of, um, one way of, um, you know, one axis of you know, one axis of you know, one axis of quality, I would say. And then quality, I would say. And then I built I built this function to this function to this function to show what LLM is show what LLM is show what LLM is judging for each judging for each judging for each trajectory, um, and what trajectory, um, and what trajectory, um, and what your, um, you know, your, um, you know, your, um, you know, number of words in number of words in number of words in your question or your question or your question or search query is, and search query is, and search query is, and then here's your then here's your then here's your key percentage of 66% and key percentage of 66% and key percentage of 66% and so on. So what's so on. So what's so on. So what's really... You can really... You can really... You can look at this look at this look at this function later, but function later, but function later, but what's really interesting is how do you what's really interesting is how do you what's really interesting is how do you actually actually actually test whether test whether test whether the answers that the answers that the answers that the model gets are correct or the model gets are correct or the model gets are correct or not? So I came across not? So I came across not? So I came across this really this really this really interesting example, and the interesting example, and the interesting example, and the search query was search query was search query was pretty simple. It pretty simple. It pretty simple. It was, "What... what was, "What... what was, "What... what country owns US Steel?" country owns US Steel?" And I thought the And I thought the answer was the US. And answer was the US. And answer was the US. And so, um, I, you know, I, I so, um, I, you know, I, I so, um, I, you know, I, I think the agent think the agent think the agent predicted US in the predicted US in the predicted US in the training training training data set. And, um, my data set. And, um, my data set. And, um, my verifier said verifier said verifier said that was correct.

that was correct. that was correct. But then I went and But then I went and But then I went and Googled it, and apparently US Googled it, and apparently US Googled it, and apparently US sold US Steel to Japan, I think, sold US Steel to Japan, I think, a couple of years ago, if I remember correctly. So a couple of years ago, if I remember correctly. So even my original even my original even my original data from Wikipedia data from Wikipedia data from Wikipedia could be out of date, could be out of date, could be out of date, right? And so the real right? And so the real right? And so the real problem is problem is problem is how do how do how do you actually evaluate and, um, uh, you actually evaluate and, um, uh, you actually evaluate and, um, uh, determine whether the determine whether the determine whether the data you've created is, data you've created is, data you've created is, um, good, accurate, um, good, accurate, um, good, accurate, um, relevant, etc. um, relevant, etc. . Um, I think if you . Um, I think if you . Um, I think if you implement a implement a implement a deterministic, um, deterministic, um, deterministic, um, I think, simple I think, simple I think, simple matching function like I did, you're going matching function like I did, you're going matching function like I did, you're going to get about to get about to get about 24%, and then you're like, " 24%, and then you're like, " 24%, and then you're like, " Well, my data is terrible." Well, my data is terrible." Well, my data is terrible." Why did I do all this? " Why did I do all this? " But that's not really the case, But that's not really the case, But that's not really the case, because the because the because the expected expected expected percentages are low, because I don't know percentages are low, because I don't know how to actually how to actually how to actually create this function. create this function. create this function. So this is an active area So this is an active area So this is an active area of ​​research, how do you of ​​research, how do you of ​​research, how do you actually evaluate this actually evaluate this actually evaluate this data? So I think data? So I think data? So I think evaluation from my evaluation from my evaluation from my perspective is a very perspective is a very perspective is a very multifaceted multifaceted multifaceted problem. So you can't problem. So you can't problem. So you can't just take LLM just take LLM just take LLM as an estimator, put it in as an estimator, put it in as an estimator, put it in there and there and there and hope that it, hope that it, hope that it, you know, gives you everything.

you know, gives you everything. you know, gives you everything. Evaluation is a Evaluation is a Evaluation is a problem where if you problem where if you problem where if you check 100 problems check 100 problems , and if you see that , and if you see that , and if you see that those 100 problems actually those 100 problems actually those 100 problems actually exist, all you exist, all you exist, all you can say after can say after can say after checking 100 checking 100 checking 100 problems is that those problems is that those problems is that those 100 problems don't exist in 100 problems don't exist in 100 problems don't exist in your data set. your data set. your data set. But there could be a thousand But there could be a thousand But there could be a thousand other problems. So here's the other problems. So here's the other problems. So here's the problem: problem: problem: how do you actually evaluate how do you actually evaluate how do you actually evaluate the data? We have a couple of the data? We have a couple of the data? We have a couple of different ways different ways different ways to do that. One of them to do that. One of them , as I mentioned , as I mentioned , as I mentioned earlier, is LLM as an estimator earlier, is LLM as an estimator , which is generic. Uh, , which is generic. Uh, , which is generic. Uh, something very interesting that I something very interesting that I something very interesting that I did is did is did is actually do LLM actually do LLM actually do LLM as an estimator, uh, I as an estimator, uh, I as an estimator, uh, I took the judgments and took the judgments and took the judgments and sent them to sent them to sent them to human reviewers, and human reviewers, and human reviewers, and those reviewers those reviewers checked the checked the LLM judgments and essentially gave LLM judgments and essentially gave LLM judgments and essentially gave me confidence that me confidence that me confidence that LLM was scoring LLM was scoring LLM was scoring my answers correctly. So that's my answers correctly. So that's my answers correctly. So that's one way one way one way to do that. And then in to do that. And then in to do that. And then in search, you could search, you could search, you could look at look at look at the information in the information in the information in real time. So real time. So real time. So you need to find a you need to find a you need to find a combination of all these combination of all these combination of all these different, uh, levels of different, uh, levels of different, uh, levels of scoring that will scoring that will scoring that will essentially give you essentially give you essentially give you enough confidence enough confidence enough confidence in your dataset in your dataset . That's all for the code. I . That's all for the code. I . That's all for the code. I know I went know I went know I went through this quickly, but through this quickly, but through this quickly, but hopefully I've, you know, hopefully I've, you know, hopefully I've, you know, uh, made uh, made uh, made all of this available in my all of this available in my all of this available in my GitHub repository so GitHub repository so GitHub repository so you can all you can all you can all take a look. Uh, take a look. Uh, take a look. Uh, let's talk let's talk let's talk about a few other about a few other use cases. As I use cases. As I mentioned earlier, mentioned earlier, mentioned earlier, we're, uh, part of a we're, uh, part of a we're, uh, part of a much larger much larger much larger team that's building team that's building team that's building NeMo Tron, right? So, uh NeMo Tron, right? So, uh , my focus or , my focus or , my focus or my team's focus my team's focus my team's focus is on is on is on building building building datasets or building a datasets or building a datasets or building a framework for framework for framework for building building building datasets, or, I think,

datasets, or, I think, datasets, or, I think, both of those aspects. both of those aspects. both of those aspects. So we've built So we've built So we've built all sorts of datasets. all sorts of datasets. all sorts of datasets. The first one that we The first one that we The first one that we talked about today talked about today is the is the is the search tool call. search tool call. search tool call. Uh, and then we Uh, and then we Uh, and then we also have a dataset also have a dataset also have a dataset for multi-turn for multi-turn tool calls. So, I tool calls. So, I think the search was think the search was think the search was very similar to the very similar to the very similar to the single-turn tool call, single-turn tool call, single-turn tool call, right? We're going right? We're going right? We're going from query to from query to from query to response. But what response. But what response. But what if you had if you had if you had multiple turns? Or multiple turns? Or multiple turns? Or what if you had what if you had what if you had multiple tools? multiple tools? multiple tools? Um, or you know, not Um, or you know, not Um, or you know, not just just just search tools, but maybe search tools, but maybe you have, I don't know, a you have, I don't know, a weather forecasting tool weather forecasting tool or a or a or a coding tool, etc. Um, coding tool, etc. Um, coding tool, etc. Um, so we're working on so we're working on so we're working on a couple of different a couple of different a couple of different projects. Another projects. Another projects. Another really interesting project really interesting project is is is structured structured structured output generation. I output generation. I output generation. I talked about that briefly. There's a talked about that briefly. There's a talked about that briefly. There's a benchmark called benchmark called benchmark called Struct Eval that measures Struct Eval that measures Struct Eval that measures your performance your performance your performance in TOML, JSON, YAML, CSV, XML. And in TOML, JSON, YAML, CSV, XML. And in TOML, JSON, YAML, CSV, XML. And they're all incredibly they're all incredibly they're all incredibly important because now everyone important because now everyone important because now everyone wants to wants to wants to use, use, use, you know, agents for you know, agents for you know, agents for everything. I'm sure everything. I'm sure everything. I'm sure you've all seen the you've all seen the you've all seen the blowtorch meme on blowtorch meme on blowtorch meme on LinkedIn or something. But LinkedIn or something. But LinkedIn or something. But if we're going to if we're going to if we're going to use use use agents as, um, the agents as, um, the agents as, um, the centerpiece centerpiece centerpiece of software of software of software applications, then applications, then applications, then they need to reliably they need to reliably they need to reliably provide us with, um, provide us with, um, provide us with, um, JSON or YAML, etc.

JSON or YAML, etc. JSON or YAML, etc. So that's what we've So that's what we've So that's what we've been focusing on. Um, we've been focusing on. Um, we've been focusing on. Um, we've also been working on also been working on also been working on SQL, which someone SQL, which someone SQL, which someone asked about here. And, um, we've also asked about here. And, um, we've also asked about here. And, um, we've also been working on been working on been working on scientific thinking. scientific thinking. scientific thinking. Um, there's also Um, there's also pre- pre- training datasets that we've training datasets that we've training datasets that we've created created created with Data Designer for with Data Designer for with Data Designer for STEM productivity, STEM productivity, STEM productivity, etc. Um, and so I can etc. Um, and so I can etc. Um, and so I can talk about talk about talk about best practices for best practices for best practices for data generation. Um, I'm data generation. Um, I'm data generation. Um, I'm not going to go not going to go not going to go into that in detail because you into that in detail because you into that in detail because you can always can always can always revisit the slides revisit the slides revisit the slides later. But, essentially, later. But, essentially, later. But, essentially, variety variety variety over volume. I over volume. I over volume. I think Kimi think Kimi think Kimi K2's paper showed something K2's paper showed something K2's paper showed something similar, which is 10 similar, which is 10 similar, which is 10 strategies, one epoch is strategies, one epoch is strategies, one epoch is better than one epoch, better than one epoch, better than one epoch, sorry, 10 strategies, sorry, 10 strategies, sorry, 10 strategies, one epoch is better than one epoch is better than one epoch is better than 10 epochs, one 10 epochs, one 10 epochs, one strategy. And, um, you strategy. And, um, you strategy. And, um, you always want to always want to always want to use use use cheapest cheapest cheapest correct correct correct column type. So column type. So column type. So using LLM to using LLM to using LLM to create a sampler create a sampler create a sampler doesn't make sense, does doesn't make sense, does doesn't make sense, does it? It's a complete waste of it? It's a complete waste of it? It's a complete waste of resources. But resources. But resources. But using LLM to using LLM to using LLM to generate text generate text generate text data makes a lot of data makes a lot of data makes a lot of sense. So our sense. So our sense. So our samplers are of course samplers are of course samplers are of course written in Python. Um, and written in Python. Um, and written in Python. Um, and then you generate, then you generate, then you generate, you process the data, you process the data, you process the data, and then you filter it and then you filter it , as I mentioned , as I mentioned , as I mentioned earlier. And then you earlier. And then you earlier. And then you check what you check what you check what you can. Um, can. Um, can. Um, you re- you re- you re- preview.

preview. preview. I think the best I think the best I think the best way to know if way to know if way to know if your data is good is to your data is good is to your data is good is to actually look actually look actually look at the data. That's the best at the data. That's the best at the data. That's the best way. If you can, way. If you can, way. If you can, look at some look at some look at some samples. And of course, what's the samples. And of course, what's the samples. And of course, what's the idea of ​​a idea of ​​a idea of ​​a data designer? Essentially, you data designer? Essentially, you data designer? Essentially, you take the process take the process take the process of creating the data and of creating the data and of creating the data and make it, well, make it, well, make it, well, something, you know, something, you know, something, you know, version-controlled, version-controlled, version-controlled, well, that you can well, that you can well, that you can share like code. share like code. share like code. So, essentially, we So, essentially, we So, essentially, we create code that create code that create code that can create the data. can create the data. can create the data. Now you can Now you can Now you can share that code share that code share that code with anyone, because, you know, with anyone, because, you know, with anyone, because, you know, we've already built all we've already built all we've already built all these tools for these tools for code sharing, code sharing, version control, and version control, and version control, and everything else. So everything else. So everything else. So if you're trying if you're trying if you're trying to do a search, but to do a search, but to do a search, but not exactly like me, but not exactly like me, but not exactly like me, but a little bit differently, you a little bit differently, you a little bit differently, you take my code, you take my code, you take my code, you modify it, and modify it, and modify it, and you can create your you can create your you can create your own dataset from it own dataset from it own dataset from it . So, something . So, something . So, something really interesting, I'm really interesting, I'm really interesting, I'm going to go to some going to go to some going to go to some additional slides additional slides additional slides that I hadn't thought about. that I hadn't thought about. that I hadn't thought about. I think I have another 10 I think I have another 10 I think I have another 10 minutes, right? minutes, right? So, this is something really So, this is something really interesting. We created interesting. We created interesting. We created something called something called something called personas. So, Neumotron personas. So, Neumotron personas. So, Neumotron personas USA, um, also personas USA, um, also personas USA, um, also inspired by this inspired by this inspired by this other paper that other paper that other paper that scales scales scales synthetic data generation with 1 synthetic data generation with 1 synthetic data generation with 1 billion personas.

billion personas. billion personas. So what is a persona, So what is a persona, So what is a persona, right? Persona is right? Persona is right? Persona is essentially, um, we're essentially, um, we're essentially, um, we're trying to solve the trying to solve the trying to solve the problem of how do problem of how do problem of how do you know if the you know if the you know if the data that you're creating is data that you're creating is data that you're creating is actually representative of actually representative of actually representative of demographics demographics demographics or, or, or, you know, based you know, based you know, based on certain on certain on certain real-world distributions, real-world distributions, real-world distributions, right? That's really right? That's really right? That's really important. Uh, if I were to important. Uh, if I were to important. Uh, if I were to create a dataset of create a dataset of create a dataset of medical records in medical records in medical records in San Francisco, San Francisco, San Francisco, California, well, I want to California, well, I want to California, well, I want to make sure that the make sure that the actual actual demographics of demographics of demographics of San San Francisco Francisco Francisco are reflected in are reflected in are reflected in my dataset. data. my dataset. data. my dataset. data. Right? So, like Right? So, like Right? So, like the number of, um, you know, the number of, um, you know, the number of, um, you know, all kinds of all kinds of demographics. Uh, demographics. Uh, we created a we created a we created a dataset called Neiman dataset called Neiman dataset called Neiman Tron personas, um, using Tron personas, um, using data designer, but that's also the data designer, but that's also the seed in seed in seed in data designer for creating data designer for creating data designer for creating more more more datasets. And, um, datasets. And, um, datasets. And, um, it essentially allows it essentially allows it essentially allows you to train on you to train on you to train on user data user data user data without actually without actually without actually revealing any revealing any revealing any information about any information about any information about any individual. So now you individual. So now you individual. So now you can train an can train an can train an AI model AI model AI model and learn and learn and learn from a collection of, you from a collection of, you from a collection of, you know, know, know, people's information, but people's information, but people's information, but you don't actually you don't actually you don't actually have to have to have to use, use, use, like, my like, my like, my medical records that are medical records that are medical records that are directly directly used to used to train the train the train the AI ​​model. I AI ​​model. I AI ​​model. I keep keep keep using the using the using the healthcare example healthcare example healthcare example because I want to because I want to because I want to find a sensitive find a sensitive find a sensitive industry that's, you know, industry that's, you know, industry that's, you know, HIPAA compliant, there's HIPAA compliant, there's HIPAA compliant, there's all kinds of rules there, all kinds of rules there, all kinds of rules there, right? That we have to right? That we have to right? That we have to respect. So this is a respect. So this is a respect. So this is a persona, this person—a 50-

persona, this person—a 50- year-old pharmacist with a year-old pharmacist with a year-old pharmacist with a free spirit who free spirit who free spirit who balances her balances her balances her relaxed relaxed relaxed approach, blah blah blah. approach, blah blah blah. But Basically, there's a persona , there's a professional , there's a professional , there's a professional persona, well, first name, persona, well, first name, persona, well, first name, last name, middle name last name, middle name , gender, age, well, all , gender, age, well, all , gender, age, well, all that. And we have a that. And we have a that. And we have a million of them. You can million of them. You can million of them. You can take that, you can take that, you can take that, you can use that, use that, use that, well, whatever, as a well, whatever, as a well, whatever, as a starting point starting point starting point to create to create to create more more more data. So yeah, I data. So yeah, I data. So yeah, I think think think I'll probably stop there with personas I'll probably stop there with personas I'll probably stop there with personas . We have another 2 . We have another 2 . We have another 2 minutes. I'll talk minutes. I'll talk minutes. I'll talk about these other about these other about these other tools, and then tools, and then tools, and then we'll have time for we'll have time for we'll have time for questions. Or do you questions. Or do you questions. Or do you want a question? Yeah want a question? Yeah . Yeah. What for? Yeah. Yeah. . Yeah. What for? Yeah. Yeah. . Yeah. What for? Yeah. Yeah. Yeah. That's why Yeah. That's why Yeah. That's why we we we built it, to have a built it, to have a built it, to have a way to learn from way to learn from way to learn from aggregate aggregate aggregate information without, you information without, you information without, you know, individual people know, individual people . So, for example, . So, for example, . So, for example, you could you could you could create create create synthetic synthetic synthetic social social social security numbers. They security numbers. They security numbers. They wouldn't actually wouldn't actually wouldn't actually look like your look like your look like your social social social security number, would security number, would security number, would they? But, I think the idea is they? But, I think the idea is they? But, I think the idea is that that that if there's, like, um, if there's, like, um, if there's, like, um, um, I don't know. What's a um, I don't know. What's a um, I don't know. What's a good example I can good example I can give... Yeah. Yeah. Yes . Yes. That's right.

. Yes. That's right. For example, we For example, we created some created some created some tax data for tax data for tax data for a client. And I a client. And I a client. And I realized that for this realized that for this realized that for this tax data, tax data, tax data, there are exactly seven people in the United States there are exactly seven people in the United States there are exactly seven people in the United States under the age of 18 who under the age of 18 who under the age of 18 who make over make over make over $10 million a $10 million a $10 million a year on their year on their year on their tax tax tax return. But there are return. But there are return. But there are a lot more a lot more a lot more older people older people older people who make, who make, who make, you know, X, Y, or Z amount of you know, X, Y, or Z amount of you know, X, Y, or Z amount of money. So money. So money. So the tax data that the tax data that the tax data that we're creating, let's say we're creating, let's say , we want to create a , we want to create a , we want to create a synthetic W-2 form. synthetic W-2 form. synthetic W-2 form. We want to We want to We want to make sure that this make sure that this make sure that this form is not just form is not just form is not just random random random values ​​from LLM, values ​​from LLM, values ​​from LLM, because LLM is not going to give you... because LLM is not going to give you... because LLM is not going to give you... You asked LLM You asked LLM You asked LLM to tell you a joke? to tell you a joke? to tell you a joke? It's going to tell It's going to tell It's going to tell you the same you the same you the same joke every time. So, joke every time. So, joke every time. So, similarly, it's not going to similarly, it's not going to similarly, it's not going to give you give you give you realistic numbers, realistic numbers, realistic numbers, right? It's always right? It's always right? It's always going to revert to the going to revert to the going to revert to the mean. mean. mean. And so when you And so when you And so when you create data, you create data, you create data, you want to make sure want to make sure want to make sure that this synthetic that this synthetic that this synthetic W-2 form or 1040 form W-2 form or 1040 form W-2 form or 1040 form or whatever or whatever or whatever actually matches actually matches actually matches the distribution in the the distribution in the the distribution in the real world. So real world. So real world. So we created this we created this we created this personal data set.

personal data set. personal data set. We have a bunch of We have a bunch of We have a bunch of tools. Just tools. Just tools. Just 2 minutes and then more 2 minutes and then more 2 minutes and then more questions. Sorry about questions. Sorry about questions. Sorry about that. Um... So we have a that. Um... So we have a that. Um... So we have a bunch of tools bunch of tools bunch of tools that we've created. Data that we've created. Data that we've created. Data Designer is just the tip of Designer is just the tip of Designer is just the tip of the iceberg, I think. Um the iceberg, I think. Um , but it's our , but it's our , but it's our most successful most successful most successful tool so far tool so far tool so far because it's our because it's our because it's our oldest oldest oldest tool. Uh, we tool. Uh, we tool. Uh, we also have, uh, you know, also have, uh, you know, also have, uh, you know, NVIDIA has tools for NVIDIA has tools for NVIDIA has tools for curating data. curating data. curating data. For example, NeMo Curator, For example, NeMo Curator, For example, NeMo Curator, which I'm sure there are which I'm sure there are which I'm sure there are other other other discussions on this discussions on this discussions on this topic. I think I topic. I think I topic. I think I saw in the schedule. Uh, saw in the schedule. Uh, saw in the schedule. Uh, that's really interesting. that's really interesting. that's really interesting. You know, you need to You know, you need to You know, you need to take the data, uh, take the data, uh, take the data, uh, real data, real data, real data, curate it, curate it, curate it, modify it, post- modify it, post- process it, get it process it, get it process it, get it to a state where it's to a state where it's to a state where it's suitable for suitable for suitable for LLM training. Or you LLM training. Or you LLM training. Or you can create can create can create synthetic data. So, synthetic data. So, synthetic data. So, I think it depends I think it depends I think it depends on your on your on your use case. I'm not use case. I'm not use case. I'm not going to stand here and going to stand here and going to stand here and say that synthetic say that synthetic say that synthetic data is the best data is the best data is the best option for everything. option for everything. option for everything. But, I think there are But, I think there are But, I think there are specific specific specific use cases where you use cases where you use cases where you want to curate want to curate want to curate real data or when real data or when real data or when you want to, you know, you want to, you know, you want to, you know, generate generate generate synthetic data. Uh, synthetic data. Uh, synthetic data. Uh, we have we have we have data curation tools. data curation tools. data curation tools. We have Data Designer, our We have Data Designer, our synthetic data generation tool. synthetic data generation tool. We have a new We have a new We have a new tool tool tool called Anonymizer. And by called Anonymizer. And by called Anonymizer. And by the way, all of these the way, all of these the way, all of these tools are tools are tools are completely open completely open completely open source. So source. So source. So everything that our everything that our everything that our organization does at NVIDIA organization does at NVIDIA organization does at NVIDIA is completely is completely is completely open open open source, including NeMo Tron, source, including NeMo Tron, source, including NeMo Tron, including all of these including all of these including all of these tools. So you tools. So you tools. So you can can can use it use it use it however you want. So this data however you want. So this data however you want. So this data tool tool is called NeMo Anonymizer.

is called NeMo Anonymizer. The goal is, The goal is, The goal is, can we can we can we actually actually actually anonymize the data? I anonymize the data? I anonymize the data? I don't know about you guys, don't know about you guys, don't know about you guys, but I don't know if but I don't know if but I don't know if I should talk I should talk I should talk about this when the conversation about this when the conversation about this when the conversation is recorded, but I is recorded, but I is recorded, but I put all kinds of things in put all kinds of things in put all kinds of things in my agents. my agents. my agents. Like, you know, I Like, you know, I Like, you know, I accidentally put accidentally put accidentally put my API keys in there, or, my API keys in there, or, my API keys in there, or, you know, some other you know, some other you know, some other sensitive sensitive sensitive information, right? And all of information, right? And all of information, right? And all of this log data for this log data for this log data for all of your agents all of your agents all of your agents that you're that you're that you're using, you using, you using, you need some need some need some way to anonymize way to anonymize way to anonymize them before they them before they them before they become available, become available, become available, right? So that's what right? So that's what right? So that's what Anonymizer does. This is just an Anonymizer does. This is just an Anonymizer does. This is just an example, but example, but example, but obviously there are many obviously there are many obviously there are many other ways other ways other ways to use Anonymizer. And to use Anonymizer. And to use Anonymizer. And then we have the NeMo Safe Synthesizer then we have the NeMo Safe Synthesizer , which allows us , which allows us , which allows us to generate safe to generate safe to generate safe synthetic versions of synthetic versions of synthetic versions of sensitive sensitive sensitive datasets, like datasets, like datasets, like before. Yeah. before. Yeah. before. Yeah. Come talk Come talk Come talk to me if you have to me if you have to me if you have any questions. Here's any questions. Here's any questions. Here's my LinkedIn if you my LinkedIn if you my LinkedIn if you want to stay in want to stay in want to stay in touch. Again, touch. Again, touch. Again, you can you can you can use Air Designer. use Air Designer. use Air Designer. I'd really like for you I'd really like for you I'd really like for you guys to try out guys to try out guys to try out these notebooks. I'm these notebooks. I'm these notebooks. I'm really sorry we had to really sorry we had to really sorry we had to , we , we , we only had an hour, so only had an hour, so only had an hour, so I had to just I had to just I had to just fly through all the fly through all the fly through all the code. But you can code. But you can code. But you can try out the notebooks. try out the notebooks. try out the notebooks. We also have blogs We also have blogs We also have blogs if you're interested in a if you're interested in a follow-up text or follow-up text or some other some other some other use case. So use case. So use case. So this, uh, shows us this, uh, shows us this, uh, shows us how to use how to use how to use personas. Uh, I think there personas. Uh, I think there 's a blog somewhere about 's a blog somewhere about 's a blog somewhere about how we created the how we created the how we created the personas dataset. For the US, personas dataset. For the US, personas dataset. For the US, we took the we took the we took the census information and census information and census information and used PGM, a used PGM, a used PGM, a probabilistic graphical probabilistic graphical probabilistic graphical a model to, essentially, a model to, essentially, a model to, essentially, create, uh, a connection create, uh, a connection create, uh, a connection between all of the... essentially, between all of the... essentially, between all of the... essentially, take this take this take this demographic demographic demographic information, process information, process information, process it, and then it, and then it, and then try to, uh, try to, uh, try to, uh, create a dataset from it create a dataset from it create a dataset from it , uh, essentially.

, uh, essentially. That's what we did. Uh, That's what we did. Uh, and every country, uh, and every country, uh, and every country, uh, publishes this, uh, publishes this, uh, publishes this, uh, demographic demographic demographic information. So it's not information. So it's not information. So it's not exactly... uh, I'm exactly... uh, I'm exactly... uh, I'm looking at other looking at other looking at other slides now. We've had a slides now. We've had a slides now. We've had a bunch of these bunch of these bunch of these workshops recently. Uh, but workshops recently. Uh, but workshops recently. Uh, but yeah, there are all kinds of yeah, there are all kinds of yeah, there are all kinds of personas. If you're not personas. If you're not personas. If you're not trying to build trying to build trying to build something outside of the US, you're something outside of the US, you're something outside of the US, you're building something for building something for building something for Japan, India, Japan, India, Japan, India, Singapore, Brazil, Singapore, Brazil, Singapore, Brazil, I'm sure we I'm sure we I'm sure we have more now. So have more now. So have more now. So we have all kinds of we have all kinds of we have all kinds of persona datasets. persona datasets. So, uh, this gives us, uh, a So, uh, this gives us, uh, a way to build a way to build a way to build a custom AI custom AI custom AI for any country for any country for any country that you're interested in.