Beyond RAG: Build a Relational Context Engine from Scratch — Peter Werry, Unblocked

So, I'm Peter. I'm from So, I'm Peter. I'm from Unblocked. And here is Unblocked. And here is Unblocked. And here is my colleague my colleague my colleague Brandon. And today, Brandon. And today, Brandon. And today, well, first of all, I want to well, first of all, I want to well, first of all, I want to talk a little bit about what talk a little bit about what talk a little bit about what we do. we do. we do. We are creating a We are creating a We are creating a contextual engine. contextual engine. contextual engine. Our company Our company Our company is called Unblocked. We is called Unblocked. We is called Unblocked. We provide provide provide organizational organizational organizational context to people and, context to people and, increasingly, to increasingly, to agents. The session agents. The session agents. The session was originally was originally was originally scheduled to last scheduled to last scheduled to last several hours. several hours. several hours. We only have an hour now, so the We only have an hour now, so the We only have an hour now, so the material will be very material will be very material will be very concise. This means concise. This means concise. This means that I will be going through the that I will be going through the that I will be going through the workshop quite workshop quite workshop quite quickly. If you don't have quickly. If you don't have quickly. If you don't have time to complete time to complete time to complete all the steps, don't all the steps, don't all the steps, don't worry, because worry, because worry, because the repository is the repository is the repository is available online and available online and available online and you can you can you can work on it work on it work on it later at a time convenient for later at a time convenient for later at a time convenient for you. The thesis is you. The thesis is you. The thesis is that RAG—and I'm going to that RAG—and I'm going to that RAG—and I'm going to use use use the term RAG as the term RAG as the term RAG as semantic retrieval semantic retrieval semantic retrieval or vector or vector or vector data retrieval, data retrieval, data retrieval, okay? RAG is only okay? RAG is only okay? RAG is only half of the search, and half of the search, and half of the search, and today we will create the today we will create the today we will create the other half: other half: other half: converting converting converting natural language into natural language into natural language into actual actual actual database queries. So, here's the database queries. So, here's the database queries. So, here's the QR code for QR code for QR code for setup again. If setup again. If setup again. If anyone hasn't done so yet anyone hasn't done so yet anyone hasn't done so yet , I'll move on to the , I'll move on to the , I'll move on to the next item and next item and next item and open it briefly, open it briefly, open it briefly, hopefully it will hopefully it will hopefully it will load. No, load. No, load. No, of course it won't of course it won't of course it won't load. Give load. Give load. Give me a second to me a second to me a second to switch. Oh.

switch. Oh. I hope you see I hope you see this, here is the repository. The this, here is the repository. The this, here is the repository. The setup steps are setup steps are setup steps are quite simple. There are quite simple. There are quite simple. There are a few dependencies a few dependencies a few dependencies and requirements you and requirements you and requirements you need to fulfill. need to fulfill. Throughout the Throughout the workshop, we will workshop, we will workshop, we will work with a number of work with a number of work with a number of pull pull pull requests. requests. requests. The network seems to have disappeared. The network seems to have disappeared. At least At least it doesn't work it doesn't work it doesn't work properly. Let's try properly. Let's try properly. Let's try this. The good thing is that this. The good thing is that we don't we don't need the Internet for this story. I hope need the Internet for this story. I hope need the Internet for this story. I hope you have everything ready to you have everything ready to you have everything ready to start. Because that start. Because that start. Because that obviously won't obviously won't obviously won't work. But in work. But in work. But in any case, we any case, we any case, we will follow a will follow a will follow a series of six series of six overlapping merge requests overlapping merge requests . So, I'll... That's it . So, I'll... That's it . I . I . I will refer to them as I will refer to them as I will refer to them as I work. I'll work. I'll work. I'll come back here for come back here for come back here for everyone to see. And back to everyone to see. And back to everyone to see. And back to the presentation. Now the presentation. Now the presentation. Now I'll have to I'll have to I'll have to expand expand expand it again. Sorry. That's it.

it again. Sorry. That's it. Okay, I hope Okay, I hope everyone sees this. Perfectly. everyone sees this. Perfectly. everyone sees this. Perfectly. So, our agents So, our agents So, our agents need context. And need context. And need context. And the context the context the context is scattered across is scattered across is scattered across all the tools all the tools all the tools that that that engineering teams use. You will engineering teams use. You will engineering teams use. You will find it in PRs, find it in PRs, find it in PRs, review comments, review comments, review comments, tickets in Linear or Jira, tickets in Linear or Jira, tickets in Linear or Jira, documentation and wikis, documentation and wikis, documentation and wikis, Notion, Confluence, and, of course, in Notion, Confluence, and, of course, in Notion, Confluence, and, of course, in discussions in Slack or discussions in Slack or discussions in Slack or Teams. All of this can be Teams. All of this can be Teams. All of this can be queried queried queried semantically, but it semantically, but it semantically, but it also has structure. also has structure. also has structure. There are authors, dates, There are authors, dates, There are authors, dates, statuses, tags, statuses, tags, statuses, tags, relationships, etc. relationships, etc. You know, the instinctive You know, the instinctive desire that I desire that I desire that I like to talk about— like to talk about— even two years ago, which is a long time ago even two years ago, which is a long time ago in the field of AI —was to —was to —was to just load just load just load everything into a vector database, everything into a vector database, everything into a vector database, extract it, and extract it, and extract it, and reorder it in reorder it in reorder it in the hopes of getting the the hopes of getting the the hopes of getting the desired result. desired result. But it won't But it won't work like that. So I'll take a work like that. So I'll take a work like that. So I'll take a step back and step back and step back and connect this to a larger connect this to a larger connect this to a larger discussion about discussion about discussion about context. In the context of context. In the context of context. In the context of an engine that provides an engine that provides an engine that provides organizational organizational organizational context, it's context, it's context, it's not just about not just about not just about code, as I said.

code, as I said. We tie everything We tie everything together. So, together. So, the context is scattered on the left, all the context is scattered on the left, all your code, PRs, your code, PRs, your code, PRs, documentation, etc. documentation, etc. documentation, etc. are fed into the input. are fed into the input. are fed into the input. The idea is to The idea is to get a get a reasonable reasonable reasonable context at the output, where all the data is context at the output, where all the data is context at the output, where all the data is interconnected interconnected interconnected and adapted to a and adapted to a and adapted to a specific task. In specific task. In specific task. In this model, it is this model, it is this model, it is important for us to consider not important for us to consider not important for us to consider not only semantic only semantic only semantic relationships, but also relationships, but also relationships, but also structural ones that structural ones that structural ones that unite all the data. unite all the data. Again, RAG is Again, RAG is about indexing about indexing about indexing a query, finding a query, finding a query, finding documents with similar documents with similar documents with similar content, which we then content, which we then content, which we then add to the prompt, right? add to the prompt, right? This works great for content questions: find code, content questions: find code, find PRs where it was find PRs where it was find PRs where it was changed, find changed, find changed, find discussions in Slack. And discussions in Slack. And discussions in Slack. And if you add an if you add an if you add an agent cycle to this, you can agent cycle to this, you can agent cycle to this, you can achieve significant achieve significant achieve significant results. results. results. You can also connect You can also connect You can also connect tools for tools for tools for recursive recursive recursive file browsing, which file browsing, which file browsing, which would be a good would be a good would be a good starting point. You starting point. You starting point. You get a good get a good get a good semantic start, and semantic start, and semantic start, and then your agent can then your agent can then your agent can recursively recursively recursively explore the file explore the file explore the file system in GitHub, and that system in GitHub, and that system in GitHub, and that is often is often is often enough. But, from enough. But, from enough. But, from our experience, in our experience, in our experience, in practice, things don't work out that practice, things don't work out that way. Engineers way. Engineers often ask often ask often ask questions like: questions like: questions like: what PRs did I or my what PRs did I or my what PRs did I or my team freeze team freeze team freeze last week? They are last week? They are last week? They are actually looking for actually looking for actually looking for something specific, something specific, something specific, knowing the starting knowing the starting knowing the starting point but not knowing point but not knowing point but not knowing where the where the where the destination is. But they

destination is. But they destination is. But they know about the know about the know about the connections. So, connections. So, connections. So, as an example, the PRs I as an example, the PRs I as an example, the PRs I drained last week, drained last week, drained last week, let's think let's think let's think about it in the context of RAG. about it in the context of RAG. about it in the context of RAG. You would, you know, You would, you know, You would, you know, embed that sentence. embed that sentence. embed that sentence. You would find You would find You would find the closest vectors, the closest vectors, the closest vectors, rerank them, rerank them, rerank them, choose the top-K. You choose the top-K. You choose the top-K. You get a PR get a PR get a PR mentioning the merge mentioning the merge mentioning the merge regarding regarding regarding last week's conversation. But last week's conversation. But last week's conversation. But you wouldn't get the you wouldn't get the you wouldn't get the actual links, actual links, actual links, like the author, the like the author, the like the author, the merge status, and whether merge status, and whether merge status, and whether it was actually merged it was actually merged it was actually merged last week. So as last week. So as last week. So as a result you may a result you may a result you may get an incorrect get an incorrect get an incorrect answer where a answer where a answer where a PR is extracted that is PR is extracted that is PR is extracted that is semantically semantically semantically related but related but related but merged 3 years ago. And merged 3 years ago. And merged 3 years ago. And that doesn't really that doesn't really that doesn't really answer the answer the answer the question, because there question, because there question, because there 's a structural, 's a structural, 's a structural, temporal aspect to it. So, temporal aspect to it. So, temporal aspect to it. So, the search here has two the search here has two the search here has two halves. Semantic halves. Semantic halves. Semantic and, again, and, again, and, again, today we'll today we'll today we'll talk about this, talk about this, talk about this, structured queries structured queries . And that's the point. We're . And that's the point. We're . And that's the point. We're going to take… going to take… the model doesn't actually the model doesn't actually the model doesn't actually answer the answer the answer the question in this question in this question in this paradigm, it writes a paradigm, it writes a paradigm, it writes a query that query that query that goes to the goes to the goes to the database. Okay, so it would be database. Okay, so it would be database. Okay, so it would be naive to say that naive to say that naive to say that query generation is the query generation is the query generation is the only thing we have to only thing we have to only thing we have to do. To create do. To create do. To create a query reliably and a query reliably and a query reliably and perhaps even perhaps even perhaps even take care of take care of take care of security issues, we need to security issues, we need to security issues, we need to consider all consider all consider all four points here.

four points here. So, first of all, LLM So, first of all, LLM needs to know the schema before needs to know the schema before needs to know the schema before creating a query. creating a query. She needs to understand She needs to understand some of the relationships some of the relationships some of the relationships in the data. For example, in the data. For example, in the data. For example, if you pass if you pass if you pass a query in natural a query in natural a query in natural language: "Show language: "Show language: "Show Peter's open PRs." Peter's open PRs." Then she won't know Then she won't know what to do because she what to do because she what to do because she 'll create a request with the 'll create a request with the 'll create a request with the name "Peter" even though the name "Peter" even though the name "Peter" even though the GitHub login may be GitHub login may be GitHub login may be different. So, she different. So, she different. So, she should be able to should be able to should be able to compare it. And, compare it. And, compare it. And, of course, you'll of course, you'll of course, you'll get a few get a few get a few unreliable requests at the end. unreliable requests at the end. unreliable requests at the end. Perhaps using the Perhaps using the $where operator in $where operator in [ __ ] terminology, [ __ ] terminology, [ __ ] terminology, which can execute which can execute which can execute JavaScript directly JavaScript directly JavaScript directly on the server, which we on the server, which we on the server, which we probably don't probably don't probably don't want. And finally, want. And finally, want. And finally, if if if mistakes occur the first time, we mistakes occur the first time, we mistakes occur the first time, we want to be want to be want to be able to able to able to fix them ourselves fix them ourselves . This is what it looks like in . This is what it looks like in . This is what it looks like in one holistic one holistic one holistic process. We want to process. We want to process. We want to get something like get something like get something like this. this. this. It compares my It compares my It compares my colleague, his name is colleague, his name is colleague, his name is Rashin. His username is Rashin. His username is Rasharab. We want to be Rasharab. We want to be Rasharab. We want to be able to able to able to compare state, as well compare state, as well compare state, as well as get as get as get operators like operators like operators like limit and sort. So this is what limit and sort. So this is what limit and sort. So this is what we're going to we're going to we're going to build. We will build. We will build. We will start with a complete UX, a start with a complete UX, a start with a complete UX, a stub engine that stub engine that stub engine that initially initially initially does nothing, and we will does nothing, and we will does nothing, and we will gradually gradually gradually build it up. And build it up. And build it up. And then, in the third then, in the third then, in the third phase, everything will come to life and we phase, everything will come to life and we phase, everything will come to life and we will be able to send

will be able to send will be able to send requests and see requests and see requests and see how it works. If you have how it works. If you have how it works. If you have n't seen it yet, n't seen it yet, n't seen it yet, now is the time now is the time now is the time to scan and to scan and to scan and add it. Yes. Yes, add it. Yes. Yes, add it. Yes. Yes, any any any repository will do. It repository will do. It repository will do. It doesn't matter which doesn't matter which doesn't matter which one. And if you want to one. And if you want to one. And if you want to take the path of take the path of take the path of least resistance, least resistance, least resistance, just point just point just point the agent at this thing and the agent at this thing and the agent at this thing and he'll set everything up. It he'll set everything up. It he'll set everything up. It works. OK? works. OK? works. OK? Let's go. So, here's what Let's go. So, here's what Let's go. So, here's what we're going to we're going to we're going to do at launch. Um, do at launch. Um, do at launch. Um, by this point you should by this point you should by this point you should probably have already probably have already probably have already started the started the started the ingestion process. Don't ingestion process. Don't ingestion process. Don't worry if not worry if not . Again, you . Again, you . Again, you can come back can come back can come back to this at the end. to this at the end. to this at the end. I'll just quickly I'll just quickly I'll just quickly show you what show you what show you what this basic this basic this basic interface looks like. So, interface looks like. So, interface looks like. So, now this thing works now this thing works . If I type . If I type . If I type something here, it won't work, of course something here, it won't work, of course something here, it won't work, of course . Um, and you . Um, and you . Um, and you should see should see should see something like this. There is also a something like this. There is also a something like this. There is also a chat tab. This is a chat tab. This is a chat tab. This is a full-fledged agent full-fledged agent full-fledged agent that can do the that can do the that can do the work for you. So we'll work for you. So we'll work for you. So we'll come back to that in a come back to that in a come back to that in a moment. Here it is— moment. Here it is— the contract in code, if the contract in code, if the contract in code, if you want to you want to you want to read it. We won't read it. We won't read it. We won't ... Never mind that

... Never mind that ... Never mind that . We will miss . We will miss . We will miss this moment. But, this moment. But, this moment. But, essentially, this is the entry point essentially, this is the entry point essentially, this is the entry point around which we around which we around which we will build everything. So, the will build everything. So, the will build everything. So, the first thing first thing first thing I want to talk about I want to talk about I want to talk about is pattern recognition. is pattern recognition. During ingestion, During ingestion, if you look at how if you look at how if you look at how the script works, it the script works, it the script works, it actually had to actually had to actually had to build a schema to build a schema to build a schema to download and download and download and save the content. But save the content. But save the content. But you can imagine a you can imagine a you can imagine a situation where you don't situation where you don't situation where you don't know the scheme. know the scheme. know the scheme. Ingestion can Ingestion can Ingestion can happen in another happen in another happen in another service, and you service, and you service, and you need to build a need to build a need to build a dynamic dynamic dynamic schema discovery engine. [ __ ] schema discovery engine. [ __ ] schema discovery engine. [ __ ] has no schema, so there is has no schema, so there is has no schema, so there is nothing to read, nothing to read, nothing to read, unlike PG or unlike PG or unlike PG or other systems where we other systems where we other systems where we could read could read could read table definitions. table definitions. table definitions. So, we will So, we will So, we will sample documents sample documents sample documents to derive the schema. That's the to derive the schema. That's the to derive the schema. That's the whole whole whole trick. We're just trick. We're just trick. We're just going to going to going to reconstruct it. We reconstruct it. We reconstruct it. We will take all the nodes and will take all the nodes and will take all the nodes and merge them. So, merge them. So, merge them. So, this will take a sample of this will take a sample of this will take a sample of 300 documents and 300 documents and 300 documents and collapse them into a single collapse them into a single collapse them into a single tree. Null values tree. Null values tree. Null values are skipped are skipped are skipped because missing because missing because missing fields will not be able to fields will not be able to fields will not be able to constrain the type. And then constrain the type. And then constrain the type. And then we'll do something we'll do something we'll do something interesting here with interesting here with interesting here with enum types. So enum types. So enum types. So now now now we can we can we can do an interesting do an interesting do an interesting procedural trick with enumeration types. This is procedural trick with enumeration types. This is procedural trick with enumeration types. This is just one thing. I would just one thing. I would just one thing. I would advise advise advise using using using procedural tricks for procedural tricks for procedural tricks for things like this, because you can things like this, because you can things like this, because you can run it through run it through run it through LLM to extract the LLM to extract the LLM to extract the data and it works,

data and it works, data and it works, but the result is quite but the result is quite but the result is quite non-deterministic. non-deterministic. Therefore, you always Therefore, you always need some kind of need some kind of need some kind of structural framework structural framework structural framework that executes that executes that executes procedural code to procedural code to procedural code to validate what you are validate what you are validate what you are extracting, or to do the extracting, or to do the extracting, or to do the extraction itself. And in extraction itself. And in extraction itself. And in this example this example this example project, it is the project, it is the project, it is the procedural code that procedural code that procedural code that handles handles handles the removal. Let's the removal. Let's the removal. Let's go ahead and show go ahead and show go ahead and show it. So, we'll do it and it. So, we'll do it and it. So, we'll do it and run it again. And run it again. And run it again. And now we now we now we get this. get this. get this. The schema ran, the The schema ran, the The schema ran, the discovery process discovery process discovery process passed, and it passed, and it passed, and it created a schema for us created a schema for us that outlines what a that outlines what a pull request schema looks like on GitHub. pull request schema looks like on GitHub. He even created He even created He even created a few indexes for us a few indexes for us a few indexes for us . This is done . This is done . This is done in markdown format in markdown format in markdown format intentionally, because we will intentionally, because we will intentionally, because we will pass this schema to LLM pass this schema to LLM when we when we when we synthesize the synthesize the synthesize the query itself. Now the second query itself. Now the second query itself. Now the second step is what I step is what I step is what I mentioned a few mentioned a few mentioned a few slides ago: slides ago: slides ago: resolving resolving resolving identity. This is because identity. This is because , for example, my , for example, my , for example, my GitHub login is similar, GitHub login is similar, GitHub login is similar, but not exactly the but not exactly the but not exactly the same. So if I were to same. So if I were to same. So if I were to ask the question, " ask the question, " Hey, find all of Hey, find all of Hey, find all of Peter's PRs." Without some kind of Peter's PRs." Without some kind of identity matching, it would identity matching, it would simply try simply try simply try to find a match for the name

to find a match for the name to find a match for the name "Peter" and "Peter" and "Peter" and find nothing. So, we find nothing. So, we find nothing. So, we must be able to must be able to must be able to solve this. And again solve this. And again solve this. And again , at first , at first , at first glance it seems like glance it seems like glance it seems like we're getting a little too we're getting a little too we're getting a little too attached to the attached to the attached to the scheme here. But it is entirely scheme here. But it is entirely scheme here. But it is entirely possible to determine this possible to determine this possible to determine this dynamically, and you dynamically, and you dynamically, and you can let can let can let your LLM essentially your LLM essentially your LLM essentially synthesize these synthesize these synthesize these mappings for you. mappings for you. So in our case, So in our case, we'll just apply we'll just apply we'll just apply these little simple these little simple these little simple tricks. A very simple tricks. A very simple tricks. A very simple fuzzy comparison. The fuzzy comparison. The fuzzy comparison. The Löwenstein distance is Löwenstein distance is Löwenstein distance is quite suitable. You quite suitable. You quite suitable. You can just can just can just match the names. And match the names. And match the names. And everything is ready. Oh, everything is ready. Oh, everything is ready. Oh, actually I want to actually I want to actually I want to show you one thing show you one thing before we before we before we get to that. get to that. get to that. So, let's do the same thing So, let's do the same thing So, let's do the same thing again. We can again. We can again. We can increase this. Oh, yes, increase this. Oh, yes, increase this. Oh, yes, sorry. Undoubtedly. sorry. Undoubtedly. sorry. Undoubtedly. Tell me when it's Tell me when it's Tell me when it's big enough big enough big enough . So good?

. So good? . So good? Thank you. A valid Thank you. A valid Thank you. A valid remark. Good. remark. Good. remark. Good. So, let's So, let's So, let's start ours again. start ours again. And now, um Oh, And now, um Oh, I actually I actually I actually need to wait until the need to wait until the need to wait until the next step. next step. next step. Sorry. This is not Sorry. This is not Sorry. This is not connected yet. So, connected yet. So, connected yet. So, let's move on to what will let's move on to what will let's move on to what will really give us really give us really give us results—this is step results—this is step results—this is step three. This is actually the three. This is actually the query planning stage, and it's probably the query planning stage, and it's probably the easiest part. easiest part. easiest part. This is the part where This is the part where This is the part where LLM is involved. We LLM is involved. We LLM is involved. We pass the schema, pass the schema, pass the schema, provide the tool, and have it provide the tool, and have it call that call that tool to tool to tool to extract and synthesize the extract and synthesize the extract and synthesize the names. And then he names. And then he names. And then he will carry out this plan. There is will carry out this plan. There is will carry out this plan. There is one more thing—the time one more thing—the time one more thing—the time aspect. LLMs aspect. LLMs aspect. LLMs generate generate generate time queries quite well if time queries quite well if time queries quite well if you pass them the current you pass them the current you pass them the current date and time. So this is another date and time. So this is another date and time. So this is another important point to important point to important point to consider in consider in consider in the request. This is where the the request. This is where the name extraction happens. In name extraction happens. In this case, we this case, we this case, we will force the tool call to be executed will force the tool call to be executed will force the tool call to be executed . And . And . And now I'll show you the now I'll show you the now I'll show you the result of this. So result of this. So . Oh. Here. Now I can

. Oh. Here. Now I can . Oh. Here. Now I can do this. Open do this. Open do this. Open PR from Peter. And boom. As PR from Peter. And boom. As PR from Peter. And boom. As you can see, he you can see, he you can see, he generated this generated this generated this magic sentence. It's magic sentence. It's not visible from here, but not visible from here, but he matched "Peter he matched "Peter " with my GitHub login " with my GitHub login , few worries. I generated a , few worries. I generated a , few worries. I generated a query, and here is the query, and here is the query, and here is the result. This is cool, result. This is cool, result. This is cool, but we need to but we need to but we need to move to the move to the move to the next step, because next step, because next step, because while we can already while we can already while we can already query the query the query the data, there are some data, there are some security concerns. It may happen that security concerns. It may happen that it will give it will give it will give unreliable results unreliable results . It can . It can . It can use where clauses use where clauses use where clauses or functions and or functions and or functions and run JavaScript run JavaScript run JavaScript directly on the directly on the directly on the server, making it a server, making it a request injection point. Your schema request injection point. Your schema may also have may also have may also have hidden fields, such as hidden fields, such as hidden fields, such as IDs or metadata, that IDs or metadata, that IDs or metadata, that reveal reveal reveal internal data. So internal data. So internal data. So we need to we need to we need to do a cleanup, and do a cleanup, and do a cleanup, and we're not creating a we're not creating a we're not creating a blacklist, we're creating a blacklist, we're creating a blacklist, we're creating a whitelist. By whitelist. By whitelist. By default, everything default, everything default, everything is prohibited. You is prohibited. You is prohibited. You need to create a need to create a need to create a whitelist of whitelist of whitelist of allowed allowed allowed operators. And operators. And operators. And perhaps a whitelist of perhaps a whitelist of perhaps a whitelist of allowed fields. In allowed fields. In allowed fields. In our case, this is our case, this is our case, this is dynamic dynamic dynamic schema discovery, so we schema discovery, so we schema discovery, so we won't do that. But won't do that. But won't do that. But we can have some we can have some we can have some known fields,

known fields, known fields, for example for example for example metadata fields. This is useful metadata fields. This is useful metadata fields. This is useful for isolating for isolating for isolating tenants. If you tenants. If you tenants. If you have a [ __ ] collection that have a [ __ ] collection that have a [ __ ] collection that contains multiple contains multiple contains multiple tenants, I would tenants, I would tenants, I would n't recommend this. It's probably a n't recommend this. It's probably a n't recommend this. It's probably a good idea to separate good idea to separate good idea to separate collections by collections by collections by tenant, but if tenant, but if tenant, but if you did that, you you did that, you you did that, you could add a could add a could add a metadata field with the metadata field with the tenant ID. And you would tenant ID. And you would like to hide this... like to hide this... um, information from LLM, um, information from LLM, um, information from LLM, but also verify the but also verify the but also verify the original data. So, original data. So, original data. So, if the system is outputting this if the system is outputting this if the system is outputting this metadata field, it metadata field, it metadata field, it should be removed. And should be removed. And should be removed. And then you need to then you need to then you need to insert your own insert your own insert your own metadata field at the bottom. And metadata field at the bottom. And metadata field at the bottom. And again, again, again, technically it's technically it's technically it's actually not that actually not that actually not that difficult to do. difficult to do. difficult to do. You just need to You just need to You just need to write the code. This is all write the code. This is all write the code. This is all procedural work. Hmm procedural work. Hmm , I wouldn't recommend , I wouldn't recommend , I wouldn't recommend using LLM using LLM using LLM for this part. And for this part. And for this part. And again, again, again, use a whitelist use a whitelist use a whitelist . Don't . Don't . Don't forbid it. More precisely, forbid it. More precisely, forbid it. More precisely, I should say: I should say: I should say: ban everything.

ban everything. Now the last Now the last part, which, um, part, which, um, part, which, um, smoothly transitions into the smoothly transitions into the smoothly transitions into the next step. next step. next step. At the end of the check, At the end of the check, At the end of the check, we will, of course, return an we will, of course, return an we will, of course, return an error. And she will be error. And she will be error. And she will be brought out. And now this brought out. And now this brought out. And now this error simply exists. error simply exists. error simply exists. We We We won't do anything with it yet. won't do anything with it yet. won't do anything with it yet. But I'll show But I'll show But I'll show you what it looks like now, you what it looks like now, you what it looks like now, um. Oh. Let's leave it at um. Oh. Let's leave it at um. Oh. Let's leave it at that. And let's launch that. And let's launch that. And let's launch our service. So, our service. So, our service. So, now. I'm going to now. I'm going to now. I'm going to run another run another run another request that checks for request that checks for request that checks for authentication in the authentication in the authentication in the header. And we header. And we header. And we banned regular banned regular banned regular expressions. So now we expressions. So now we expressions. So now we see that the request is see that the request is see that the request is denied. And finally, denied. And finally, denied. And finally, this is what ties it all this is what ties it all this is what ties it all together—the cycle of together—the cycle of together—the cycle of retrying. The plan retrying. The plan retrying. The plan can go wrong in can go wrong in can go wrong in a number of ways, a number of ways, a number of ways, right? Hmm, it might right? Hmm, it might right? Hmm, it might not pass validation, not pass validation, not pass validation, maybe LM is actually maybe LM is actually maybe LM is actually issuing things that are

issuing things that are issuing things that are invalid according invalid according invalid according to our validator. to our validator. to our validator. Or it may Or it may Or it may actually throw actually throw actually throw an error at an error at an error at runtime. Um, runtime. Um, runtime. Um, for example, let's imagine for example, let's imagine for example, let's imagine he mixed up the he mixed up the he mixed up the inclusion and inclusion and inclusion and exclusion project. Um, then exclusion project. Um, then exclusion project. Um, then of course it won't work of course it won't work , but we can , but we can fix it ourselves. We fix it ourselves. We can take that can take that can take that error and pass it error and pass it error and pass it back to LLM, saying, " back to LLM, saying, " Hey, that was Hey, that was Hey, that was wrong." Hmm, wrong." Hmm, wrong." Hmm, maybe you can do maybe you can do maybe you can do better. Of course, up to a better. Of course, up to a better. Of course, up to a certain number of certain number of certain number of maximum attempts. maximum attempts. So, I'll show you So, I'll show you what it looks like now. what it looks like now. Let's get back to Let's get back to our project. So, our project. So, our project. So, we can raise the we can raise the we can raise the level. Oh, sorry. level. Oh, sorry. Excuse me. I Excuse me. I actually have to actually have to actually have to read this from the screen read this from the screen read this from the screen here. OK. So here. OK. So here. OK. So if we run the if we run the if we run the same query, same query, Great. So, we Great. So, we can see that it can see that it can see that it took a few took a few took a few tries to get tries to get tries to get everything right. Ahem.

everything right. Ahem. However, by the third However, by the third attempt, there is a attempt, there is a attempt, there is a small problem small problem small problem because we said: because we said: because we said: no regular no regular no regular expressions, no FTS. expressions, no FTS. expressions, no FTS. Um, so now he Um, so now he Um, so now he said: We'll just said: We'll just said: We'll just use match. But use match. But use match. But now it's an exact match. now it's an exact match. now it's an exact match. So, it tries So, it tries So, it tries to find to find to find authentication and authentication and authentication and fails. This fails. This fails. This means there is still something means there is still something means there is still something we can do. we can do. And that's it. We will add And that's it. We will add full-text search full-text search . It's pretty simple. Um, . It's pretty simple. Um, . It's pretty simple. Um, of course we need to of course we need to of course we need to create an index. And we'll create an index. And we'll create an index. And we'll combine it with the combine it with the combine it with the author and author and author and state filters in the same state filters in the same state filters in the same query. And so, now query. And so, now we can we can answer this answer this answer this question. Let's question. Let's question. Let's see what happens see what happens see what happens and if he and if he and if he really does what I really does what I really does what I said he would. Father. That's all.

said he would. Father. That's all. And now the end And now the end result is result is result is that if we that if we that if we go to the go to the go to the chat tab, we can chat tab, we can chat tab, we can ask more complex ask more complex ask more complex questions that the questions that the questions that the agent can actually agent can actually agent can actually think about behind think about behind think about behind the scenes. So, the scenes. So, the scenes. So, you could say something you could say something you could say something like: did like: did like: did we work on anything we work on anything we work on anything related to related to related to authentication authentication authentication last week? Here last week? Here last week? Here we see that it we see that it we see that it generated a request. generated a request. generated a request. Now he receives Now he receives Now he receives data and reflects data and reflects data and reflects on it. That's all. That's on it. That's all. That's on it. That's all. That's all. So, a quick all. So, a quick all. So, a quick summary of what we summary of what we summary of what we built. We are built. We are built. We are almost at the almost at the almost at the end. This will leave end. This will leave end. This will leave plenty of time for plenty of time for plenty of time for questions. So, to questions. So, to questions. So, to summarize, there are summarize, there are summarize, there are questions that RAG questions that RAG questions that RAG cannot cover. All these cannot cover. All these cannot cover. All these structural questions structural questions structural questions about things like about things like about things like state or time state or time state or time parameters—you won't parameters—you won't parameters—you won't get the right get the right get the right data by asking these data by asking these data by asking these types of questions with types of questions with traditional RAG. So,

traditional RAG. So, the same mechanism the same mechanism the same mechanism behind the behind the behind the request platform request platform request platform and the chat agent—they are and the chat agent—they are and the chat agent—they are the same. So if you the same. So if you the same. So if you want to want to want to experiment with experiment with experiment with this later, please do this later, please do this later, please do . It's the same . It's the same . It's the same foundation. The only difference foundation. The only difference , of course, is that a , of course, is that a , of course, is that a chat agent actually chat agent actually chat agent actually reasoned, while a reasoned, while a reasoned, while a query interface was query interface was query interface was designed only for designed only for designed only for queries. So, queries. So, queries. So, to summarize, here are the to summarize, here are the to summarize, here are the main ideas that main ideas that main ideas that can be can be can be reused. reused. reused. Schema-aware prompting, Schema-aware prompting, Schema-aware prompting, dynamic dynamic dynamic schema detection, schema detection, schema detection, identifier detection, identifier detection, identifier detection, validation, and validation, and error retries. The error retries. The error handling cycle is error handling cycle is error handling cycle is actually one of the actually one of the actually one of the most important things. most important things. We built a We built a system that can system that can system that can filter, filter, filter, aggregate, aggregate, aggregate, has time limits, has time limits, has time limits, full-text search full-text search , is validated, , is validated, , is validated, self-correcting, and self-correcting, and self-correcting, and supports a supports a supports a full-fledged chat full-fledged chat agent. You can agent. You can agent. You can take this into your take this into your take this into your organization and organization and organization and roll it out, although I would roll it out, although I would roll it out, although I would advise doing a little more advise doing a little more advise doing a little more work on work on validation tools before validation tools before internal internal internal implementation, but implementation, but implementation, but yes, this will answer yes, this will answer yes, this will answer all your questions all your questions all your questions about PRs. And that's it.

about PRs. And that's it. about PRs. And that's it. So, we left So, we left So, we left plenty of time for plenty of time for plenty of time for questions. Did you manage to set it up questions. Did you manage to set it up , or did everyone manage , or did everyone manage , or did everyone manage to set it up? Just to set it up? Just to set it up? Just curious. No, okay. curious. No, okay. curious. No, okay. If you want, you can If you want, you can If you want, you can come and pick up come and pick up come and pick up Brandon and me Brandon and me Brandon and me later. We'll just later. We'll just later. We'll just wait here, wait here, wait here, nearby. Oh, and also at nearby. Oh, and also at nearby. Oh, and also at stand P16. Yes, stand P16. Yes, stand P16. Yes, but it starts but it starts but it starts later today, later today, later today, around 4:00, and we'll around 4:00, and we'll around 4:00, and we'll be there all be there all be there all week. Steeply. Have week. Steeply. Have week. Steeply. Have a question? Yes. a question? Yes. a question? Yes. Yes, I have a Yes, I have a Yes, I have a question. I think you've laid out question. I think you've laid out question. I think you've laid out everything perfectly everything perfectly everything perfectly , the basics. I , the basics. I , the basics. I often think, "Okay, but often think, "Okay, but often think, "Okay, but how does this work at how does this work at how does this work at scale?" For example, scale?" For example, scale?" For example, you create a diagram and you create a diagram and you create a diagram and then add it to the then add it to the then add it to the system prompt. How does system prompt. How does system prompt. How does it work when it work when it work when scaling? It scaling? It scaling? It also depends on also depends on also depends on the size of your the size of your the size of your company, how do you company, how do you company, how do you fit all this data? fit all this data? fit all this data? How do you choose which How do you choose which How do you choose which data? And how data? And how data? And how long does it take to process long does it take to process long does it take to process all this? What is all this? What is all this? What is the cost? That's where the cost? That's where the cost? That's where my thoughts are headed. my thoughts are headed. my thoughts are headed. Yes, that's a good Yes, that's a good Yes, that's a good question. So, question. So, question. So, you know, I'll just you know, I'll just you know, I'll just repeat the question repeat the question repeat the question so everyone can so everyone can so everyone can hear it. The demo, hear it. The demo, hear it. The demo, of course, showed a of course, showed a of course, showed a rather artificial rather artificial rather artificial example where you example where you example where you only focus only focus only focus on merge queries.

on merge queries. on merge queries. So the question is, So the question is, So the question is, how do you scale this? how do you scale this? How do I do this for How do I do this for many different many different many different collections, tables, collections, tables, collections, tables, etc.? etc.? etc.? Another LLM level could be added, where the Another LLM level could be added, where the Another LLM level could be added, where the schema collections schema collections schema collections would have short would have short would have short summaries and descriptions. And summaries and descriptions. And summaries and descriptions. And then, depending on then, depending on then, depending on what the request is, what the request is, what the request is, you can you can you can download the desired download the desired download the desired schema. LLMs do schema. LLMs do schema. LLMs do this surprisingly well this surprisingly well this surprisingly well . . . It wasn't like that a year ago, It wasn't like that a year ago, It wasn't like that a year ago, but now they're very but now they're very but now they're very good at it. So this is a good at it. So this is a good at it. So this is a way to reduce way to reduce way to reduce the context, and this is the the context, and this is the the context, and this is the answer to your answer to your answer to your question about question about question about scaling. Was there scaling. Was there scaling. Was there any other aspect any other aspect any other aspect of scaling that of scaling that of scaling that interested you? interested you? interested you? This is getting all the This is getting all the This is getting all the data. And I think data. And I think data. And I think part of it is the part of it is the part of it is the cost. cost. cost. Yes, it's also Yes, it's also Yes, it's also data relevance, data relevance, data relevance, etc. etc. etc. How long does it How long does it How long does it take? For example, take? For example, take? For example, what data should we what data should we what data should we expect expect expect to be placed there? to be placed there? to be placed there? Yes, that's a good Yes, that's a good Yes, that's a good question. At Unblocked, we are question. At Unblocked, we are question. At Unblocked, we are constantly adding constantly adding constantly adding new content. We new content. We new content. We get get get webhook callbacks for webhook callbacks for webhook callbacks for all events, for example all events, for example , in PR. So every time there , in PR. So every time there , in PR. So every time there is a change, we get a is a change, we get a is a change, we get a webhook event, webhook event, webhook event, add content, add content, add content, rebuild it, rebuild it, rebuild it, and store it in the and store it in the and store it in the database. It's really a database. It's really a database. It's really a distributed distributed distributed systems question about how systems question about how systems question about how to scale this at the to scale this at the to scale this at the data collection level. It's the data collection level. It's the data collection level. It's the same with storage.

same with storage. same with storage. So yes. So yes. So yes. Yes, maybe that Yes, maybe that Yes, maybe that will make things more will make things more will make things more concrete. So, concrete. So, concrete. So, if you had to if you had to if you had to name the top three name the top three name the top three sources of data if sources of data if sources of data if someone came into this someone came into this someone came into this company today? company today? company today? Probably GitHub, maybe Probably GitHub, maybe Probably GitHub, maybe Confluence or something Confluence or something Confluence or something like that. like that. like that. Yes. Um, well, Slack for sure. Yes. Um, well, Slack for sure. Yes. Um, well, Slack for sure. A lot of the "why" questions and A lot of the "why" questions and A lot of the "why" questions and decision-making decision-making decision-making happens in happens in happens in Slack. In the context of Slack. In the context of Slack. In the context of such a such a such a structured structured structured query, people usually query, people usually query, people usually recall conversations they recall conversations they recall conversations they had in the past. And had in the past. And had in the past. And they can make a they can make a they can make a request to Unblocked and request to Unblocked and request to Unblocked and say, "Oh yeah, say, "Oh yeah, say, "Oh yeah, about 3 weeks about 3 weeks about 3 weeks ago I was chatting with ago I was chatting with ago I was chatting with so-and-so." And this is an so-and-so." And this is an so-and-so." And this is an ideal option ideal option ideal option for a structured for a structured for a structured query. Um, yeah, I would query. Um, yeah, I would query. Um, yeah, I would say Slack is say Slack is say Slack is most important for most important for most important for "why" questions. Many of the "why" questions. Many of the "why" questions. Many of the "why" solutions are also "why" solutions are also "why" solutions are also embedded in PR. PR has embedded in PR. PR has embedded in PR. PR has all this organizational all this organizational all this organizational history in the comments. history in the comments. And this is essentially a black And this is essentially a black hole, right? So you hole, right? So you hole, right? So you can view can view can view this data before this data before this data before processing and extract processing and extract processing and extract interesting ideas.

interesting ideas. For example, at Unblocked we For example, at Unblocked we create what we call " create what we call " best best best practice memories" that are practice memories" that are practice memories" that are pulled from pulled from pulled from PR comments in the PR comments in the PR comments in the past. Does this make sense? past. Does this make sense? past. Does this make sense? Yes. Yes. Yes. Aha. Anyone else? Yes. Aha. Anyone else? Yes. Aha. Anyone else? Yes. Um, question. So, Um, question. So, Um, question. So, you allow the model you allow the model you allow the model to generate comments to generate comments to generate comments and access the and access the database directly. How is that database directly. How is that better than, for example, better than, for example, better than, for example, creating an MCP and creating an MCP and creating an MCP and building that level, building that level, building that level, because you have that because you have that because you have that approach that you give approach that you give approach that you give it more access? it more access? Why not just Why not just give MCP control give MCP control give MCP control over what it can over what it can over what it can do instead of do instead of do instead of letting it letting it letting it create its own create its own create its own experience? experience? Yes, I mean Yes, I mean you can definitely you can definitely you can definitely do it. So, if do it. So, if do it. So, if you have a table with you have a table with you have a table with structured structured structured queries, you can queries, you can queries, you can express it through MCP. express it through MCP. express it through MCP. You can even You can even You can even do this dynamically, do this dynamically, do this dynamically, updating the tool updating the tool updating the tool every time the every time the every time the schema changes. This schema changes. This schema changes. This type of approach shows type of approach shows type of approach shows what is happening on the what is happening on the what is happening on the other side of the other side of the other side of the MCP call. You could have an MCP call. You could have an MCP call. You could have an MCP interface that MCP interface that MCP interface that simply accepts simply accepts simply accepts natural natural natural language requests, for example, and language requests, for example, and language requests, for example, and then translates that then translates that then translates that into the backend. Or you into the backend. Or you into the backend. Or you can provide a direct can provide a direct can provide a direct interface to the interface to the interface to the database. It depends database. It depends database. It depends on what the on what the on what the client interface looks like.

client interface looks like. client interface looks like. But yes. Yes. Yes. But yes. Yes. Yes. But yes. Yes. Yes. Yes, I have a Yes, I have a Yes, I have a question. question. question. Yes. Yes. Yes. Yes. Yes. Yes. I understand this a little I understand this a little I understand this a little better. Um, first better. Um, first better. Um, first question: I actually didn't question: I actually didn't question: I actually didn't quite understand how quite understand how quite understand how you created a schema from you created a schema from you created a schema from essentially schemaless essentially schemaless essentially schemaless data. data. Yes. Uh, what do you Yes. Uh, what do you mean by "how mean by "how mean by "how to combine", or... to combine", or... Let me make sure Let me make sure I I I understood your understood your understood your last question correctly. last question correctly. last question correctly. Uh, and how, and like... Uh, and how, and like... Yes. No, not at all. No, no , they completely , they completely , they completely complement each complement each complement each other. Yes. Yes. other. Yes. Yes. other. Yes. Yes. So, you can have a So, you can have a So, you can have a full-text search full-text search that does a kind of that does a kind of that does a kind of , you know, , you know, , you know, full-text search full-text search , right? So, it will , right? So, it will , right? So, it will identify identify identify keywords. You keywords. You keywords. You still need still need still need some mechanism to some mechanism to some mechanism to perform the actual perform the actual perform the actual semantic semantic semantic similarity search, because there are similarity search, because there are similarity search, because there are encoded ideas and encoded ideas and encoded ideas and meanings that meanings that meanings that are not necessarily are not necessarily are not necessarily textual matches.

textual matches. So, you need So, you need this conceptual this conceptual this conceptual mapping. Yes? So, mapping. Yes? So, mapping. Yes? So, as I said at the beginning, as I said at the beginning, as I said at the beginning, similarity search is similarity search is similarity search is actually very actually very actually very powerful for the powerful for the powerful for the initial stages. You initial stages. You initial stages. You can, for example, can, for example, can, for example, start with vector start with vector start with vector search, and then search, and then search, and then provide access to provide access to provide access to such tools, such tools, such tools, because then it will because then it will because then it will have interesting have interesting have interesting initial data and initial data and initial data and will be able to form will be able to form will be able to form queries to obtain queries to obtain queries to obtain information. So yeah, that's what information. So yeah, that's what information. So yeah, that's what we do. we do. we do. So, sorry, the first So, sorry, the first So, sorry, the first part of your part of your part of your question was...Yes. question was...Yes. Yes. So you take, Yes. So you take, you know, in [ __ ] these are all you know, in [ __ ] these are all you know, in [ __ ] these are all documents, and they're documents, and they're documents, and they're kind of kind of kind of formatted in JSON. So formatted in JSON. So formatted in JSON. So you can combine you can combine you can combine these things and look for these things and look for these things and look for certain tokens. And then, certain tokens. And then, certain tokens. And then, you know, pull out the you know, pull out the you know, pull out the tokens, and that's how you tokens, and that's how you tokens, and that's how you define the types. You define the types. You define the types. You can look at the can look at the can look at the source code and source code and source code and see that see that see that at least this at least this at least this iteration of the algorithm is iteration of the algorithm is iteration of the algorithm is pretty crude, right?

pretty crude, right? It does things like It does things like checking the checking the checking the number of characters number of characters number of characters from two to eight from two to eight from two to eight for for for type information. And for type information. And for type information. And for calculations, it calculations, it calculations, it also looks for certain also looks for certain also looks for certain patterns. Yes. Yes. Yes. patterns. Yes. Yes. Yes. patterns. Yes. Yes. Yes. Yes. Yes. as hybrid search This is a very This is a very ambiguous term ambiguous term . So I want to . So I want to . So I want to make sure I make sure I make sure I understand exactly what you're understand exactly what you're understand exactly what you're talking about. Do you talking about. Do you talking about. Do you mean hybrid mean hybrid mean hybrid search as a provider search as a provider search as a provider that performs that performs that performs semantic similarity as well semantic similarity as well semantic similarity as well as lexical as lexical as lexical search and then search and then search and then rank aggregation? rank aggregation? Yes, yes. Actually, with Yes, yes. Actually, with Atlas you can, as far as the Atlas you can, as far as the aggregation pipeline is concerned, aggregation pipeline is concerned, Yes.

Yes. Yes. have a full-text have a full-text have a full-text stage and a semantic stage and a semantic stage and a semantic stage. And this is for my own stage. And this is for my own stage. And this is for my own use. use. use. Yes, sorry. I work Yes, sorry. I work Yes, sorry. I work at Omni. This... at Omni. This... at Omni. This... Yes, oh yes, yes. No, so that's Yes, oh yes, yes. No, so that's Yes, oh yes, yes. No, so that's exactly what we're doing. exactly what we're doing. exactly what we're doing. Yes. The answer is yes, Yes. The answer is yes, Yes. The answer is yes, of course. Yes. of course. Yes. of course. Yes. Half was for the Half was for the Half was for the scheme sample. 300—that scheme sample. 300—that scheme sample. 300—that was was was Yes. Yes. Yes. Do you How did you arrive at Do you How did you arrive at Do you How did you arrive at this number as a this number as a this number as a really good really good really good set for sampling? set for sampling? Um, just an empirical Um, just an empirical trial and trial and trial and error method. So, I error method. So, I error method. So, I mean, there's mean, there's mean, there's no simple solution here. Um, no simple solution here. Um, no simple solution here. Um, behind the scenes, like in behind the scenes, like in behind the scenes, like in Unblocks, we Unblocks, we Unblocks, we use use use a lot of static a lot of static a lot of static schemes because we want to schemes because we want to schemes because we want to achieve absolute achieve absolute achieve absolute accuracy. It was accuracy. It was accuracy. It was just a technique that I just a technique that I just a technique that I wanted to show wanted to show wanted to show because, you know, for because, you know, for because, you know, for data exploration or data exploration or data exploration or experiments, or experiments, or experiments, or for connecting for connecting for connecting arbitrary databases, arbitrary databases, it's a good approach. We it's a good approach. We it's a good approach. We came to this came to this came to this through testing through testing through testing and iterations with and iterations with and iterations with customers. We were customers. We were customers. We were uploading a bunch of uploading a bunch of uploading a bunch of content from the content from the content from the document repository and document repository and document repository and getting feedback getting feedback getting feedback from customers: "Oh, you from customers: "Oh, you from customers: "Oh, you missed what I was missed what I was missed what I was looking for." We looking for." We looking for." We analyzed this, analyzed this, analyzed this, figured out the reason, and that's how we figured out the reason, and that's how we figured out the reason, and that's how we arrived at the arrived at the arrived at the sample size that sample size that sample size that made the most sense.

made the most sense. made the most sense. Yes, that definitely makes sense Yes, that definitely makes sense . Yes, because I think I've . Yes, because I think I've . Yes, because I think I've also encountered this with also encountered this with also encountered this with our clients, we our clients, we our clients, we have to have to have to adjust adjust adjust the number of sample the number of sample the number of sample documents. documents. Yes. This is absolutely Yes. This is absolutely true. Yes, true. Yes, true. Yes, definitely. You will always definitely. You will always definitely. You will always miss something. There will miss something. There will miss something. There will always be a always be a always be a document with some document with some document with some unusual unusual unusual aspect. Yes. Anyone aspect. Yes. Anyone aspect. Yes. Anyone else? I thought I else? I thought I else? I thought I saw a question there. saw a question there. saw a question there. Oh, yes. Excuse me, Oh, yes. Excuse me, Oh, yes. Excuse me, can you repeat the can you repeat the can you repeat the first part? Yes, first part? Yes, first part? Yes, we force we force we force the tool to be called the tool to be called the tool to be called not for not for not for pattern detection, because that pattern detection, because that pattern detection, because that happens at the very happens at the very happens at the very beginning. This is beginning. This is beginning. This is for extracting names.

for extracting names. for extracting names. We have a validator that We have a validator that We have a validator that monitors what the monitors what the monitors what the agent does. So, agent does. So, agent does. So, if it doesn't call the if it doesn't call the if it doesn't call the first part, which is first part, which is first part, which is name extraction, name extraction, name extraction, there are actually two there are actually two there are actually two techniques for doing that. techniques for doing that. The first is to create a The first is to create a tool that tool that tool that executes a query and executes a query and executes a query and takes a username takes a username takes a username as an as an as an argument. So you argument. So you argument. So you can abort the can abort the can abort the process if the username is process if the username is incorrect or malformed. If you malformed. If you want to perform want to perform want to perform name extraction via a name extraction via a name extraction via a tool call, tool call, tool call, you have to monitor you have to monitor you have to monitor the agent, and if it the agent, and if it the agent, and if it doesn't call it, you doesn't call it, you doesn't call it, you simply add simply add simply add a message from a message from a message from the user: "You the user: "You the user: "You forgot to call the forgot to call the forgot to call the extraction tool extraction tool ." "Do it now." ." "Do it now." Yes. Yes. How do you Yes. Yes. How do you handle change? For handle change? For handle change? For example, changes example, changes example, changes in the scheme. For example, the in the scheme. For example, the in the scheme. For example, the parameter name is Yes. So, parameter name is Yes. So, parameter name is Yes. So, remember that in this remember that in this remember that in this particular example, particular example, particular example, the schema is unknown at all the schema is unknown at all the schema is unknown at all . So as . So as . So as documents are added, documents are added, documents are added, this schema can this schema can this schema can be updated. Your be updated. Your be updated. Your choice is to choice is to re-generate

re-generate the pattern at a certain sampling interval. In our the pattern at a certain sampling interval. In our the pattern at a certain sampling interval. In our case, in Unblock, the schema is case, in Unblock, the schema is case, in Unblock, the schema is static and static and static and tied to our tied to our tied to our codebase. And we codebase. And we codebase. And we ourselves control this ourselves control this ourselves control this scheme. So if GitHub scheme. So if GitHub scheme. So if GitHub decides to change decides to change decides to change the schema, which is of course the schema, which is of course the schema, which is of course rare, rare, rare, it would have serious it would have serious it would have serious consequences for their consequences for their consequences for their customers. But if customers. But if customers. But if they did, we would they did, we would they did, we would just statically just statically just statically accept any accept any accept any changes and then changes and then changes and then obviously deploy obviously deploy obviously deploy them via CI. Anyone else? them via CI. Anyone else? Perfectly. I think Perfectly. I think we can we can we can save some save some save some time in that case. Thank you.