Better Search Took This AI Agent From 36% to 98% Accuracy - Aamir Shakir, Mixedbread

[music] So quickly about mixread and what we're doing before I jump into the interesting things. So mixread started as a research lab around retrieval since we were really sad about the state how the space was evolving. So we have been doing research for the two and a half years over how we can make retrieval really really good and we come ended up with a product where you can ingest any kind of data and make it searchable for agents meaning PDF, spreadsheets, images, video, audio in over 100 languages. So it's a true multimodel multilingual search where we are owning the models. So we built the models from scratch specifically for search. We build our own database so we can provide zero data retention and all the privacy privacy requirements you need and also we build injection pipeline so you don't have to hustle to managing injection managing OCR and knowing what to do and instead you just dump your data into it rely on our models working good enough and you know so you can build really really powerful agents and I'm really happy that over the last three months like we're live since 5 months but over the last three months we onboarded banks and hedge funds into mixread and alive with it with billions of documents and the main reason is we are so ISO compliant and we have this bring your own bucket mode meaning that we can really offer all the security requirements you need with zero data retention so when we talk

about evidence layer or using really really powerful agents right it mostly comes down to working on your data so if you have like oh sorry um how to move this forward so assume that you're building an agent or you have an equity analyst this is just like an example query because Intel yesterday released their new GPU cards, right? Like the D270 and somehow it has a really weird design that it's using a lot of LPDDR5 memory. So, you know, like you're an analyst sitting there and what the Why is Intel publishing this, right? Why why does it have so much LPDDR5 and not HBM memory? So, it would ask your system a question like, hey, why does the new Intel D270 use LDR5 memory? And since you guys probably as you're working in finance, you have a lot of data brokers and data providers. They send you the podcast, they send you interviews, they send you materials, research uh stuff and so on and so forth. So you have this huge knowledge base and now your analyst is asking this question, right? So when you use cloud code or cloud cowwork, what it normally does when you have your ingest data locally, it uses the terminal and basically uh lexical search like grab to figure out, hey, where do we find terms like ADDPR5 or Intel and GPUs to just figure out, hey, what documents can be the closest and the most relevant ones? And as you have a lot of PDFs and a lot of documents and interviews, you will get a lot of hits and you have no relevancy and you don't know where to look into.

So what the model will end up doing is reading every document and like this, you know, like your token cost will just go up and up and up and the more documents you have, you will end up with per query costing 10 or $20 to just ask a really basic question. And the second downside of that is also that most PDFs sadly are not really grapable because you know like there are binary formats sometimes there are scans in there images in there so you can't really exploit this multimodel data relying in there to work for you. So these are like the key bottlenecks and then you think okay what are people doing and what people have done previously in the spade space who's familiar with the term rack retrie augmented generation just hands up really quickly. So I think most people so uh I think you will it will resonate what I'm going to say now. So then we were like okay good let's let's build a sophisticated pipeline where can make all this data really really usable for us. So we have this billions of PDFs podcast whatsoever. So first thing we do is we do parsing right. So if you have um audio we do transcription um if you have PDF we do OCR and what every of the step does it introduces noise. So if you can if you don't process tables properly, if you don't pass diagrams properly, what will happen is that you will just lose information and then you do the chunking. So you already lost information because your table is not passed properly or characters are not processed properly, then you do some chunking, right? So if you have figures,

table, pages, where's really the chunk boundary? So you're introducing more noise and then you're giving it to a pure embedding model which you know like takes this information and then tries to guess a point in the space to you know like what this information represents and since it already lost so much information you are losing then having this point right and this is then a super lossy process from going from this really rich information to hey here's just a point and then you put it into a vector database do some retrieval and rerank. So since every step is lossy and you have to add a vendor for every process, you will end up hey what is the best OCR? What is the best chunking strategy? What is the best embedding model? Uh what is the best vector database? What is the best ranking? So you will just evaluate it for six or eight months different vendors run through security to just go live and you have to allocate four to five people to that to just build something from my perspective super basic which won't work in the end because there's some inherent limits of how this things are done and we internally call it the Frankenstein stack and there like some proofs why you know like the single vector models or like just putting all this information into embedding model and one point actually doesn't really work and we can discuss it later if you guys are interested why and I can go on a rant and explain it a bit more in depth but this is actually right now the state-of-the-art what people use and

this is also the reason like all the bottlenecks I described right now why rack actually never worked so you know like I think you all remember this promise of hey here's spine cone or here's any vector database just put it in and it will just work but it never really worked out and it comes down to that every step is super lossy and that also the models we have the embedding models with the technologies they're using at the moment are not good enough so you you can't really use this information properly and the downside when I talk about this embedding models is you have to imagine like how this embedding model work right you have an information like I love bread you put it into a model and then it just takes this sentence and puts it somewhere in the space and later if you ask hey who loves bread we'll put also into the model and we'll put it somewhere into space as well and then just search around the query. Hey, where's the similar point to basically describe that but now let's think about a financial report like a sec filling.

If you have like complex tables with all this different information like net income, revenue yada yada yada what does this point actually represent? Right? So maybe it uh represents information about Nvidia. So it's Nvidia's earnings report, right? But if you have the earnings report from 1996 or 2000 and 2026 the points will look really really similar. That's why when you have a lot of information which look very similar rack and all this vector database approaches don't work and we see with a lot of customers in their classic approach if you have just few documents let's say tens of hundreds of documents the quality is pretty good but as you start scaling and going into production the answer quality will just degrade significantly because this type of vector search and this type of technology is not able to really distinguish between this fine grained information and the issue is you have a pure a poor agent, right, who wants to really help you. He he will do everything his to serve you and do everything for you. So what will this agent start doing? It will just query aggressively. It will spin up tens of sub agents. Um we'll do more and more queries. So you know it will just fill up the context window and every query will become expensive. You can answer it like this by just putting a lot of agents on that. But the question is hey is it really the way you want to go into production? If you also imagine at one point that maybe your fund or your VC will be mostly run by agents and the volume of agent traffic will just increase. So the cost of

running the systems will just skyrocket. So this is why we came to the conclusion pretty early when we're working with our customers that intelligence is there but the real real bottleneck is actually access to your data and to your knowledge and we believe that we will see a unbundling of intelligence and data where you have intelligence where you can choose between different intelligence models but you need to have the right access for this intelligence for your data so you can make the most of that and for this we built um mix bread. So as I mentioned mixread started as a research lab and this is a really interesting graphic and this is a preview of what's coming next week. Um is that retrieval how it changed like over the last from 1994 to now it's like what 32 years I think my if my mental math is not wrong. So we started with BM25. This is what elastic search is doing like simple keyword matching and then we just started coming up with smarter method. So the first really good embedding models were published in 2022.

So it took the space 27 years to come something up for text search better than BM25. But since then we have more like a linear um like improvement. And this is also what we meant with rack not working because there was never a step function change like we saw with chatbt and so and so forth and we published our first model by beginning uh we finished the training at end of last year published it beginning of this year it was pretty solid but the thing we're publishing next week and we already live with a lot of our customers we think is a step function change which overcomes most of the challenges I described before just in short what this model does is it's a late interaction model Meaning instead of having just one embedding, you have multiple embeddings for your documents and you have multiple embeddings for your queries like this. You can really describe fine grained information on a page. So going back to the SEC filling for Nvidia for different years, right? So you would have the model would figure out an embedding for net income for net revenue etc etc but since it's like creating it in one pass it also know it's about Nvidia. So every embedding is contextualized and with the information which are relevant to you and all this technique and scaling up the model and better data gives us significant jump in quality uh which allows to do some cool things which allows to actually build this product uh mix where you just upload your data mix takes care of indexing it for you like every page as it looks because it's multimodel I will show what it means and you then just can

give it agent access to mixread and the agent does the querying the reading the grabbing everything for you and you can just focus on on what's fun actually building the real application and actually building real RARI right so um let me show you what it means so this is mixre and what I did here is I uploaded some um Intel um wait hold on where is it exactly so I uploaded some Intel reports here for example the Intel earning stack so what you do in mix is just upload the uh page. The internet is pretty slow. I [snorts] hope it will be fine. So, it's it's failed to load because the internet is slow here. I'm so sorry about that. Um I should have recorded a video. Um so, let's do this. Cool. So, what it does here is so we have a slide. Come on, dude. So, you see there's a uh is there's a figure, right? So normally if you would just do OCR and the figure and you want to know the actually the revenue of Intel in Q2 2025 if you just would do OCR you would just all lose all information it would never really able to match the page because you need to understand what's going on there but if you create the embedding on the visual perspection like looking let the model look at the page the screenshot and create the embedding based on that you are getting rid of all this useless stuff I was talking before but since not all LM are multimodel mixread still provides stuff like layout information

and OCR for your downstream thing where you can decide to put in the screenshot or the text yourself to do the right thing and let's just try it out together. So let's say we want to know hey what was in Intel's foundry segment operation mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar mar margin in Q2 2025 it goes out I hope it will work now normally it's like sub 200 milliseconds but you saw the thing before um so we see hey it finds somehow this page here which is talking about in Foundry and down here come on you can do it here you go it's talking about Q2 to 2025 and it understands also hey it's somehow about segment revenue and can really pass all this information and since we have this visual layout stuff we can actually reference this with bounding boxes and when your analyst or your agent is deriving to a decision you can actually backtrack and site where it is coming from and we're providing all the APIs and all the primitives for you to build this experience yourself and if you have like really really complex queries for example you want to know like something more qualitative Hey uh is Intel's foundry uh prof uh profitably improving and what is driving it? So if you just query it for example our system realize hey this is a more complex query let's do a genetic search a genetic search is a model we trained toast one it's like a agent which does multiple queries for you multihop

reasoning and it's as good as Astra and fable while being 40 times faster and cheaper which then goes out and does like multiple queries um and can really decompose everything and come back with the right data and uh this data can then fed into an LLM to actually explain what what what is happening there and we just also expose it as a primitive. This is just a demo, a playground to show you what's possible, but you can just use it, give it to an agent and let the agent decide if you want to use grab, normal search or the agentic search depending on the complexity. Exactly. And yeah, um so this is uh was as a backup if it wouldn't have worked. Um so the agent can do everything. And here a small case study which we will publish also in two weeks from now. It's like a really big hedge fund. We on boarded within like one and a half weeks. um because they have like over 1 billion documents right now and they are building a lot of internal agent platform for the analyst and for their fund managers to derive qualitative uh analysis and also come up with should we is it a good decision or not and before they have built what I described before this chunking betting OCR some random embedding model some vector database and they had a answer accuracy like how often the agent is right of 36% where the human experts knew the answer and graded And once they move to mix spread the answer accuracy moved to 98% because the again the issue is not intelligence it's again getting the right data and just by providing way better search to

your agents you can first make it really good um here's a breakdown on multimodality and text questions and as you can see multimodality is just bumps it quite significantly but second is you use also wafier tokens so the LM spend is going down by 3x because again most of the tokens are used to find, pass, and read information. And as you just fit in really good context, you know, like you just need way less LLM cost and the LM can just focus on what's relevant. Um, exactly. So, this is my talk in a nutshell. Um, if you want to try out MixPread, uh, this we have toast, uh, flyers downstairs, so if you want to take them, I think they're pretty funny. Um, and you can sign up. We have free credits. Uh, if you are concerned about security, you can just write us. We activate bring on bucket within 5 minutes for all of our customers and then you can just start trying it and if you have like really large corpuses we provide a lot of free credits so you can benchmark it before really onboarding it.