Anthropic Rebuilt Git (yes really)

For as long as I've been writing code, For as long as I've been writing code, I've been using Git and GitHub. Funny I've been using Git and GitHub. Funny I've been using Git and GitHub. Funny enough, I actually started using Git enough, I actually started using Git enough, I actually started using Git before I wrote code cuz I used it to before I wrote code cuz I used it to before I wrote code cuz I used it to manage plugins for my Minecraft servers manage plugins for my Minecraft servers manage plugins for my Minecraft servers way way back when I was in like early way way back when I was in like early way way back when I was in like early high school. Git is core to the way that high school. Git is core to the way that high school. Git is core to the way that I build and work with software. But I I build and work with software. But I I build and work with software. But I have been starting to notice rough have been starting to notice rough have been starting to notice rough edges. A lot of the way that Git was edges. A lot of the way that Git was edges. A lot of the way that Git was built was around code being expensive built was around code being expensive built was around code being expensive and humans needing to review the code, and humans needing to review the code, and humans needing to review the code, approve the code, and control where this approve the code, and control where this approve the code, and control where this code goes because losing code or merging code goes because losing code or merging code goes because losing code or merging the wrong code was a very expensive and the wrong code was a very expensive and the wrong code was a very expensive and painful thing. The reality of how Git painful thing. The reality of how Git painful thing. The reality of how Git works has influenced the whole industry works has influenced the whole industry works has influenced the whole industry from how we work with our team to how from how we work with our team to how from how we work with our team to how products like GitHub are built to the products like GitHub are built to the products like GitHub are built to the way that we think about things that way that we think about things that way that we think about things that should be basic like secrets management. should be basic like secrets management. should be basic like secrets management. Shout out to my viral clip saying that Shout out to my viral clip saying that Shout out to my viral clip saying that we should be able to commit our we should be able to commit our we should be able to commit our environment variables. Seriously though, environment variables. Seriously though, environment variables. Seriously though, Git is not the right primitive for the Git is not the right primitive for the Git is not the right primitive for the way that things are moving right now. way that things are moving right now. way that things are moving right now. I'll be real with y'all. A lot of my I'll be real with y'all. A lot of my I'll be real with y'all. A lot of my problems with Git are coming from GitHub problems with Git are coming from GitHub problems with Git are coming from GitHub because it's just a slow and miserable because it's just a slow and miserable because it's just a slow and miserable platform. External estimates are platform. External estimates are platform. External estimates are measuring as much as 7% downtime across measuring as much as 7% downtime across measuring as much as 7% downtime across different things within GitHub. I can't different things within GitHub. I can't different things within GitHub. I can't tell you how many times we've had tell you how many times we've had tell you how many times we've had releases blocked because actions were releases blocked because actions were releases blocked because actions were failing or I haven't been able to merge failing or I haven't been able to merge failing or I haven't been able to merge an important change because the page an important change because the page an important change because the page wasn't loading. The fact that GitHub's wasn't loading. The fact that GitHub's wasn't loading. The fact that GitHub's uptime looks more like a Christmas tree uptime looks more like a Christmas tree uptime looks more like a Christmas tree than a green bar like it's supposed to than a green bar like it's supposed to than a green bar like it's supposed to should say everything you need to know.

should say everything you need to know. should say everything you need to know. But what if I told you GitHub wasn't the But what if I told you GitHub wasn't the But what if I told you GitHub wasn't the only problem here? that git itself is only problem here? that git itself is only problem here? that git itself is not the right solution for working not the right solution for working not the right solution for working heavily in parallel across a lot of heavily in parallel across a lot of heavily in parallel across a lot of people and even more agents. I've been people and even more agents. I've been people and even more agents. I've been thinking about this a lot because back thinking about this a lot because back thinking about this a lot because back in December I explored JJ and I loved it in December I explored JJ and I loved it in December I explored JJ and I loved it and ever since then I've been thinking and ever since then I've been thinking and ever since then I've been thinking about what would it look like to build about what would it look like to build about what would it look like to build something new for shipping really really something new for shipping really really something new for shipping really really fast with agents. It turns out I'm not fast with agents. It turns out I'm not fast with agents. It turns out I'm not the only one who's been thinking about the only one who's been thinking about the only one who's been thinking about this. Anthropic just put up an article this. Anthropic just put up an article this. Anthropic just put up an article about their new in-house source control about their new in-house source control about their new in-house source control system. Back in May, they were merging system. Back in May, they were merging system. Back in May, they were merging 2500 changes per day, but now in 2500 changes per day, but now in 2500 changes per day, but now in September, they're doing more than September, they're doing more than September, they're doing more than 30,000 changes merged a day. They are 30,000 changes merged a day. They are 30,000 changes merged a day. They are now bottlenecked by developer now bottlenecked by developer now bottlenecked by developer infrastructure that was built for humans infrastructure that was built for humans infrastructure that was built for humans and not for agents. And they've decided and not for agents. And they've decided and not for agents. And they've decided to build something of their own. It to build something of their own. It to build something of their own. It seems kind of crazy that agents are now seems kind of crazy that agents are now seems kind of crazy that agents are now smart enough to do things like this, but smart enough to do things like this, but smart enough to do things like this, but they really are. And I love this idea of they really are. And I love this idea of they really are. And I love this idea of challenging the fundamentals of how challenging the fundamentals of how challenging the fundamentals of how you're building to make the right you're building to make the right you're building to make the right solution for yourself. I have a feeling solution for yourself. I have a feeling solution for yourself. I have a feeling this one's going to be a really fun deep this one's going to be a really fun deep this one's going to be a really fun deep dive, especially now that I'm ripping dive, especially now that I'm ripping dive, especially now that I'm ripping things out like TypeScript for my own things out like TypeScript for my own things out like TypeScript for my own vibecoded alternatives. That said, I vibecoded alternatives. That said, I vibecoded alternatives. That said, I don't think everything should be ripped don't think everything should be ripped don't think everything should be ripped out for your own vibecoded alternatives.

out for your own vibecoded alternatives. out for your own vibecoded alternatives. There's a lot of places where it makes There's a lot of places where it makes There's a lot of places where it makes sense to use a vendored product. Like sense to use a vendored product. Like sense to use a vendored product. Like for example, today's sponsor. It kind of for example, today's sponsor. It kind of for example, today's sponsor. It kind of feels like AI has solved most of modern feels like AI has solved most of modern feels like AI has solved most of modern webdev, but there are some things it webdev, but there are some things it webdev, but there are some things it hasn't solved. In particular, mobile. As hasn't solved. In particular, mobile. As hasn't solved. In particular, mobile. As good as it is at starting up an app and good as it is at starting up an app and good as it is at starting up an app and making small changes here and there, I making small changes here and there, I making small changes here and there, I find myself just fighting the model find myself just fighting the model find myself just fighting the model constantly when I'm trying to actually constantly when I'm trying to actually constantly when I'm trying to actually get my apps ready to ship, fix real get my apps ready to ship, fix real get my apps ready to ship, fix real bugs, deal with Apple reviews, and all bugs, deal with Apple reviews, and all bugs, deal with Apple reviews, and all these other pieces. As I ran into more these other pieces. As I ran into more these other pieces. As I ran into more and more walls as we tried shipping the and more walls as we tried shipping the and more walls as we tried shipping the apps we were building, I hit up my apps we were building, I hit up my apps we were building, I hit up my friend Jamon at Infinite Red, and he has friend Jamon at Infinite Red, and he has friend Jamon at Infinite Red, and he has made life so much easier. If you're not made life so much easier. If you're not made life so much easier. If you're not familiar with these guys, they're familiar with these guys, they're familiar with these guys, they're industryleading experts in all things industryleading experts in all things industryleading experts in all things mobile, especially React Native and mobile, especially React Native and mobile, especially React Native and Expo, but also Swift and Cotlin as well. Expo, but also Swift and Cotlin as well. Expo, but also Swift and Cotlin as well. There's a reason that everyone from There's a reason that everyone from There's a reason that everyone from Domino's to Microsoft has called these Domino's to Microsoft has called these Domino's to Microsoft has called these guys in. They really know how to ship. guys in. They really know how to ship. guys in. They really know how to ship. I'm gonna be real with you guys. The I'm gonna be real with you guys. The I'm gonna be real with you guys. The role of a consultancy has fundamentally role of a consultancy has fundamentally role of a consultancy has fundamentally changed as a result of AI. You're not changed as a result of AI. You're not changed as a result of AI. You're not just hiring these guys to come in and do just hiring these guys to come in and do just hiring these guys to come in and do grunt code work anymore. You have agents grunt code work anymore. You have agents grunt code work anymore. You have agents for that. What you bring them in for is for that. What you bring them in for is for that. What you bring them in for is the expertise. Making sure you're not the expertise. Making sure you're not the expertise. Making sure you're not making simple mistakes. Getting your making simple mistakes. Getting your making simple mistakes. Getting your pipeline set up so you can actually pipeline set up so you can actually pipeline set up so you can actually build and ship the apps. Making it build and ship the apps. Making it build and ship the apps. Making it possible for your agents to actually possible for your agents to actually possible for your agents to actually verify their changes and know things are verify their changes and know things are verify their changes and know things are working. And just making sure everything working. And just making sure everything working. And just making sure everything start to finish is ready to go. In a start to finish is ready to go. In a start to finish is ready to go. In a world where we're all shipping faster, world where we're all shipping faster, world where we're all shipping faster, these guys can come in and fix anything these guys can come in and fix anything these guys can come in and fix anything from your team to your app. They can from your team to your app. They can from your team to your app. They can build a whole new app from scratch. They build a whole new app from scratch. They build a whole new app from scratch. They can take over development, or they can can take over development, or they can can take over development, or they can just give you the pipelines you need to just give you the pipelines you need to just give you the pipelines you need to do it yourselves. If you want to ship do it yourselves. If you want to ship do it yourselves. If you want to ship better apps sooner with less bugs, look better apps sooner with less bugs, look better apps sooner with less bugs, look no further than/infinite no further than/infinite no further than/infinite red. Man, I'm actually really excited to red. Man, I'm actually really excited to red. Man, I'm actually really excited to dive into this article. As soon as I saw dive into this article. As soon as I saw dive into this article. As soon as I saw it drop, I wanted to read it, but I it drop, I wanted to read it, but I it drop, I wanted to read it, but I decided to put it off for this filming.

decided to put it off for this filming. decided to put it off for this filming. So, you're getting my organic take on So, you're getting my organic take on So, you're getting my organic take on this. If you're wondering why the site this. If you're wondering why the site this. If you're wondering why the site looks gross, it's because when I leave looks gross, it's because when I leave looks gross, it's because when I leave it on the default light mode that the it on the default light mode that the it on the default light mode that the blog is in, everyone yells at me. So, blog is in, everyone yells at me. So, blog is in, everyone yells at me. So, I'm using the dark reader extension to I'm using the dark reader extension to I'm using the dark reader extension to make it not just bright white. And this make it not just bright white. And this make it not just bright white. And this is the poop green brown color it chose. is the poop green brown color it chose. is the poop green brown color it chose. Redesigning source control at Anthropic. Redesigning source control at Anthropic. Redesigning source control at Anthropic. As our merges per day grew more than As our merges per day grew more than As our merges per day grew more than 10fold, source control became our 10fold, source control became our 10fold, source control became our biggest development bottleneck. Here we biggest development bottleneck. Here we biggest development bottleneck. Here we share how we migrated to an in-house share how we migrated to an in-house share how we migrated to an in-house version control system in 5 months. I version control system in 5 months. I version control system in 5 months. I won't lie, the five months feels like won't lie, the five months feels like won't lie, the five months feels like one of those clawed timeline one of those clawed timeline one of those clawed timeline hallucinations when you ask it how long hallucinations when you ask it how long hallucinations when you ask it how long will this take and it's like oh two to will this take and it's like oh two to will this take and it's like oh two to three days and then it finishes the work three days and then it finishes the work three days and then it finishes the work in an hour. Regardless, this is in an hour. Regardless, this is in an hour. Regardless, this is impressive. Replacing Git at a company impressive. Replacing Git at a company impressive. Replacing Git at a company as big as Anthropic sounds insane. Last as big as Anthropic sounds insane. Last as big as Anthropic sounds insane. Last I checked, they had around 2,000 I checked, they had around 2,000 I checked, they had around 2,000 employees. So yeah, at Anthropic, coding employees. So yeah, at Anthropic, coding employees. So yeah, at Anthropic, coding is no longer the bottleneck to building is no longer the bottleneck to building is no longer the bottleneck to building systems or doing research. Claude is systems or doing research. Claude is systems or doing research. Claude is able to autonomously iterate on research able to autonomously iterate on research able to autonomously iterate on research loops, microoptimize complex programs loops, microoptimize complex programs loops, microoptimize complex programs and build a new services from the ground and build a new services from the ground and build a new services from the ground up at a speed and scale we've never seen up at a speed and scale we've never seen up at a speed and scale we've never seen before. The bottlenecks now all of the before. The bottlenecks now all of the before. The bottlenecks now all of the developer info that was built for human developer info that was built for human developer info that was built for human scale source control builds CI things scale source control builds CI things scale source control builds CI things like that. In May of 2026, Anthropic like that. In May of 2026, Anthropic like that. In May of 2026, Anthropic averaged around 2500 code changes merged averaged around 2500 code changes merged averaged around 2500 code changes merged a day. As of September, it's over 30,000 a day. As of September, it's over 30,000 a day. As of September, it's over 30,000 a day. We don't expect this rate of a day. We don't expect this rate of a day. We don't expect this rate of growth to slow anytime soon. The latent growth to slow anytime soon. The latent growth to slow anytime soon. The latent demand for merging code is practically demand for merging code is practically demand for merging code is practically infinite and every bottleneck we remove infinite and every bottleneck we remove infinite and every bottleneck we remove immediately leads to further increasing immediately leads to further increasing immediately leads to further increasing the amount of code that we produce. The the amount of code that we produce. The the amount of code that we produce. The way I've framed this in the past is way I've framed this in the past is way I've framed this in the past is derisking the merge button, making it so derisking the merge button, making it so derisking the merge button, making it so by the time a person's looking at merge,

by the time a person's looking at merge, by the time a person's looking at merge, it's pretty likely the code is safe to it's pretty likely the code is safe to it's pretty likely the code is safe to merge. But also the other side where if merge. But also the other side where if merge. But also the other side where if you merge something you shouldn't have, you merge something you shouldn't have, you merge something you shouldn't have, the blast radius and the damage that it the blast radius and the damage that it the blast radius and the damage that it can do is reduced massively by systems can do is reduced massively by systems can do is reduced massively by systems to make it easy to test the code once it to make it easy to test the code once it to make it easy to test the code once it goes out with a limited set of users and goes out with a limited set of users and goes out with a limited set of users and revert the things if they do actually revert the things if they do actually revert the things if they do actually end up breaking. And I've seen the end up breaking. And I've seen the end up breaking. And I've seen the difference when we introduce these types difference when we introduce these types difference when we introduce these types of things. We just ship faster. For of things. We just ship faster. For of things. We just ship faster. For example, I stayed up a bit too late last example, I stayed up a bit too late last example, I stayed up a bit too late last night because I was annoyed with some night because I was annoyed with some night because I was annoyed with some niche performance issues when you had a niche performance issues when you had a niche performance issues when you had a really big database in T3 code. So, I really big database in T3 code. So, I really big database in T3 code. So, I decided to fix as much of it as I decided to fix as much of it as I decided to fix as much of it as I possibly could in one sitting, as well possibly could in one sitting, as well possibly could in one sitting, as well as clearing out some old issues in PRs as clearing out some old issues in PRs as clearing out some old issues in PRs that were annoying me, as well as taking that were annoying me, as well as taking that were annoying me, as well as taking over the PRs that weren't quite over the PRs that weren't quite over the PRs that weren't quite implemented how I wanted. And between implemented how I wanted. And between implemented how I wanted. And between the hours of like midnight and I think the hours of like midnight and I think the hours of like midnight and I think around 3:00 or 4:00 a.m., I merged from around 3:00 or 4:00 a.m., I merged from around 3:00 or 4:00 a.m., I merged from here through to the top of this page to here, to here, another full page of just Theo merges. another full page of just Theo merges. another full page of just Theo merges. You'll see t3.gg on the left for I think You'll see t3.gg on the left for I think You'll see t3.gg on the left for I think every single one of these. Some of them every single one of these. Some of them every single one of these. Some of them are multiple people, but they are me.

are multiple people, but they are me. are multiple people, but they are me. We're still going We're still going We're still going all the way up. Okay, these were two I all the way up. Okay, these were two I all the way up. Okay, these were two I merged from others to here. merged from others to here. merged from others to here. That is three full pages of GitHub merge That is three full pages of GitHub merge That is three full pages of GitHub merge history that I did in 4 hours. I know it history that I did in 4 hours. I know it history that I did in 4 hours. I know it was four hours because I did it between was four hours because I did it between was four hours because I did it between nightly releases. We do every six hours nightly releases. We do every six hours nightly releases. We do every six hours because I wanted to have as much in as because I wanted to have as much in as because I wanted to have as much in as possible for the nightly. So before I possible for the nightly. So before I possible for the nightly. So before I went to bed, I could download it, test went to bed, I could download it, test went to bed, I could download it, test it, make sure all was good, fix anything it, make sure all was good, fix anything it, make sure all was good, fix anything that was broken, and revoke the build if that was broken, and revoke the build if that was broken, and revoke the build if it was bad, and then go to bed smiling, it was bad, and then go to bed smiling, it was bad, and then go to bed smiling, knowing that performance would be way knowing that performance would be way knowing that performance would be way better for everybody on the nightly when better for everybody on the nightly when better for everybody on the nightly when I woke up. The fact that I could I woke up. The fact that I could I woke up. The fact that I could confidently merge 50 plus PRs in a confidently merge 50 plus PRs in a confidently merge 50 plus PRs in a sitting, it just feels insane. Like sitting, it just feels insane. Like sitting, it just feels insane. Like imagine going even a year back and imagine going even a year back and imagine going even a year back and telling yourself that you were merging telling yourself that you were merging telling yourself that you were merging 50 plus PRs a day. Hard to fathom. And 50 plus PRs a day. Hard to fathom. And 50 plus PRs a day. Hard to fathom. And if Anthropic was just saying these if Anthropic was just saying these if Anthropic was just saying these things and I hadn't experienced it things and I hadn't experienced it things and I hadn't experienced it myself, I'd be calling [ __ ] right myself, I'd be calling [ __ ] right myself, I'd be calling [ __ ] right now. But this is reality. If you're not now. But this is reality. If you're not now. But this is reality. If you're not moving this fast yet, you are either in moving this fast yet, you are either in moving this fast yet, you are either in like a place where you absolutely can't like a place where you absolutely can't like a place where you absolutely can't because people will die or you just because people will die or you just because people will die or you just don't realize how powerful these things don't realize how powerful these things don't realize how powerful these things are and you haven't taken the time to are and you haven't taken the time to are and you haven't taken the time to set up the tooling to make the stuff set up the tooling to make the stuff set up the tooling to make the stuff less risky. So let's dive in a bit here.

less risky. So let's dive in a bit here. less risky. So let's dive in a bit here. This is interesting. They have blue and This is interesting. They have blue and This is interesting. They have blue and orange. Blue is pull requests. Orange is orange. Blue is pull requests. Orange is orange. Blue is pull requests. Orange is antique change requests. That must be antique change requests. That must be antique change requests. That must be their new thing. It lines up with the their new thing. It lines up with the their new thing. It lines up with the way they name [ __ ] Antique seems like way they name [ __ ] Antique seems like way they name [ __ ] Antique seems like their system for all of this. If you their system for all of this. If you their system for all of this. If you think this is anthropic being uniquely think this is anthropic being uniquely think this is anthropic being uniquely weird and silly here, there's a long weird and silly here, there's a long weird and silly here, there's a long history of companies reinventing source history of companies reinventing source history of companies reinventing source control for their specific needs. For control for their specific needs. For control for their specific needs. For example, Meta built a crazy system example, Meta built a crazy system example, Meta built a crazy system around Mercurial that I think is around Mercurial that I think is around Mercurial that I think is actually a hard fork of Mercurial. They actually a hard fork of Mercurial. They actually a hard fork of Mercurial. They built all of that to make their stacked built all of that to make their stacked built all of that to make their stacked diff system as strong as possible for diff system as strong as possible for diff system as strong as possible for when people were working on large arcs when people were working on large arcs when people were working on large arcs and approving pieces of it to merge it and approving pieces of it to merge it and approving pieces of it to merge it all in at once. Graphite is a whole all in at once. Graphite is a whole all in at once. Graphite is a whole company that was built on top of GitHub company that was built on top of GitHub company that was built on top of GitHub to bring the way meta worked with source to bring the way meta worked with source to bring the way meta worked with source control to people who were on GitHub control to people who were on GitHub control to people who were on GitHub because it was so far ahead of how we because it was so far ahead of how we because it was so far ahead of how we were building if we you were shipping were building if we you were shipping were building if we you were shipping fast. At my company, Ping Labs, we were fast. At my company, Ping Labs, we were fast. At my company, Ping Labs, we were heavy users of graphite. Now they're heavy users of graphite. Now they're heavy users of graphite. Now they're part of cursor and they're building an part of cursor and they're building an part of cursor and they're building an alternative to GitHub into cursor with alternative to GitHub into cursor with alternative to GitHub into cursor with origin. The point I'm trying to make origin. The point I'm trying to make origin. The point I'm trying to make here is that this isn't as unprecedented here is that this isn't as unprecedented here is that this isn't as unprecedented as it might seem. And many companies as it might seem. And many companies as it might seem. And many companies have found meaningfully better ways to have found meaningfully better ways to have found meaningfully better ways to work with and manage their source work with and manage their source work with and manage their source outside of Git. And Jold here, who used outside of Git. And Jold here, who used outside of Git. And Jold here, who used to work at Meta and also works at to work at Meta and also works at to work at Meta and also works at Graphite and now cursor pointed out that Graphite and now cursor pointed out that Graphite and now cursor pointed out that a lot of the reason Meta built their a lot of the reason Meta built their a lot of the reason Meta built their systems and themselves instead of using systems and themselves instead of using systems and themselves instead of using off-the-shelf things is because their off-the-shelf things is because their off-the-shelf things is because their massive checkouts just didn't work well massive checkouts just didn't work well massive checkouts just didn't work well with Git. I vaguely remember the history with Git. I vaguely remember the history with Git. I vaguely remember the history here of Meta trying to convince the here of Meta trying to convince the here of Meta trying to convince the teams working on Git to make it work teams working on Git to make it work teams working on Git to make it work better for giant monor repos instead of better for giant monor repos instead of better for giant monor repos instead of doing separate repos for everything. The doing separate repos for everything. The doing separate repos for everything. The git maintainers didn't want to budge.

git maintainers didn't want to budge. git maintainers didn't want to budge. So, they ended up going their own way. So, they ended up going their own way. So, they ended up going their own way. And that's fine. You shouldn't have to And that's fine. You shouldn't have to And that's fine. You shouldn't have to force the industry standard into your force the industry standard into your force the industry standard into your workflow if it doesn't fit. And it is workflow if it doesn't fit. And it is workflow if it doesn't fit. And it is nice seeing more companies challenge nice seeing more companies challenge nice seeing more companies challenge this cuz Git is not perfect and it this cuz Git is not perfect and it this cuz Git is not perfect and it certainly is not perfect for agents. So, certainly is not perfect for agents. So, certainly is not perfect for agents. So, let's learn more about what Anthropic let's learn more about what Anthropic let's learn more about what Anthropic has built. In order to support their has built. In order to support their has built. In order to support their insane growth, they built Scion, an insane growth, they built Scion, an insane growth, they built Scion, an in-house version control system, and in-house version control system, and in-house version control system, and they fully migrated their monor repo they fully migrated their monor repo they fully migrated their monor repo from Git in five months. Ready for a from Git in five months. Ready for a from Git in five months. Ready for a really silly fun fact? Both Anthropic really silly fun fact? Both Anthropic really silly fun fact? Both Anthropic and OpenAI are built in monor repos. So and OpenAI are built in monor repos. So and OpenAI are built in monor repos. So everything they're working on is everything they're working on is everything they're working on is different projects and folders inside of different projects and folders inside of different projects and folders inside of this one giant repo. If you've ever been this one giant repo. If you've ever been this one giant repo. If you've ever been curious why the sidebar in the chat GPT curious why the sidebar in the chat GPT curious why the sidebar in the chat GPT and Codex app is so bad, it's because of and Codex app is so bad, it's because of and Codex app is so bad, it's because of this. The vast majority of employees at this. The vast majority of employees at this. The vast majority of employees at OpenAI have one project, OpenAI, and OpenAI have one project, OpenAI, and OpenAI have one project, OpenAI, and they're still using github.com, by the they're still using github.com, by the they're still using github.com, by the way. Pretty much all of OpenAI's code way. Pretty much all of OpenAI's code way. Pretty much all of OpenAI's code lives in OpenAI/OPAI, which 404 is if lives in OpenAI/OPAI, which 404 is if lives in OpenAI/OPAI, which 404 is if you're not an employee. So for those you're not an employee. So for those you're not an employee. So for those saying I'm secretly employed, here's saying I'm secretly employed, here's saying I'm secretly employed, here's your proof. That makes a lot of the way your proof. That makes a lot of the way your proof. That makes a lot of the way they think and work and build strange to they think and work and build strange to they think and work and build strange to people who have more than one project. people who have more than one project. people who have more than one project. So yeah, just a silly fun fact I wanted So yeah, just a silly fun fact I wanted So yeah, just a silly fun fact I wanted to include. Back to what Anthropic was to include. Back to what Anthropic was to include. Back to what Anthropic was saying here. Not only did they build saying here. Not only did they build saying here. Not only did they build Scion, which is their in-house version Scion, which is their in-house version Scion, which is their in-house version control system, they also built and control system, they also built and control system, they also built and shipped a code review surface called shipped a code review surface called shipped a code review surface called Antique alongside code search, a rich Antique alongside code search, a rich Antique alongside code search, a rich source control AM system, novel ways of source control AM system, novel ways of source control AM system, novel ways of managing and landing large amounts of managing and landing large amounts of managing and landing large amounts of reviewed code, and more, all in that reviewed code, and more, all in that reviewed code, and more, all in that same 5month period. In this post, same 5month period. In this post, same 5month period. In this post, they're going to focus on how they built they're going to focus on how they built they're going to focus on how they built Scion specifically. It makes for an Scion specifically. It makes for an Scion specifically. It makes for an interesting case study. Claude's ability interesting case study. Claude's ability interesting case study. Claude's ability to write code was both what put them in to write code was both what put them in to write code was both what put them in the situation in the first place and how the situation in the first place and how the situation in the first place and how they're able to build a highquality they're able to build a highquality they're able to build a highquality scalable system to deal with it. growing

scalable system to deal with it. growing scalable system to deal with it. growing our monor repo. Before we explore what our monor repo. Before we explore what our monor repo. Before we explore what we built, we should acknowledge how we built, we should acknowledge how we built, we should acknowledge how unusual our workload is. Unsurprisingly, unusual our workload is. Unsurprisingly, unusual our workload is. Unsurprisingly, Anthropic uses Claude to write software Anthropic uses Claude to write software Anthropic uses Claude to write software far more than almost any other company. far more than almost any other company. far more than almost any other company. You might be surprised, guys. Okay, You might be surprised, guys. Okay, You might be surprised, guys. Okay, seriously though, they're uh they are seriously though, they're uh they are seriously though, they're uh they are definitely doing it more than we are, definitely doing it more than we are, definitely doing it more than we are, although we are doing quite a lot. But although we are doing quite a lot. But although we are doing quite a lot. But since they're using it so heavily, since they're using it so heavily, since they're using it so heavily, they're often the first to see where they're often the first to see where they're often the first to see where tools built for human scale start to tools built for human scale start to tools built for human scale start to strain. Source control is just one strain. Source control is just one strain. Source control is just one example of this. While we don't expect example of this. While we don't expect example of this. While we don't expect many organizations to have these many organizations to have these many organizations to have these problems in the very near term, we problems in the very near term, we problems in the very near term, we suspect it won't be too long before suspect it won't be too long before suspect it won't be too long before others who use AI code tools experience others who use AI code tools experience others who use AI code tools experience these same pressure points, which is why these same pressure points, which is why these same pressure points, which is why we want to share what we've built and we want to share what we've built and we want to share what we've built and what we learned. It's a decentralized what we learned. It's a decentralized what we learned. It's a decentralized system. Every clone is a complete system. Every clone is a complete system. Every clone is a complete self-contained repository that can self-contained repository that can self-contained repository that can compute diffs, read history, and blame, compute diffs, read history, and blame, compute diffs, read history, and blame, and check out any revision while fully and check out any revision while fully and check out any revision while fully offline. For the vast majority of repos, offline. For the vast majority of repos, offline. For the vast majority of repos, that is the right design, and it is a that is the right design, and it is a that is the right design, and it is a large part of why Git became the large part of why Git became the large part of why Git became the standard. But as their monor repo has standard. But as their monor repo has standard. But as their monor repo has grown, it increasingly doesn't fit that grown, it increasingly doesn't fit that grown, it increasingly doesn't fit that design space. The history of their main design space. The history of their main design space. The history of their main branch alone is almost 100 gigs and a branch alone is almost 100 gigs and a branch alone is almost 100 gigs and a compressed working copy is over 25 gigs.

compressed working copy is over 25 gigs. compressed working copy is over 25 gigs. They perform on the order of 25 million They perform on the order of 25 million They perform on the order of 25 million code checkouts per day for CI before code checkouts per day for CI before code checkouts per day for CI before counting developer environments, agent counting developer environments, agent counting developer environments, agent sandboxes, and research jobs. Fetched sandboxes, and research jobs. Fetched sandboxes, and research jobs. Fetched natively, that would be 600 pabytes of natively, that would be 600 pabytes of natively, that would be 600 pabytes of Git traffic a day. Git offers tools for Git traffic a day. Git offers tools for Git traffic a day. Git offers tools for working with a slice of a repository working with a slice of a repository working with a slice of a repository like shallow clones, sparse checkouts, like shallow clones, sparse checkouts, like shallow clones, sparse checkouts, and promiser remotes. Beyond these, and promiser remotes. Beyond these, and promiser remotes. Beyond these, there are standard workarounds for large there are standard workarounds for large there are standard workarounds for large monor repos like internal git mirrors monor repos like internal git mirrors monor repos like internal git mirrors and repository snapshots like seeds that and repository snapshots like seeds that and repository snapshots like seeds that are kept in blob storage. They've tried are kept in blob storage. They've tried are kept in blob storage. They've tried and used all of them at their peak. They and used all of them at their peak. They and used all of them at their peak. They operated several hundred internal git operated several hundred internal git operated several hundred internal git mirrors to deal with the load. This mirrors to deal with the load. This mirrors to deal with the load. This bought them time, but it became harder bought them time, but it became harder bought them time, but it became harder to operate as merge rates were to operate as merge rates were to operate as merge rates were increasing. Ces are falling further from increasing. Ces are falling further from increasing. Ces are falling further from the tip of Maine between refreshes and the tip of Maine between refreshes and the tip of Maine between refreshes and it took more and more care to keep it took more and more care to keep it took more and more care to keep operations that depend on history like operations that depend on history like operations that depend on history like diffs and blame working on these partial diffs and blame working on these partial diffs and blame working on these partial clones. Taking a step back, we noticed clones. Taking a step back, we noticed clones. Taking a step back, we noticed that like most companies, we were using that like most companies, we were using that like most companies, we were using Git as a centralized system with one Git as a centralized system with one Git as a centralized system with one authoritative copy, but nearly all of authoritative copy, but nearly all of authoritative copy, but nearly all of our clients were agents in jobs that are our clients were agents in jobs that are our clients were agents in jobs that are online by definition. This is a very online by definition. This is a very online by definition. This is a very interesting and worthwhile call out. You interesting and worthwhile call out. You interesting and worthwhile call out. You can't use claude if you're offline can't use claude if you're offline can't use claude if you're offline because claude is being served from a because claude is being served from a because claude is being served from a giant data center that has GPUs you giant data center that has GPUs you giant data center that has GPUs you don't. And unless you are plugged into don't. And unless you are plugged into don't. And unless you are plugged into those directly and somehow the internet those directly and somehow the internet those directly and somehow the internet for that data centers offline, you're for that data centers offline, you're for that data centers offline, you're online. Building everything around this online. Building everything around this online. Building everything around this idea that everyone needs everything idea that everyone needs everything idea that everyone needs everything synced just doesn't make that much synced just doesn't make that much synced just doesn't make that much sense. The properties that make Git sense. The properties that make Git sense. The properties that make Git excellent for distributed offline work excellent for distributed offline work excellent for distributed offline work were actively undermining their ability were actively undermining their ability were actively undermining their ability to scale as the repo grew. So they asked to scale as the repo grew. So they asked to scale as the repo grew. So they asked themselves if they designed source

themselves if they designed source themselves if they designed source control with the fact that agents are control with the fact that agents are control with the fact that agents are always online in mind from the start always online in mind from the start always online in mind from the start what would it look like how would it be what would it look like how would it be what would it look like how would it be different with agentic coding many different with agentic coding many different with agentic coding many common assumptions about how software is common assumptions about how software is common assumptions about how software is developed are evolving they embrace developed are evolving they embrace developed are evolving they embrace these changes when they built scion these changes when they built scion these changes when they built scion which allowed them to take a which allowed them to take a which allowed them to take a fundamentally different approach than fundamentally different approach than fundamentally different approach than git on several core principles source git on several core principles source git on several core principles source control for huge monor repos does have control for huge monor repos does have control for huge monor repos does have prior art like google's piper and meta's prior art like google's piper and meta's prior art like google's piper and meta's sapling I touching on meta stuff before sapling I touching on meta stuff before sapling I touching on meta stuff before so they were able to build scion with so they were able to build scion with so they were able to build scion with their hard one lessons in mind It is their hard one lessons in mind It is their hard one lessons in mind It is worth noting just how many people at worth noting just how many people at worth noting just how many people at Anthropic are ex Meta and ex Google and Anthropic are ex Meta and ex Google and Anthropic are ex Meta and ex Google and they were often working on these types they were often working on these types they were often working on these types of things at those companies like I know of things at those companies like I know of things at those companies like I know Boris was deep in developer tooling at Boris was deep in developer tooling at Boris was deep in developer tooling at Meta and now he is making cloud code. Meta and now he is making cloud code. Meta and now he is making cloud code. The first core change they made is again The first core change they made is again The first core change they made is again this idea that clients are always online this idea that clients are always online this idea that clients are always online since nearly all of the clients are since nearly all of the clients are since nearly all of the clients are agents in jobs. They designed Scion agents in jobs. They designed Scion agents in jobs. They designed Scion client and protocols around a server client and protocols around a server client and protocols around a server that's always reachable and they have that's always reachable and they have that's always reachable and they have clients fetch data when they need it. clients fetch data when they need it. clients fetch data when they need it. also clearly not using GitHub because also clearly not using GitHub because also clearly not using GitHub because even when you're online, you can't even when you're online, you can't even when you're online, you can't assume GitHub's online. The next point assume GitHub's online. The next point assume GitHub's online. The next point is this idea of client server asymmetry.

is this idea of client server asymmetry. is this idea of client server asymmetry. In Git, every client is capable of being In Git, every client is capable of being In Git, every client is capable of being a full-fledged server. This is one of a full-fledged server. This is one of a full-fledged server. This is one of those things that makes Git Git. It is those things that makes Git Git. It is those things that makes Git Git. It is truly decentralized. Anything can be a truly decentralized. Anything can be a truly decentralized. Anything can be a Git server. But in Scion, the server Git server. But in Scion, the server Git server. But in Scion, the server answers questions about file history and answers questions about file history and answers questions about file history and blame, creates commits, performs merges blame, creates commits, performs merges blame, creates commits, performs merges and rebases, and so on. The client and rebases, and so on. The client and rebases, and so on. The client maintains a bounded local object cache. maintains a bounded local object cache. maintains a bounded local object cache. It lays out the working copy files on It lays out the working copy files on It lays out the working copy files on disk. It monitors for file system disk. It monitors for file system disk. It monitors for file system changes and it gives users ergonomic changes and it gives users ergonomic changes and it gives users ergonomic ways to work with that server. It can ways to work with that server. It can ways to work with that server. It can compute very little on its own, but in compute very little on its own, but in compute very little on its own, but in Exchange, it requires way less data Exchange, it requires way less data Exchange, it requires way less data locally. Next, I talk about how the locally. Next, I talk about how the locally. Next, I talk about how the source control service can power many source control service can power many source control service can power many novel application types. With git, a CI novel application types. With git, a CI novel application types. With git, a CI job or tool that needs to ask a question job or tool that needs to ask a question job or tool that needs to ask a question about the repo, like who last changed about the repo, like who last changed about the repo, like who last changed the line or how two commits are related, the line or how two commits are related, the line or how two commits are related, it would have to maintain the whole it would have to maintain the whole it would have to maintain the whole local copy of that repo or hope that local copy of that repo or hope that local copy of that repo or hope that some vendor specific API is able to some vendor specific API is able to some vendor specific API is able to provide the answer. With Scion, all the provide the answer. With Scion, all the provide the answer. With Scion, all the source control queries that the client source control queries that the client source control queries that the client uses are also exposed via API. So in uses are also exposed via API. So in uses are also exposed via API. So in many cases, those jobs don't need the many cases, those jobs don't need the many cases, those jobs don't need the local repository at all. Many internal local repository at all. Many internal local repository at all. Many internal tools like their automated bisect and tools like their automated bisect and tools like their automated bisect and revert pipelines wouldn't be practical revert pipelines wouldn't be practical revert pipelines wouldn't be practical at the scale they're operating at at the scale they're operating at at the scale they're operating at without this cheap programmatic access without this cheap programmatic access without this cheap programmatic access to the source. They also call out that to the source. They also call out that to the source. They also call out that everything must be horizontally everything must be horizontally everything must be horizontally scalable. At their peak, they ran scalable. At their peak, they ran scalable. At their peak, they ran several hundred git mirrors which several hundred git mirrors which several hundred git mirrors which significantly strained capacity and no significantly strained capacity and no significantly strained capacity and no longer scaled once the repo outgrew the longer scaled once the repo outgrew the longer scaled once the repo outgrew the mirrors discs or memory limits. You know mirrors discs or memory limits. You know mirrors discs or memory limits. You know it's bad when your git mirrors are it's bad when your git mirrors are it's bad when your git mirrors are running out of disc space. And to be running out of disc space. And to be running out of disc space. And to be fair, discs did get a lot more fair, discs did get a lot more fair, discs did get a lot more expensive. I still can't believe that expensive. I still can't believe that expensive. I still can't believe that hard drives increased in price as much hard drives increased in price as much hard drives increased in price as much as they did. Thanks, Anthropic. In

as they did. Thanks, Anthropic. In as they did. Thanks, Anthropic. In Scion, the repo's data is distributed Scion, the repo's data is distributed Scion, the repo's data is distributed and every component is independently and every component is independently and every component is independently scalable. There is no materialized copy scalable. There is no materialized copy scalable. There is no materialized copy of the repo anywhere, even on the of the repo anywhere, even on the of the repo anywhere, even on the server. There is no equivalent of pack server. There is no equivalent of pack server. There is no equivalent of pack files, just gets compressed bundles of files, just gets compressed bundles of files, just gets compressed bundles of objects, so there's no repo maintenance objects, so there's no repo maintenance objects, so there's no repo maintenance needed either. And then a very important needed either. And then a very important needed either. And then a very important piece, commits must be cheap, piece, commits must be cheap, piece, commits must be cheap, incremental, and lock free. Lock free is incremental, and lock free. Lock free is incremental, and lock free. Lock free is particularly important. One person particularly important. One person particularly important. One person committing shouldn't block somebody committing shouldn't block somebody committing shouldn't block somebody else. Or more realistically, 500 agents else. Or more realistically, 500 agents else. Or more realistically, 500 agents committing shouldn't block another 500 committing shouldn't block another 500 committing shouldn't block another 500 agents. In Git, you commit locally and agents. In Git, you commit locally and agents. In Git, you commit locally and then you push in order to share your then you push in order to share your then you push in order to share your work. In Scion, every commit, rebase, or work. In Scion, every commit, rebase, or work. In Scion, every commit, rebase, or merge happens on the server. So, there's merge happens on the server. So, there's merge happens on the server. So, there's nothing to push. But each of these nothing to push. But each of these nothing to push. But each of these operations needs to be fast. The cost of operations needs to be fast. The cost of operations needs to be fast. The cost of creating a commit and updating its creating a commit and updating its creating a commit and updating its downstream data structures has to scale downstream data structures has to scale downstream data structures has to scale with the size of the change, not the with the size of the change, not the with the size of the change, not the overall size of the repo, or the server overall size of the repo, or the server overall size of the repo, or the server won't be able to keep up. The won't be able to keep up. The won't be able to keep up. The development life cycle in Scion development life cycle in Scion development life cycle in Scion discourages refs pointers to commits discourages refs pointers to commits discourages refs pointers to commits such as branches and tags except for the such as branches and tags except for the such as branches and tags except for the main ref or release branches. The reason main ref or release branches. The reason main ref or release branches. The reason for that is because global state that for that is because global state that for that is because global state that every server has to agree on is always every server has to agree on is always every server has to agree on is always going to cause scale issues. Globals are going to cause scale issues. Globals are going to cause scale issues. Globals are bad when you're operating this fast.

bad when you're operating this fast. bad when you're operating this fast. Always. And then the most important Always. And then the most important Always. And then the most important piece, the working copy is optional and piece, the working copy is optional and piece, the working copy is optional and it is sparse by design. Because the repo it is sparse by design. Because the repo it is sparse by design. Because the repo is so large and they're checking out is so large and they're checking out is so large and they're checking out code tens of millions of times a day for code tens of millions of times a day for code tens of millions of times a day for CI, they want to avoid checkouts CI, they want to avoid checkouts CI, they want to avoid checkouts wherever they can. With Scion, a CI job wherever they can. With Scion, a CI job wherever they can. With Scion, a CI job that needs the diff between a commit and that needs the diff between a commit and that needs the diff between a commit and its merge base, for example, can ask the its merge base, for example, can ask the its merge base, for example, can ask the server just for that. The same goes for server just for that. The same goes for server just for that. The same goes for writes. Many write workloads can skip writes. Many write workloads can skip writes. Many write workloads can skip the checkout and function entirely over the checkout and function entirely over the checkout and function entirely over API. Scion's protocol is centered around API. Scion's protocol is centered around API. Scion's protocol is centered around sparse checkouts, which only contain the sparse checkouts, which only contain the sparse checkouts, which only contain the directories the client asked for. And directories the client asked for. And directories the client asked for. And there's also virtualized checkouts, there's also virtualized checkouts, there's also virtualized checkouts, which materialize files on demand. So which materialize files on demand. So which materialize files on demand. So when the job needs the file on the disk, when the job needs the file on the disk, when the job needs the file on the disk, the client only pays for what it's the client only pays for what it's the client only pays for what it's actually reading. I was playing with actually reading. I was playing with actually reading. I was playing with some of these ideas real late last year some of these ideas real late last year some of these ideas real late last year and early this year, trying to make it and early this year, trying to make it and early this year, trying to make it easier to manage state of like a easier to manage state of like a easier to manage state of like a codebase in a way where you don't have codebase in a way where you don't have codebase in a way where you don't have to have it all on your computer, but to have it all on your computer, but to have it all on your computer, but your agents can still access it. It's your agents can still access it. It's your agents can still access it. It's dope to see them doing the real version dope to see them doing the real version dope to see them doing the real version of this cuz mine was a chaotic of this cuz mine was a chaotic of this cuz mine was a chaotic experiment. So, how does it actually experiment. So, how does it actually experiment. So, how does it actually work? Now that we've seen the work? Now that we've seen the work? Now that we've seen the principles, we can hopefully understand principles, we can hopefully understand principles, we can hopefully understand how it all comes together. These how it all comes together. These how it all comes together. These insights can apply to other source insights can apply to other source insights can apply to other source control systems or more broadly to other control systems or more broadly to other control systems or more broadly to other stages of the software development life stages of the software development life stages of the software development life cycle. Oh boy, we're now in the data cycle. Oh boy, we're now in the data cycle. Oh boy, we're now in the data structure section. This is where we're structure section. This is where we're structure section. This is where we're going to get into the fun details. The going to get into the fun details. The going to get into the fun details. The unit of work is a change set. I I see unit of work is a change set. I I see unit of work is a change set. I I see the meta inspiration here. And change the meta inspiration here. And change the meta inspiration here. And change sets are entries and write ahead logs. A sets are entries and write ahead logs. A sets are entries and write ahead logs. A change set stores most of what you'd change set stores most of what you'd change set stores most of what you'd expect from a commit in git. Where a git expect from a commit in git. Where a git expect from a commit in git. Where a git commit stores a pointer to the tree, a commit stores a pointer to the tree, a commit stores a pointer to the tree, a change set stores a list of the files change set stores a list of the files change set stores a list of the files that changed relative to its first that changed relative to its first that changed relative to its first parent. But where a git commit stores a parent. But where a git commit stores a parent. But where a git commit stores a pointer to the tree, a change set stores pointer to the tree, a change set stores pointer to the tree, a change set stores a list of the files that changed

a list of the files that changed a list of the files that changed relative to the first parent along with relative to the first parent along with relative to the first parent along with content address pointers to the content address pointers to the content address pointers to the compressed file contents in blob compressed file contents in blob compressed file contents in blob storage. Change sets are created server storage. Change sets are created server storage. Change sets are created server side and are available everywhere side and are available everywhere side and are available everywhere instantly. Fascinating. I knew they were instantly. Fascinating. I knew they were instantly. Fascinating. I knew they were doing blob storage for a lot of this, doing blob storage for a lot of this, doing blob storage for a lot of this, but kind of crazy to see the like direct but kind of crazy to see the like direct but kind of crazy to see the like direct call out that the source code does not call out that the source code does not call out that the source code does not exist in the system and when you request exist in the system and when you request exist in the system and when you request the source it has to go fetch that from the source it has to go fetch that from the source it has to go fetch that from blob store and it's effectively just a blob store and it's effectively just a blob store and it's effectively just a database pointing at those blobs. Seems database pointing at those blobs. Seems database pointing at those blobs. Seems like all the metadata is in Spanner. like all the metadata is in Spanner. like all the metadata is in Spanner. Makes sense. Again, very Google Makes sense. Again, very Google Makes sense. Again, very Google inspired. Very interesting that they inspired. Very interesting that they inspired. Very interesting that they have separate author and committer have separate author and committer have separate author and committer fields. makes sense because the author fields. makes sense because the author fields. makes sense because the author is the person who could who triggered it is the person who could who triggered it is the person who could who triggered it and the committer is whatever agent did and the committer is whatever agent did and the committer is whatever agent did it or where it happened if you have like it or where it happened if you have like it or where it happened if you have like different systems that are doing these different systems that are doing these different systems that are doing these things. Very interesting. And other things. Very interesting. And other things. Very interesting. And other usual fields like author date, committer usual fields like author date, committer usual fields like author date, committer date, message, the parent, which we date, message, the parent, which we date, message, the parent, which we talked about before, that is the thing talked about before, that is the thing talked about before, that is the thing that this is in reference to. And then that this is in reference to. And then that this is in reference to. And then the file changes which specifies the the file changes which specifies the the file changes which specifies the file that changed and then has pointers file that changed and then has pointers file that changed and then has pointers to the blobs in their blob storage. The to the blobs in their blob storage. The to the blobs in their blob storage. The structure lets them accept change sets structure lets them accept change sets structure lets them accept change sets with very high throughput because with very high throughput because with very high throughput because there's no pack files they have to deal there's no pack files they have to deal there's no pack files they have to deal with. No locking or crossover with. No locking or crossover with. No locking or crossover coordination and no refs to move around.

coordination and no refs to move around. coordination and no refs to move around. What this is trying to say is you just What this is trying to say is you just What this is trying to say is you just have to append. You only are adding to have to append. You only are adding to have to append. You only are adding to the data set. You don't have to go the data set. You don't have to go the data set. You don't have to go through the whole history before through the whole history before through the whole history before appending it. You don't have to move appending it. You don't have to move appending it. You don't have to move pointers from here to there. You don't pointers from here to there. You don't pointers from here to there. You don't have to restructure the tree. You don't have to restructure the tree. You don't have to restructure the tree. You don't have to sync all of the data back and have to sync all of the data back and have to sync all of the data back and forth to make sure it's all in the right forth to make sure it's all in the right forth to make sure it's all in the right place. You just put up the change set place. You just put up the change set place. You just put up the change set and store the blobs in blob storage and and store the blobs in blob storage and and store the blobs in blob storage and you're done. Oh, is it not even storing you're done. Oh, is it not even storing you're done. Oh, is it not even storing the diffs for the file change? It's just the diffs for the file change? It's just the diffs for the file change? It's just storing the new state of the file. That storing the new state of the file. That storing the new state of the file. That would make sense. So, it's saying like would make sense. So, it's saying like would make sense. So, it's saying like these files changed. Here is where they these files changed. Here is where they these files changed. Here is where they point. But this thing in the storage is point. But this thing in the storage is point. But this thing in the storage is the full content of that file, not a the full content of that file, not a the full content of that file, not a diff. Because if it was a diff, diff. Because if it was a diff, diff. Because if it was a diff, recalculating would be hell. Change sets recalculating would be hell. Change sets recalculating would be hell. Change sets are recording only what changed. So on are recording only what changed. So on are recording only what changed. So on their own, they can't quickly answer their own, they can't quickly answer their own, they can't quickly answer questions like who edited a line or what questions like who edited a line or what questions like who edited a line or what commit affected a file. To answer those, commit affected a file. To answer those, commit affected a file. To answer those, Scion builds additional data structures Scion builds additional data structures Scion builds additional data structures from the change sets, including from the change sets, including from the change sets, including dedicated structures for file blame and dedicated structures for file blame and dedicated structures for file blame and path history, as well as skip lists for path history, as well as skip lists for path history, as well as skip lists for the commit graph. They call everything the commit graph. They call everything the commit graph. They call everything that isn't directly part of the change that isn't directly part of the change that isn't directly part of the change set derived data. And they compute it set derived data. And they compute it set derived data. And they compute it all through one generic mechanism. Every all through one generic mechanism. Every all through one generic mechanism. Every type of derived data is defined type of derived data is defined type of derived data is defined recursively. So its value for a change recursively. So its value for a change recursively. So its value for a change set is computed from its values for the set is computed from its values for the set is computed from its values for the change sets parents. The mechanism has change sets parents. The mechanism has change sets parents. The mechanism has three benefits. First, they can easily three benefits. First, they can easily three benefits. First, they can easily add new types of derived data when new add new types of derived data when new add new types of derived data when new data structures would make some data structures would make some data structures would make some operations faster. Second, for each operations faster. Second, for each operations faster. Second, for each type, they can choose whether to compute type, they can choose whether to compute type, they can choose whether to compute it eagerly or defer the work, so they it eagerly or defer the work, so they it eagerly or defer the work, so they can avoid materializing data they don't can avoid materializing data they don't can avoid materializing data they don't need. And third, because the client need. And third, because the client need. And third, because the client never sees the internals of the never sees the internals of the never sees the internals of the structures, they can optimize them structures, they can optimize them structures, they can optimize them without changing the client. They've

without changing the client. They've without changing the client. They've added new types of derived data many added new types of derived data many added new types of derived data many times as the system has grown, such as times as the system has grown, such as times as the system has grown, such as new indices for performing copy new indices for performing copy new indices for performing copy detection or path-based file search. detection or path-based file search. detection or path-based file search. makes a ton of sense to try and dumb makes a ton of sense to try and dumb makes a ton of sense to try and dumb this down. We're deep in the weeds here, this down. We're deep in the weeds here, this down. We're deep in the weeds here, which I I love. A big part of what makes which I I love. A big part of what makes which I I love. A big part of what makes Git so powerful is that you have all of Git so powerful is that you have all of Git so powerful is that you have all of the data when you clone the repo. So, if the data when you clone the repo. So, if the data when you clone the repo. So, if you want to find something like when you want to find something like when you want to find something like when this file changed, all of the things this file changed, all of the things this file changed, all of the things this person did, stuff like that, you this person did, stuff like that, you this person did, stuff like that, you can go through all the data yourself. can go through all the data yourself. can go through all the data yourself. The problem with this is if you want to The problem with this is if you want to The problem with this is if you want to add something new to Git or make some add something new to Git or make some add something new to Git or make some type of action faster, you're kind of type of action faster, you're kind of type of action faster, you're kind of screwed because you'd have to upend screwed because you'd have to upend screwed because you'd have to upend everybody's git. But if people don't everybody's git. But if people don't everybody's git. But if people don't have to clone the whole thing and have to clone the whole thing and have to clone the whole thing and they're just hitting an API for this, they're just hitting an API for this, they're just hitting an API for this, you can choose what you do and don't you can choose what you do and don't you can choose what you do and don't expose. And since the server here is expose. And since the server here is expose. And since the server here is full of cache layers, overprovisioned full of cache layers, overprovisioned full of cache layers, overprovisioned CPU, and so many other things, they look CPU, and so many other things, they look CPU, and so many other things, they look for the things clients want to request for the things clients want to request for the things clients want to request to get more information. They built some to get more information. They built some to get more information. They built some way to compute that that's efficient way to compute that that's efficient way to compute that that's efficient without having to change the data without having to change the data without having to change the data structure and now the clients can do the structure and now the clients can do the structure and now the clients can do the thing and the clients aren't going to thing and the clients aren't going to thing and the clients aren't going to cheat and do something weird which I've cheat and do something weird which I've cheat and do something weird which I've seen a lot. When you give Claude access seen a lot. When you give Claude access seen a lot. When you give Claude access to the internals of something, it will to the internals of something, it will to the internals of something, it will invent some crazy novel and also crazy invent some crazy novel and also crazy invent some crazy novel and also crazy slow and CPU inensive solutions to get slow and CPU inensive solutions to get slow and CPU inensive solutions to get what it's looking for. This hard API what it's looking for. This hard API what it's looking for. This hard API boundary prevents that which is almost boundary prevents that which is almost boundary prevents that which is almost certainly a huge benefit. So how did certainly a huge benefit. So how did certainly a huge benefit. So how did they scale this? starts with a small they scale this? starts with a small they scale this? starts with a small fleet of Scion servers providing source fleet of Scion servers providing source fleet of Scion servers providing source control APIs for any operation a client control APIs for any operation a client control APIs for any operation a client needs like getting change sets, needs like getting change sets, needs like getting change sets, computing diffs, finding merge bases, computing diffs, finding merge bases, computing diffs, finding merge bases, and fetching blobs. Computing diffs is a

and fetching blobs. Computing diffs is a and fetching blobs. Computing diffs is a fun thing to realize here because again fun thing to realize here because again fun thing to realize here because again it's storing the file content at the end it's storing the file content at the end it's storing the file content at the end of a change. So when you are asking what of a change. So when you are asking what of a change. So when you are asking what changed between these two like change changed between these two like change changed between these two like change sets, you don't just look at the diff sets, you don't just look at the diff sets, you don't just look at the diff because you don't have a diff. The diff because you don't have a diff. The diff because you don't have a diff. The diff is computed when you say here are two is computed when you say here are two is computed when you say here are two change sets. change between them and it change sets. change between them and it change sets. change between them and it does that by grabbing the blobs for all does that by grabbing the blobs for all does that by grabbing the blobs for all the files that are affected and seeing the files that are affected and seeing the files that are affected and seeing how they differ between those states. how they differ between those states. how they differ between those states. All source control data is content All source control data is content All source control data is content addressed. So a piece of data never addressed. So a piece of data never addressed. So a piece of data never changes once it's stored. This is also changes once it's stored. This is also changes once it's stored. This is also important. The ability to override important. The ability to override important. The ability to override history in git isn't too bad because history in git isn't too bad because history in git isn't too bad because whenever you force push, you're changing whenever you force push, you're changing whenever you force push, you're changing all the commit hashes. But if you had all the commit hashes. But if you had all the commit hashes. But if you had the ability to edit the contents of a the ability to edit the contents of a the ability to edit the contents of a given hash, that would fundamentally given hash, that would fundamentally given hash, that would fundamentally break git at its core. Immutability is break git at its core. Immutability is break git at its core. Immutability is always more scalable than mutable stuff always more scalable than mutable stuff always more scalable than mutable stuff pretty much 100% of the time. So pretty much 100% of the time. So pretty much 100% of the time. So building the whole system around these building the whole system around these building the whole system around these change sets being fixed and permanent change sets being fixed and permanent change sets being fixed and permanent and all of the flexible stuff being and all of the flexible stuff being and all of the flexible stuff being calculations done on top. This is calculations done on top. This is calculations done on top. This is touching my functional programming heart touching my functional programming heart touching my functional programming heart deep at its core. And since they have deep at its core. And since they have deep at its core. And since they have these immutable content addressed change these immutable content addressed change these immutable content addressed change sets, all servers can cach them without sets, all servers can cach them without sets, all servers can cach them without checking with the others. As soon as checking with the others. As soon as checking with the others. As soon as it's sent up to one server, it knows it's sent up to one server, it knows it's sent up to one server, it knows that change set's unique. it can just that change set's unique. it can just that change set's unique. it can just blindly trust that. As a result of this, blindly trust that. As a result of this, blindly trust that. As a result of this, the servers are horizontally scalable the servers are horizontally scalable the servers are horizontally scalable and they require very little and they require very little and they require very little coordination. They're essentially large coordination. They're essentially large coordination. They're essentially large caches that can also compute derived caches that can also compute derived caches that can also compute derived data on the fly and write the results data on the fly and write the results data on the fly and write the results back to Spanner. Scion's many consumers back to Spanner. Scion's many consumers back to Spanner. Scion's many consumers no longer need to maintain local repos.

no longer need to maintain local repos. no longer need to maintain local repos. Fetching and maintaining the repo used Fetching and maintaining the repo used Fetching and maintaining the repo used to be a significant maintenance burden to be a significant maintenance burden to be a significant maintenance burden for the CI jobs and tools that needed to for the CI jobs and tools that needed to for the CI jobs and tools that needed to ask source control questions, especially ask source control questions, especially ask source control questions, especially around commit, ancestry, and blame. around commit, ancestry, and blame. around commit, ancestry, and blame. Early in the migration, before they even Early in the migration, before they even Early in the migration, before they even had their own client, repo format and had their own client, repo format and had their own client, repo format and protocols, Scion was only used to answer protocols, Scion was only used to answer protocols, Scion was only used to answer these types of metadata queries. these types of metadata queries. these types of metadata queries. Fetching file contents and materializing Fetching file contents and materializing Fetching file contents and materializing working copies was far more data working copies was far more data working copies was far more data inensive operations. So they needed to inensive operations. So they needed to inensive operations. So they needed to design something of their own there. design something of their own there. design something of their own there. Scion's SCN client deviates from the Scion's SCN client deviates from the Scion's SCN client deviates from the Gits fetch protocols in several Gits fetch protocols in several Gits fetch protocols in several important ways and as a result it lends important ways and as a result it lends important ways and as a result it lends itself to highly scalable architectures itself to highly scalable architectures itself to highly scalable architectures that require significantly less CPU than that require significantly less CPU than that require significantly less CPU than git servers do. Science negotiation git servers do. Science negotiation git servers do. Science negotiation doesn't tell the server what the client doesn't tell the server what the client doesn't tell the server what the client already has. Instead, it operates already has. Instead, it operates already has. Instead, it operates exclusively on trees and blobs that the exclusively on trees and blobs that the exclusively on trees and blobs that the client wants. Starting from a root tree, client wants. Starting from a root tree, client wants. Starting from a root tree, the client checks its local store for the client checks its local store for the client checks its local store for missing trees and requests them missing trees and requests them missing trees and requests them recursively. Then it repeats the process recursively. Then it repeats the process recursively. Then it repeats the process for the blobs. So, it starts by finding for the blobs. So, it starts by finding for the blobs. So, it starts by finding the change sets that it needs to do the the change sets that it needs to do the the change sets that it needs to do the thing it wants to do. And once it has thing it wants to do. And once it has thing it wants to do. And once it has those, it goes and collects all of the those, it goes and collects all of the those, it goes and collects all of the blobs to have the files it needs. And blobs to have the files it needs. And blobs to have the files it needs. And then it can actually start working.

then it can actually start working. then it can actually start working. Every request is for a specific object Every request is for a specific object Every request is for a specific object that never changes. So, any cache can that never changes. So, any cache can that never changes. So, any cache can answer it. And the architecture closely answer it. And the architecture closely answer it. And the architecture closely resembles traditional tiered caches. The resembles traditional tiered caches. The resembles traditional tiered caches. The process is also efficient and resumable process is also efficient and resumable process is also efficient and resumable because there's no single multi- because there's no single multi- because there's no single multi- gigabyte pack file response. Sparse and gigabyte pack file response. Sparse and gigabyte pack file response. Sparse and even virtualized checkouts don't need even virtualized checkouts don't need even virtualized checkouts don't need special support because the client special support because the client special support because the client simply doesn't ask for trees it doesn't simply doesn't ask for trees it doesn't simply doesn't ask for trees it doesn't want. This paragraph is using a bunch of want. This paragraph is using a bunch of want. This paragraph is using a bunch of terms that they did not define and they terms that they did not define and they terms that they did not define and they are using in unique ways. So I will do are using in unique ways. So I will do are using in unique ways. So I will do my best to translate from claudeisms to my best to translate from claudeisms to my best to translate from claudeisms to English here. The clients don't connect English here. The clients don't connect English here. The clients don't connect directly to the Scion servers because directly to the Scion servers because directly to the Scion servers because those are managing all of the change those are managing all of the change those are managing all of the change sets. They're talking to a Damon over a sets. They're talking to a Damon over a sets. They're talking to a Damon over a socket and that Damon stores objects for socket and that Damon stores objects for socket and that Damon stores objects for the whole node. So, it's effectively the whole node. So, it's effectively the whole node. So, it's effectively keeping a cache of like what was the keeping a cache of like what was the keeping a cache of like what was the most recent version of a given file in a most recent version of a given file in a most recent version of a given file in a given file path. This allows it to given file path. This allows it to given file path. This allows it to respond to requests for files without respond to requests for files without respond to requests for files without having to go down the whole tree to having to go down the whole tree to having to go down the whole tree to compute the current state of things. I compute the current state of things. I compute the current state of things. I really hate how they use node here. Each really hate how they use node here. Each really hate how they use node here. Each of these dammons that does this caching of these dammons that does this caching of these dammons that does this caching gets periodically reset to be on the gets periodically reset to be on the gets periodically reset to be on the latest snapshot so that it isn't like latest snapshot so that it isn't like latest snapshot so that it isn't like recalculating all the things that have recalculating all the things that have recalculating all the things that have changed in the last 3 days. They update changed in the last 3 days. They update changed in the last 3 days. They update it constantly. And let's be real with it constantly. And let's be real with it constantly. And let's be real with the way people are spinning up these the way people are spinning up these the way people are spinning up these agents, spinning up CI, all of these agents, spinning up CI, all of these agents, spinning up CI, all of these things. They're probably starting that things. They're probably starting that things. They're probably starting that work from latest. So it makes a lot of work from latest. So it makes a lot of work from latest. So it makes a lot of sense to get the cache up to date that sense to get the cache up to date that sense to get the cache up to date that way and then just resolve things that way and then just resolve things that way and then just resolve things that have changed since the cache server have changed since the cache server have changed since the cache server stream objects directly from memory stream objects directly from memory stream objects directly from memory instead of like storing it in the hard instead of like storing it in the hard instead of like storing it in the hard drive which makes it way faster to get drive which makes it way faster to get drive which makes it way faster to get that data when you're on a client on the

that data when you're on a client on the that data when you're on a client on the other side. And it stores this in the other side. And it stores this in the other side. And it stores this in the same format that it would have been on same format that it would have been on same format that it would have been on disk so that it doesn't have to change disk so that it doesn't have to change disk so that it doesn't have to change the shape or format of the data when it the shape or format of the data when it the shape or format of the data when it comes out of memory and gets onto a comes out of memory and gets onto a comes out of memory and gets onto a drive after going over the network. When drive after going over the network. When drive after going over the network. When caches are missing objects, they ask the caches are missing objects, they ask the caches are missing objects, they ask the authoritative scion servers, which is authoritative scion servers, which is authoritative scion servers, which is the layer we were talking about before, the layer we were talking about before, the layer we were talking about before, which will read from Spanner or cloud which will read from Spanner or cloud which will read from Spanner or cloud storage as needed in order to get the storage as needed in order to get the storage as needed in order to get the updated data and the updated files in updated data and the updated files in updated data and the updated files in that cache layer. The Scion Damon, Scion that cache layer. The Scion Damon, Scion that cache layer. The Scion Damon, Scion cache, and Scion server all respond to cache, and Scion server all respond to cache, and Scion server all respond to identical RPCs for this protocol. So, identical RPCs for this protocol. So, identical RPCs for this protocol. So, the client doesn't need to know the the client doesn't need to know the the client doesn't need to know the cache topology. Any tier could cache topology. Any tier could cache topology. Any tier could theoretically talk to any other, but theoretically talk to any other, but theoretically talk to any other, but they strictly enforce the tiering to they strictly enforce the tiering to they strictly enforce the tiering to isolate failure domains. They want to isolate failure domains. They want to isolate failure domains. They want to avoid a situation where a failure in a avoid a situation where a failure in a avoid a situation where a failure in a regional scion cache overwhelms the regional scion cache overwhelms the regional scion cache overwhelms the upstream servers and brings down the upstream servers and brings down the upstream servers and brings down the entire system. So again they have that entire system. So again they have that entire system. So again they have that cluster of scion servers that are the cluster of scion servers that are the cluster of scion servers that are the like proper management of the real world like proper management of the real world like proper management of the real world state of things the authoritative layer state of things the authoritative layer state of things the authoritative layer that is the layer that stores in spanner that is the layer that stores in spanner that is the layer that stores in spanner for the change sets and the cloud for the change sets and the cloud for the change sets and the cloud storage for the actual contents of the storage for the actual contents of the storage for the actual contents of the files. There is then a cache layer that files. There is then a cache layer that files. There is then a cache layer that is what gets hit often for your actual is what gets hit often for your actual is what gets hit often for your actual work that is regional. So you don't have work that is regional. So you don't have work that is regional. So you don't have to like fetch across the world to hit to like fetch across the world to hit to like fetch across the world to hit this server. This is what will hit the this server. This is what will hit the this server. This is what will hit the scion server to get a reasonable state scion server to get a reasonable state scion server to get a reasonable state in the cache. Developer machines in the cache. Developer machines in the cache. Developer machines directly hit the scion cache when you're directly hit the scion cache when you're directly hit the scion cache when you're like pulling down the code to do work.

like pulling down the code to do work. like pulling down the code to do work. And if the scan cache is missing things And if the scan cache is missing things And if the scan cache is missing things that you need, it will go pull it from that you need, it will go pull it from that you need, it will go pull it from the server. But if you're working on CI, the server. But if you're working on CI, the server. But if you're working on CI, even this isn't the right abstraction. even this isn't the right abstraction. even this isn't the right abstraction. And that's where this Damon comes in And that's where this Damon comes in And that's where this Damon comes in that we were just talking about that that we were just talking about that that we were just talking about that gets periodic snapshots so that it's gets periodic snapshots so that it's gets periodic snapshots so that it's ready to spin up CI jobs as quickly as ready to spin up CI jobs as quickly as ready to spin up CI jobs as quickly as possible. Each tier of this system possible. Each tier of this system possible. Each tier of this system maintains a cache hit rate of over 99% maintains a cache hit rate of over 99% maintains a cache hit rate of over 99% and it doesn't require longived state. and it doesn't require longived state. and it doesn't require longived state. So it's easily scaled up if needed. So it's easily scaled up if needed. So it's easily scaled up if needed. Their infrastructure footprint is 200x Their infrastructure footprint is 200x Their infrastructure footprint is 200x smaller than their previous git smaller than their previous git smaller than their previous git mirroring infra in terms of CPU and mirroring infra in terms of CPU and mirroring infra in terms of CPU and memory requirements. That's insane. memory requirements. That's insane. memory requirements. That's insane. again like and this is like the crazy again like and this is like the crazy again like and this is like the crazy idea that has been like hurting me a idea that has been like hurting me a idea that has been like hurting me a bit. The realization that like all bit. The realization that like all bit. The realization that like all software was built to be somewhat software was built to be somewhat software was built to be somewhat generalizable. Now we can build software generalizable. Now we can build software generalizable. Now we can build software for our specific needs. And like it's a for our specific needs. And like it's a for our specific needs. And like it's a silly thing to pivot to, but hear me silly thing to pivot to, but hear me silly thing to pivot to, but hear me out. One of my weird YOLO projects right out. One of my weird YOLO projects right out. One of my weird YOLO projects right now is called Lust Lust Lang to be now is called Lust Lust Lang to be now is called Lust Lust Lang to be specific. on trying to take a subset of specific. on trying to take a subset of specific. on trying to take a subset of Rust that gets you most of the benefits Rust that gets you most of the benefits Rust that gets you most of the benefits of Rust, but make it so that it could be of Rust, but make it so that it could be of Rust, but make it so that it could be compiled and built a hundred times compiled and built a hundred times compiled and built a hundred times faster because the build times for TS faster because the build times for TS faster because the build times for TS Rust, which is my rewrite of the Tyer Rust, which is my rewrite of the Tyer Rust, which is my rewrite of the Tyer compiler in Rust, are too slow. This is compiler in Rust, are too slow. This is compiler in Rust, are too slow. This is an insane project that I didn't think an insane project that I didn't think an insane project that I didn't think would make any progress, but it's would make any progress, but it's would make any progress, but it's already able to compile over 40% of the already able to compile over 40% of the already able to compile over 40% of the files from TS Rust, and I am expecting files from TS Rust, and I am expecting files from TS Rust, and I am expecting by the end of the week it might actually by the end of the week it might actually by the end of the week it might actually work. Would this work for most Rust work. Would this work for most Rust work. Would this work for most Rust projects? [ __ ] no. Absolutely not. That projects? [ __ ] no. Absolutely not. That projects? [ __ ] no. Absolutely not. That is not what this is for. This is a is not what this is for. This is a is not what this is for. This is a bespoke experiment to see if I take a bespoke experiment to see if I take a bespoke experiment to see if I take a given codebase that I'm not happy with

given codebase that I'm not happy with given codebase that I'm not happy with compile times on. Can I invent a new compile times on. Can I invent a new compile times on. Can I invent a new language to make it compile faster? It's language to make it compile faster? It's language to make it compile faster? It's like insane that this is even a thing like insane that this is even a thing like insane that this is even a thing you can think, much less do. But I'm you can think, much less do. But I'm you can think, much less do. But I'm using a model that's a lot dumber than using a model that's a lot dumber than using a model that's a lot dumber than what they have access to. I'm using the what they have access to. I'm using the what they have access to. I'm using the distilled version of two model distilled version of two model distilled version of two model generations away from what they have generations away from what they have generations away from what they have internally. They're rocking Fable Ultra internally. They're rocking Fable Ultra internally. They're rocking Fable Ultra Max 7 internally and I'm here with Opus Max 7 internally and I'm here with Opus Max 7 internally and I'm here with Opus 55 more than happy to do these types of 55 more than happy to do these types of 55 more than happy to do these types of things. The fact that we can build this things. The fact that we can build this things. The fact that we can build this way now is just crazy. And while this way now is just crazy. And while this way now is just crazy. And while this system might not be 200 times smaller system might not be 200 times smaller system might not be 200 times smaller and faster for some cases, it is for and faster for some cases, it is for and faster for some cases, it is for theirs. And I would expect more and more theirs. And I would expect more and more theirs. And I would expect more and more companies that have these types of big companies that have these types of big companies that have these types of big painful scaling things to go reinvent painful scaling things to go reinvent painful scaling things to go reinvent the painful pieces that are slowing them the painful pieces that are slowing them the painful pieces that are slowing them down. And uh fun fact, SJ, the borrow down. And uh fun fact, SJ, the borrow down. And uh fun fact, SJ, the borrow checker is not why build times suck in checker is not why build times suck in checker is not why build times suck in Rust. It's a bunch of weird LLVM Rust. It's a bunch of weird LLVM Rust. It's a bunch of weird LLVM [ __ ] Anyways, CI and research jobs [ __ ] Anyways, CI and research jobs [ __ ] Anyways, CI and research jobs are able to check out code two orders of are able to check out code two orders of are able to check out code two orders of magnitude faster, seconds rather than magnitude faster, seconds rather than magnitude faster, seconds rather than minutes, which frees up a sizable amount minutes, which frees up a sizable amount minutes, which frees up a sizable amount of capacity across the fleet. CI alone of capacity across the fleet. CI alone of capacity across the fleet. CI alone had a 25% CPU capacity when neither had a 25% CPU capacity when neither had a 25% CPU capacity when neither clients nor caches need to perform repo clients nor caches need to perform repo clients nor caches need to perform repo maintenance, which simplifies the life maintenance, which simplifies the life maintenance, which simplifies the life cycle management of our jobs. Out of cycle management of our jobs. Out of cycle management of our jobs. Out of curiosity, I'm trying to get the hard curiosity, I'm trying to get the hard curiosity, I'm trying to get the hard numbers for how much time T3 code builds numbers for how much time T3 code builds numbers for how much time T3 code builds and CI have spent dealing with GitHub and CI have spent dealing with GitHub and CI have spent dealing with GitHub and Git like clones. It's going to be a and Git like clones. It's going to be a and Git like clones. It's going to be a number that hurts me a lot. We'll come number that hurts me a lot. We'll come number that hurts me a lot. We'll come back and check on that in a bit, but I back and check on that in a bit, but I back and check on that in a bit, but I have a feeling I'm going to feel a lot have a feeling I'm going to feel a lot have a feeling I'm going to feel a lot of pain as soon as I see that. So, what of pain as soon as I see that. So, what of pain as soon as I see that. So, what did they learn building Scion? They

did they learn building Scion? They did they learn building Scion? They started building it around the same time started building it around the same time started building it around the same time as Mythos Preview and Claude Tag as Mythos Preview and Claude Tag as Mythos Preview and Claude Tag becoming available internally. Both of becoming available internally. Both of becoming available internally. Both of these releases simultaneously amplified these releases simultaneously amplified these releases simultaneously amplified the need for Scion and meaningfully the need for Scion and meaningfully the need for Scion and meaningfully accelerated how quickly they'd be able accelerated how quickly they'd be able accelerated how quickly they'd be able to build it. Here's a particularly to build it. Here's a particularly to build it. Here's a particularly striking example. Earlier in the year, striking example. Earlier in the year, striking example. Earlier in the year, they had largely handwritten the first they had largely handwritten the first they had largely handwritten the first version of the Git to Scion ingestion version of the Git to Scion ingestion version of the Git to Scion ingestion pipeline as a prototype. They thought it pipeline as a prototype. They thought it pipeline as a prototype. They thought it was good until they got their hands on was good until they got their hands on was good until they got their hands on early internal snapshots of Mythos early internal snapshots of Mythos early internal snapshots of Mythos preview. They asked it to rewrite the preview. They asked it to rewrite the preview. They asked it to rewrite the whole thing and with no further whole thing and with no further whole thing and with no further instructions, it produced a version that instructions, it produced a version that instructions, it produced a version that consumed 20x less memory and was four consumed 20x less memory and was four consumed 20x less memory and was four times faster while preserving all of the times faster while preserving all of the times faster while preserving all of the implicit invariations around ingestion, implicit invariations around ingestion, implicit invariations around ingestion, ordering, and data structures. This is ordering, and data structures. This is ordering, and data structures. This is the history of changes to Scion and the history of changes to Scion and the history of changes to Scion and Antique. I will say worth noting here, Antique. I will say worth noting here, Antique. I will say worth noting here, they were bragging about 30k changes a they were bragging about 30k changes a they were bragging about 30k changes a day, but they've only made 12,250 day, but they've only made 12,250 day, but they've only made 12,250 changes to this architecture total. So changes to this architecture total. So changes to this architecture total. So their daily velocity is more than double their daily velocity is more than double their daily velocity is more than double the amount of like requests, commits, the amount of like requests, commits, the amount of like requests, commits, change sets, whatever you want to call change sets, whatever you want to call change sets, whatever you want to call them, that have landed for the whole of them, that have landed for the whole of them, that have landed for the whole of their source control. Kind of crazy. But their source control. Kind of crazy. But their source control. Kind of crazy. But they learned building and operating they learned building and operating they learned building and operating Scion offers some interesting lessons Scion offers some interesting lessons Scion offers some interesting lessons for what building agent scale software for what building agent scale software for what building agent scale software will look like. Rapid development with will look like. Rapid development with will look like. Rapid development with small teams is going to become more small teams is going to become more small teams is going to become more common. Can confirm. A team of seven common. Can confirm. A team of seven common. Can confirm. A team of seven people built both Scion and Antique, people built both Scion and Antique, people built both Scion and Antique, their code review tool, which they could their code review tool, which they could their code review tool, which they could have easily written a whole separate have easily written a whole separate have easily written a whole separate blog post about considering how awful blog post about considering how awful blog post about considering how awful code review is. I feel it. I'm thankful code review is. I feel it. I'm thankful code review is. I feel it. I'm thankful that we have so many wonderful sponsors that we have so many wonderful sponsors that we have so many wonderful sponsors working on that now. They also follow working on that now. They also follow working on that now. They also follow with not to mention half a dozen related with not to mention half a dozen related with not to mention half a dozen related systems that they fully migrated all of

systems that they fully migrated all of systems that they fully migrated all of in 5 months. They did this all while the in 5 months. They did this all while the in 5 months. They did this all while the volume of code merged per day was volume of code merged per day was volume of code merged per day was increasing 10x. In the past, developing increasing 10x. In the past, developing increasing 10x. In the past, developing and deploying source control systems and deploying source control systems and deploying source control systems like this would have taken years of like this would have taken years of like this would have taken years of iterative migrations with many more iterative migrations with many more iterative migrations with many more engineers. We think the sort of fast, engineers. We think the sort of fast, engineers. We think the sort of fast, nimble migration will become more common nimble migration will become more common nimble migration will become more common in the future. Yeah, it just will. in the future. Yeah, it just will. in the future. Yeah, it just will. Switching cost has gone down more than Switching cost has gone down more than Switching cost has gone down more than almost anything. Historically, people almost anything. Historically, people almost anything. Historically, people adopt a technology, a solution, a adopt a technology, a solution, a adopt a technology, a solution, a vendor, and then they kind of are stuck vendor, and then they kind of are stuck vendor, and then they kind of are stuck there because switching off is more there because switching off is more there because switching off is more expensive than whatever cost they're expensive than whatever cost they're expensive than whatever cost they're incurring staying. Now, it's easier to incurring staying. Now, it's easier to incurring staying. Now, it's easier to write your own alternative and move to write your own alternative and move to write your own alternative and move to it. it. it. It's insane. I personally threw together It's insane. I personally threw together It's insane. I personally threw together a solution to clone and mirror and a solution to clone and mirror and a solution to clone and mirror and backfill all of our analytics data that backfill all of our analytics data that backfill all of our analytics data that I have on one particular analytics I have on one particular analytics I have on one particular analytics provider so that I have it directly on provider so that I have it directly on provider so that I have it directly on my own managed click house on top of R2. my own managed click house on top of R2. my own managed click house on top of R2. It ended up being more than 10x cheaper It ended up being more than 10x cheaper It ended up being more than 10x cheaper and way easier for my agents to and way easier for my agents to and way easier for my agents to interface with. And I did this part-time interface with. And I did this part-time interface with. And I did this part-time with a handful of threads on the side with a handful of threads on the side with a handful of threads on the side while I was working on my real while I was working on my real while I was working on my real day-to-day work. And these types of day-to-day work. And these types of day-to-day work. And these types of things are miserable to do without AI.

things are miserable to do without AI. things are miserable to do without AI. Like this sucked long ago and now it's Like this sucked long ago and now it's Like this sucked long ago and now it's relatively trivial to set this stuff up relatively trivial to set this stuff up relatively trivial to set this stuff up yourself. Like truly crazy. As Chad is yourself. Like truly crazy. As Chad is yourself. Like truly crazy. As Chad is pointing out, I literally wrote my own pointing out, I literally wrote my own pointing out, I literally wrote my own TypeScript compiler and it's now running TypeScript compiler and it's now running TypeScript compiler and it's now running thousands of times a day in T3 Code. thousands of times a day in T3 Code. thousands of times a day in T3 Code. It's over 40,000 weekly downloads. And I It's over 40,000 weekly downloads. And I It's over 40,000 weekly downloads. And I threw this together kind of as like a threw this together kind of as like a threw this together kind of as like a joke to see if it was possible. Truly joke to see if it was possible. Truly joke to see if it was possible. Truly unbelievable. Oh, I got my numbers. Of unbelievable. Oh, I got my numbers. Of unbelievable. Oh, I got my numbers. Of all of my CI, almost a thousand hours of all of my CI, almost a thousand hours of all of my CI, almost a thousand hours of the CI on T3 code was spent waiting for the CI on T3 code was spent waiting for the CI on T3 code was spent waiting for Git checkouts. That's 42 days of waiting Git checkouts. That's 42 days of waiting Git checkouts. That's 42 days of waiting for your git pull. And that's just CI. for your git pull. And that's just CI. for your git pull. And that's just CI. That's not counting people doing it That's not counting people doing it That's not counting people doing it locally. Yeah. The estimates just for T3 locally. Yeah. The estimates just for T3 locally. Yeah. The estimates just for T3 code time spent waiting for git pull is code time spent waiting for git pull is code time spent waiting for git pull is between $350 and $600 just for the between $350 and $600 just for the between $350 and $600 just for the renting of a CPU pulling down the renting of a CPU pulling down the renting of a CPU pulling down the changes. Insane. And that's for a small changes. Insane. And that's for a small changes. Insane. And that's for a small project by a startup with fiveish project by a startup with fiveish project by a startup with fiveish employees depending on how you cut. employees depending on how you cut. employees depending on how you cut. Imagine how much crazier that could get Imagine how much crazier that could get Imagine how much crazier that could get at this level of scale. I do like this at this level of scale. I do like this at this level of scale. I do like this section. This is a thing I've never section. This is a thing I've never section. This is a thing I've never heard from Anthropic. Engineers are no heard from Anthropic. Engineers are no heard from Anthropic. Engineers are no less valuable, but their work is less valuable, but their work is less valuable, but their work is changing. Thank you, Anthropic, for not changing. Thank you, Anthropic, for not changing. Thank you, Anthropic, for not saying engineers don't matter anymore.

saying engineers don't matter anymore. saying engineers don't matter anymore. Nice change. All the engineers who Nice change. All the engineers who Nice change. All the engineers who worked on the system would now rate worked on the system would now rate worked on the system would now rate Claude as better at writing code than Claude as better at writing code than Claude as better at writing code than them. But the engineers still supplied them. But the engineers still supplied them. But the engineers still supplied the invaluable understanding of the the invaluable understanding of the the invaluable understanding of the highlevel interystem components in the highlevel interystem components in the highlevel interystem components in the design level trade-offs, making informed design level trade-offs, making informed design level trade-offs, making informed choices around the problems they wanted choices around the problems they wanted choices around the problems they wanted to solve and the scale they knew they to solve and the scale they knew they to solve and the scale they knew they needed to achieve. Very often they found needed to achieve. Very often they found needed to achieve. Very often they found that the role of engineers was to tell that the role of engineers was to tell that the role of engineers was to tell Claude no. Agents will sometimes violate Claude no. Agents will sometimes violate Claude no. Agents will sometimes violate core system constraints to satisfy a core system constraints to satisfy a core system constraints to satisfy a pattern that should never have been pattern that should never have been pattern that should never have been allowed in the first place. Okay, maybe allowed in the first place. Okay, maybe allowed in the first place. Okay, maybe they are using the same Claude that we they are using the same Claude that we they are using the same Claude that we are. This was particularly common in are. This was particularly common in are. This was particularly common in their migrations. For example, there their migrations. For example, there their migrations. For example, there were multiple systems that tried to were multiple systems that tried to were multiple systems that tried to brute force looking up when a change brute force looking up when a change brute force looking up when a change merged domain by linearly reading the merged domain by linearly reading the merged domain by linearly reading the commit log instead of asking the code commit log instead of asking the code commit log instead of asking the code review tool. Anthropic also makes review tool. Anthropic also makes review tool. Anthropic also makes extensive use of adversarial review. And extensive use of adversarial review. And extensive use of adversarial review. And while this is incredibly effective in while this is incredibly effective in while this is incredibly effective in many ways, can also be myopic. So many ways, can also be myopic. So many ways, can also be myopic. So skilled engineers need to steer Claude skilled engineers need to steer Claude skilled engineers need to steer Claude away from overengineering parts of the away from overengineering parts of the away from overengineering parts of the system in response to hypothetical system in response to hypothetical system in response to hypothetical concerns. Yep, this is even worse when concerns. Yep, this is even worse when concerns. Yep, this is even worse when you have feedback from other model you have feedback from other model you have feedback from other model families because they're worse at families because they're worse at families because they're worse at convincing each other. Making sure that convincing each other. Making sure that convincing each other. Making sure that your AI review systems don't your AI review systems don't your AI review systems don't accidentally cause huge overengineering accidentally cause huge overengineering accidentally cause huge overengineering and scope creep is a genuine real and scope creep is a genuine real and scope creep is a genuine real challenge. For example, one such review challenge. For example, one such review challenge. For example, one such review was overly defensive about trusting was overly defensive about trusting was overly defensive about trusting other internal services resulted in other internal services resulted in other internal services resulted in Claude adding excessive validation and Claude adding excessive validation and Claude adding excessive validation and complexity. Our team members still complexity. Our team members still complexity. Our team members still review every change request for the review every change request for the review every change request for the system, but they now examine the intent system, but they now examine the intent system, but they now examine the intent of the change as opposed to reading of the change as opposed to reading of the change as opposed to reading every line of code. Oh, how the world every line of code. Oh, how the world every line of code. Oh, how the world has changed. We were able to has changed. We were able to has changed. We were able to substantially improve optimization and substantially improve optimization and substantially improve optimization and quality by virtue of Claude's ability to

quality by virtue of Claude's ability to quality by virtue of Claude's ability to write immense amounts of code much write immense amounts of code much write immense amounts of code much faster than they could themselves. Scion faster than they could themselves. Scion faster than they could themselves. Scion is among the most well- tested and is among the most well- tested and is among the most well- tested and well-defended code bases they've ever well-defended code bases they've ever well-defended code bases they've ever worked on. Even on their tight worked on. Even on their tight worked on. Even on their tight timelines, they were able to do hermetic timelines, they were able to do hermetic timelines, they were able to do hermetic simulations of every non-trivial simulations of every non-trivial simulations of every non-trivial infrastructure and source control infrastructure and source control infrastructure and source control interaction that they've ever interaction that they've ever interaction that they've ever encountered, as well as fuzzing and encountered, as well as fuzzing and encountered, as well as fuzzing and sophisticated rate limiting. It still sophisticated rate limiting. It still sophisticated rate limiting. It still blows me away that I can be like, "Hey, blows me away that I can be like, "Hey, blows me away that I can be like, "Hey, Claude, build a fuzzer for this system. Claude, build a fuzzer for this system. Claude, build a fuzzer for this system. Find bugs." and it just does. Surreal. Find bugs." and it just does. Surreal. Find bugs." and it just does. Surreal. Claude also found significant Claude also found significant Claude also found significant optimizations. For example, algorithmic optimizations. For example, algorithmic optimizations. For example, algorithmic optimizations to ancestry queries that optimizations to ancestry queries that optimizations to ancestry queries that missed one of their indexes, which would missed one of their indexes, which would missed one of their indexes, which would have taken an expert days or weeks of have taken an expert days or weeks of have taken an expert days or weeks of analyzing and tuning and then did it all analyzing and tuning and then did it all analyzing and tuning and then did it all again the next day. This one's again the next day. This one's again the next day. This one's particularly funny cuz I just filmed my particularly funny cuz I just filmed my particularly funny cuz I just filmed my video about Gemini 4 argan. Gemini 4 was video about Gemini 4 argan. Gemini 4 was video about Gemini 4 argan. Gemini 4 was hallucinating columns in my SQLite hallucinating columns in my SQLite hallucinating columns in my SQLite tables. After reading the file that is tables. After reading the file that is tables. After reading the file that is my whole schema, it will then fire a my whole schema, it will then fire a my whole schema, it will then fire a query for columns that don't exist. query for columns that don't exist. query for columns that don't exist. Meanwhile, Claude is doing algorithmic Meanwhile, Claude is doing algorithmic Meanwhile, Claude is doing algorithmic optimizations to get around missing optimizations to get around missing optimizations to get around missing indexes. Anybody who says Gemini is in a indexes. Anybody who says Gemini is in a indexes. Anybody who says Gemini is in a good state right now is not trustable as good state right now is not trustable as good state right now is not trustable as an engineer. It's pretty rough.

an engineer. It's pretty rough. an engineer. It's pretty rough. Yeah, Gemini's imagination is Yeah, Gemini's imagination is Yeah, Gemini's imagination is incredible. It hallucinates so well. incredible. It hallucinates so well. incredible. It hallucinates so well. It's truly a creative model. You can see It's truly a creative model. You can see It's truly a creative model. You can see here the difference in everyday source here the difference in everyday source here the difference in everyday source control commands when they run their SCN control commands when they run their SCN control commands when they run their SCN system instead of git. Pulls are 25x system instead of git. Pulls are 25x system instead of git. Pulls are 25x faster. Checkouts are 14x faster. Rebase faster. Checkouts are 14x faster. Rebase faster. Checkouts are 14x faster. Rebase is 5x faster. Status is 207 times is 5x faster. Status is 207 times is 5x faster. Status is 207 times faster. Commits are only 30% faster. faster. Commits are only 30% faster. faster. Commits are only 30% faster. Disappointing. But diffs are six times Disappointing. But diffs are six times Disappointing. But diffs are six times faster. New work trees. Oh god, this is faster. New work trees. Oh god, this is faster. New work trees. Oh god, this is one of the biggest issues. And like we one of the biggest issues. And like we one of the biggest issues. And like we have to deal this in T3 code. Making have to deal this in T3 code. Making have to deal this in T3 code. Making work trees when you have to clone work trees when you have to clone work trees when you have to clone everything is really expensive. 84x everything is really expensive. 84x everything is really expensive. 84x faster is a huge win there. Pull latest faster is a huge win there. Pull latest faster is a huge win there. Pull latest is 20 times faster. Commits are 24 times is 20 times faster. Commits are 24 times is 20 times faster. Commits are 24 times faster. And status for agent faster. And status for agent faster. And status for agent environments is 20x faster. Huge. Oh, environments is 20x faster. Huge. Oh, environments is 20x faster. Huge. Oh, this first section was developer this first section was developer this first section was developer environments versus agent environments. environments versus agent environments. environments versus agent environments. That makes sense. Yeah. All of these That makes sense. Yeah. All of these That makes sense. Yeah. All of these wins are massive. Alongside all the wins are massive. Alongside all the wins are massive. Alongside all the performance wins, they also laid the performance wins, they also laid the performance wins, they also laid the groundwork for new forms of monor repo groundwork for new forms of monor repo groundwork for new forms of monor repo security. Managing safety, security, and security. Managing safety, security, and security. Managing safety, security, and compliance in massive monor repos is as compliance in massive monor repos is as compliance in massive monor repos is as important as managing the size and important as managing the size and important as managing the size and scale. They built Scion with these in scale. They built Scion with these in scale. They built Scion with these in mind from the start. It supports rich mind from the start. It supports rich mind from the start. It supports rich access controls on repositories as well access controls on repositories as well access controls on repositories as well as the ability to apply permissions to as the ability to apply permissions to as the ability to apply permissions to individual directories in order to individual directories in order to individual directories in order to protect the most sensitive code without protect the most sensitive code without protect the most sensitive code without losing the benefits of the monor repo.

losing the benefits of the monor repo. losing the benefits of the monor repo. Look who was right about committing your Look who was right about committing your Look who was right about committing your environment variables, [ __ ] environment variables, [ __ ] environment variables, [ __ ] The theo wins never end. can't wait for The theo wins never end. can't wait for The theo wins never end. can't wait for another animate profile picture on another animate profile picture on another animate profile picture on Twitter to say I'm wrong because they Twitter to say I'm wrong because they Twitter to say I'm wrong because they don't have a job. Furthermore, the code don't have a job. Furthermore, the code don't have a job. Furthermore, the code review tools that they built can express review tools that they built can express review tools that they built can express complex semantic constraints on complex semantic constraints on complex semantic constraints on approvals which allows them to approvals which allows them to approvals which allows them to differentiate simple things like comment differentiate simple things like comment differentiate simple things like comment only changes from sensitive logical only changes from sensitive logical only changes from sensitive logical changes so that human reviewers can changes so that human reviewers can changes so that human reviewers can focus on the most consequential parts. focus on the most consequential parts. focus on the most consequential parts. You [ __ ] you don't need to do You [ __ ] you don't need to do You [ __ ] you don't need to do this. You don't have to do this this. You don't have to do this this. You don't have to do this programmatically. You don't have to programmatically. You don't have to programmatically. You don't have to build code to change the categorization build code to change the categorization build code to change the categorization of the change sets. I don't know if you of the change sets. I don't know if you of the change sets. I don't know if you guys have heard about this. There's this guys have heard about this. There's this guys have heard about this. There's this thing called claude. You can give it thing called claude. You can give it thing called claude. You can give it code and ask it a question and you get code and ask it a question and you get code and ask it a question and you get an answer. This actually might be the an answer. This actually might be the an answer. This actually might be the thing I'm most bearish on in this whole thing I'm most bearish on in this whole thing I'm most bearish on in this whole thing. If they are compute constrained thing. If they are compute constrained thing. If they are compute constrained enough that they're not running claude enough that they're not running claude enough that they're not running claude to summarize and filter their diffs and to summarize and filter their diffs and to summarize and filter their diffs and they feel like they need to make they feel like they need to make they feel like they need to make programmatic solutions for this instead, programmatic solutions for this instead, programmatic solutions for this instead, maybe they are out of comput. Maybe we maybe they are out of comput. Maybe we maybe they are out of comput. Maybe we are screwed. With all of this, they are screwed. With all of this, they are screwed. With all of this, they claim that a new level of operational claim that a new level of operational claim that a new level of operational excellence is possible. And you know excellence is possible. And you know excellence is possible. And you know what? To their [ __ ] credit, it is.

what? To their [ __ ] credit, it is. what? To their [ __ ] credit, it is. Their uptime is significantly better Their uptime is significantly better Their uptime is significantly better than it was in the past. It was bad bad than it was in the past. It was bad bad than it was in the past. It was bad bad before. I can't remember the last time before. I can't remember the last time before. I can't remember the last time cloud was down. The website goes down cloud was down. The website goes down cloud was down. The website goes down decently often, but like cloud code and decently often, but like cloud code and decently often, but like cloud code and the cloud APIs, those are pretty the cloud APIs, those are pretty the cloud APIs, those are pretty reliable lately. Funny enough, I think reliable lately. Funny enough, I think reliable lately. Funny enough, I think I've had more problems with OpenAI and I've had more problems with OpenAI and I've had more problems with OpenAI and codeex outages than I have with cloud codeex outages than I have with cloud codeex outages than I have with cloud recently, which is unbelievable recently, which is unbelievable recently, which is unbelievable considering how big the gap was before. considering how big the gap was before. considering how big the gap was before. So yeah, if they're saying that this is So yeah, if they're saying that this is So yeah, if they're saying that this is helping make operational excellence more helping make operational excellence more helping make operational excellence more possible, I'll believe them because they possible, I'll believe them because they possible, I'll believe them because they are they are fixing their downtime are they are fixing their downtime are they are fixing their downtime problems. Their team uses Claude Tag, problems. Their team uses Claude Tag, problems. Their team uses Claude Tag, which if you don't know, that's their which if you don't know, that's their which if you don't know, that's their Slackbot that they're all obsessed with. Slackbot that they're all obsessed with. Slackbot that they're all obsessed with. They use it heavily and it helps them They use it heavily and it helps them They use it heavily and it helps them monitor each component of the system. monitor each component of the system. monitor each component of the system. For example, they have Slack channels For example, they have Slack channels For example, they have Slack channels where they tasked Claude with being where they tasked Claude with being where they tasked Claude with being obsessively focused on the health of obsessively focused on the health of obsessively focused on the health of single components. They call these super single components. They call these super single components. They call these super fans. Also, they refer to them as fans. Also, they refer to them as fans. Also, they refer to them as claudes plural. That hurts me a bit. claudes plural. That hurts me a bit. claudes plural. That hurts me a bit. Guys, why why do you have to do this Guys, why why do you have to do this Guys, why why do you have to do this anthropic? I I was defending you and it anthropic? I I was defending you and it anthropic? I I was defending you and it felt good. Now you're doing cringe [ __ ] felt good. Now you're doing cringe [ __ ] felt good. Now you're doing cringe [ __ ] like this. We call these Claude super like this. We call these Claude super like this. We call these Claude super fans. I'm like imagining the person fans. I'm like imagining the person fans. I'm like imagining the person saying this to me and somehow their face saying this to me and somehow their face saying this to me and somehow their face is more punchable than mine. They have a is more punchable than mine. They have a is more punchable than mine. They have a Scion Damon superfan, a Scion cash Scion Damon superfan, a Scion cash Scion Damon superfan, a Scion cash superfan, an antique merge Q superfan, superfan, an antique merge Q superfan, superfan, an antique merge Q superfan, and so on. They have access to the and so on. They have access to the and so on. They have access to the telemetry and logs, and they act like telemetry and logs, and they act like telemetry and logs, and they act like experienced engineers who have nothing experienced engineers who have nothing experienced engineers who have nothing better to do than refresh charts and better to do than refresh charts and better to do than refresh charts and look for anything strange. Are you look for anything strange. Are you look for anything strange. Are you talking [ __ ] That's my job. They do talking [ __ ] That's my job. They do talking [ __ ] That's my job. They do call out they have traditional alerts as call out they have traditional alerts as call out they have traditional alerts as well, but the super fans detect subtle well, but the super fans detect subtle well, but the super fans detect subtle and novel issues that make those alerts and novel issues that make those alerts and novel issues that make those alerts seem one-dimensional by comparison. That seem one-dimensional by comparison. That seem one-dimensional by comparison. That does actually seem kind of cool. Another does actually seem kind of cool. Another does actually seem kind of cool. Another thing earlier I said where they were

thing earlier I said where they were thing earlier I said where they were reducing compute with the programmatic reducing compute with the programmatic reducing compute with the programmatic thing. This is the opposite. This is thing. This is the opposite. This is thing. This is the opposite. This is them burning compute for fun. And here's them burning compute for fun. And here's them burning compute for fun. And here's what's probably the most important what's probably the most important what's probably the most important lesson in all of this. We're completely lesson in all of this. We're completely lesson in all of this. We're completely rethinking our priors. Engineers, for rethinking our priors. Engineers, for rethinking our priors. Engineers, for good reasons, get pretty fixed in their good reasons, get pretty fixed in their good reasons, get pretty fixed in their way of doing things because in order to way of doing things because in order to way of doing things because in order to be a worldclass engineer, you need to be a worldclass engineer, you need to be a worldclass engineer, you need to have a system in your head on how to do have a system in your head on how to do have a system in your head on how to do the thing. And generally speaking, the the thing. And generally speaking, the the thing. And generally speaking, the hot, hyped up, cool, new thing is not hot, hyped up, cool, new thing is not hot, hyped up, cool, new thing is not the best long-term solution. So, a lot the best long-term solution. So, a lot the best long-term solution. So, a lot of the best engineers in the world have of the best engineers in the world have of the best engineers in the world have gotten used to this and are inherently gotten used to this and are inherently gotten used to this and are inherently skeptical by default. They trust what skeptical by default. They trust what skeptical by default. They trust what they know and what they know works and they know and what they know works and they know and what they know works and they're hesitant to try new things they're hesitant to try new things they're hesitant to try new things because they've been burned so many because they've been burned so many because they've been burned so many times. You got to get over that. Things times. You got to get over that. Things times. You got to get over that. Things are changing too fast. You have to flip are changing too fast. You have to flip are changing too fast. You have to flip the way you think about this almost like the way you think about this almost like the way you think about this almost like upside down where you have this upside down where you have this upside down where you have this knowledge of what does and doesn't work knowledge of what does and doesn't work knowledge of what does and doesn't work which leads you to picking certain which leads you to picking certain which leads you to picking certain tools. Now you need to do the opposite. tools. Now you need to do the opposite. tools. Now you need to do the opposite. You need to know what does and doesn't You need to know what does and doesn't You need to know what does and doesn't work and where it breaks and where it work and where it breaks and where it work and where it breaks and where it functions and use that to reinvent the functions and use that to reinvent the functions and use that to reinvent the things you've trusted your whole life to things you've trusted your whole life to things you've trusted your whole life to be perfect for your needs. I have been be perfect for your needs. I have been be perfect for your needs. I have been writing Typescript for almost a decade writing Typescript for almost a decade writing Typescript for almost a decade at this point and I rewrote TypeScript at this point and I rewrote TypeScript at this point and I rewrote TypeScript itself for fun. Let's see how anthropic itself for fun. Let's see how anthropic itself for fun. Let's see how anthropic explained this phenomena. This project explained this phenomena. This project explained this phenomena. This project prompted them to take a very hard look prompted them to take a very hard look prompted them to take a very hard look at other workflows that were designed at other workflows that were designed at other workflows that were designed for human scale. The proper merge Q for for human scale. The proper merge Q for for human scale. The proper merge Q for example just doesn't make sense at agent example just doesn't make sense at agent example just doesn't make sense at agent scale. What does CI even mean when scale. What does CI even mean when scale. What does CI even mean when hundreds or even thousands of changes hundreds or even thousands of changes hundreds or even thousands of changes can merge while tests are running? We can merge while tests are running? We can merge while tests are running? We haven't been able to effectively haven't been able to effectively haven't been able to effectively replicate the industry standard of a replicate the industry standard of a replicate the industry standard of a global repo lock on the merge queue. So

global repo lock on the merge queue. So global repo lock on the merge queue. So instead, they've invested in postmerge instead, they've invested in postmerge instead, they've invested in postmerge in pre-eploy automated testing. Look at in pre-eploy automated testing. Look at in pre-eploy automated testing. Look at that. Theo was right again. Sorry to that. Theo was right again. Sorry to that. Theo was right again. Sorry to freak out about these things again, but freak out about these things again, but freak out about these things again, but you guys have no idea how long I've been you guys have no idea how long I've been you guys have no idea how long I've been harassed for thinking it's more harassed for thinking it's more harassed for thinking it's more important to be able to undo a bad important to be able to undo a bad important to be able to undo a bad change than to add thousands of layers change than to add thousands of layers change than to add thousands of layers in front of the changes to make it in front of the changes to make it in front of the changes to make it slightly harder to make a bad change. So slightly harder to make a bad change. So slightly harder to make a bad change. So much of engineering has been focused on much of engineering has been focused on much of engineering has been focused on increasing friction more and more and increasing friction more and more and increasing friction more and more and more to lower the likelihood something more to lower the likelihood something more to lower the likelihood something breaks. I've been focused on make the breaks. I've been focused on make the breaks. I've been focused on make the friction as low as possible. Both to get friction as low as possible. Both to get friction as low as possible. Both to get code out, but more importantly to fix code out, but more importantly to fix code out, but more importantly to fix something when it breaks. Because no something when it breaks. Because no something when it breaks. Because no matter how strict your processes are, matter how strict your processes are, matter how strict your processes are, you will ship something broken. I don't you will ship something broken. I don't you will ship something broken. I don't care who the [ __ ] you are. If you care who the [ __ ] you are. If you care who the [ __ ] you are. If you actually think you will never ship actually think you will never ship actually think you will never ship broken code or that you have built your broken code or that you have built your broken code or that you have built your system in a way where it will never ever system in a way where it will never ever system in a way where it will never ever have bugs, I hope that your mom's have bugs, I hope that your mom's have bugs, I hope that your mom's basement doesn't smell too bad because basement doesn't smell too bad because basement doesn't smell too bad because in the real world things break and when in the real world things break and when in the real world things break and when things break you need to fix them and things break you need to fix them and things break you need to fix them and focusing on ways to fix things and focusing on ways to fix things and focusing on ways to fix things and reduce this damage that broken things reduce this damage that broken things reduce this damage that broken things can do is incredibly important. They do can do is incredibly important. They do can do is incredibly important. They do call out that by working this way call out that by working this way call out that by working this way sometimes concurrent changes conflict sometimes concurrent changes conflict sometimes concurrent changes conflict but Claude can typically fix these but Claude can typically fix these but Claude can typically fix these conflicts faster than humans can even conflicts faster than humans can even conflicts faster than humans can even identify them. I've been thinking a lot identify them. I've been thinking a lot identify them. I've been thinking a lot about this. The idea I've had is what if about this. The idea I've had is what if about this. The idea I've had is what if the merge Q wasn't PRs that need to be the merge Q wasn't PRs that need to be the merge Q wasn't PRs that need to be merged one after another? What if the merged one after another? What if the merged one after another? What if the merge Q was a super general here are merge Q was a super general here are merge Q was a super general here are things that we're happy to have go in.

things that we're happy to have go in. things that we're happy to have go in. Hey Claude, land all of this. And it Hey Claude, land all of this. And it Hey Claude, land all of this. And it would try to combine like once something would try to combine like once something would try to combine like once something has the the merge thumbs up, Claude has the the merge thumbs up, Claude has the the merge thumbs up, Claude tries to squash as many of those chunks tries to squash as many of those chunks tries to squash as many of those chunks into individual things as possible like into individual things as possible like into individual things as possible like in groups and then pull them into main in groups and then pull them into main in groups and then pull them into main as effectively as it can dealing with as effectively as it can dealing with as effectively as it can dealing with the conflicts as it goes because Claude the conflicts as it goes because Claude the conflicts as it goes because Claude is now better at dealing with merge is now better at dealing with merge is now better at dealing with merge conflicts than I am. And now we got an conflicts than I am. And now we got an conflicts than I am. And now we got an interesting end here contributing to the interesting end here contributing to the interesting end here contributing to the wider ecosystem. We see Scion as just wider ecosystem. We see Scion as just wider ecosystem. We see Scion as just one part of the answer to the scaling one part of the answer to the scaling one part of the answer to the scaling concerns that increasingly capable AI concerns that increasingly capable AI concerns that increasingly capable AI agents have raised where sharing these agents have raised where sharing these agents have raised where sharing these details is an example of how teams can details is an example of how teams can details is an example of how teams can tackle similar challenges. Though they tackle similar challenges. Though they tackle similar challenges. Though they don't expect their solution or their don't expect their solution or their don't expect their solution or their considerations to suit every case. They considerations to suit every case. They considerations to suit every case. They specify this is in no means a suitable specify this is in no means a suitable specify this is in no means a suitable replacement for Git and general use. The replacement for Git and general use. The replacement for Git and general use. The concepts I've outlined here really only concepts I've outlined here really only concepts I've outlined here really only make sense for truly enormous monor make sense for truly enormous monor make sense for truly enormous monor repos with either tens of thousands of repos with either tens of thousands of repos with either tens of thousands of engineers or swarms of agents engineers or swarms of agents engineers or swarms of agents prolifically writing code. Plus, prolifically writing code. Plus, prolifically writing code. Plus, Anthropic's unique in that their ability Anthropic's unique in that their ability Anthropic's unique in that their ability to spend tokens for internal use far to spend tokens for internal use far to spend tokens for internal use far exceeds what any other business would exceeds what any other business would exceeds what any other business would realistically want to spend. Never realistically want to spend. Never realistically want to spend. Never thought I'd see the day. To be fair, as thought I'd see the day. To be fair, as thought I'd see the day. To be fair, as I've talked about many times, tokens are I've talked about many times, tokens are I've talked about many times, tokens are getting cheaper, way cheaper. A given getting cheaper, way cheaper. A given getting cheaper, way cheaper. A given level of intelligence is achievable at level of intelligence is achievable at level of intelligence is achievable at lower and lower prices constantly. The lower and lower prices constantly. The lower and lower prices constantly. The fable rewrite of bun into Rust was like fable rewrite of bun into Rust was like fable rewrite of bun into Rust was like 300k plus of Fable tokens. I was able to 300k plus of Fable tokens. I was able to 300k plus of Fable tokens. I was able to do a bolder rewrite which was Typescript do a bolder rewrite which was Typescript do a bolder rewrite which was Typescript in Rust myself in around 20 to $25,000 in Rust myself in around 20 to $25,000 in Rust myself in around 20 to $25,000 of Opus 55 tokens. The speed that these of Opus 55 tokens. The speed that these of Opus 55 tokens. The speed that these prices are going down is insane. So, it prices are going down is insane. So, it prices are going down is insane. So, it is absolutely possible by this time next is absolutely possible by this time next is absolutely possible by this time next year, you'd be able to write your own

year, you'd be able to write your own year, you'd be able to write your own similar thing to Scion, your own similar thing to Scion, your own similar thing to Scion, your own alternative to Git for your company for alternative to Git for your company for alternative to Git for your company for just a few thousand in tokens. They also just a few thousand in tokens. They also just a few thousand in tokens. They also call out a thing that is important to call out a thing that is important to call out a thing that is important to understand. They are doing this both to understand. They are doing this both to understand. They are doing this both to address their own software development address their own software development address their own software development needs, but also to test Claude's needs, but also to test Claude's needs, but also to test Claude's capabilities to see what it can do. capabilities to see what it can do. capabilities to see what it can do. Practically speaking, it's difficult to Practically speaking, it's difficult to Practically speaking, it's difficult to offer Scion in a form that would be offer Scion in a form that would be offer Scion in a form that would be broadly useful to other organizations. broadly useful to other organizations. broadly useful to other organizations. It diverges significantly from git, It diverges significantly from git, It diverges significantly from git, meaning companies would need to migrate meaning companies would need to migrate meaning companies would need to migrate every piece of the development life every piece of the development life every piece of the development life cycle and it requires its own code cycle and it requires its own code cycle and it requires its own code review, code search, and CI. Their review, code search, and CI. Their review, code search, and CI. Their internal version of these components are internal version of these components are internal version of these components are highly bespoke and they would be highly bespoke and they would be highly bespoke and they would be difficult for other organizations to difficult for other organizations to difficult for other organizations to adopt and they worry that attempting to adopt and they worry that attempting to adopt and they worry that attempting to build a public version that tries to build a public version that tries to build a public version that tries to please everybody wouldn't actually be please everybody wouldn't actually be please everybody wouldn't actually be very useful. Okay, I know what this very useful. Okay, I know what this very useful. Okay, I know what this paragraph is here for. There was a paragraph is here for. There was a paragraph is here for. There was a conversation internally about whether or conversation internally about whether or conversation internally about whether or not they could sell what they built. The not they could sell what they built. The not they could sell what they built. The people who built it said, "No, [ __ ] you. people who built it said, "No, [ __ ] you. people who built it said, "No, [ __ ] you. We're not going to do that. Here's all We're not going to do that. Here's all We're not going to do that. Here's all the reasons why it doesn't make sense." the reasons why it doesn't make sense." the reasons why it doesn't make sense." Enterprise sales finally probably caved. Enterprise sales finally probably caved. Enterprise sales finally probably caved. It's like, "Okay, whatever. We don't It's like, "Okay, whatever. We don't It's like, "Okay, whatever. We don't have to sell this." And then they wanted have to sell this." And then they wanted have to sell this." And then they wanted to talk about it. And comms is like, to talk about it. And comms is like, to talk about it. And comms is like, "Wait, wait, wait. People are going to "Wait, wait, wait. People are going to "Wait, wait, wait. People are going to ask for this." So, they had to take all ask for this." So, they had to take all ask for this." So, they had to take all the arguments they had internally and do the arguments they had internally and do the arguments they had internally and do this so that I can justify now as I read this so that I can justify now as I read this so that I can justify now as I read this to you why they are not open this to you why they are not open this to you why they are not open sourcing this for others to use. Okay, sourcing this for others to use. Okay, sourcing this for others to use. Okay, more justification. Let's see this part.

more justification. Let's see this part. more justification. Let's see this part. However, they anticipate that this class However, they anticipate that this class However, they anticipate that this class of problems will become more commonplace of problems will become more commonplace of problems will become more commonplace over the next 6 to 12 months and it over the next 6 to 12 months and it over the next 6 to 12 months and it would be an inefficient use of would be an inefficient use of would be an inefficient use of engineering resources and tokens every engineering resources and tokens every engineering resources and tokens every company using Git to face similar company using Git to face similar company using Git to face similar challenges alone. Given that, we think challenges alone. Given that, we think challenges alone. Given that, we think the most impactful thing we can do is the most impactful thing we can do is the most impactful thing we can do is share our thinking so that other devs share our thinking so that other devs share our thinking so that other devs can apply some of the most successful can apply some of the most successful can apply some of the most successful parts of Scion to Git without departing parts of Scion to Git without departing parts of Scion to Git without departing from the overall model. A Git server from the overall model. A Git server from the overall model. A Git server doesn't have to work the same way as a doesn't have to work the same way as a doesn't have to work the same way as a Git client does. Novel server Git client does. Novel server Git client does. Novel server architectures exist that provide architectures exist that provide architectures exist that provide significant scaling improvements. Git significant scaling improvements. Git significant scaling improvements. Git platforms can provide comprehensive platforms can provide comprehensive platforms can provide comprehensive serverside RPCs like Scion, perhaps even serverside RPCs like Scion, perhaps even serverside RPCs like Scion, perhaps even in a standardized way across forges that in a standardized way across forges that in a standardized way across forges that the Git clients can learn to talk to. the Git clients can learn to talk to. the Git clients can learn to talk to. Negotiation protocol doesn't have to Negotiation protocol doesn't have to Negotiation protocol doesn't have to look exactly like it does today either look exactly like it does today either look exactly like it does today either in the various sparse and shallow in the various sparse and shallow in the various sparse and shallow features in Git can be made to compose features in Git can be made to compose features in Git can be made to compose well. Lastly, they'd like to acknowledge well. Lastly, they'd like to acknowledge well. Lastly, they'd like to acknowledge that the Git community has spent nearly that the Git community has spent nearly that the Git community has spent nearly two decades scaling the platform for two decades scaling the platform for two decades scaling the platform for millions of devs. Their contribution is millions of devs. Their contribution is millions of devs. Their contribution is far narrower. They're sharing their far narrower. They're sharing their far narrower. They're sharing their early views into how agentic software early views into how agentic software early views into how agentic software development interacts with these development interacts with these development interacts with these systems. They can partner with GitHub, systems. They can partner with GitHub, systems. They can partner with GitHub, GitLab, and the open source community to GitLab, and the open source community to GitLab, and the open source community to support the changing needs of devs.

support the changing needs of devs. support the changing needs of devs. We're all in it together as the software We're all in it together as the software We're all in it together as the software development life cycle intersects with development life cycle intersects with development life cycle intersects with the AI exponential. It's crazy how the AI exponential. It's crazy how the AI exponential. It's crazy how something like this would have felt something like this would have felt something like this would have felt comical a year ago, and now it's almost comical a year ago, and now it's almost comical a year ago, and now it's almost like, yeah, of course, makes a lot of like, yeah, of course, makes a lot of like, yeah, of course, makes a lot of sense. I'm going to go fight the urge to sense. I'm going to go fight the urge to sense. I'm going to go fight the urge to make my own git clone. It's going to be make my own git clone. It's going to be make my own git clone. It's going to be hard, but I'll do my best. I hope you hard, but I'll do my best. I hope you hard, but I'll do my best. I hope you enjoyed this deep dive. I know I did. I enjoyed this deep dive. I know I did. I enjoyed this deep dive. I know I did. I love these types of nerdy rants and this love these types of nerdy rants and this love these types of nerdy rants and this was a very fun one for me getting to go was a very fun one for me getting to go was a very fun one for me getting to go deep on how source control works and how deep on how source control works and how deep on how source control works and how to make it scale the way that agents to make it scale the way that agents to make it scale the way that agents need. You still enjoy these deep dives need. You still enjoy these deep dives need. You still enjoy these deep dives or do you just want to see me vibe code or do you just want to see me vibe code or do you just want to see me vibe code more? Let me know in the comments. Until more? Let me know in the comments. Until more? Let me know in the comments. Until next time, he sns.