Changing how Engineering works with Ironclad CTO Sunita Verma

I think I think last year everyone last year everyone last year everyone told me AI told me AI told me AI would make me 20% more would make me 20% more productive, and I productive, and I productive, and I assumed that assumed that assumed that meant I'd get meant I'd get meant I'd get Friday off. Friday off. Friday off. But I guess But I guess But I guess you're working harder you're working harder you're working harder than ever. So than ever. So than ever. So how is it how is it that AI was supposed to that AI was supposed to that AI was supposed to take away the tedious take away the tedious take away the tedious work, but now I work, but now I work, but now I can generate 10 can generate 10 can generate 10 things and they all things and they all things and they all need to be verified? Yes. Yes. need to be verified? Yes. Yes. How do we How do we balance this? I balance this? I balance this? I understand that the basis understand that the basis understand that the basis is capitalism and is capitalism and is capitalism and growth, growth, growth. But growth, growth, growth. But growth, growth, growth. But how can you develop your how can you develop your how can you develop your brain, become brain, become brain, become smarter and not burn out, smarter and not burn out, smarter and not burn out, because AI was supposed to because AI was supposed to because AI was supposed to prevent prevent prevent burnout, and now burnout, and now burnout, and now everyone feels everyone feels everyone feels pretty burned out pretty burned out ? ? ? So, So, So, hello friends, hello friends, hello friends, thanks to Text Control for thanks to Text Control for thanks to Text Control for sponsoring the show. sponsoring the show. sponsoring the show. The developers ask a The developers ask a The developers ask a new question. Do we still need new question. Do we still need third-party libraries if AI can just if AI can just if AI can just generate all the generate all the generate all the code? For typical code? For typical UI tasks, maybe UI tasks, maybe not always, but not always, but not always, but deep deep deep functionality functionality is a different matter. is a different matter. Document editing, Document editing, PDF generation, PDF generation, PDF generation, rendering, rendering, rendering, conversion, and conversion, and conversion, and enterprise enterprise document workflows are the document workflows are the result of result of result of years of years of years of engineering, engineering, engineering, extensive extensive extensive testing, and millions of testing, and millions of testing, and millions of real-world deployments. real-world deployments. real-world deployments. Artificial Artificial Artificial intelligence can intelligence can intelligence can generate a lot of generate a lot of generate a lot of code, but it can't code, but it can't code, but it can't instantly create instantly create decades-old document processing technologies. And TX Text document processing technologies. And TX Text Control provides Control provides Control provides developers with this developers with this developers with this foundation so you foundation so you foundation so you can can can focus on focus on focus on building your building your building your application, not application, not application, not rebuilding the rebuilding the document engine. Learn document engine. Learn more at textcontrol.com.

more at textcontrol.com. more at textcontrol.com. Hi, I'm Scott Hi, I'm Scott Hi, I'm Scott Hanselman. This is Hanselman. This is Hanselman. This is another edition of another edition of another edition of Hanselminutes. Today I Hanselminutes. Today I Hanselminutes. Today I have the pleasure of have the pleasure of have the pleasure of chatting with Sunita chatting with Sunita chatting with Sunita Verma. She is the Verma. She is the Verma. She is the CTO of CTO of CTO of Ironclad. How are you? Ironclad. How are you? Ironclad. How are you? Fine. Very good. Fine. Very good. Fine. Very good. Thank you. Glad to be Thank you. Glad to be Thank you. Glad to be here. here. here. Yes, thanks for Yes, thanks for Yes, thanks for chatting. You've chatting. You've chatting. You've been working for been working for been working for quite a long time, 18 years quite a long time, 18 years quite a long time, 18 years at Google. at Google. at Google. Many, many years ago Many, many years ago Many, many years ago you worked at Symantec. You you worked at Symantec. You you worked at Symantec. You received your received your received your master's degree from the master's degree from the University of California, Santa Barbara. So I Barbara. So I Barbara. So I feel some feel some feel some kinship because kinship because kinship because I feel like we've I feel like we've I feel like we've all seen it. But you all seen it. But you all seen it. But you decided to stay in decided to stay in decided to stay in this this this AI-driven moment and AI-driven moment and AI-driven moment and help development teams help development teams navigate it. Yes. navigate it. Yes. And I would say that And I would say that And I would say that we have seen a lot, we have seen a lot, we have seen a lot, maybe not everything. maybe not everything. maybe not everything. Yes, yes. Yes, yes. Yes, yes. And the reason And the reason And the reason I'm still in the game is because I'm still in the game is because I've I've been working in the AI ​​field for many years. I been working in the AI ​​field for many years. I been working in the AI ​​field for many years. I survived the "winter". I survived the "winter". I survived the "winter". I went through a phase went through a phase went through a phase where machine where machine where machine learning hadn't yet become learning hadn't yet become learning hadn't yet become deep learning, had deep learning, had deep learning, had n't yet become, you know n't yet become, you know n't yet become, you know , generative AI. And , generative AI. And , generative AI. And for me, given for me, given for me, given that I'm a that I'm a that I'm a technology technology technology optimist and I believe in the optimist and I believe in the optimist and I believe in the potential of technology potential of technology potential of technology for positive for positive for positive transformation, and this is the transformation, and this is the transformation, and this is the biggest biggest biggest transformation transformation transformation we can we can we can hope to experience hope to experience hope to experience in our lifetime, in our lifetime, in our lifetime, so I wanted to be so I wanted to be so I wanted to be there. Yes. Is it there. Yes. Is it hard to be a hard to be a technological technological technological optimist? I optimist? I optimist? I encounter this a little bit.

encounter this a little bit. encounter this a little bit. I was recently called I was recently called I was recently called toxically positive. toxically positive. toxically positive. I think it's a compliment I think it's a compliment , but perhaps a , but perhaps a , but perhaps a veiled one. But at veiled one. But at veiled one. But at this moment, when this moment, when this moment, when everyone is saying that AI everyone is saying that AI everyone is saying that AI will ruin everything, I will ruin everything, I will ruin everything, I try to be try to be try to be positive and positive and positive and say that in say that in say that in fact, everything will get fact, everything will get fact, everything will get better if we decide to better if we decide to better if we decide to improve it. improve it. improve it. I agree with this I agree with this I agree with this formulation: if formulation: if formulation: if we decide to improve it we decide to improve it . And I think a lot . And I think a lot . And I think a lot depends on us, depends on us, depends on us, on how we on how we on how we use use use technology, for what technology, for what technology, for what purposes and for what, purposes and for what, purposes and for what, so to speak, end so to speak, end so to speak, end results. And this is results. And this is results. And this is no different, no different, no different, Scott, than any Scott, than any Scott, than any other technology curve other technology curve we've we've seen before. seen before. seen before. Take social Take social Take social media, mobile media, mobile media, mobile devices, devices, devices, infrastructure—everything infrastructure—everything infrastructure—everything has gone through a has gone through a has gone through a similar curve. similar curve. similar curve. So I don't expect that to be the case So I don't expect that to be the case So I don't expect that to be the case with this with this with this technology development curve technology development curve technology development curve . Although the . Although the . Although the technology itself is technology itself is technology itself is significantly significantly significantly different. different. different. Yes, it doesn't Yes, it doesn't Yes, it doesn't feel like just feel like just feel like just another another another tool. Just like we tool. Just like we tool. Just like we have iPhone, cellular have iPhone, cellular have iPhone, cellular communications, the cloud. It communications, the cloud. It communications, the cloud. It seemed like something very seemed like something very seemed like something very important. This seems important. This seems important. This seems even more important than the even more important than the even more important than the cloud. And the cloud was cloud. And the cloud was cloud. And the cloud was important. important. important. Yes, and this is a more important Yes, and this is a more important Yes, and this is a more important event, in my event, in my event, in my opinion, because opinion, because opinion, because the technology at first the technology at first the technology at first glance seems very glance seems very glance seems very simple, very simple, very simple, very clear, because clear, because clear, because natural language natural language is the interface. But under the is the interface. But under the is the interface. But under the hood, it works hood, it works hood, it works completely completely differently, and this may differently, and this may differently, and this may come as a surprise come as a surprise come as a surprise to many people. So to many people. So to many people. So this technology has a this technology has a this technology has a very...I don't know what to very...I don't know what to very...I don't know what to say. She has a very say. She has a very say. She has a very strange dynamic.

strange dynamic. strange dynamic. Yeah, I think people Yeah, I think people Yeah, I think people don't talk about how don't talk about how don't talk about how it makes it makes it makes us feel. We us feel. We us feel. We communicate with a huge communicate with a huge communicate with a huge numerical matrix, with numerical matrix, with numerical matrix, with mathematics. But if mathematics. But if mathematics. But if you add a voice or a you add a voice or a you add a voice or a feature to an open AI feature to an open AI , like making , like making , like making it breathe, giggle, it breathe, giggle, it breathe, giggle, or make some kind of or make some kind of or make some kind of noise, they noise, they noise, they try their best try their best try their best to anthropomorphize to anthropomorphize to anthropomorphize those numbers. those numbers. And sometimes I think that And sometimes I think that creates a creates a creates a certain feeling certain feeling , especially in , especially in software development teams, where people software development teams, where people might think, “This might think, “This might think, “This object, this person, object, this person, object, this person, this entity is going to replace this entity is going to replace this entity is going to replace me. He is smarter than me." me. He is smarter than me." me. He is smarter than me." But I don't believe that he is But I don't believe that he is But I don't believe that he is smarter than us. smarter than us. smarter than us. He may know something we don't He may know something we don't He may know something we don't , but I don't believe , but I don't believe , but I don't believe he's smarter than us. he's smarter than us. he's smarter than us. So, So, So, at least as I at least as I at least as I thought about it, thought about it, thought about it, despite despite despite knowing the inner knowing the inner knowing the inner workings of the workings of the workings of the technology, it can be technology, it can be technology, it can be considered that it was considered that it was considered that it was able to compress able to compress able to compress human knowledge. human knowledge. human knowledge. Everything she Everything she Everything she learned, she learned learned, she learned learned, she learned from us. Both on the scale of from us. Both on the scale of from us. Both on the scale of all humanity that all humanity that all humanity that created this created this created this knowledge, and in the knowledge, and in the knowledge, and in the time dimension.

time dimension. Our brain, like any individual Our brain, like any individual person person person , is , is , is capable of containing capable of containing capable of containing much more much more much more knowledge. Yes, in that knowledge. Yes, in that knowledge. Yes, in that sense it sense it sense it definitely contains definitely contains definitely contains more knowledge. But more knowledge. But more knowledge. But I'm I'm I'm not sure about intelligence, because I not sure about intelligence, because I not sure about intelligence, because I recently came across a recently came across a recently came across a paper on recursive paper on recursive paper on recursive superintelligence from superintelligence from superintelligence from Shingwa University, Shingwa University, Shingwa University, where, among other things, where, among other things, where, among other things, the authors used the authors used the authors used artificial artificial artificial intelligence to intelligence to intelligence to train AI. They train AI. They train AI. They found that the AI found that the AI found that the AI continually continually continually optimized its optimized its optimized its performance within the performance within the performance within the given strategy and did given strategy and did given strategy and did not change it. This is not change it. This is not change it. This is quite logical, quite logical, quite logical, given the principle of its given the principle of its given the principle of its operation, since it has already operation, since it has already operation, since it has already found, how can I found, how can I found, how can I put it, a local put it, a local put it, a local optimum. And it optimum. And it optimum. And it strives to stay strives to stay strives to stay in this local in this local in this local optimum. While optimum. While optimum. While humans are much more humans are much more humans are much more creative, we think creative, we think creative, we think outside the box. If in a outside the box. If in a outside the box. If in a given situation we don't given situation we don't given situation we don't get the desired get the desired get the desired result, we result, we result, we say, "You know what? say, "You know what? say, "You know what? Let's just Let's just Let's just rethink the whole rethink the whole rethink the whole paradigm." And that's not something paradigm." And that's not something paradigm." And that's not something I can do I can do . That's why I still . That's why I still . That's why I still associate associate associate critical thinking critical thinking , imagination, , imagination, , imagination, creativity with creativity with creativity with people.

people. This is an excellent This is an excellent argument. I think argument. I think argument. I think distillation is distillation is distillation is different from different from different from remixing. And remixing. And remixing. And if I have a if I have a if I have a mixtape, I can mixtape, I can mixtape, I can make a copy of it, and make a copy of it, and make a copy of it, and then it might be of then it might be of then it might be of lower quality. lower quality. lower quality. But as a person, I But as a person, I But as a person, I will want to will want to will want to remake it, change it, remake it, change it, remake it, change it, do something different with it do something different with it do something different with it . So I . So I . So I think you're right. I think you're right. I think you're right. I recently gave a recently gave a recently gave a session to a group of session to a group of session to a group of non-technical non-technical non-technical AI users, and I AI users, and I AI users, and I talked about JPEG talked about JPEG formatting of formatting of formatting of knowledge, right? And knowledge, right? And someone said, “Oh, all of someone said, “Oh, all of someone said, “Oh, all of Wikipedia in this Wikipedia in this Wikipedia in this one file?” And here's one file?” And here's one file?” And here's what I'm thinking: "Well, no, what I'm thinking: "Well, no, what I'm thinking: "Well, no, they just said they just said they just said the sky is blue the sky is blue the sky is blue instead of instead of instead of showing me every showing me every showing me every single blue single blue single blue pixel and all the little pixel and all the little pixel and all the little details. It's like we details. It's like we details. It's like we chose "Save chose "Save chose "Save File As," chose JPEG, File As," chose JPEG, File As," chose JPEG, and did a very, very and did a very, very and did a very, very small thing. small thing. small thing. Compression. Compression. Compression. We have thrown away a lot. We have thrown away a lot. We have thrown away a lot. This is with losses. This is with losses. This is with losses. So we had a So we had a So we had a whole conversation about whole conversation about whole conversation about lossy compression. But lossy compression. But lossy compression. But it doesn't feel like it doesn't feel like it doesn't feel like lossy compression. And I lossy compression. And I lossy compression. And I think that's the think that's the think that's the secret. That's the secret. That's the secret. I was going to secret. I was going to say that this is the say that this is the say that this is the secret. Yes. Yes. secret. Yes. Yes. secret. Yes. Yes. So, let So, let So, let me ask you this me ask you this me ask you this . You, you, you know . You, you, you know . You, you, you know we're we're we're in a in a in a branding moment right now, in the branding moment right now, in the branding moment right now, in the sense that it's funny to sense that it's funny to sense that it's funny to talk about AI, and LLMs have talk about AI, and LLMs have talk about AI, and LLMs have made people made people made people think that deep think that deep think that deep learning and machine learning and machine learning and machine learning are just learning are just learning are just predicting the predicting the predicting the next token, next token, next token, even though the even though the even though the machine learning, machine learning, machine learning, deep learning, and deep learning, and deep learning, and AI ecosystem in general has been around for AI ecosystem in general has been around for AI ecosystem in general has been around for decades. But decades. But decades. But now that we now that we now that we have these bots, and everyone's have these bots, and everyone's have these bots, and everyone's stopped writing code stopped writing code stopped writing code and is now talking to a and is now talking to a and is now talking to a bot, is there... Something bot, is there... Something bot, is there... Something or anything that you or anything that you or anything that you once believed about once believed about once believed about engineering engineering engineering organizations 5 years organizations 5 years organizations 5 years ago, and now your ago, and now your ago, and now your beliefs have changed beliefs have changed , and you believe something

, and you believe something , and you believe something different? different? Yes, one thing has Yes, one thing has definitely definitely definitely changed, I would changed, I would changed, I would say two things. I will say two things. I will say two things. I will tell you what has changed tell you what has changed and what has remained the and what has remained the and what has remained the same for any same for any same for any organization. I think organization. I think organization. I think what has changed is what has changed is what has changed is where engineering where engineering where engineering organizations get organizations get organizations get their leverage. their leverage. And what I mean is And what I mean is this: if you this: if you this: if you think about, say, the think about, say, the think about, say, the pre-agent era of AI, pre-agent era of AI, you needed you needed a group of a group of a group of engineers to achieve something; and the speed of engineers to achieve something; and the speed of , say, generating , say, generating , say, generating code and getting code and getting code and getting results was largely results was largely limited by limited by how much code people how much code people how much code people could write, could write, could write, how much good how much good how much good code they could code they could code they could write. But write. But write. But writing code is writing code is writing code is no longer a no longer a no longer a bottleneck. So bottleneck. So bottleneck. So that part of the that part of the that part of the leverage leverage leverage has shifted, and that's what has shifted, and that's what has shifted, and that's what I think I think I think makes small makes small makes small teams just as teams just as teams just as attractive in attractive in attractive in terms of terms of terms of achieving achieving achieving results as results as results as large teams. What has large teams. What has large teams. What has n't changed is the n't changed is the n't changed is the part I part I part I really really really believe in. As for believe in. As for believe in. As for the job of an engineer, I the job of an engineer, I the job of an engineer, I never thought, at never thought, at never thought, at least personally I least personally I least personally I never thought, that the never thought, that the never thought, that the job of an engineer is job of an engineer is job of an engineer is only about only about only about writing code.

writing code. Writing code was Writing code was only part of the job. only part of the job. Much of the work Much of the work involved involved involved thinking about the problem thinking about the problem , coming up with creative , coming up with creative , coming up with creative solutions, evaluating solutions, evaluating solutions, evaluating alternatives, alternatives, alternatives, deciding whether the deciding whether the proposed solution was proposed solution was good or not, and good or not, and good or not, and brainstorming with brainstorming with brainstorming with different people. And I do different people. And I do different people. And I do think think think that with the advent of that with the advent of that with the advent of tools, all of these tools, all of these tools, all of these aspects that I'm aspects that I'm aspects that I'm talking about have become even more talking about have become even more talking about have become even more important. So important. So important. So maybe that part of the maybe that part of the maybe that part of the job has changed for the job has changed for the job has changed for the better in that sense better in that sense better in that sense for me. So I for me. So I for me. So I think engineers, think engineers, think engineers, who have always who have always who have always looked for creative looked for creative looked for creative ways to solve ways to solve ways to solve problems, still problems, still problems, still get a get a get a lot of enjoyment out of it lot of enjoyment out of it . Yes. . Yes. . Yes. I like it too, I like it too, I like it too, but I think I'm but I think I'm but I think I'm grieving for what grieving for what grieving for what was lost, although I'm was lost, although I'm was lost, although I'm not quite sure if not quite sure if not quite sure if I lost the I lost the I lost the ability to solve ability to solve ability to solve problems, if problems, if problems, if I lost the work ethic. A I lost the work ethic. A I lost the work ethic. A Microsoft engineer Microsoft engineer Microsoft engineer recently posted a recently posted a recently posted a provocative tweet provocative tweet provocative tweet declaring that the declaring that the declaring that the typing phase of typing phase of typing phase of programming is programming is programming is over. It's over. It's clear. I think clear. I think I disagree with the typesetting I disagree with the typesetting I disagree with the typesetting for the for the for the following reason. It would be following reason. It would be following reason. It would be possible possible possible to add a voice to add a voice to add a voice interface to an interface to an interface to an existing existing existing editor, even editor, even editor, even though the though the earlier editors were earlier editors were developed first, and developed first, and developed first, and if the voice if the voice if the voice interface was interface was interface was good enough, it good enough, it good enough, it could, in a could, in a could, in a sense, record sense, record sense, record dictation and write code dictation and write code dictation and write code for you. But I think what's for you. But I think what's for you. But I think what's changed now is the changed now is the changed now is the ability to ability to create code yourself

create code yourself using these using these using these tools. tools. Okay, interesting. But Okay, interesting. But people still people still people still feel like feel like feel like something has been taken away from them. I something has been taken away from them. I see it as, I would say, a see it as, I would say, a bimodal bimodal bimodal distribution in distribution in distribution in some sense, where some sense, where some sense, where there's a group of there's a group of there's a group of engineers for engineers for engineers for whom whom whom programming programming programming was a very enjoyable was a very enjoyable was a very enjoyable activity, and for activity, and for activity, and for some, it some, it some, it really is. Yes. really is. Yes. They miss it, but They miss it, but for others it was for others it was for others it was just a tool and just a tool and just a tool and they enjoyed the they enjoyed the problem solving part more. This is a truly This is a truly thoughtful analysis. I thoughtful analysis. I thoughtful analysis. I think you are right. And one think you are right. And one think you are right. And one of the analogies that of the analogies that of the analogies that I tested was that I tested was that I tested was that I was in Japan and I I was in Japan and I I was in Japan and I saw Japanese saw Japanese saw Japanese carpentry, a carpentry, a carpentry, a thousand-year-old thousand-year-old thousand-year-old practice where practice where practice where you know how to fit you know how to fit you know how to fit wood, and they would wood, and they would wood, and they would spend months spend months spend months building a house. building a house. building a house. But then I came back to But then I came back to But then I came back to America, went to IKEA and America, went to IKEA and America, went to IKEA and bought some shelves for bought some shelves for bought some shelves for my kid my kid my kid who's going to who's going to who's going to college. college. college. So how can I live in a So how can I live in a So how can I live in a world where there is both IKEA and world where there is both IKEA and world where there is both IKEA and Japanese carpentry Japanese carpentry Japanese carpentry , and appreciate both , and appreciate both , and appreciate both and their place in and their place in and their place in this world?

this world? Yes, I think it's a Yes, I think it's a craft, that is, how to craft, that is, how to craft, that is, how to assemble a solution, how to assemble a solution, how to assemble a solution, how to build it, build it, build it, make it make it make it the best, the best, the best, optimized, optimized, optimized, scalable and scalable and scalable and reliable. I don't think any of reliable. I don't think any of reliable. I don't think any of that that that has changed. It may has changed. It may has changed. It may just be that the mechanical just be that the mechanical just be that the mechanical parts of the job parts of the job have definitely changed or been have definitely changed or been eliminated thanks to eliminated thanks to eliminated thanks to these tools. these tools. So yes, I So yes, I do do do feel that feel that feel that engineers, and I'm one of engineers, and I'm one of engineers, and I'm one of those who feels those who feels those who feels the joy and the joy and satisfaction of satisfaction of solving a solving a solving a problem well, feel problem well, feel problem well, feel it even more. In it even more. In it even more. In today's today's today's environment, I environment, I environment, I actually actually actually feel more feel more feel more confident. confident. confident. Yes, I used to Yes, I used to Yes, I used to feel feel feel limited in terms of limited in terms of limited in terms of input/output input/output input/output because my brain already knew the because my brain already knew the because my brain already knew the answer and all I had to do was answer and all I had to do was wait for the wait for the input/ input/ output process, which was output process, which was output process, which was my fingers, to complete. And I'm my fingers, to complete. And I'm my fingers, to complete. And I'm curious, as an curious, as an curious, as an experienced specialist, experienced specialist, experienced specialist, as a very experienced as a very experienced as a very experienced person, you person, you person, you return to return to return to programming, programming, programming, despite the fact that you are a despite the fact that you are a despite the fact that you are a technical director technical director technical director and do PR?

and do PR? and do PR? Because I see Because I see Because I see more and more more and more more and more really really really good good good CTOs who are CTOs who are CTOs who are doing PR for their doing PR for their doing PR for their teams. teams. teams. I have tested very I have tested very I have tested very few PRs so far, but have created few PRs so far, but have created few PRs so far, but have created many prototypes. I many prototypes. I many prototypes. I created a lot of created a lot of created a lot of prototypes to prototypes to prototypes to demonstrate the demonstrate the demonstrate the concepts that concepts that concepts that came to my mind came to my mind , and like you, I , and like you, I , and like you, I felt like felt like felt like I had a lot of ideas in my I had a lot of ideas in my I had a lot of ideas in my head and I was head and I was head and I was limited by limited by limited by how many of them I could how many of them I could how many of them I could program. And program. And program. And now I can now I can now I can code code code a lot of these ideas, a lot of these ideas, a lot of these ideas, create prototypes, create prototypes, create prototypes, show them to my show them to my show them to my team, showing how team, showing how team, showing how I at least I at least I at least thought I would solve this thought I would solve this thought I would solve this problem. Does this problem. Does this problem. Does this coincide with your coincide with your coincide with your opinion? opinion? opinion? Worldview? And we Worldview? And we Worldview? And we can have a much can have a much can have a much more, I would say, more, I would say, more, I would say, much much much deeper conversation. Yes. deeper conversation. Yes. deeper conversation. Yes. Well, I also Well, I also Well, I also notice that we ask ourselves notice that we ask ourselves notice that we ask ourselves : : : if we can if we can if we can build anything, build anything, build anything, what is the right thing to what is the right thing to what is the right thing to build? Because build? Because build? Because I see some I see some I see some organizations just organizations just organizations just throwing spaghetti at the throwing spaghetti at the throwing spaghetti at the wall, just wall, just wall, just coding whatever comes to hand. coding whatever comes to hand. coding whatever comes to hand. I'm guessing that in I'm guessing that in I'm guessing that in your job, deciding your job, deciding your job, deciding which which which direction direction direction to go is a lot more to go is a lot more to go is a lot more interesting than interesting than interesting than just writing everything just writing everything just writing everything down. down. down. Exactly. Exactly. This is how Exactly. Exactly. This is how Exactly. Exactly. This is how I I I use my time.

use my time. use my time. Yes, it's interesting. So, Yes, it's interesting. So, Yes, it's interesting. So, as a master of your as a master of your as a master of your craft, are you a craft, are you a craft, are you a tastemaker? Do you help tastemaker? Do you help tastemaker? Do you help your teams your teams your teams develop a sense of develop a sense of code smell? And what do you code smell? And what do you code smell? And what do you think about junior think about junior think about junior engineers? Because engineers? Because engineers? Because a lot of junior a lot of junior a lot of junior engineers are just engineers are just engineers are just coming out of school, just coming out of school, just coming out of school, just when AI came along when AI came along , and they're a little bit like, , and they're a little bit like, "What was the point of "What was the point of "What was the point of my education?" my education?" And I try to remind And I try to remind And I try to remind them that you learned to them that you learned to them that you learned to smell and smell and smell and taste during your education. taste during your education. Yes, and here we Yes, and here we purposefully and purposefully and purposefully and consciously decided that we consciously decided that we consciously decided that we would hire would hire would hire both experienced and both experienced and both experienced and entry-level entry-level entry-level specialists to join the team. We were specialists to join the team. We were specialists to join the team. We were one of those companies one of those companies that didn't say that didn't say that didn't say all the all the all the entry-level jobs entry-level jobs entry-level jobs were gone because were gone because were gone because who was going to be the, so to who was going to be the, so to who was going to be the, so to speak, senior speak, senior speak, senior engineers, the lead engineers, the lead engineers, the lead engineers engineers engineers of tomorrow? Those of tomorrow? Those of tomorrow? Those who have just who have just who have just completed their studies. And I completed their studies. And I completed their studies. And I see two different see two different see two different sides: senior sides: senior sides: senior engineers, or very engineers, or very engineers, or very experienced engineers, experienced engineers, experienced engineers, sometimes have a hard time sometimes have a hard time sometimes have a hard time unleashing old unleashing old unleashing old habits. But habits. But habits. But they are the ones who have strong they are the ones who have strong they are the ones who have strong analytical analytical analytical thinking. They are the ones who have the thinking. They are the ones who have the thinking. They are the ones who have the strong taste strong taste to evaluate to evaluate to evaluate the results of AI work.

the results of AI work. the results of AI work. Beginning engineers are Beginning engineers are Beginning engineers are completely immersed completely immersed completely immersed in the tools, but in the tools, but in the tools, but they don't have the they don't have the they don't have the analytical analytical analytical thinking or sense thinking or sense thinking or sense of smell of smell of smell you talked about. They you talked about. They you talked about. They don't have don't have pattern recognition skills yet because pattern recognition skills yet because they haven't gained they haven't gained they haven't gained that experience yet. That's why we that experience yet. That's why we that experience yet. That's why we pair people up pair people up pair people up ." Um, I'm not saying ." Um, I'm not saying ." Um, I'm not saying that we don't have people in our company that we don't have people in our company who use who use tools to tools to tools to record record record agent actions. No, we do agent actions. No, we do agent actions. No, we do n't have them anymore. But I think n't have them anymore. But I think that this combination that this combination that this combination helps a lot helps a lot helps a lot because junior because junior because junior engineers learn to engineers learn to engineers learn to spot bugs, spot bugs, spot bugs, since AI is not the since AI is not the only thing we only thing we teach our team. teach our team. teach our team. One of the things One of the things One of the things I encourage people to I encourage people to I encourage people to do is to always do is to always do is to always challenge challenge challenge AI actions. Don't take what AI actions. Don't take what AI actions. Don't take what he says as he says as he says as truth. Therefore, truth. Therefore, truth. Therefore, challenge this with challenge this with challenge this with your arguments your arguments your arguments or, in fact, or, in fact, or, in fact, refute them with refute them with refute them with your your your formulations. And this is where formulations. And this is where formulations. And this is where junior junior junior engineers can engineers can engineers can learn from senior ones. And I learn from senior ones. And I learn from senior ones. And I will give you a very will give you a very will give you a very specific example. specific example. specific example. One of the tools One of the tools One of the tools for decoding for decoding for decoding agent actions agent actions agent actions wrote a SQL query wrote a SQL query wrote a SQL query that was extremely that was extremely that was extremely unoptimized.

unoptimized. unoptimized. If it had gotten into If it had gotten into If it had gotten into production, it would production, it would production, it would likely have crashed likely have crashed likely have crashed our our our production production production system. And the engineer system. And the engineer system. And the engineer who checked it who checked it who checked it obviously had experience, obviously had experience, obviously had experience, understood how understood how understood how our our our production production production system worked, and said, “ system worked, and said, “ This is not the request. We This is not the request. We This is not the request. We would have written it would have written it would have written it differently." "And differently." "And differently." "And we actually provided a we actually provided a we actually provided a pretty detailed pretty detailed pretty detailed review to another review to another review to another engineer who engineer who engineer who wrote or wrote or wrote or generated some of the generated some of the generated some of the code. So, code. So, code. So, I think this ability I think this ability I think this ability to evaluate, this “nose” to evaluate, this “nose” , this ability , this ability , this ability to evaluate is to evaluate is to evaluate is even more important today than even more important today than even more important today than before. Absolutely. And I before. Absolutely. And I before. Absolutely. And I think it's important to note think it's important to note that people early in their that people early in their that people early in their careers, in my careers, in my careers, in my opinion, will acquire opinion, will acquire opinion, will acquire this "sense" through this "sense" through this "sense" through life experience, life experience, life experience, through making mistakes in through making mistakes in through making mistakes in production, or, in production, or, in production, or, in this case, almost making this case, almost making this case, almost making mistakes in mistakes in mistakes in production. production. production. Otherwise, they will Otherwise, they will Otherwise, they will never develop never develop never develop these skills. And I'm these skills. And I'm these skills. And I'm a little worried a little worried a little worried about the developers about the developers about the developers who used who used who used only AI, only AI, only AI, talked only talked only talked only to themselves in the AI ​​mirror, to themselves in the AI ​​mirror, to themselves in the AI ​​mirror, who will who will who will support them and support them and support them and help them? Because help them? Because help them? Because we came into being at a we came into being at a we came into being at a time when we were time when we were time when we were working in tandem with working in tandem with working in tandem with senior senior senior specialists.

specialists. specialists. Mmm. Mmm. Mmm. So I'm really So I'm really So I'm really pleased to hear that pleased to hear that pleased to hear that you're actually you're actually you're actually pairing people up pairing people up pairing people up with young with young with young professionals professionals professionals because it's a because it's a because it's a tough job market right now tough job market right now tough job market right now and a lot of young and a lot of young and a lot of young professionals are having a professionals are having a professionals are having a hard time hard time hard time finding work. finding work. finding work. I know that, and by the way, I know that, and by the way, I know that, and by the way, I'm very grateful. I'm very grateful. I'm very grateful. Our Our Our CEO was completely on CEO was completely on CEO was completely on board with this board with this board with this strategy. strategy. strategy. Yeah. Yeah. Yeah. And that's what I told him And that's what I told him , why we're doing this , why we're doing this , and he completely , and he completely , and he completely agreed with it. In agreed with it. In agreed with it. In fact, he even fact, he even fact, he even posted about it posted about it posted about it on LinkedIn. on LinkedIn. on LinkedIn. Oh, I like it. Oh, I like it. Oh, I like it. I like it. I like it. I like it. Hey guys, it's Hey guys, it's Hey guys, it's Scott and I want to Scott and I want to Scott and I want to say hi to say hi to say hi to one of our new one of our new one of our new sponsors, SerpApi. This is s e sponsors, SerpApi. This is s e sponsors, SerpApi. This is s e r p a p i. It is a web r p a p i. It is a web search API that search API that search API that provides provides provides developers with developers with developers with structured structured structured access to access to real-time search results. real-time search results. Works great with AI Works great with AI agents and AI agents and AI assistants. I assistants. I assistants. I use it on use it on use it on my personal my personal my personal instance of OpenClaw. It is instance of OpenClaw. It is instance of OpenClaw. It is suitable for suitable for suitable for research research research tools, tools, market analysis platforms, market analysis platforms, Rag workflows, anything Rag workflows, anything Rag workflows, anything that requires fresh that requires fresh that requires fresh information from the information from the information from the internet. You can internet. You can internet. You can get get get web search results as web search results as web search results as clean, clean, clean, structured structured structured Markdown, which on Markdown, which on Markdown, which on average uses average uses average uses about 50% fewer about 50% fewer about 50% fewer tokens than JSON. Very tokens than JSON. Very , very clean and , very clean and , very clean and simple integration into simple integration into simple integration into your existing your existing your existing AI workflows.

AI workflows. AI workflows. In fact, there In fact, there In fact, there are over a hundred are over a hundred are over a hundred such Serp APIs such Serp APIs such Serp APIs available to you. available to you. available to you. So, instead of So, instead of So, instead of building or building or building or maintaining maintaining data collection infrastructure and data collection infrastructure and trying to scrape trying to scrape websites, the Serp API takes care of websites, the Serp API takes care of websites, the Serp API takes care of all the complexity all the complexity all the complexity by providing by providing by providing search results as search results as structured structured data that your data that your data that your application can application can application can simply consume. simply consume. simply consume. As I said, As I said, As I said, there are over a hundred there are over a hundred there are over a hundred APIs. You have access to APIs. You have access to APIs. You have access to Google, Bing, Amazon, Walmart, YouTube. Google, Bing, Amazon, Walmart, YouTube. Google, Bing, Amazon, Walmart, YouTube. Many different Many different Many different sources. They sources. They sources. They have have have security and security and security and compliance with compliance with compliance with corporate corporate corporate standards, all the standards, all the standards, all the necessary necessary necessary certifications, certifications, certifications, making them a making them a making them a truly truly truly strong option strong option strong option for organizations where for organizations where for organizations where security and security and compliance are compliance are important. The important. The important. The best? You get best? You get best? You get 250 search queries 250 search queries 250 search queries every month for every month for every month for free. You can free. You can free. You can check them out at check them out at check them out at serpapi.com/hanselminutes. This is serpapi.com/hanselminutes. This is serpapi.com/hanselminutes for 250 serpapi.com/hanselminutes for 250 free free free searches per month. searches per month. searches per month. Another point Another point Another point you mentioned, you mentioned, you mentioned, which is a bit which is a bit which is a bit provocative, is provocative, is provocative, is your opinion that the your opinion that the your opinion that the team should be team should be team should be a little smaller. So a little smaller. So a little smaller. So , you could , you could , you could essentially split your essentially split your essentially split your teams. So teams. So teams. So instead of instead of instead of having a having a having a giant team, giant team, giant team, you would have many you would have many you would have many small teams, small teams, small teams, each each each responsible for its own responsible for its own responsible for its own work. What do you work. What do you work. What do you think is the think is the think is the optimal optimal optimal team size for an AI team right team size for an AI team right now?

now? I would probably answer I would probably answer I would probably answer this question a little this question a little this question a little differently. differently. differently. Yeah. Yeah. Yeah. And I thought about this a lot And I thought about this a lot And I thought about this a lot , because I , because I , because I worked in large worked in large worked in large teams. teams. teams. Yeah. Previously, Yeah. Previously, Yeah. Previously, since people were, since people were, since people were, so to speak, the main so to speak, the main so to speak, the main factor, and factor, and factor, and human human human results were, so to results were, so to results were, so to speak, the “long speak, the “long speak, the “long pole,” pole,” teams, especially in teams, especially in large companies, large companies, large companies, were organized according to were organized according to were organized according to specialization. And, in specialization. And, in specialization. And, in my opinion, some of the my opinion, some of the my opinion, some of the specialization will specialization will specialization will remain. So remain. So remain. So things like things like things like fault tolerance, fault tolerance, fault tolerance, reliability, reliability, reliability, scalability, scalability, scalability, especially because you especially because you especially because you have to take have to take have to take deployment data into account deployment data into account . It will . It will take quite a take quite a long time for artificial intelligence to long time for artificial intelligence to long time for artificial intelligence to achieve this, in my achieve this, in my achieve this, in my opinion. But I think opinion. But I think opinion. But I think that at other levels that at other levels that at other levels people are better at generalizing, people are better at generalizing, people are better at generalizing, they are better at they are better at they are better at doing more doing more doing more general work. And as general work. And as general work. And as a result, you don't a result, you don't a result, you don't need some of need some of need some of these these these specialized specialized specialized teams that you used to teams that you used to need, and so the need, and so the team sizes team sizes get smaller as a result. And to get smaller as a result. And to some extent, some extent, some extent, it also eliminates it also eliminates it also eliminates or reduces the or reduces the communication and communication and coordination overhead that coordination overhead that coordination overhead that naturally naturally naturally exists in large exists in large exists in large groups. So I groups. So I groups. So I think that small think that small think that small teams, if teams, if teams, if well organized, well organized, well organized, will have an will have an will have an advantage in the advantage in the advantage in the future. And I'm not future. And I'm not future. And I'm not necessarily saying necessarily saying necessarily saying that I completely that I completely that I completely agree with that agree with that agree with that point of view, but I point of view, but I point of view, but I think Sam Altman think Sam Altman think Sam Altman has said that, you know, has said that, you know, has said that, you know, in in in one- one- one- person companies, I don't completely person companies, I don't completely person companies, I don't completely share that point of share that point of share that point of view, but view, but view, but certainly certainly certainly team sizes are shrinking.

team sizes are shrinking. team sizes are shrinking. Yes, another way to look at Yes, another way to look at Yes, another way to look at this, which may this, which may this, which may be a little more be a little more be a little more positive, is the positive, is the positive, is the idea of ​​collective idea of ​​collective idea of ​​collective ownership. ownership. ownership. You know, when systems You know, when systems You know, when systems were smaller and weren't were smaller and weren't were smaller and weren't these huge these huge these huge distributed distributed distributed cloud systems like they are cloud systems like they are cloud systems like they are today, we talked today, we talked today, we talked about a full-stack engineer. It about a full-stack engineer. It about a full-stack engineer. It was a was a was a common common common phrase. And then we phrase. And then we phrase. And then we decided, “Well, the stack is decided, “Well, the stack is decided, “Well, the stack is too deep. There is no way I too deep. There is no way I too deep. There is no way I can know the can know the can know the whole stack." But now whole stack." But now whole stack." But now I feel like I can, I feel like I can, I feel like I can, and I can be the and I can be the and I can be the owner of the owner of the owner of the team. And I team. And I team. And I can also...You know, before can also...You know, before can also...You know, before I might have had a I might have had a server build engineer, a server build engineer, a quality assurance engineer, quality assurance engineer, quality assurance engineer, or a database engineer or a database engineer or a database engineer . Now I . Now I . Now I notice that people are more notice that people are more notice that people are more willing to take on willing to take on willing to take on things and say, “I’ve things and say, “I’ve things and say, “I’ve never done this before never done this before never done this before . I'll give it a try . I'll give it a try . I'll give it a try and see if I can and see if I can and see if I can handle this handle this handle this pull request." pull request." pull request." This is a little out of This is a little out of This is a little out of my my my comfort zone. “And I think comfort zone. “And I think comfort zone. “And I think that if we have this kind of that if we have this kind of growth and growth and learning mindset, I'll be able to learning mindset, I'll be able to learning mindset, I'll be able to try something try something try something new that's not in new that's not in new that's not in my area of my area of my area of expertise expertise expertise because the bot will because the bot will because the bot will support me and support me and support me and my team, and the my team, and the my team, and the mentors and mentors and mentors and mentees on my mentees on my mentees on my team will team will team will support me. support me. support me. You know, in a small You know, in a small You know, in a small team there can be a team there can be a team there can be a deeper deeper deeper collective collective collective responsibility. Yes. Yes.

responsibility. Yes. Yes. responsibility. Yes. Yes. And we see And we see And we see this in our environment too. Are this in our environment too. Are you changing you changing organizations? How do you organizations? How do you organizations? How do you do it? Because do it? Because do it? Because we are not we are not we are not necessarily talking about necessarily talking about necessarily talking about staff reduction, but staff reduction, but staff reduction, but about changing its about changing its about changing its structure. You structure. You structure. You might have hundreds of might have hundreds of might have hundreds of engineers, and now engineers, and now engineers, and now you're going to you're going to you're going to announce that you're a announce that you're a group of five group of five group of five people people people doing doing doing startups. startups. startups. Does this change Does this change Does this change the situation structurally? And I'm also the situation structurally? And I'm also the situation structurally? And I'm also curious, as a second curious, as a second curious, as a second question, do you change question, do you change question, do you change job titles? job titles? job titles? We see more and We see more and We see more and more technical more technical more technical specialists, which specialists, which specialists, which means that means that means that everyone has their own everyone has their own everyone has their own talent. Yes. talent. Yes. talent. Yes. So the answer to So the answer to So the answer to this question has this question has this question has two parts. First of all two parts. First of all , we have been around , we have been around , we have been around for about, I think, 10 for about, I think, 10 for about, I think, 10 years. More than years. More than years. More than 10 years have passed. So 10 years have passed. So 10 years have passed. So there is a there is a there is a certain certain certain culture, some culture, some culture, some aspects of it are quite aspects of it are quite aspects of it are quite good, and other aspects, in good, and other aspects, in good, and other aspects, in general, need to general, need to general, need to change, and we have already change, and we have already change, and we have already made those changes. made those changes. made those changes. But I think we want to But I think we want to But I think we want to preserve the good preserve the good preserve the good aspects of the culture, aspects of the culture, aspects of the culture, particularly those particularly those particularly those related to related to related to collaboration, collaboration, collaboration, knowledge sharing, and so knowledge sharing, and so knowledge sharing, and so on. Part of this is on. Part of this is on. Part of this is due to the fact that we have due to the fact that we have due to the fact that we have already made changes already made changes already made changes to the organization of teams, to the organization of teams, to the organization of teams, to their size. I would to their size. I would to their size. I would say it wasn't say it wasn't say it wasn't easy. This easy. This easy. This takes time, and it takes time, and it takes time, and it was more about was more about was more about changing mental changing mental changing mental models than any models than any other other other changes. But I changes. But I changes. But I think once think once think once those mental those mental those mental models changed, models changed, models changed, once people once people once people understood how working with understood how working with understood how working with this technology this technology this technology could significantly could significantly could significantly improve results, improve results, improve results, I think that I think that I think that change was change was change was beneficial.

beneficial. It took time, It took time, but now we are on the but now we are on the but now we are on the other side of this other side of this other side of this process. process. process. Now I'm 99% sure Now I'm 99% sure Now I'm 99% sure you won't tell me you won't tell me you won't tell me that that that performance in performance in performance in AI is measured in AI is measured in AI is measured in lines of code. lines of code. lines of code. No, I have never been a No, I have never been a No, I have never been a subscriber. To this. subscriber. To this. subscriber. To this. Even before AI came along, Even before AI came along, Even before AI came along, I never really I never really I never really shared that shared that shared that view. view. view. Great. So we Great. So we Great. So we agree that agree that agree that pull requests, pull requests, pull requests, lines of code, and all of these lines of code, and all of these lines of code, and all of these things can be things can be things can be used for used for used for malicious purposes. What malicious purposes. What malicious purposes. What is is is performance measurement? What performance measurement? What performance measurement? What is ROI? Are these is ROI? Are these is ROI? Are these results for results for results for clients? Is this the clients? Is this the clients? Is this the number of closed number of closed number of closed incidents? Is this time incidents? Is this time incidents? Is this time for training? Yes. for training? Yes. for training? Yes. First of all, I do First of all, I do First of all, I do n't really...Again n't really...Again n't really...Again , I also don't , I also don't , I also don't share the point of share the point of share the point of view about view about view about efficiency, efficiency, efficiency, because this is my because this is my because this is my interpretation, and interpretation, and interpretation, and interpretations of interpretations of interpretations of efficiency can efficiency can only be minimized to a only be minimized to a certain certain certain point, because the point, because the point, because the end result is end result is end result is zero. Basically, zero. Basically, zero. Basically, no waste, zero no waste, zero no waste, zero waste. waste. waste. Exactly. That Exactly. That Exactly. That 's why I don't think about 's why I don't think about 's why I don't think about efficiency. I efficiency. I efficiency. I think...Again, think...Again, think...Again, going back to your going back to your going back to your point about point about point about growth mindset, I growth mindset, I growth mindset, I think in terms of think in terms of think in terms of growth. And growth can growth. And growth can growth. And growth can be endless. be endless. be endless. If you want, it If you want, it If you want, it can be an can be an can be an endless game. This is endless game. This is endless game. This is how I think we how I think we how I think we think about it. And again think about it. And again think about it. And again , I've never , I've never , I've never measured measured measured performance in performance in performance in lines of code. I think lines of code. I think lines of code. I think we look at we look at we look at what results we what results we what results we can achieve at a can achieve at a can achieve at a given point in given point in given point in time.

time. time. Timeframes depend on Timeframes depend on Timeframes depend on the size of our the size of our the size of our team. And is the team. And is the team. And is the situation improving? It is also situation improving? It is also situation improving? It is also important to consider the important to consider the important to consider the quality of the result, quality of the result, quality of the result, because this is also an because this is also an because this is also an important aspect. You important aspect. You important aspect. You can write can write can write any code, there are no any code, there are no any code, there are no restrictions, but restrictions, but restrictions, but the quality of the result the quality of the result the quality of the result and whether and whether and whether it actually solves it actually solves it actually solves our clients' problem is an our clients' problem is an our clients' problem is an important criterion. important criterion. important criterion. So we So we So we look at both look at both look at both the results and the results and the results and what value what value what value we can deliver to clients we can deliver to clients with a given with a given team over a team over a team over a given period of given period of given period of time. time. time. I understand that this is a I understand that this is a I understand that this is a humanistic humanistic humanistic approach. The question is: have approach. The question is: have approach. The question is: have we improved anyone's we improved anyone's we improved anyone's life? Was the product life? Was the product life? Was the product sold? Has the sold? Has the sold? Has the user user user story been released? And story been released? And story been released? And was it released with was it released with was it released with quality? And was quality? And was quality? And was it released well? it released well? it released well? You know what? This is the You know what? This is the You know what? This is the right artifact, right artifact, right artifact, not metrics, not metrics, not metrics, token maximization token maximization token maximization and all that nonsense. and all that nonsense. and all that nonsense. I've never I've never I've never been into been into token maximization. Yes. token maximization. Yes. So yes, I So yes, I So yes, I understand that understand that understand that there is a trap in there is a trap in there is a trap in measuring measuring measuring productivity, productivity, productivity, because trying to because trying to because trying to reduce it to zero reduce it to zero reduce it to zero just leads to... just leads to... It's not... It's not... And you It's not... It's not... And you It's not... It's not... And you mentioned earlier... mentioned earlier... mentioned earlier... Ultimately, it's a Ultimately, it's a person's attention that's person's attention that's most most most valuable right now. I write code valuable right now. I write code valuable right now. I write code faster than I can faster than I can faster than I can check it. And check it. And check it. And it's easy to fall into it's easy to fall into it's easy to fall into the trap of just the trap of just the trap of just saying, "Okay, okay. " saying, "Okay, okay. " saying, "Okay, okay. " Merge the changes Merge the changes ." “I'm curious ." “I'm curious ." “I'm curious if you've encountered if you've encountered if you've encountered any any any difficulties in this difficulties in this difficulties in this regard, where people, regard, where people, regard, where people, I think someone I think someone I think someone called them 'meat called them 'meat called them 'meat proxies,' where I proxies,' where I proxies,' where I act as a act as a act as a proxy between the AI ​​and it.

proxy between the AI ​​and it. proxy between the AI ​​and it. So, you're my boss, So, you're my boss, So, you're my boss, you ask me a question you ask me a question , I ask the AI ​​a question, and , I ask the AI ​​a question, and , I ask the AI ​​a question, and I tell you the answer. What am I tell you the answer. What am I tell you the answer. What am I needed for? Yes. Yes. I needed for? Yes. Yes. I needed for? Yes. Yes. I have a I have a I have a joke. Maybe joke. Maybe joke. Maybe if we have if we have if we have time, I'll tell you time, I'll tell you time, I'll tell you about it. But, uh, to about it. But, uh, to about it. But, uh, to get back to your get back to your get back to your question, uh, one of the question, uh, one of the question, uh, one of the approaches that we've approaches that we've approaches that we've taken from the taken from the taken from the very beginning is that when very beginning is that when we introduce AI into our software development process, we software development process, we introduce it introduce it introduce it ubiquitously, that is, throughout ubiquitously, that is, throughout ubiquitously, that is, throughout all stages of the all stages of the software development life cycle, not software development life cycle, not just in the just in the just in the generation process, not just generation process, not just generation process, not just in the review process, in the review process, in the review process, but also in other parts of the but also in other parts of the but also in other parts of the process. Because it process. Because it process. Because it was very clear was very clear was very clear that by speeding up one that by speeding up one that by speeding up one part, you would essentially part, you would essentially part, you would essentially create create create bottlenecks in other bottlenecks in other bottlenecks in other areas. And that's areas. And that's areas. And that's going to happen because going to happen because going to happen because you have a you have a you have a lot of machines lot of machines lot of machines producing code on producing code on producing code on one side, and if one side, and if one side, and if you have humans, you have humans, you have humans, no matter no matter no matter how many humans you how many humans you how many humans you put in, you can't put in, you can't put in, you can't …" In a …" In a …" In a sense, outrunning a sense, outrunning a sense, outrunning a machine is the same thing machine is the same thing machine is the same thing . . . Mmm. Mmm. Mmm. But I must say, But I must say, But I must say, despite all our despite all our despite all our good intentions, and good intentions, and good intentions, and again, again, again, changing the mental changing the mental changing the mental model here takes model here takes model here takes some time, some time, some time, some parts of the code some parts of the code are still checked manually. are still checked manually. And even this sometimes proves to be an And even this sometimes proves to be an overwhelming task overwhelming task for the team. And we for the team. And we for the team. And we are currently in the are currently in the are currently in the process of working on it. We process of working on it. We process of working on it. We are exploring other are exploring other are exploring other ways to conduct ways to conduct ways to conduct code review.

code review. code review. Mmm. Mmm. Mmm. And one of the approaches And one of the approaches And one of the approaches we're thinking about we're thinking about we're thinking about is, "Can we is, "Can we is, "Can we use a use a risk-based approach to risk-based approach to code review? So that code review? So that code review? So that riskier parts of riskier parts of riskier parts of the code would be reviewed by a the code would be reviewed by a the code would be reviewed by a larger number of larger number of larger number of people, while people, while people, while more standard more standard more standard parts of the code could parts of the code could parts of the code could be reviewed by be reviewed by be reviewed by another AI and simply another AI and simply another AI and simply pass the review." “And pass the review." “And pass the review." “And one of the things one of the things one of the things we're very we're very we're very careful about is careful about is careful about is that the agent doing that the agent doing that the agent doing the checking has to be the checking has to be the checking has to be working with a different working with a different working with a different model than the agent model than the agent creating the code, creating the code, creating the code, because because because otherwise they're going to otherwise they're going to otherwise they're going to introduce the same introduce the same introduce the same biases on both biases on both biases on both sides. sides. sides. Yes, we do a lot of Yes, we do a lot of Yes, we do a lot of adversarial adversarial adversarial testing using testing using multiple models. I multiple models. I even take the model of the even take the model of the even take the model of the three major systems three major systems three major systems and compare it to the and compare it to the and compare it to the Claude, GPT, and Gemini models, Claude, GPT, and Gemini models, Claude, GPT, and Gemini models, and ask them all and ask them all and ask them all to vote on to vote on to vote on some parameters some parameters some parameters to try to to try to to try to avoid avoid avoid bias, bias, bias, because we find because we find because we find that, in my opinion, that, in my opinion, that, in my opinion, all of the Claude systems all of the Claude systems all of the Claude systems have bias, and have bias, and have bias, and that leads to... How do you that leads to... How do you that leads to... How do you say this? Blind say this? Blind say this? Blind spots. spots. spots. We do this in the We do this in the We do this in the assessment phase, so here's assessment phase, so here's assessment phase, so here's how it's done how it's done how it's done in a qualitative in a qualitative in a qualitative assessment. assessment. assessment. Oh, great. Great. Oh, great. Great. Oh, great. Great. You also mentioned You also mentioned You also mentioned burnout. I feel a burnout. I feel a burnout. I feel a bit like bit like bit like last year last year last year everyone told me everyone told me everyone told me AI would make me 20% more AI would make me 20% more productive, and I productive, and I productive, and I assumed that assumed that assumed that meant I'd meant I'd meant I'd get get get Friday off. But I Friday off. But I Friday off. But I guess you guess you guess you work harder than work harder than ..." Ever. So, ..." Ever. So, ..." Ever. So, how...AI is supposed to how...AI is supposed to how...AI is supposed to take away the mundane take away the mundane take away the mundane work, but now I work, but now I work, but now I can generate 10 can generate 10 can generate 10 things, and they all things, and they all things, and they all need to be checked. Yes. Yes.

need to be checked. Yes. Yes. How do we How do we balance this? I balance this? I balance this? I understand that understand that understand that capitalism and capitalism and capitalism and growth are at the core, but how do you growth are at the core, but how do you growth are at the core, but how do you develop your brain, develop your brain, develop your brain, become smarter, and become smarter, and become smarter, and not burn out? Because not burn out? Because not burn out? Because AI was supposed to AI was supposed to AI was supposed to prevent prevent prevent burnout, and now burnout, and now burnout, and now everyone feels everyone feels everyone feels pretty burned out pretty burned out . . So at least that's what So at least that's what I want us to I want us to I want us to think about: there's the think about: there's the think about: there's the rate of production of rate of production of rate of production of technology, and there's the rate of technology, and there's the rate of technology, and there's the rate of diffusion of that diffusion of that diffusion of that technology into our technology into our technology into our actual work actual work actual work processes, the processes, the processes, the workflows in the companies workflows in the companies workflows in the companies where we sell, and so where we sell, and so where we sell, and so on. on. Yeah. And there are a And there are a lot of lot of lot of existing systems existing systems existing systems and work that and work that and work that needs to be taken into account needs to be taken into account , and I think , and I think , and I think again I'm not one of those people again I'm not one of those people again I'm not one of those people who would say that AI who would say that AI who would say that AI will solve all the problems. will solve all the problems. Yeah. Yeah. We still have to We still have to We still have to participate. We participate. We participate. We still have to still have to still have to use our use our use our creativity and creativity and creativity and our judgment, and I our judgment, and I our judgment, and I think the pace of think the pace of think the pace of diffusion is diffusion is diffusion is slower, and I think slower, and I think slower, and I think ultimately ultimately ultimately the pace of the pace of the pace of technology... In my technology... In my technology... In my opinion, the pace of opinion, the pace of technology diffusion has to technology diffusion has to match the match the match the pace of technology pace of technology pace of technology adoption. adoption. adoption. Mmm. Mmm.

And maybe this is And maybe this is a little controversial on my a little controversial on my a little controversial on my part, but I think it's part, but I think it's part, but I think it's premature to premature to premature to say that the rate of say that the rate of technology production will technology production will largely largely largely match the match the match the rate of technology adoption. rate of technology adoption. We are not seeing We are not seeing results at the same results at the same results at the same speed, and speed, and speed, and there are many there are many there are many studies studies studies that show that show that show that for whatever that for whatever that for whatever reason, reason, reason, organizational organizational organizational inertia, the inertia, the inertia, the diffusion model, diffusion model, diffusion model, the results are not the results are not the results are not keeping up with the pace of keeping up with the pace of technology production. Yes, and I think there is great great value in simply acknowledging that. When value in simply acknowledging that. When value in simply acknowledging that. When leadership says things leadership says things leadership says things out loud that are out loud that are out loud that are n't usually said, and tells n't usually said, and tells n't usually said, and tells people that it's people that it's people that it's okay okay okay to feel a to feel a to feel a certain way certain way , and that it's okay , and that it's okay to focus on whether to focus on whether customers got what they customers got what they needed, needed, I think that makes the organization healthier. I think that makes the organization healthier. I agree. I I agree. I I agree. I agree that we agree that we agree that we want to continue to want to continue to want to continue to focus on focus on customer outcomes. customer outcomes. Yes, that's exactly it. So there are Yes, that's exactly it. So there are companies that do nothing with AI at all, as well as do nothing with AI at all, as well as companies with an companies with an companies with an immature software immature software immature software development lifecycle development lifecycle . I know . I know companies that do companies that do companies that do n't even have a formal n't even have a formal CICD system, and it's 2026 CICD system, and it's 2026 , and those people , and those people , and those people should be very should be very should be very concerned. If concerned. If concerned. If you came into a you came into a you came into a typical typical software development organization, software development organization, for example, as a for example, as a for example, as a new new new CTO, and asked CTO, and asked CTO, and asked : what do we need to : what do we need to : what do we need to change first change first change first ? How do we ? How do we ? How do we implement this new implement this new implement this new approach to the work of people approach to the work of people who are who are who are years behind?

years behind? I also I also adhere to two adhere to two adhere to two principles, and this is what principles, and this is what principles, and this is what we did at Iron Cloud. we did at Iron Cloud. we did at Iron Cloud. When I arrived, Iron Cloud When I arrived, Iron Cloud When I arrived, Iron Cloud was using was using was using earlier versions of AI, not earlier versions of AI, not earlier versions of AI, not generative AI. generative AI. generative AI. And one of the things And one of the things And one of the things that we actually that we actually that we actually made a conscious made a conscious made a conscious choice about, and our choice about, and our choice about, and our CTO CTO CTO and CEO were very much on board with this, is that we're going to make it and CEO were very much on board with this, is that we're going to make it easier for people to easier for people to access technology access technology access technology or give them the or give them the or give them the choice, and we're going choice, and we're going choice, and we're going to view to view to view technology adoption technology adoption technology adoption as a learning as a learning as a learning and development process. We will and development process. We will and development process. We will say that this is a say that this is a learning opportunity. And by the way, learning opportunity. And by the way, one of the things I think you one of the things I think you one of the things I think you know, and I know, is that in the know, and I know, is that in the know, and I know, is that in the technology technology technology space, space, adaptability works best. You adaptability works best. You adapt to adapt to adapt to new new new technological technological technological advances, otherwise you advances, otherwise you advances, otherwise you essentially become obsolete. essentially become obsolete. So, we said So, we said that this is an opportunity that this is an opportunity that this is an opportunity for growth, the company for growth, the company for growth, the company supports supports supports the training that we the training that we the training that we provide, and this provide, and this provide, and this training will help you training will help you training will help you do your do your do your job better.

job better. job better. Last year we Last year we Last year we ran a 20-day ran a 20-day ran a 20-day program. I program. I program. I taught there taught there taught there because I had a deep because I had a deep because I had a deep understanding of understanding of understanding of technology and helping technology and helping technology and helping people understand how to people understand how to people understand how to use it in their use it in their use it in their everyday work. everyday work. We ran this 20- day program day program day program before a hackathon we before a hackathon we before a hackathon we wanted to wanted to wanted to organize. And now, organize. And now, organize. And now, almost 10 almost 10 almost 10 months later, we as a months later, we as a months later, we as a company are in a company are in a company are in a very good very good very good position in position in position in terms of implementation. terms of implementation. terms of implementation. The implementation was The implementation was The implementation was fairly fairly fairly voluntary, and that's the voluntary, and that's the voluntary, and that's the part about part about part about measuring measuring measuring results results results that I was talking about. that I was talking about. that I was talking about. I was asked the same I was asked the same I was asked the same questions. The team questions. The team questions. The team that said, "Are you that said, "Are you that said, "Are you going to measure going to measure going to measure the number of lines of the number of lines of the number of lines of my code even though my code even though my code even though I don't write it?" And I I don't write it?" And I I don't write it?" And I replied: “I’ve never replied: “I’ve never replied: “I’ve never been a fan of the been a fan of the been a fan of the number of lines of number of lines of number of lines of code.” code.” I guess the short I guess the short answer to your answer to your answer to your question is that I would question is that I would question is that I would start with changing the start with changing the start with changing the mental model, mental model, mental model, even though that seems like a even though that seems like a even though that seems like a slower slower slower approach and approach and approach and the results will come the results will come the results will come a little later than an a little later than an a little later than an approach like, "Hey, approach like, "Hey, approach like, "Hey, starting starting starting today, we're today, we're today, we're all essentially all essentially all essentially moving to AI." “Um, moving to AI." “Um, moving to AI." “Um, because I don’t think the because I don’t think the because I don’t think the mental mental mental model in a model in a model in a company like the one company like the one company like the one you’re talking about is going to be you’re talking about is going to be you’re talking about is going to be set up to set up to set up to accept that, in accept that, in accept that, in a sense, a sense, a sense, once you once you once you launch launch launch educational educational educational programs or some kind of programs or some kind of training program. Yes.

training program. Yes. I'm really glad to I'm really glad to I'm really glad to hear that you're so hear that you're so hear that you're so focused on focused on focused on education, education, education, because I know that in 5 because I know that in 5 because I know that in 5 years, or when I years, or when I years, or when I retire, people who retire, people who retire, people who are at the beginning of their are at the beginning of their are at the beginning of their careers will have been careers will have been careers will have been working for 5 years or 10 working for 5 years or 10 working for 5 years or 10 years, and they'll be years, and they'll be years, and they'll be senior senior senior professionals, and they'll be professionals, and they'll be professionals, and they'll be living in a living in a living in a completely different completely different completely different world than we are. And I world than we are. And I world than we are. And I hope that the systems hope that the systems hope that the systems we leave them we leave them will not be fragile, and the will not be fragile, and the will not be fragile, and the organizations organizations organizations we leave them will be we leave them will be we leave them will be resilient, and we resilient, and we resilient, and we really really really focus on focus on focus on artificial artificial artificial intelligence. intelligence. intelligence. Intelligence Intelligence Intelligence is not intelligence is not intelligence , it is artificial, and , it is artificial, and , it is artificial, and it is here to it is here to it is here to serve us and make serve us and make serve us and make us smarter. So I us smarter. So I us smarter. So I hope that the hope that the hope that the organizations organizations organizations we leave them will be we leave them will be we leave them will be made up of made up of made up of smarter people. smarter people. smarter people. I agree with you, I agree with you, I agree with you, Scott, on that point, Scott, on that point, Scott, on that point, and that's one of the reasons and that's one of the reasons and that's one of the reasons we said we we said we wanted to have a wanted to have a wanted to have a mixed mixed mixed workforce." workforce." workforce." The company, the people The company, the people The company, the people who will become, who will become, who will become, you know, the you know, the you know, the managers and managers and managers and leaders of tomorrow leaders of tomorrow leaders of tomorrow . Yes. . Yes. . Yes. Well, I really appreciate Well, I really appreciate Well, I really appreciate you and thank you for taking the you and thank you for taking the time to talk to me today. time to talk to me today. No thanks. Thanks No thanks. Thanks No thanks. Thanks for all the for all the for all the insightful insightful insightful questions and it was a lot of questions and it was a lot of questions and it was a lot of fun. fun. fun. We spoke with We spoke with We spoke with Sunita Verma. She's the Sunita Verma. She's the Sunita Verma. She's the CTO at CTO at CTO at Ironclad, and this has been Ironclad, and this has been Ironclad, and this has been another edition of another edition of another edition of Hanselminutes, and we'll see you again Hanselminutes, and we'll see you again Hanselminutes, and we'll see you again next next next week.