Why you're not falling behind on AI - Barry O'Reilly (Author, Artificial Organizations)

Barry O’Reilly is an entrepreneur, author, and founder of Nobody Studios, an early-stage venture studio focused on building AI companies. Over the last six years he has worked with founders, executives and enterprise leadership teams to rethink how organisations operate in the age of generative AI, while simultaneously building and launching companies inside the studio model.

A former startup advisor and executive coach, Barry has spent the last several years studying why most AI transformations fail despite enormous investment. Through his coaching and advisory work with leaders from companies including American Airlines, Skyscanner, and Slack, Barry has developed practical frameworks for improving decision-making, reducing administrative overhead, and increasing what he calls "decision velocity".

In this episode, Barry explains why AI adoption fails when companies focus on tools instead of behaviour change, why judgment is becoming the most important human skill, and how teams can use AI to improve collaboration rather than replace people.

Key takeaways
 — Most AI transformations fail because organisations start with tools instead of behaviours. Installing AI software does not change how people work, make decisions or collaborate.
 — The most effective AI use cases amplify a person’s natural way of working. Barry realised he produced better writing by talking through ideas and using transcription tools instead of forcing himself into traditional writing workflows.
 — Capturing meetings, conversations and decisions as structured data creates long-term organisational intelligence. Every interaction becomes a reusable asset that improves preparation, follow-through, and future decision-making.
 — Leaders must role-model AI adoption themselves. Organisations see better outcomes when executives openly experiment with tools, share lessons learned, and create psychological safety around adoption.
 — Decision velocity matters more than raw productivity. Teams improve when they arrive prepared, make decisions faster, reduce reversals, and spend more time solving meaningful problems instead of handling administration.
 — AI should be used to challenge thinking, not replace it. The most valuable prompts ask for blind spots, alternative scenarios, and pressure tests rather than definitive answers.
 — Teams working with AI outperform individuals working with AI. Barry cites research showing that collaborative ideation with AI produces significantly stronger outcomes than isolated use.
 — Productivity gains are meaningless if they simply create more exhaustion. The real opportunity is creating space for reflection, slow thinking, and better judgment.
 — Judgment is the critical human capability organisations cannot outsource. If people stop exercising judgment and rely entirely on AI-generated answers, they gradually erode their ability to make decisions under uncertainty.

Chapters
 1:03 — Building AI companies at Nobody Studios
 3:16 — Why AI transformations fail
 5:05 — The danger of focusing on tools
 6:35 — Discovering natural workflows with AI
 8:51 — Turning conversations into data assets
 12:02 — Measuring successful AI adoption
 13:14 — Why leaders must role-model behaviour change
 18:39 — Decision velocity as a leadership metric
 21:33 — Escaping administrative overload
 23:02 — Why leaders need time to think
 26:54 — What CFOs are worried about
 28:08 — Can AI replace startup teams?
 29:45 — Why distribution still matters most
 33:13 — Capturing and synthesising ideas with AI
 34:38 — Using AI to challenge your thinking
 37:11 — Avoiding top-down AI-driven strategy
 39:00 — Why teams plus AI outperform individuals
 42:31 — The problem with AI-generated certainty
 43:12 — Preserving human judgment
 44:55 — Hiring for judgment and decision-making
 47:19 — Final reflections on leadership and AI

Our Hosts
Lily Smith enjoys working as a consultant product manager with early-stage and growing startups and as a mentor to other product managers. She’s currently Chief Product Officer at BBC Maestro, and has spent 13 years in the tech industry working with startups in the SaaS and mobile space. She’s worked on a diverse range of products – leading the product teams through discovery, prototyping, testing and delivery. Lily also founded ProductTank Bristol and runs ProductCamp in Bristol and Bath.

Randy Silver is a Leadership & Product Coach and Consultant. He gets teams unstuck, helping you to supercharge your results. Randy's held interim CPO and Leadership roles at scale-ups and SMEs, advised start-ups, and been Head of Product at HSBC and Sainsbury’s. He participated in Silicon Valley Product Group’s Coaching the Coaches forum, and speaks frequently at conferences and events. You can join one of communities he runs for CPOs (CPO Circles), Product Managers (Product In the {A}ether) and Product Coaches. He’s the author of What Do We Do Now? A Product Manager’s Guide to Strategy in the Time of COVID-19. A recovering music journalist and editor, Randy also launched Amazon’s music stores in the US & UK.

Transcript

SPEAKER_00 0:00

If you start with tools, you're going to fail. It's that simple. They don't tell you if their research is any good. They don't tell you if their emails are any good. They're just like, you need to use this tool or you're falling behind. It's designed to agree with you. Yes, it's designed to sort of surprise you. It's going to tell you the answer you want to hear. 85% of Gen AI projects fail. 83% of transformations fail because people make it about tools. What it's really about is behavior change. Judgment is the last human sort of skill that we need to hold on to. And if you are not exercising your judgment, you're eroding it. If a human pairs up with an AI, they can almost get parity to what a team is capable of doing. If teams pair up with AI, they're three times more productive in the quality of the ideation that they do. It's called thinking. It's called actually having space in your day to sort of marinate in these problems.

SPEAKER_01 0:59

Barry, thank you so much for joining us today. How are you doing?

SPEAKER_00 1:03

Yeah, good to be back again, Randy. Always fun to spend time with you. I love all the work at Mind the Product. I love your show. I love how you dig into helping people get better. So it's always a pleasure to have a chat.

SPEAKER_01 1:14

Flattery will get you everywhere, as you as you well know. Thank you so much. And we love having you on. It's wonderful to welcome you back on the occasion of your new book. Uh, but it's been a couple of years since you've been here. So besides the new book, what are you up to these days?

SPEAKER_00 1:28

Yeah, so maybe six years ago, I decided I was going to start an early stage uh incubator or venture studio, primarily focused on building AI companies. It's called Nobody Studios. Um and that's been quite a journey. It's probably been the biggest and hardest thing that I've done ever, but so much learning around what these tools and technologies actually mean and how you can apply them not only to building businesses, but also uh how you can start to you know launch companies around them. So a lot of the book is really a lessons learned, if you will, of the last six years building that studio. We've done everything from equity crowdfunding and had a thousand people invest to get the studio off the ground to we just literally cleared, closed our our seed round and opened our angel round or our series A round actually this week. So it's been an amazing journey, built a ton of companies, learnt a ton of stuff, made so many mistakes that I'm happy to share with everybody why you should never do this. Um so that's really been me for the last six years. And I guess the book is sort of a retrospective of those learnings.

SPEAKER_01 2:30

Well, congratulations on the series A. That's that's always really exciting, or or it's opening around. But it's it's it's this lovely new book that we're gonna be focusing our chat about today. And I want to start with the the very basic because you know, everybody in our world, whether they're product or tech or leadership of of some sort, they're going through the responding to the transformation or the acceleration that AI is bringing on. And it's both at the personal level and at the corporate level. And they're either wondering if they're you know, if they're fully on board or if they're missing out. And I think what I see from people most of the time is they're focusing on the technical aspects of that.

SPEAKER_00 3:10

Yeah.

SPEAKER_01 3:11

What might be the the what are they missing out if they're focusing purely on the tools and the tech?

SPEAKER_00 3:16

Yeah, so um, here's the news flash. If you start with tools, you're gonna fail. It's that simple. Um, you know, one one of the most interesting things about any sort of transformation, and and we've seen this whether it was mobile, cloud, digital transformation, but the the whole point is most people, their feed is just full of everyone going, but it's Instagram, uh, LinkedIn, you know, I woke up this morning, I I launched 7,000 agents, you know, I'm doing so much work, and you know, I'm gonna be okay, but you're gonna lose your job because you don't know how to do this. And it's it's this sort of like fear selling, and it's I call it productivity flex. And all people talk about is the tool. And the reason they do that is because it's easy to talk about. Like I use Claude to do this. You, you know, you should use Claude for email, you should use perplexity for research. Like they don't tell you if their research is any good, they don't tell you if their emails are any good, they're just like, you need to use this tool or you're falling behind, you know, and it's it's it's also the language of the LLMs as well, right? They've massive marketing budgets, so they're just scaring people um into submission, sadly, for the most part. And then when people jump into this and they're like, oh, I better download GBT or Claude, and um, okay, and now I've got the tool, and what am I going to even use this for? What tasks is it useful for? Oh, I don't know. Um, maybe I'll just try and write an email or I'll write a blog. That seems like a thing I should do, you know, and they start like talking to it or typing to it, and it's like you know, fumbling their way around this tool for a task they don't even know if it's the right thing for, and never really asking themselves how do I do my best work. So it invariably happens is that people just sort of feel like, oh, this isn't for me, and they sort of you know regurgitate and and and fall away. And and that's what happens why most of these transformations fail. 85% of Gen AI projects fail, 83% of uh transformations fail because people make it about tools. That what it's really about is behavior change. It's challenging yourself to really like work in a different way. And the one of the big aha's for me was actually when I was writing my book on Learn six years ago, you know, I had this sort of notion that writers like sort of sit at their desk. You know, I was always told that like writers sort of sit down and type, and they're beside a roaring fire with a purple velvet jacket on and drinking a glass of wine, and they're just tearing out pages as they're like pouring it. So obviously I sat by the fire and drank a lot of wine, but I still didn't create a lot of uh of content because I was doing um using a tool like a keyboard and a behavior that I thought that writers were meant to do, but none of that was my behavior. Uh and then when I reframed it, and I was like, actually, how do I do my best work? Actually, and and what's the task I'm actually trying to do here? Oh, right, the task is create content. And my natural trait for creating content was not typing, it was actually talking, and so I had this idea that, oh, right, well, my my the way I do my best work is talking, like debating ideas back and forth. The task is to create content, so why don't I just use a tool then that will capture me talking to create content? And that's when I first downloaded a transcription service. So I hired a journalist to interview me for On Learn, and we literally I would bullet point out each chapter, and we would just talk like you and I are for 40 minutes, and I'd end up with like 10,000 words, and then the journalist would take the transcription and they would sort of copy edit it, and then an hour later, they would send me a minimum viable product or chapter. So suddenly I went from blank page to like first version of a chapter in two hours, and I was like iterating really, really fast. And so I was using my natural traits for a task to create value, and then the tool amplified and accelerated that the best of how I did my natural work, and it literally felt like cheating, right? I was like, people were sitting there going, Oh yeah, writing sucks. And I'm like, writing? What are you writing? Like I just talk, and then I end up with an MVP of a chapter in two hours, and then I'm out there iterating and improving. And you know, I was finishing chapters in a week that used to take me a month. So that was sort of a big reframe for me about when I started to use these tools. And then I just started extrapolating it into well, what what other high-stakes tasks do I have? Oh, one-on-ones when I'm like coaching CEOs or founders of teams. Like that, that's a high-stakes task. And we're talking. So why don't we just have like a meeting co-pilot in it and turn every conversation into a data asset? And we can really focus on the conversation and and talking through these problems, knowing that in real time we're creating these data assets that can compound. Every conversation becomes a data asset. So every one-on-one with a founder becomes a data asset. And you you build up, like I basically have now seven years of conversations that I've been having with people that is my own unique database that I can leverage, that I can ask, how can I run my one-on-ones better? How can I prepare for these meetings better? You know, and then it just propagates through all of my work systems, then. So when I would have meetings with people, you know, literally, I'd be I'd be sitting there chatting to them about like a problem that we're working on, or really listening to them and like getting into a lot of detail. And then two minutes after that meeting, I could send them, all right, uh, Randy, you know, here's the key things we talked about. These are the three issues we need to resolve, two actions. I've got one, you've got two. And then this is our next step. And here's a quote that sort of summarizes our conversation. So next time we meet, these are this is the information we should have to make the next decision. And people were like freaked out initially, going, like, how did you how did you do that? Because I felt so heard in the meeting. And then as we go to meet again, I'd be sending them like, you know, literally 24 hours before we meet. All right, Randy, we need to make a decision on, you know, are we opening in uh South Korea or are we opening in um Indonesia? Here's all the information I've got to support us making that decision. I I want us to meet and try and make that decision. And here's the agenda, and here's all the things we're going to talk about. People were like literally going, like, holy crap, if I meet Barry, I better be ready because he's showing up like prepared and he wants, and I better be prepared too. So suddenly, like the people I was working with were like, like, this is great. You know, like I first of all, you've got an agenda for your meeting. That that's exciting. Most people never even have an agenda. Two, we've all the information that we think is useful to help us make this decision. So I don't meet people for an hour. I we were meeting until we made the decision and then we got back to work, you know. So all of this stuff just starts compounding where you have these what I call decision velocity as a key metric, like how soon are we able to make these decisions? Decision advantage, like what information do we have to support decision making? And then that compounds. So you have less reversals, right? Like you and I are meeting and we're solving, right? Do we go to um South Korea or do we go to Indonesia? Right, we're deciding uh Indonesia, right? Now, next question. Well, what part of Indonesia are we going for? Are we going for Bali? Are we going for Lombok? Are we so we're moving forward, like we're progressing with our decisions, right? And that creates velocity, speed, and ultimately outcome. And then your world changes. And I think that's really the encapsulation of what this book is about, is showing people these very sort of simple frameworks and systems that help me not only improve productivity and not only get performance, but the gift of all of this, I think, is executive presence, is actually showing up with a clear mind, with a calm mind, with your head not full of 20 actions from the last meeting. So you're prepared to make high-stakes decisions in the face of uncertainty, and you're not in a highly anxious state. You're actually in a calm state. And that is like the gift of all this stuff.

SPEAKER_01 12:02

Let me let me pause you there. Besides, you said so many things there that I want to follow up on. Uh, and one of the things that the key through line there that I'm hearing is that this is a tool. It is useful when it changes a behavior, when it has a purpose, when it has a measurable outcome. And you you mentioned early on in that answer, uh, the people who are just posting on LinkedIn or Instagram or whatever, talking about token maxing and things like that. And every time I see that, I think of uh David Attenborough talking about some uh animal that is doing a mating dance and maximizing its plumage kind of thing, rather than what uh is this one actually successful? Is this resulting in something? And you're you're talking about the results. But the other side of this is uh one of the statistics you you talked about when you're here in London the other week and and in the book, only 5% of organizations are getting any measurable gain from AI. And the executives are either uh embracing it or they're being left behind, and there's a gap that's widening. So, why what is the difference between someone who is successfully using this uh at a personal level and a company that's starting to embrace this technology?

SPEAKER_00 13:14

Yeah, so um there's some really good stats out there, right? Um one of one of the best ones is that as you said, even the companies that are successful at this, and it's it's in the range of 12 to 18 percent, depending on listen who you listen to, like from Gartner to McKinsey to BGC or whoever, right? But even the successful ones are seeing less than 5% of EBADA improvement, right? So they're they're only making 5% more profit in their business, right? So so it it it is it is marginal for the spend you can imagine is going into AI at the moment, right? And that's where I'm actually next week I'm going to the Gartner CFO conference in Washington, DC, to give a keynote around this to CFOs, because they're all sitting there going, what are we spending this money on? Because no one's everyone's doing AI, but no one's showing results. Like this is this is the big concern for all these folks right now. Now, to your question about what makes uh success or leads to better outcomes than others, one of the most important things is actually the leaders role modeling this for themselves. The the companies that are successful is as opposed to digital transformation, where the CEO could say, hey CTO, we need to digitally transform. And CTO would go, let's go do that. I'm gonna build a new mobile app, I'm gonna build a cloud-based architecture platform of some description, and we're gonna digitize a lot of our channels about how we connect with customers, right? So the CTO could essentially go off and tick those boxes off in isolation, really. This technology is not the CTO's responsibility. It impacts finance, it's impacting HR, legal, every department in the business. So, therefore, it's not just uh give it to IT or technology to digitally transform us, it's the whole company that has to re-evaluate how are we working? How are we making decisions? So, this is why it's really, really important that the CEO and the entire executive team are starting to role model this. Now, one of the examples in the book is with Pete Avienski, who's the CEO of Progeny. Progeny are Nasdaq listed, they're the largest provider of healthcare or fertility benefits in the US. He was the former CFO of WebMD. He grew that company over 14 years, obviously, and the massive success they had. Now, Pete said, which was a really, really important part of this, I want to use AI to elevate people, not eliminate people in the company. So instantly, this statement creates psychological safety in the company. That I I the the the contributors are not sitting there going, I'm training a tool to take my job. No, you're you're you're training a tool to do low-level tasks so you can focus on higher order tasks in your job, right? That it's a that's a fundamental, that's a directive, right, which creates safety for people to experiment. Then the other thing Pete did is he was one of the first persons that I started working with in coaching. So he's role modeling. Hey folks, this week I learned how to use capturing information using a transcription service for every meeting. So I can capture it, I can transcribe it, I can synthesize it, and then take action about what I'm going to do. So he he got into this sort of compounding, turning everything into a data asset, all his meetings with his one-on-ones and execs. And then he's talking about that experience, right? Here's what was easy, here's what sucked, here's what I struggled with. So you've got the most senior person role modeling the behavior changes about how they're changing how they work, right? And then that trickles down into the CTO, the CHR role, all these, all these different leaders, then role modeling individually what they're learning. And then when you role model individually as a leader, it inspires your teams. And then suddenly your teams are like, hey, as a HR department, what are what are some of the processes that we could maybe use these tools for? Like, could we create HR bots to answer sort of rudimentary questions about when the next public holiday is or how much annual days leave people get? Like, you know, like all these companies then start going, wow, there's ways that we can create and better customer experiences, I guess their customers, employees, by using these tools to automate like low-level administrative tasks that eat up loads of our time so we can focus on the more complex, like problem solving. So this is sort of the way you know, success that I've certainly seen in companies is about creating the safe space, like telling people we're we're going to elevate you, not eliminate you. Role model this in yourself as a leader, like here's what I'm trying and figuring out how I'm using this tool. And then that inspires your teams, and then that sets the wildfire going across your organization. And again, there's a bunch of case studies from uh Andrew Phillips, the CTO of Sky Scanner, doing the same. And then even a community I set up with Stephen Franchetti, the CIO of Slack, where we were bringing like you know, 50 to 100 execs together every two weeks to share lessons learned with one another. Um, and you know, that is a really powerful mechanism.

SPEAKER_01 18:39

So, one of the things I really liked is you're focusing on this almost on leading metrics in terms of what is success look like for an organization. And it's about things like decision velocity, decision frequency, things like that. Did talk a little bit more about what those those metrics are. What are the things you're trying to optimize for in the short term that will then lead to compounding business success?

SPEAKER_00 19:02

Yeah, so what one of the things people get quite nervous about is the metrics, right? It's like how how do I communicate I'm making progress? Because I'm trying really hard. And you know, at the moment, you know, let's just think about the classic metrics people use, like time, you know, are you saving time? All right, like did the amount of uh meetings that you held this week, were they faster? Were they more productive? Did you move ahead with them? Did you get more stuff done? Right. Then people talk about money instantly, they want to equate your time to money, right? Like we're we're spending $30 a person on a license, we're spending X amount on tokens. You know, how much time and therefore money did you save this week uh to help the company be more successful, right? All of these things start that, but they're all lagging indicators to what you're describing, right? Leading indicators are always changes in behavior. It's always changes in behavior, right? It's a signal that people are doing things differently and it's going to lead to a better outcome later. So when I started like doing this with all with all the execs, like very simple things, I would ask them when they're having meetings with their team, is like, before you kick off this meeting, just do a simple thing. On a scale of one to 10, how prepared does everybody feel to make a decision in this meeting? Like straight away, you're getting like you're getting measures, right? People are like, oh, do we have on a scale of one to 10, do we have all the information we need in front of us to make this decision in the next 35 minutes? Like, like these are these are all your leading indicators, right? Where people are still going, uh actually, I feel like we're missing something. Or actually, people go, no, I I think we're good. Let's let's dive in. At the end of the meetings, you know, do you like working with me? What would you know, what would make like, you know, all of this, these things, right? Like they're they're simple ways that you can get qualitative feedback, but very quickly, there that's the sort of leading indicator that you're you're moving in the right direction, right? That people are enjoying, you know, and another one I love to ask people, what percentage of your time is spent on really the sort of creative problem solving, like high priority, really tough problems versus what percentage is spent on the necessary but administrative part of your job? Right? I'm sure I asked you that, you know, what what are you what are you what's your split, Randy, in your head right now? What how what percentage is creative problem solving versus admin?

SPEAKER_01 21:33

It yeah, it's it's not the optimal, it's probably more admin than like I'm trying to to increase efficiency on it, but there's only I've only been so successful so far.

SPEAKER_00 21:42

Well well, that's it, right? Like most people sort of go 80 20. They're like 20% is that like, you know, those really nutty problems that I can dive into, and 80% of my time is all this admin, especially if you're a manager, right? And again, the promise of AI for me is not that the admin just like Magically disappears. What the promise is that you go from 20% to maybe 40% of your time on creative problem solving. That's still like a hundred percent increase in creative problem solving time.

SPEAKER_01 22:13

One of the challenges around that though is you know, you're spending a lot more time on the hard things. Uh, and that's great. It's very efficient, but it's also exhausting. And it means that you're not necessarily spending as much time prepping for it. You're more efficient in your prepping, but you're expected to you're be we're being expected to make more and better decisions faster. So the only things that come up to me, you know, it's like being the CEO or the president of a very large corporation. Anything that can be decided by someone else is being decided by someone else. So the only things that come to me are really, really difficult. So now I've got pressure and I've got context whiplash because I'm just being whipped back and forth between hard things. And I may have better prep in front of me, but even still, how do you deal with that?

SPEAKER_00 23:02

All right, now we're getting into it around the eleven. Right. Okay. So this is sort of again another one of the inertias I think that is it happens, right? Is that we we are such like uh beat productivity is like hammered into us, right? It's pressurized into us. And especially if you're working in the tech sector, the Silicon Valley sort of flex of I'm measured on how much impact I've had every year. That's how I get my bonus. I have to demonstrate the impact I've had. So everyone's like, you know, competing to be like, oh, look how much work I've done, right? So the the whole dynamic, even of if if I do work that will automate some of my administrative work, so my capacity for problem solving goes up. The instant sort of re like sort of reflex is like, oh, that means I'm gonna get more work to do. That that capacity is gonna be filled. It's not gonna be my admin goes down. It's like, as you say, more hard work is gonna appear. But the the way I would actually frame it to people is this what you're actually creating capacity and space for is actually time. Time that can be spent doing a really, really important part of solving hard problems. It's called thinking, it's called actually having space in your day to sort of marinate in these problems, right? Because you don't want to be making hard decisions just like bang, bang, bang, bang, bang. The whole point is that leadership is that you there are questions that you need to consider for a period of time. Like that you need you need time to think about them. Yeah, you need to marinate. You need to marinate, right? And this is the thing where we've we've sort of almost forgotten that thinking is one of the most important jobs of good judgment and judgment of leadership. So I I've almost got to this point now where when I design my week, I literally am like, I need thinking time. So, and that thinking time might be me saying, I'm gonna go for a walk for an hour around around the uh you know the area I live in. I'm gonna make sure I go out for lunch and actually go for 90 minutes. And you know, maybe I'll go for lunch or randy for half an hour or that, and maybe I'll just I'll just sort of go and wander around and think about this for a problem. I'm gonna walk around Central Park and just just marinate in this problem for a while and and change the way and action that my brain is operating from constantly, I think Daniel Caterman calls it like system one, these fast response sort of um thinking to actually doing slow thinking. This system two model he describes, where you you again, you as you said, you marinate in the problem and let your brain just sort of let those ideas come to you. Because you don't have your best ideas in the one-hour innovation workshop. It's often you bump into a customer problem, you you know, you you see something, you have a eureka moment, but I know you know this 95% of the time that is not in the innovation workshop. It's when you're doing something else, right? So I think we have to be even more aware about the the goal, is not just to be more productivity, more activity, more execution, more, more, more, more, more. It's actually um, and I think Bezos is is famous for this. I think he calls it puttering. Like he he would putter in the morning between sort of, you know, waking up until eight or nine o'clock, where he just sort of marinates in his day, right? Doesn't have devices. Like that was one of his sort of management hacks, you know. And I think we need to remind ourselves of that a bit too, as well.

SPEAKER_01 26:54

But you're you're going off to talk to these CFOs, as you said, next week. And the response that we're seeing in in the in industry at the moment is we should be able to do more with less. We're laying people off and we're buying more token, increasing our token budget. And I'm seeing two trends that are going on in general. Either the size of teams is shrinking, so it's not a two-pizza team. Sometimes it's a sandwich and a pack of crisps team, and it's one person and some agents. It's a meal deal. It's a meal deal. It's a meal deal team, exactly. Um, and I hate that. I mean, I will I've interviewed people and talked to people who work that way, and them with a bunch of agents, you can be efficient, you can run experiments quickly, it's brilliant, but you're missing out on the the friction that comes from talking to other people. Um, and you need that as well. Or the other trend I'm seeing is team size isn't necessarily shrinking, but their control surface gets a lot bigger. They're being asked to, I still have six or eight people on my team, but we're taking on two or three more things, we're doing a lot more than we did before. So, is it is this the right way? What are you going to tell the CFOs? And what are you hearing from from that part of you know, the COO, CFO side of the executive suite?

SPEAKER_00 28:08

Yeah, no, it's a great question. And so here's what I've learned in the studio, right? When we when we first started the studio six years ago, even the way we modeled our startup teams was generally in the region of sort of six to eight people, right? And in in the classic sort of startup style, where you would have some engineering capacity, you need you need a CEO of some description, someone has to do sales, someone has to do marketing, someone has to do operations, someone has to do customer support, and then maybe three engineers, right? That that was like our model, right? And then about halfway through the whole experience, these tools started to show a lot of promise, but not promise that they were like production ready. So what we were learning is that we could sort of shrink maybe the teams a little bit, but they were still only building prototypes, they weren't building like fully functional, you know, uh regulator compliant applications that I would stick into the middle of a bank and go, we're good. Right. So you know what we could see is that you could get speed, you could get output quickly, but output did not guarantee quality. Same as what you're describing with the thinking, right? You can have one or two people build a lot of stuff as entrepreneurs, but like maybe they're really great at engineering and marketing, but they suck at customer service, right? The the magic of a team is still just as important as ever. So, what we've constantly found is that you need all those people who are especially you know great at selling, great at marketing, because distribution has never mattered more because everybody can now build relatively cheaply, right?

SPEAKER_01 29:55

So distribution is like the ultimate music industry years ago, where it used to be through physical distribution, if you owned NCAPs and record stores and now it's infinite distribution, but it's marketing and attention is the the the differentiator, yeah.

SPEAKER_00 30:09

Right, you know, so like the these were the things we were learning is that like we can create loads of stuff, but if it doesn't go into the right people's hands or you don't have channels to get to them, it doesn't matter what you're building, you could have the best product in the world, you know. So there a lot of this start we started learning that that we sure we could have smaller teams, but we still needed excellence in certain areas to be successful. You still you don't like skip all the steps. And again, this is sort of where people go to the extremes where you know you're looking at your Instagram, LinkedIn feed, and the news people are just looking to say, oh, it's a one-person unicorn, or look at this business. It's like he started it in his garage after he tripped over a stone, and now it's worth two billion dollars, and he's done it all with agents. You know, it's like they just want this narrative that is not true, and maybe there is an exception, maybe there's one or two people that do that, but that's one or two people in the in the myriad of startups that are happening on a on a planet with eight billion people on it, right? So that is not the norm, and it's just so I think it's helping people reset a little bit of the expectation because we're peak hype at the moment, and oh you know, you can see the the skepticism now starting to appear because again, CFOs to your question, they're the ones that at the end of the day, the brass tax shows up at on the balance sheet where they're either like, hey, we just got everybody a um uh an LLM license, $30 a month for you know a staff of a thousand people, and then my token bill doesn't seem to be doing much. So are people even using this tool? Or, you know, there's people using the tool wildly and they're getting massive token bills and they don't understand why. Or to your other point, some companies are jumping, and again, it's very hard to sort of be generalist on this, but a lot of companies I think are letting go of people without the capability proved in place that they can deliver on what they think the promise of reducing their headcount is. And to your point, then that plays into the both sides of the surface area question you you you put as well, which is you either have the people who stay being expected to do more and leading to burnout, or this productivity sort of expectation where there's still the same staff and you're expecting them to produce 50% more. Like it's just not realistic, right? And we're still only at the beginning of this. So, yes, um, there's efficiencies. I think the best research we've seen in engineering has been somewhere between five to twenty percent uptick in engineering capability, but that's you know, it it's it's great, it's not earth shattering, it's not like fire half of the engineering department because we're sort of 10% better, but all the other functions are still learning how to use this stuff.

SPEAKER_01 33:13

Let's get into the learning how to use it because I think there's uh the the first step that you talk about is capture your work as data. And I think that that hopefully for most people here, I think we've covered that. It's about the conversations, it's about the documents, it's it's lots of things, and there's lots in the book about doing that. But I think one of the really critical things, and it it speaks to having multiple people and multiple viewpoints on the team, is what do you do with the data? How do you synthesize it and make it useful rather than just having you know seven years of huge amounts of transcripts that you know the some of it is a lot of fluff, a lot of good things, a lot that's open to interpretation that and you know, tone and sarcasm and and things like that. So how do you how do you make this useful?

SPEAKER_00 34:01

Yeah, so I think one of the great case studies in the book is is uh Misty Safer, and she's the VP of customer technology at American Airlines, the world's largest airline, right? So she's responsible for AA.com, the mobile app, like anything that that touches customers, like that's she's responsible for delivering that. Now, Misty is like uh just amazing anyway. She again was one of the first people when I said I was going to do this coaching thing. She was like the first person that emailed me. And then she she's so she is a talker, right? Like she externalizes her thinking, she loves Socratic debate, she loves sort of getting ideas out there and capturing them in the moment, right? She's instinctual with this stuff. But for someone like that, for for years, like she'd have an idea and then you know, she's a she's super busy, and then she'd go to the next meeting, she'd be like, What was that idea again? It's almost like it had disappeared, right? So one of the simple things that we just started doing is that anytime she had an idea, she would capture it, literally like just talk it into a voice note, right? And that gets transcribed. And then the end of her day, her synthesis would be taking all of these like half-baked ideas, little fleeting moments of a hunch, and putting them into an LLM. And then the real magic here is using it as a teammate. So starting to ask the LLM, what's a what's a blind spot with this idea? Give me three different scenarios. Um, if the price of oil is $200 a barrel, and we're trying to open five new routes between North America and Asia, and um the you know, the market economics of this are so she can literally start playing around with these scenarios, right? Pressure testing them, looking at her decisions and saying, and and again, not looking for answers. She's asking harder questions to make her thinking more resilient, right? Like that's where this tool, uh, when you do synthesis, becomes the ultimate teammate. Because it's like that person, if you use it correctly, that you don't ask it for an answer, ask it to challenge your thinking. What's the blind spot here? What's what's a different perspective I could take on this problem? Um, what would the CEO, who's really detail-oriented and analytical, say about this strategy that I'm about to present to them? Like this this is the power of synthesis, you know, and then the last step, which is the of the uh capture, transcribe, synthesize, and act loop, is ultimately it's the decision. It's like you as a human, machines are great at capture, transcribing, and synthesizing stuff, but you need to hold on to the action, the judgment step, and make the choice after you've looked at 15 different scenarios. Which one do you think bears potentially the most fruit? Or you've you've actually got rid of that hunch. You know what? I thought it was a great idea to open a taco stand in Mexico City, but it turns out there's 7,000 taco stands there. So it's probably not a good idea, Barry. You know, is there room for one more? Are you telling me, Randy?

SPEAKER_01 37:11

I I could always have another taco stand, but um so I love the the the way this optimizes each person individually, but there's and I don't know Misty, so I'm I don't want to cast any aspersions on her, but when I was reading about this in the book and hearing you talk about it, my mind goes to the one extreme of this behavior, which is this executive being a limiting factor, that every the ideas come through them, and it's the ultimate version of the executive swoop and poop of I've got an idea, throw this on the roadmap, and now it's a well-developed idea. So you absolutely have to do it. So, how do we how do we change the culture of an organization and say, right, we're still prioritizing effectively, we're still working together, we're still giving people time to develop ideas. We want the job at the top level is not necessarily to have every idea, but to shape the strategy and help prune and develop and and refine ideas from everyone else, and then give other people the chance to do it. So, you know, most of our audience are not the ones who are meeting are not in the C-suite chair. They're people who are actively trying to make this stuff happen on a day-to-day basis and getting changes in strategy every two days, no matter how good they are, is not going to help them.

SPEAKER_00 38:28

No, it's not, right? And and uh, but that's also an unfair behavior, I think, as you're describing, right? Like any any leader to do that onto their team, you know, it it it's it's uh I think that's massively unfair because they're sort of like, don't worry, folks, I've done all the thinking, and and here here is here is my beautiful thing. Good there.

SPEAKER_01 38:50

Your LLM even no matter how much it's challenging you, is also telling you how brilliant it is when and you are coming up with something that you've now challenged and feel good about. Yes, that's gonna be hard to deal with.

SPEAKER_00 39:00

It is right. But what here here's one bit of research that I put in the book specifically for this point. And it it's it it's we re-engineered how we do ideation in the studio on the back of it. So there's a really great report uh study that Harvard did with Proctor and Gamble. They looked at 776 uh different projects within Proctor and Gamble about how people were using AI to improve their ideation and decision making. So when people ideate on their own, they've got sort of a benchmark they can get to, right? Um it turns out like if teams, teams obviously outperform individuals, if if a human pairs up with an AI, they can almost get parity to what a team is capable of doing. But here's here's the magic one. If teams pair up with AI, they're three times more productive, they're three times better outcomes in the quality of the ideation that they do. So what's really powerful about that is suddenly you start thinking of strategy development as a team sport, right? And not if if um it's not, let's say if I go off and develop my strategy with my AI, my instant reaction, if I was an IC, is I'd be like, that's awesome. Why don't we get the team together and let you why don't you take us through this idea and then let's get the AI in the room and let's let's all of us come up with this idea and start challenging it. Start asking disconfirming questions about what could make this strategy fail? What are some of the blind spots? What are four scenarios that you've thought about and not not answer? So suddenly it goes from the sort of, you know, the here is the perfect idea, Randy. It is presented to you on a platter, and you you turn it this into a team sport where one, it's not just an exec handing it down, it's exec team. But now let's all play with the AI because we know we're going to get 3x times the quality of ideas from it. And that for again is a is an invitation for everybody to sort of put their ideas out there and try and improve them. And I think again, it's one of the reports in the book. Um, again, I'll share with you to put in the show notes too as well, that it just we we totally changed how we did ideation in the studio on the back of that.

SPEAKER_01 41:12

That's my favorite approach, actually. It's because it's it goes back to the basics of everything we're doing now is no different than what we've done before in terms of philosophy. It's still build, measure, learn, it's still iterate and and everything else. But and this is a design sprint. There is no difference between this and and you know, that weekly did that one week design sprint that we talked about, uh have used for years. But it is you can do it faster, you can do a lot more with it. Your prototype can have a lot more fidelity, you can learn a lot more, you can do more iterations, but you're still doing the same basic approach, you're just accelerating the hell out of it.

SPEAKER_00 41:48

Uh right. And as you said, with access to better information and research potentially in the moment, so you can keep that sort of ideation momentum going, right? Because you've been there. Like, how many times are we like, should we open in South Korea or or Indonesia? And someone's like, I don't have the stats. Well, let's go and get the stats and I'll meet next week. And the energy just dies from that ideation moment, right? Right. Where again, if you can keep keep it moving, keep the wheel, keep the iteration wheel turning, keep challenging it with like what about that information? What about this blind spot? You know, like that is where great stuff happens, and it's and people love those environments to work in. They're fun, they're energizing, and it's creative problem solving.

SPEAKER_01 42:31

Barry, we could do this all day, and it's fascinating. I would love to, but I I know we're at time. Yeah, I've got just a couple last questions. I want one of the big ones is you know, when I use an LLM, when I use these these tools, and it and we're talking about something I know about, I can point out the bullshit. I know when it's hallucinating, I know when it's going wrong, and it's really uh powerful to go back and forth and pick at things and refine together. Um but when I don't know about it, it's very good at sounding plausible and I don't know when to when to pick at it. How do you deal with that? How do you deal with things that you know uh you're dealing with an overconfident intern at times?

SPEAKER_00 43:12

Yeah, and um I think you know you just have to remember that there's a product manager who's paid to design a system that makes you want to use it. So that's that's why, you know, one of your colleagues out there, Amanda Productors, is sitting there thinking, how am I going to make people keep using this machine? Right, because the way I make money is they keep using the machine or they keep burning the tokens, right? So, yes, it's designed to agree with you. Yes, it's designed to sort of surprise you with like, wow, this information is amazing, isn't it? And it is amazing. But I think that's the subtle distinction for me is this, and it's a it's maybe an oversimplification, but when you're looking to it for answers, it's going to tell you the answer you. Want to hear. If you ask it to challenge your thinking, to pressure test your thinking, judgment is like this. There's a real reason, Randy, I made sure judgment was in the title, because that is the last human sort of skill that we need to hold on to. And if you are not exercising your judgment, you're eroding it. You're you're losing. And then that and that's what happens is we suddenly stop making decisions and we forget how to make decisions. And that is something we can never give up. Right. So I would, I would almost say it's like going to the gym. You got to keep working those muscles so you don't get osteoporosis when you're older or whatever it is. You cannot uh give up uh judgment. You got to practice it and and and live it. Otherwise, again, you're you're sort of giving up your agency. And I feel very passionate about that.

SPEAKER_01 44:55

That leads to one fantastic last question, which is uh you're putting together working with all these teams, you're putting together new companies. And if judgment is the secret skill, the one thing that can't be replaced right now, how do you hire for that? What is the thing you're looking for in people when to test pressure test if they've really I hated when we called it product sense because I don't think it's that, but I think judgment is a really good word.

SPEAKER_00 45:20

Yeah, um, so I did I did a podcast the other day with uh Jim Highsmith, who was like, you know, he's been doing adaptive leadership for like 40 years. He was one of the first people to sign the Agile Manifesto. And Jim reminded me of the way he describes a capability. He says it's knowledge plus experience uh plus plus judgment, right? So knowledge is again the way you exercise your knowledge is you read, you become informed. Experience is actually making choices in sort of high-stakes scenarios where there's consequences at stake. Right. So, like he was describing it at mountaineering when he goes out in a mountain, you know, he has to set guardrails or but by pushing to the edge of his sort of his sort of experience, that's how he keeps growing it. Right. And then judgment again is this ability to make choices with uncertain information. You you have to learn the pattern recognition, the hunches that sort of again are uniquely human. So that's that's the kind of way where he shared that with me. I was like, that's exactly how I'm gonna like think about more about what what shows if someone's capable or not. Like, how how do they how do they create or gain knowledge? What experiences do they create for themselves to keep growing? And judgment, like what are their little patterns, their internal LLMs that they're using to make decisions under on certain conditions? And that's the way I'm sort of starting to think about it uh following that great conversation with him. And you know, hopefully that will be helpful for others.

SPEAKER_01 46:56

I like that. So yeah, you can probe for knowledge, you can probe for experience in interviews, but if you're doing tasks and things like that, uh purposefully designing them to uh not necessarily to get the answer, but to show how they use their judgment, what are they trying to do, what's their approach. That's that that sounds really strong. I like that. I'm stealing that.

SPEAKER_00 47:16

Yeah, wife stole it from Jim. So hey, let's pass it around.

SPEAKER_01 47:19

Perfect, yeah. Great artists steal, right? So this is everything we do is built on other stuff. Barry, this has been fantastic. Thank you so much for your time. Thank you for the new book. It's a really good read. Uh, there's lots of practical things in here about specifically how to get started. We didn't get into your 51530 and some of the other things that you you'll just have to read the book or or follow Barry on other stuff to get some of these practical things. But it this has been wonderful.

SPEAKER_00 47:44

And and thanks, Randy, for like one of the joyous things about doing these shows is when people read the book and allow us to go deeper. So I'm grateful for you investing your time to do that to make this more of a fun show for both of us. So thank you very much. Anytime. Thanks. Come again soon, Barry. We'll talk to you soon.

SPEAKER_01 48:02

The product experience hosts are me, Lily Smith, host by night, and chief product officer by day. And me, Randy Silver, also host by night. And I spend my days working with product and leadership teams, helping their teams to do amazing work.

SPEAKER_00 48:18

Lou Ron Pratt is our producer, and Luke Smith is our editor.