The Product Experience: a Mind the Product podcast
The Product Experience features conversations with the product people of the world, focusing on real insights of how to improve your product practice. Part of the Mind the Product network, hosts Lily Smith (ProductTank organiser and Product Consultant) & Randy Silver (Head of Product and product management trainer) “go deep” with the best speakers from ProductTank meetups all over the globe, Mind the Product conferences, and the wider product community.
The Product Experience: a Mind the Product podcast
What I learned from building, and exiting a startup — Kirsten Mann (Strategic Advisor)
Use Left/Right to seek, Home/End to jump to start or end. Hold shift to jump forward or backward.
Kirsten Mann is a board director, product leader and, with this episode, a three-time guest on the podcast. Last time she was here, she was building Vizory, an AI tool that helps company directors get through board packs. This time she's back to explain why she isn't building it any more. Talking to Randy Silver, she walks through exiting Vizory, the friction tests she ran before writing a line of code, and why she now believes distribution, not the build, is where most AI products actually die.
We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next!
Key takeaways
- Before writing a line of code, Mann tested for force rather than sentiment: she asked directors for two confidential board packs and an audio recording of themselves reviewing one, and only started building once 10 people agreed to do it. Coffee-shop enthusiasm is cheap; handing over a company's most sensitive document is not.
- Vizory's most-loved feature in demos, cross-pack search, was barely used once shipped. Directors said they wanted it and reacted strongly when they saw it, but in practice they went straight to an automated triage view instead — a clean case of stated preference diverging from revealed preference.
- Board software runs on quarterly, not weekly, cycles, and trust compounds slower than value does. Mann modelled two review cycles before directors would rely on the tool; in practice it took three, meaning some customers on quarterly boards needed close to a year before the value became obvious, a timeline no 30-day trial can accommodate.
- Pricing got caught in a double anchoring trap. Directors first benchmarked Vizory against the $20-a-month ChatGPT and Claude price point, so Mann positioned against the cost of a governance failure instead, then created a second anchor by pricing early pilots at $150 a seat against a $300 target, which made the later increase read as a 100% price rise.
- Cutting friction is the wrong instinct early on. Mann deliberately added friction, asking for sensitive documents and charging as early as possible, to separate real demand from polite interest, arguing that most founders run tests designed to confirm they're right rather than to find a reason to stop.
- Mann's main lesson for other founders is that distribution is now the product problem. AI has collapsed the cost of building software but done nothing to the cost of reaching people who trust you, which is why she pushed to add distribution, adoption and willingness to pay as a third discipline in the Makers' Manifesto, alongside building the right thing and building it right.
- Multi-model products carry a new operational risk. One of Vizory's AI "judges" silently stopped being selected because the underlying model had been deprecated overnight without warning, a reminder that model drift means the QA and evals work never really stops, however fast the initial build was.
We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next!
Our Hosts
Lily Smith enjoys working as a consultant product manager with early-stage and growing startups and as a mentor to other product managers. She’s currently Chief Product Officer at BBC Maestro, and has spent 13 years in the tech industry working with startups in the SaaS and mobile space.
Randy Silver is a Leadership & Product Coach and Consultant. He gets teams unstuck, helping you to supercharge your results. Randy's held interim CPO and Leadership roles at scale-ups and SMEs, advised start-ups, and been Head of Product at HSBC and Sainsbury’s. He participated in Silicon Valley Product Group’s Coaching the Coaches forum, and speaks frequently at conferences and events. You can join one of communities he runs for CPOs (CPO Circles), Product Managers (Product In the {A}ether) and Product Coaches. He’s the author of What Do We Do Now? A Product Manager’s Guide to Strategy in the Time of COVID-19. A recovering music journalist and editor, Randy also launched Amazon’s music stores in the US & UK.
I think what people get caught up with building, and I'm seeing this time and time again, is distribution is the product problem right now. People get so caught up in the build, and is the product desirable, usable, all of that stuff. Not can we get it to people?
SPEAKER_00Kirsten, this is your third time back on the podcast. We asked you to come back on. In fact, you you told us you had to come back on, and we agreed. Because last time we talked to you, you were building up your new business, uh Visori, and now you are no longer building that business, and you wanted to give us a bit of a retro and a look back at what happened. So what happened?
SPEAKER_02Well, I think it's it's one of those things, right? That you and it's always when you do the retrospective, I think that's where the powerful lessons and learning's come through. So I've actually documented a lot of these and they're and I've put them on my site, um, undervisory.co. So if anybody's listening to this and they're thinking, I want to read more about this, it's all there. But really, as you know, Randy, I think I never really set out to be a founder. Like I've worked in um for about over 30 years in product and worked alongside hugely successful founders. And I knew it isn't a role that you want to walk lightly into because you need this problem that you care enough about to live with 24-7 because you never end up switching off. And so I went in with my eyes wide open and I knew the challenges with things like pricing and distribution. And I wanted to treat visory like any other bet with stopping conditions built in it from day one. So my product it sticks transferred fine. What I think I underestimated wasn't the technical side, it was the human side. And we had AI agents doing a lot of the building for just context. It was a board software app that we were building, which used AI to analyze board packs. So I should give that context. And it was just the two of us, um, Toby, my co-founder, and myself. And because AI lets you move so fast, we found the pace was just relentless. And so it's brilliant for getting to market really quickly. It's also hard because suddenly you're doing all the go-to-market work at the same time as building. There's really nowhere to hide and pour. So suddenly I found myself being CEO, the PN, the UXer, the researcher, the marketer, customer success all at once. And what's different now is that I think I understand the tech from the inside, not just orchestrating it as a product leader. And I'm probably more sharper on the commercial questions as well, but we can go into some of those things. And I think that that's really helping in my visory work now.
SPEAKER_00Well, let's continue to set the context for this. So you started visory, the you and Toby, you built it up, you went to market. Let's just do the the the two-minute version. So, what happened? You're you're no longer involved in visory day-to-day. Did you shut down? Were you acquired? Did you IPO? What happened?
SPEAKER_02We've exited and realized pretty early that the channel, we couldn't own the channel, and that was a key part of what we were testing right from the outset. And I think it's one of those things that you go through, and you know, I was a board director, obviously, and I'm building a product for board directors. And so you suddenly you're the buyer, but that is a that is dangerous as well. And I want to talk a bit about why that is and what the advantages were, what the problem was that I was living there. But the visory, the product, we'll live on under another name. And um, so now I'm doing advising work under my visory brand.
SPEAKER_00So this leads to an interesting thing because you're building up a product uh when you were starting this, and then you sell it into some a bigger package. Looking back on it, was Visory a product or was it a feature of something else? I think this is uh gonna happen increasingly a lot for people, but something that we would have thought as a standalone product. Now the defensible mode around copying or building things is is you know so much lower or so much harder to defend. Um do you think that this was a feature or a product, or and is that gonna continue to be a problem for people?
SPEAKER_02Look, I think it's a really good point. It's for visory, it is a product still, and it's but it's in an ecosystem and of jobs to be done around boards. And um, so if part of that ecosystem, for example, is another AI product that creates the board pack, and then visory is the product that analyzes the board pack, but there's other ones there that do the effectiveness of the board itself. So you could say, are they features? I I don't think it's they're contained offerings in their own right that are giving value. I mean, we we define product as something that supports people in a process and and provides value. And so I think visory did that in its own right. But you're entirely right. I see some things and I think that's kind of just a feature that would need to be with something else for it to be totally valuable. For us, the visory concept was valuable on its own right. We just went down the path we did because we wouldn't have been able to get access to the user easily that needed that product. So we can kind of talk through what led to that decision as well.
SPEAKER_00I totally agree with you about that, but I think one of the key things of making a product standalone and work is it's it's feasible, it's usable, it's desirable, but it's also viable. And it sounds like the viability was the issue for you.
SPEAKER_02Yeah, it's it was viable in the sense of um it was people were paying for it. So that was all and the commercials were fine to at a certain point. But what started happening was I wanted this to be a B2B sale. So direct, but basically the people using it were the directors, and they would be the people buying it directly. And that started off fine, and that was the way that it was working, but pretty quickly it flipped to an enterprise sale. And that was because it was a different type of success. Like other people started talking about it on the boards and say, hey, can we get access to that as well? And then the minute you're dealing with a chair, a COSEC, who say, Oh, yeah, we'd like to buy this, but we want a deal across five people. And oh, by the way, you need to go through our IT person or our AI transformation person or whoever it was, oh, our legal person to suddenly you're in an enterprise sale. So it wasn't a direct to director slash B2B B2C sale anymore. And for me, that was something right from the outset. I was testing to see whether this could be. When I could see it wasn't going to be that and it needed an enterprise backing behind it, that wasn't what I wanted to be doing.
SPEAKER_00Okay, so it's not that you couldn't, it's just that it wasn't the vision that you had for where you wanted to go, it wasn't the company you wanted to be running.
SPEAKER_02Exactly. And also building that up and building the channel behind it takes a lot of time. And and I've lived through that in a number of enterprise companies now. So for me, it was like where AI is moving, the speed, all the rest of it. You have a choice, you can build it all yourself and create, try and create the channel and then, you know, raise money, do all the marketing, all the rest of it, or you can find an existing channel, and that's what I did.
SPEAKER_00Okay, let's talk about one of the other challenges that or lessons learned from this, which is something I'm seeing increasingly. It's uh something uh I see with uh Faith Forrester, who I'm doing a separate uh program with and doing some, and a lot of people that I know who have said, ooh, this is really interesting. I have tools now that allow me to build fast or prototype quickly at the very least, and maybe I can take it into production. And I have a problem that's been bugging me. Maybe I can solve this and turn it into something that's that's a uh a sellable product, I can turn it into a business. But you know, we used to fall in love with a problem, as you said, that you get obsessed with, but it was somebody else's problem that we wanted to solve for them. Now we're trying to solve our problems. What's what was that like for you? You're a board member, a board advisor, and you were trying to build you were building a business that solves your problem.
SPEAKER_02Yeah, I I think being your own customer is the best focus group you'll ever have and the most dangerous. And so the advantage is real because in my case, I'd lived the problem. You know, I've had the situation where uh it's a Friday evening, I've got a pack, a board pack that I've got to read, and that could be a hundred or five hundred pages. If you're on a listed board, by the way, that could be a thousand or two thousand pages, and you've got the weekend to get across it, and you read along and you're kind of feeling like you've seen 80% of it before, and you're knowing there's something in there that should probably be keeping us up as directors at night. And I didn't need to research that to feel that right. So that's the exact trap because demand isn't me wanting the thing I'm building, real demand is someone trying to make progress and blocked by the options that they've got. So my own conviction was the least reliable signal in the room, and I kept having to go back to the jobs to be done, not the features. And so I could see this is the mistake that people making. They're getting so caught up in we can do this, we can do this with AI. Look what I've done in you know, half an hour, it's amazing, you know. But it this was so critical from my point of view, because the example that comes to mind um for with what we were doing was the synthesis produced by AI was quite generic, for example, initially. And when we were getting feedback from directors and things, they were looking at that, and they said, one director said to me, you know, don't build a tool that flattens everyone into the same summary and the same questions. What I bring to this board is unique. So make the insight reflect that and how I think. And that became a design principle for us. But it's one of those things where it was an example for me where I treated myself as one data point and kept testing against people who weren't me. And I think it's so critical for the people building with AI to do that. And I think we got to have a bit of a chat about, you know, we used to speak about the mum test. Um, it's gone from the mum test to the me test now, I think. So that's something that's really fascinating to me as well.
SPEAKER_00I mean, I think the me test was always something that was useful. This is, you know, we wanted, didn't want to show something to customers or potential customers until we thought it was potentially viable at least. So we always had a bit of a me test. I think the problem that you're referring to is when the me test is the only test that you're doing.
SPEAKER_02Well, it's it's kind of, I mean, you would have encountered this ready. Like, how many people have you spoken to where they're suddenly saying, Oh, I vibe-coded this thing over the weekend, right? And then building it for an audience of one, which is themselves, and mistaking it in some ways for validation that this solves my problem. And so building fast and cheap for yourself, and as you said earlier, you've proven maybe feasibility and usability for exactly one person, being you, but you've done nothing to validate viability or desirability for anyone else, which is perfect right up until you decide to have a business. And then that's the time where you have to assume you're not the TAN and possibly not even the ICP, and your real ICP is whoever rips this thing out of your hands and closes this deal fast and stays happy afterwards. So assumptions is our uncomfortable truth sometimes, which is exactly why people skip these kinds of this thinking and get caught up in the vibe coding and testing because it's so immediate and you can see that. But I would say to anybody vibe coding something brilliant this weekend, whether it's any good is never really the question. The question is whether anyone who isn't you will pull this thing out of your hands to get to it.
SPEAKER_00You've already mentioned a couple of people that you've talked to that gave you uh really strong feedback along the way. But getting access to board directors uh is you know your target market in this case, it's not always the easiest thing. These are not people who are incredibly rich on time in many cases or very always very generous with time. It's a lot easier if you're in the population, I'm assuming, than just trying to get access to that population. But I know lots of people who have trouble getting access to their target customers on a regular basis. And it's not as bad as it was a few months ago, but I used to get solicitations every week for doing uh discovery with synthetic users. So what do you think the right balance is there between so you've got the me test, you've got the actual test of talking to real people, and then you've got something in between of we've coded up a bunch of agents to pretend to be these people. What what what was your experience or what's your opinion on that approach?
SPEAKER_02Yeah, it's it's it's a really interesting one, right? I I suppose part of it, I think those kind of things are good first validation steps sometimes, depending on what they've been trained on as well. So if these agents have been trained, like you've done a lot of discovery interviews and things on your potential users, and you've got those interviews, and you've then used them to train those agents so they've got a base level, then it's a nice quick early validation part. But of course, nothing can beat a real person who's talking about the reality of what they're dealing with. And and also I think there's another aspect to this, Randy, that a lot of people overlook. It's because it's a lot of when you're doing those things with agents and things, it's about usability testing and it's not really getting into the heavy viability stuff. And for me, a lot of people when they're talking about taking a product to market or at least testing it, they think of stripping out friction to prove demand, getting users and people using it as fast as possible. And I think that's actually backwards, at least when you're testing an idea, because pain is not the same as demand. You know, life is full of on your you're on fire moments, and we do absolutely nothing about it, right? So the only real test is what have you actually done about it? And if the when you're talking to somebody, the honest answer is nothing, there's no force there. So that was something I was testing to fail, not to pass. Now, most people, especially with these agent environments, are running tests to confirm they're right and not to find a reason to stop. So that is so critical. And an agent's not going to think through these things that we're talking about now with demand, because you're not pushing them down those kind of scenarios and they haven't necessarily got the context to think in that way as well. So this pull is really critical. And I'd love to kind of talk through a bit of an example of how I tested that because I think this is one of those really interesting periods.
SPEAKER_00Yeah, I would love to hear more about this because you hear uh this kind of thing all the time. We go back to the scientific method, and people think it's proving that you're correct. And really, what you're trying to do all the time is prove that you're not correct. Um the stuff.
SPEAKER_02So, and so this was for me, this was right in the stage before I actually decided to build anything because I actually had the product in a form that I developed while I was at prospection, which was at AI First Company, and the guys had helped me set some things up there so I could run an LIM across my own boardpacks and things. Now, and people had heard about that and and was kind of wanting me to productize it. But before I built anything anyone else could touch, I was talking to a lot of directors and you know, and we'd have these conversations over coffee and things, and they would be, oh, this would be amazing if I could have access to this. Of course, because everybody has this problem where they're drowning in these board packs and they've got to get the key insights and be able to come into the board meeting and have all their questions really sharp because you don't get to ask all the things that you want. So you've got to almost kind of prioritize that as well. But so what I did before I built anything else, I basically said to the to the people I was speaking to, I want two board packs and I want an audio recording of you reviewing yourself reading a board pack. And then I'm gonna be able to see whether what I can do with the product is better than what you can do on your own. And that's hard, right? Like that wasn't an easy ask because you're dealing with a board pack, it is one of the most sensitive documents a company has. So highly confidential material. And hold on, you're gonna, so I'm gonna have to ask permission to give somebody else these board packs, and then I'm gonna have to read aloud how I interpret this board pack and record it and give you that as well. They're hurdles, and for me it was important, it's a big ask, but it also told me whether there was enough pull there that people really had this problem that they were willing to go through that pain to get a solution. So, you know, the talk over coffee is cheap. Getting clearance to hand over two board packs and do that audio is expensive. And that was for me, was showing me force, not sentiment. So I wasn't interested in nice words. This is impressive, you know, all of that type. They're polite noes. I needed to hear people say, I'm gonna do this because I need this now. That's the real thing for me. They were the real signals.
SPEAKER_00Is this something that's specific to the work that you were doing, or is this more universal in that I think it's yeah, okay. But when you're doing with with with board directors, what you're paying them for, the value being given is their insight. And insight isn't something you get from just glancing at something, it's doing the hard work of the interrogation. The friction is the the critical thing. Something that might make it go a little bit faster, that might help you co collate your thoughts and test your your thinking and things like that. I can see that being really valuable, but I don't want something that summarizes it all for me and does my job and potentially replaces me uh as a board director, the miss then what value am I adding? But there are other things where where efficiency is the goal. So is it in a different type of product setting where efficiency is purely the goal? Do you think friction is also useful in that case?
SPEAKER_02So if somebody will jump over hurdles to do things that then let them use your product, that's a pretty good signal that you're onto something. So that that is where I'd be doing that friction things up front when you're testing these concepts. As you get into creating the product and stuff, reduce the friction and let them get as efficient as possible.
SPEAKER_00I thought I've also heard people talk about the, you know, once you've done a uh test or a pilot and saying, now what would you do if I took this away from you? And seeing you know that's that's the opposite way of doing the the uh validing demand through some friction.
SPEAKER_02Well, I wanted I wanted that test because I didn't want to build anything and nobody was gonna jump over those hurdles in the first place, right? So for me, if they were even gonna do that at the start, that was what I needed enough people to do that. And I said, if I can get 10 people who are willing to do this, I'll I'll go ahead and start creating the product. I got 10 people who are willing to do it. But pretty quickly then I moved into, okay, I want these people to actually pay for this, though. What do I have to do to make sure that people are paying and seeing there's enough value here? And how do I get to that point as quickly as possible?
SPEAKER_00And once you've gotten to that, or once you're once you pass the point where you've validated that there is demand and you want to do something, you start building. And one of the things you discovered in this is that you built something that people said they wanted, but they didn't actually use it, which I think we've all been there, but oh well, most of us we've all been there. Not everyone knows that they've been there, but not everyone has wired up the metrics very well. But but what was that like? And how do you how do you deal with that?
SPEAKER_02I'll tell you what, like again, you know, we all should know better. Um, and look, there was this this feature called cross pack search, and it's my favorite example because it really lured me in as well. Now, in research, it drew the strongest reaction of anything that we showed people. So every director. Describe the same thing or the same need. They basically wanted to find an issue across multiple packs, which they only had a vague memory to go on. I'm sure they talked about this three or four board meetings ago and where was that? And so it was kind of vague. And it came up so consistently, right? And Toby and I, when we were prioritizing what we were going to do and which jobs we were going to serve us first, we kind of put it to the back. And we said, Oh, look, you know, we can do this after we've released and all the rest of it. But it was coming up so much that, you know, we thought, okay, we've got to pull it forward. And we were pretty chuffed because the first director we showed it to said, Oh my God, you solved a problem that no other tool has. Right. It's we're like, anyway, and um, and basically you could type half a memory and get back the exact pack and the page. So, and then people loved it in the demo. Like when I was talking to the initial people that were coming on and things, and we and I do this demo, they would basically be saying, This is amazing. I've never been able to do this. Great. They started using the product and barely touched it, right? And I'm like, Well, why are they using this? And it comes back to the original reasons that they were drawn to the product was this whole time poor, they're just trying to get this review done as quickly as possible. And so they actually would go straight to this key signals area instead, which did the triage for them and surfaced the risk and execution gaps and the positive things. And because a director's time poor, they don't want a better search box. They want the thing to tell them what matters. So I think this was the classic case of stated versus revealed preference. And it's very brutal when you experience it yourself. But stated is what people say when they can imagine having the problem, and revealed is what they reach for at 10 p.m. at night, at 10 p.m. on a Sunday when they've got the meeting on a Tuesday. But these gaps don't show up in surveys and things. So I think what I would do differently is I again, hindsight is a wonderful bit. I'd press harder in discovery on that bit and say, you know, give me three specific times you tried to do this recently and tell me what you did instead. Because if it's annoying enough that they built themselves a workflow around that, then that's real. If they can't name a time, it isn't right. So that was something I probably should have pushed a bit more to really say and make sure it wasn't just this, you know, potential problem.
SPEAKER_00It's easy for me to say that I'm not sure I agree with you on this because and I'm gonna throw out the challenge because I wasn't the one building it or dealing with it. But I wonder if this is a classic Kano model delighter of the thing that helps in the demo and in the sale, and that one time every couple of years when you use it, it's amazing, but it's just not something that comes up all that, it's not actually a day-to-day problem. But what's your opinion on that?
SPEAKER_02100%. It was totally that. It's it's almost I see it like one of those risk features, right? Like you know it's there if you ever need it, and you've done the tick mentally to say, yep, when I've got that problem, it's gonna be there. But then if you don't have half the time, you haven't got that problem because you just try to keep the review of your pack done. The other, there was another area uh similar to this, where in the product we had this fantastic, I kind of wanted people to be able to take a persona of different directors. So, you know, because I've been in a lot of board meetings now, and and I'm the product and techie person, but a lot of the other people will go, Wow, I I, you know, I have no idea what any of that's about. And I'll think, well, you're legal stuff. I don't have any idea at the time what that's about. But what you could do was you could take the role of different directors and ask questions and things. And again, it was another area that people loved. And you could the first couple of times they would kind of use and play with the pack and get this really good interaction. And I think it actually made them really prepared for a meeting because they were taking questions from different angles and really thinking about stuff. But again, people are time poor. And so it was something that they just didn't use that frequently, but it was the Kano delight of all the rest of it.
SPEAKER_00And it's an interesting of it. I could see that there would be a different user in that case of I'm the uh CPO or I'm the CTO, and I want to prep for the meeting, so I want to take questions from practice questions from the from the board.
SPEAKER_02100%. And and that was exactly my view of this whole thing, Randy. I mean, you've nailed it, but people are type poor. So we're talking about things of perfect practice, and it seems right, but the reality is people are just pushed to the line to get the review done and be able to get into that meeting, and they're not necessarily, even though we probably should, spending a lot of additional time to take these perspectives and things. So I I call that sometimes the broccoli principle. You know, I know I meant to eat broccoli and greens and all the rest of it, but it doesn't mean I do it all the time.
SPEAKER_00So, what did you end up doing with these features? Did you keep them in and leave them because they were sold well and demoed well? Uh, or did you decide that, you know, for the uh sake of relentless efficiency and keeping the code clean and things like that, that it's you know, if people aren't using it, we're gonna cut it.
SPEAKER_02No, we end up leaving that. I mean, Toby pretty ruthless with stuff like that. Like he was like, let's get rid of it, let's go. But it was exactly as you said before, right? One, it was something that people were talking about still and wanting to see, so tick the kind of thing, and the rainy day feature, and also was demoing well, and we were probably still relatively early on our journey for uh, you know, not necessarily bloated with lots of features and things like that. We didn't build too way ahead, and that that's the kind of you know, it's the temptation, isn't it? And I've heard this with you and Faith on your weekly podcast, um, where Faith's building her app at the moment. And oh my goodness, it's tempting to do this, do that, because it's so easy. But we were pretty disciplined with that and and really said, look, these are the key jobs we're trying to support, and let's make sure we've got those. And also to keep the product simple. So, in the end, we didn't pull those because they were still seen as valuable. And again, it might be quite different. We were still quite early stage, obviously, as the product grows and is used in different environments, that they might become more important as well.
SPEAKER_00So we were just talking about a couple of features that were used infrequently. One of the challenges you also had is this is a product that isn't used daily, weekly, or even monthly in a lot of cases. You have quarterly board meetings. And I've seen this in finance, I've seen it in education products in the past. But when you're trying to build and get feedback and build habits within things, how do you handle that with products that aren't used quite as often? They're used only when the problem arises.
SPEAKER_02You know, it's a it's the tied-to-value problem, right? And it's the one that for us could quietly break our financial model. And and let me just give you a little context with that. Because in when you're dealing with SaaS products and things, you're assuming fast activation, or ideally, you have fast activation. So someone sees value in a week and the habits are formed, and board software just doesn't work like that. Exactly as you're saying, Randy, it's boards are meeting monthly, sometimes six weekly, sometimes quarterly. And the value invisery came from memory across cycles, and you can't feel that memory on day one. So cycle one, directors would run it alongside doing a manual read and checking whether it caught what they caught. And cycle two, they were still comparing, but they were more confident. And only by cycle three did they relax and start to rely on it. And when we'd done our modeling, I'd assumed two cycles in our model. So for some, but also I hadn't kind of gone through all these variations of all the various boards, um, you know, monthly to six monthly, I'd kind of assumed six weekly, not the quarterly. There was even some that was six monthly that was like, oh my God. So for someone on two quarterly boards, that's most of a year before their penny will drop and realize the value. And then you try writing a 30-day trial of a product that whose value shows up in a month, in month nine, you can't. So it rewrites how you have to think about your trial, your pricing, your churn, your assumptions. And there's a harder layer underneath it that like when demand is strong, buying cycles collapse. And I've seen products pull through procurement in weeks, but this is the opposite kind of market, and it's all about time to trust. So this runs even longer than time to value because trust needs a comparison, and the comparison takes cycles to build. So, what I'd probably do differently um next time or in another life is invest in a simulated second cycle view. And we had something like that in the demo with demo data, but people need to see it on their own packs for the penny to drop. And that moment was real, it just needed a faster way for people to get there. So there are things that you just never think about when you're when you're starting to get, and you know, we know time to value, we know that we're looking for that. I didn't necessarily think about time to trust.
SPEAKER_00Yeah, and I see this a lot when with with people who are learning to use AI tools as well. The real value that comes from an accumulation of data, an accumulation of information, you get you start to populate the MD files and you get to the point where the tool you're using has enough context to actually be useful. The more board packs and the more interrogation you've done in this case, the more value you're gonna get out of it. But that doesn't go. Would did you ever think about going back and populating with old packs and things like that? Or was that what a part of it, part of the onboarding in the first place?
SPEAKER_02Um, yeah, so the trial that we had a um a trial connected with that. And so part of it was allowing them to upload previous packs, but we put a limit on that as well, because otherwise, again, people were just uploading stuff and you're paying for all of those that and um uploads and analysis and all the rest of it. So there's that line there between how much do they need to upload and see for them to see the value. And reality is looking at past stuff, people weren't really that excited. It was the future stuff. They wanted because it's gone, right?
SPEAKER_00Like it's done, they're really in the I was wondering if the past stuff actually gave the present enough context to be more useful or not.
SPEAKER_02But uh Yeah, it they they weren't it wasn't as engaging, and I could tell they weren't as engaged as when it was the fresh pack. That's when they really get into it because they've got to go into a board meeting pretty quickly. Nothing, and it was funny, right? Like in the system, you totally knew when people's board meetings were coming up because it would just be this hammering on the weekend a couple of days before their their board meeting and things as well. So definitely, I think people are time poor, they're trying to juggle a lot of things, and they're trying to be as efficient as possible. And so, again, it's one of those practices that you think about, and people take the time and all the rest, but no, you're you're kind of hammering in there at the last minute. And it's the same thing with them trialing your product. So sometimes when I'd speak to people, they would say, My board meeting isn't until six weeks, I'm gonna circle back to you then. So, again, there's these long cycles of waiting until somebody's actually interested in using your product. It's the challenge with these types of products, like strategy products, I think are the same. If you're doing something that's only done once or two or three or four times a year, they're they're tricky products because you just haven't got people hooked and on and built that habit around them. They're valuable when they have to use them, but it takes longer for you to really prove that value.
SPEAKER_00So, one of the other things that uh people have trouble with a lot, Kirsten, especially when you're moving from being in a CPO exec role to a founder or co-founder role, is now you also have to take responsibility or full responsibility for pricing. So, how did you approach that? This is a really interesting market that you're working in. It's you know, it is people with money, but it's a new product category in some ways. It's not something that they had used before. So, how did you approach this?
SPEAKER_02Yeah, it's a great question. So I I knew the anchoring trap um going into this, and it wasn't so it wasn't a nasty surprise, it was a stubborn one. And what I mean by anchoring problem was Chat GPT and Claude have set up a reference point for many people, right? At 20 US dollars, they're paying for an AI analysis tool, and that's what they're comparing everything to. And visory is not like AI, not like um ChatGPT or Claude. It's a purpose-built, privately hosted in Australia, holds memory and directors' insights across board cycles. So, and it's completely different cost and liability structure. So we didn't try to explain how we differed from ChatGPT because nobody buys on spec comparison. We compared ourselves to the cost of a governance failure instead. So the liability a director carries into every meeting, the hours lost, as we talked about earlier, and grinding through a pack by hand. So we moved people past the GPT anchor that would be there initially, but then created another one with our pilot price. So you think you're in the clear and then you do something else. And in a niche like we're playing in, it's not a volume game. You're playing a high value and you have to charge for it. And in our model, um, we I basically modeled off $300 a board seat per month. So if you're on multiple boards and things, you're paying $600 per for those two boards, for example. But to encourage our early adopters to start using the product, we gave them access uh at $150 a month per board seat. And I was definitely going to want it to charge and I wanted them to pay as early as possible because I wanted to know whether they saw any real value in this versus saying, hey, this was great, right? And I told them plainly it was early pricing and it would rise. Now, in theory, you should be able to move people up as you position the value well enough. And exactly as you said earlier, that classic test of can you imagine if you didn't have this tomorrow, what's it worth to you to keep this, right? So when we were ready to move beyond the pilot to general release, I had those exact conversations with the pilot group and was trying to nudge them towards the 300. And it was tough, right? And because 150 had become the anchor, and the jump to 300 read as a 100% increase. So I'd spent all my time worrying about this GPT anger and then went in credit and sent myself another one. So the lesson here is either pilot a full price with a proper value case or pilot close enough that the step up doesn't feel like a penalty. And I see this with people all the time. Like they go and do pilots with people and don't charge for it, and then suddenly they try to put a price on it and it's a horrible discussion, like it never goes well. So you've got to, I didn't do it as badly.
SPEAKER_00This is the friction one again all over again, isn't it? And would you be actually be willing to pay for this?
SPEAKER_02It is, and so I was pretty clear on, you know, and we spent time, and I'll tell you what, it wasn't that easy. Stripe, if you're listening to this, your gateway is freaking hard to get going. But we um got that whole payment gateway, and and it was a little unusual, right? Our early adopters suddenly were saying, right, if you really want they had been in an environment where we had been controlling everything, and we knew the minute we switched over to them, we had to have all the privacy and security, all of that stuff. So there was investment in building that, but also the payment gateway, because you can't really test if somebody's going to pay if you don't have that gateway. So um, all of that had to be built, which again, they were things that we were kind of testing the signals in that before we'd go and invest in those things. But it was a really this anchoring thing. It we got money, but it was it was half of what I'd modeled, and it was still good. Like people would say, wow, you're getting 150 bucks a month for people. That's not bad. I it's a niche market, right? So you need these people paying more for something that's highly valuable to them as well.
SPEAKER_00Kirsten, this has been hugely interesting and really valuable. The lessons you kind of speed ran through uh an entire building a company and exiting the cycle and in doing it in just a couple of things. No, we are we are doing more. Don't worry, don't worry. Um I want to ask you about advice you would give and and the some of the bigger lessons you learned. So in two contexts. First, is you and I both worked on uh on on the Makers Manifesto together and you know, kind of an evolution of uh the Agile Manifesto, what some basic principles for building in this age. What were the things that were really important to you? What what did you really want to see in this? And what is the the more general advice that you are giving to everyone as a result of this experience?
SPEAKER_02I think what people get caught up with building, and I'm seeing this time and time again, is distribution is the product problem right now. And you can't defer it until you found product market fit. People get so caught up in the build. And is the product desirable, usable, all of that stuff? Not can we get it to people? And even uh I think it was Evan Spiegel who co-founded who Snapchat wasn't it, highlighted this really well for me. He was like, everyone's obsessed over product market fit and barely thinks about distribution. And distribution's getting harder, AI has collapsed the cost of building, but it hasn't touched the cost of reaching people who trust you. So what I did was I treated the channel as its own hypothesis with its own tests and its own stop criteria right from the beginning, because the channel had to mirror the customer. And in so in my case, you know, directors aren't browsing apps or answering to outreach. They act on recommendations from someone they trust. And so for me, a founder with a laptop, no matter how good the product is, there's a trap there, right? Um, if I think that they're gonna just buy from me based on that. So I really think it's one of these things that I'm seeing people underestimate the distribution side continuously, and they're getting caught up in the build and saying, hey, we'll do this, and everybody, it's the if we build it, they will come, which Randy, you and I have seen many times.
SPEAKER_00It's those are all outputs. If we can put build it, that's an output. We've successfully pushed something into production, that's fantastic. But the outcome is successfully getting people to use it and pay for it, creating value for them and for us.
SPEAKER_02It's totally that. So for me with the manifesto, and I was part of contributing to the work that fades the land, it's probably the part that I pushed for the hardest was around distribution. And because we're always saying build the right things and build it right, but that third discipline about getting the right things into the right hands, I think has equal weight. And it's things like distribution, adoption, willingness to pay, retention, none of that's downstream anymore. It's a learning loop you need to run as early as you're doing the rest of your product discovery. And so for me, that was a really critical part to come into the Agile Manifesto, and it's something that I felt quite passionate about.
SPEAKER_00Okay, so that's a good general lesson for everybody. If you were tempted to do a startup again, or you see a friend of yours uh who hasn't done it yet and is ready to go down the path, is there anything specific that you'd say, this is the thing that I went in knowing this? If I was doing it again, I would do this differently.
SPEAKER_02Building the thing is never it's never been easier. So you've got to do the hard part. You assume that you're not the customer and you've got to find out whether anyone is willing to pay. That's why I love you. Nothing faith all the time with that in the podcast, really out loud of how have you got anybody to pay for this? And I think what I talked about with putting friction in them in their way on purpose to see if they'll push through that to get to your product is something that I think everybody could should think about. And and what it will be different for the types of products we're dealing with, obviously, because liking is not boring. The build is cheap, it's the actual the commercial questions is where most of things, these things die quietly. And the one nobody puts on the highlight wheel. So for me, the sooner you get to testing that commercial site, don't get caught up in the bibing and the how beautiful. These things are that you can create, test the commercial side. And I do, like you, Randy, a lot of advising. And it's the thing that I constantly have to push people on because building it is very comfortable, right? Getting money out of people is it? And getting to the right people to buy your thing is hard as well. So give it equal importance. Um, the other hindsight things, I think it's it's pretty, as I said earlier, it's these environments are very relentless. And so for me, it was, I think we underestimated that we could build things so fast and do all of these things. And the toll, right? Like for me, I was pushing ahead, but Toby at one point said, I'm getting really tired. And it's not that she's a slouch or anything like that. It was it relentless. So you've got to be mindful that, you know, I suppose there's a conserving your energy and focus and things, but it was it's something that I think is only going to get worse, Randy, as you can do more and more things, you know. So it's probably something I would say to people, be a bit mindful of that in these environments.
SPEAKER_00But I also find that part of the problem with that is every time I sit down to to fix something in the stuff I'm building, it feels tantalizing and clueless. It feels like it's only going to take 10 minutes, and then hours later you're still sitting there refining it and getting it just quite right. And you it's in the past, some of the work was predictable. You know, you knew some of some of this stuff was just going to take a while to do, but every you now you're fooled by how quickly it could happen.
SPEAKER_02It's totally, it's your tantalizingly close statement is so true. And honestly, there was we bit people saw what we built. Like when I first spoke to the early adopters to when we went to market, it was about four months, right? And when we had, if I'd done this product, and I have done a lot of products, if I'd done this in a previous life, it was an 18-month build for sure. And a team of eight engineers and all the challenges that come with that. Now, so people were like, almost it was like, is this gonna be any good because it's been so quick? But that four months, it wasn't me doing a nine to five, it's exactly what you said, Randy. I was like, Oh, we're so close. If I do if I do this for another hour, we'll have this whole thing over the line and we're done. And it was really interesting. Uh, I had another friend who was kind of um bouncing with me on the side, and it is a fantastic product leader as well, Rachel Morley. And some and she wasn't necessarily in the thick of what we were building, but she was great to bounce and advise and was looking at some of the marketing side of things. And sometimes she'd get frustrated, like, why haven't you done that? Like, what's going on? Like, you talked about exactly what you talk about with faith, and you know, because something's so close, but then it's not 100% right. The other thing I'd say actually, another lesson that you've made me just think of as we're talking through, I think this space is actually getting harder and harder in AI as well, because models are drifting. And we found this as well. We had to invest more in the evals around the system and watching. So, again, I think the build part is really quick. It's the QA framework and the testing framework that you've got to get in place. If you're trying to commercialize this, that is hard and staying on top of it and all the rest of it. So that's the other part I think people forget about. And I see this a lot too. Like people go and put these products out, get people using on, and it mightn't be uh it might be something that's not mission critical. That's okay, right? If it's something that people are going to be reliant on, you have to make sure that quality is there in the testing and also testing the models because they do drift.
SPEAKER_00What I'm seeing is people talk about evals, and to me, it's still like test-driven and behavior-driven design. It's an evolution of that attitude and what you're doing. But because these are non-deterministic tools, uh, because they are black boxes, you have to treat them and not to anthropomorphize them, but you have to treat them like an employee and do regular reviews and and calibrations and and things. And people forget that we still think of it as pure software when it is not non-predictable sometimes. And we need to, yeah.
SPEAKER_02And it's always the way, right? Like if you cut a corner somewhere, it comes back to haunt you in another place. And it's so this is this is the part where people I I think they're not thinking about, oh, I've got this thing to market and blah, blah. And it's like, well, you know, the minute you do a feature, you're gonna have to go and do regression across the rest of it, right? All of this stuff that we know what we have to do in software doesn't go away. It might be easier, it might be more automated in certain parts, but you still have to watch those things. And so understanding the data and what's happening in the system, and but having ways to be able to see and test that and designing that from the outset is equally as important as building these things. The other thing I think that beat us a bit, I heard your example with Faith around, you know, she left an agent running and it dragged her bank account and everything. And we had a couple of situations, but it was slightly different where we in the product we used multiple models to test um the analysis. So one model would do an analysis of the pack, another model would do another, totally different model would do another analysis, and then a third model would adjudicate on who had done the best work, and that would be selected. And one point I was looking at the, and again, you want environments, and this is what Toby had set up where we could see what was going on with all the calls and all the rest of it. And at one point we were looking and we're saying, why is it always there was a couple of days where it was picking one model all the time, and we're like, why does it do that? What's going on there? Well, that the other model had just been retired overnight. So the saw with mobile, where we used to complain and say how hard it was to get something in the Apple store and a pain in the ass and all that, they're not existing now, right? And and these providers, they just deprecate their models. They might do this random email that nobody sees, and suddenly it doesn't work. And you got to be staying on that. Now, again, you can do feet, you can do feature work that detects all of that. We didn't necessarily want to invest in all of that type of detection initially when we're kind of trying to test viability. But it was really interesting of how quickly this space is changing and how much you have to be on it to because we keeps getting higher and higher. Oh man, and it's exhausting. So I totally empathize with anybody in this space during this work.
SPEAKER_00Kirsten, this has been fantastic. Congratulations on uh building a successful product, and thank you again for coming on to share all the lessons you learned from the process.
SPEAKER_02Thank you for having me, and hopefully I haven't been too naughty this time.
SPEAKER_01The product experience hosts are me, Lily Smith, host by night, and chief product officer by day.
SPEAKER_00And me, Randy Silver, also host by night. And I spend my days working with product and leadership teams, helping their teams to do amazing work.
SPEAKER_01Lou Ran Pratt is our producer, and Luke Smith is our editor.