What OpenAI taught me to unlearn — Blaine Billingsley (OpenAI, Slack, YouTube, Airbnb)

Blaine Billingsley is a Member of Technical Design Staff at OpenAI whose path into the field ran through music composition, spare-change website work at college, and a decade across Gmail, Airbnb, YouTube and Slack. At OpenAI he was hired to work on Presence, then redirected to solve a more immediate problem: making ChatGPT genuinely useful as a daily tool for designers. Working with a single engineer, he built the ChatGPT product design plugin — a bridge between the raw power of Codex and the day-to-day workflow of product teams. In this conversation with Randy Silver, he talks about what design actually means when your interface is a text box, why volume beats perfection in an age of infinite iteration, and why the fundamentals of good product thinking are more durable than the tools used to apply them.

Key takeaways

— The designer's job at an AI company has shifted from pixels to outputs. Deciding what "good" looks like, building the criteria to evaluate it, and heuristically assessing results is now a core part of the role — one that didn't exist in the same form at Gmail or Slack.
— LLMs don't replace structured creativity techniques; they scale them. The crazy eights exercise squeezes eight ideas from a room of people in eight minutes. A well-directed ChatGPT session can return 80 in the same window, alone, while you're at lunch.
— Volume beats perfection. The best way to make a pot isn't to try to make the best pot — it's to make a thousand pots. That logic now applies directly to prototyping: generate at scale, stay unattached, and find the nugget in the noise.
— Evals are closer to synthetic user research than quality assurance. The hardest part isn't building the rubric — it's correctly anticipating what people will actually try to do. Show it to one more person and your assumptions will immediately break.
— The second 80% problem hasn't gone away. Getting to a working prototype is dramatically faster; getting that prototype to production is still gruelling — and becomes harder still when platform direction shifts mid-sprint.
— Small teams with AI assistance need to protect the rituals that keep them aligned. When two people can each produce a week's worth of work in an afternoon, parallel drift becomes the real collaboration risk.
— Role boundaries are dissolving, but specialisations still matter. The question is less "what is your title" and more "what does the band need right now, and can you play that part?"
— Experience brings judgment; freshness brings juice. The best work often comes from junior designers unconstrained by years of accumulated assumptions — and both things need to be in the room.

Chapters 

• (00:00) Introduction
• (01:14) Blaine's background: from music composition to product design 
• (02:32) What design means at OpenAI 
• (04:21) The ChatGPT product design plugin 
• (06:12) Deciding what to build: the early exploration 
• (10:05) Structured creativity and LLM-powered ideation 
• (13:37) Volume over perfection: the thousand pots approach 
• (16:13) Designing as a two-person team 
• (20:29) What is the job now? 
• (23:33) Evals as synthetic user research 
• (27:08) Testing at scale when you can't know your users 
• (30:15) How they actually did the research 
• (33:01) The second 80%: from prototype to production 
• (35:06) Staying aligned without roadmaps 
• (38:19) Demo: the product design plugin 
• (44:52) When a prototype isn't ready to ship 
• (48:26) Design sprints, reimagined 
• (49:27) Advice for joining an AI-first product team 
• (52:17) The jazz analogy: experience, freshness and your role in the band 

Featured Links

• ChatGPT for Work: https://openai.com/chatgpt 
• Codex: https://openai.com/codex 
• Figma: https://figma.com 
• FigJam: https://www.figma.com/figjam 
• Linear: https://linear.app

We're refreshing The Product Experience and want your input. Take our two-minute survey and help shape where the show goes next! 

Our Hosts
Lily Smith enjoys working as a consultant product manager with early-stage and growing startups and as a mentor to other product managers. She’s currently Chief Product Officer at BBC Maestro, and has spent 13 years in the tech industry working with startups in the SaaS and mobile space. 

Randy Silver is a Leadership & Product Coach and Consultant. He gets teams unstuck, helping you to supercharge your results. Randy's held interim CPO and Leadership roles at scale-ups and SMEs, advised start-ups, and been Head of Product at HSBC and Sainsbury’s. He participated in Silicon Valley Product Group’s Coaching the Coaches forum, and speaks frequently at conferences and events. You can join one of communities he runs for CPOs (CPO Circles), Product Managers (Product In the {A}ether) and Product Coaches. He’s the author of What Do We Do Now? A Product Manager’s Guide to Strategy in the Time of COVID-19. A recovering music journalist and editor, Randy also launched Amazon’s music stores in the US & UK.

Transcript

Copy Transcript Copied

SPEAKER_00 0:00

That's pretty cool to come back from lunch and have like 80 ideas mocked up for you. I've uh I've worked at I don't know Gmail, Airbnb, YouTube, Slack, and now here at OpenAI. Where we're focusing now is like going beyond building what you already want and helping you actually like think through and ideate new ideas and make better ideas. What I think is really cool is all of the ideas on Earth are available to an LLM. What makes Chat GPT for work really useful is you can have it cover a much wider range of techniques. We focus a lot more on the present moment because the kind of fear of time has gone away a little bit. Like we know we can whip stuff up fast. We know what the company's goal is. And everybody, no matter where you're connected to that goal, it might be really far away from that goal. You're all contributing to that. The thing I've been working on is product design plugin. I just asked ChatGPT to go browse the web and uh find some cool ideas. And in about 10 minutes, you know, it pulled a bunch of ideas, a bunch of references.

SPEAKER_01 0:57

What kind of advice would you give us on day one about what do we need to unlearn and what do we need to learn?

SPEAKER_00 1:01

The titles don't make sense anymore. Things are shifting. We're in a moment of transition. Just focus on who you're working with and what you're doing, and good things will happen for you.

SPEAKER_01 1:13

Blaine, how are you doing today? It's great to see you.

SPEAKER_00 1:16

Yeah, it's good to see you too. I'm doing great. So thank you for having me.

SPEAKER_01 1:20

Yeah, thank you so much for joining us today. So we've had a brief chat before this, so we got to know each other a little bit. But for anyone who doesn't know you already, do you mind just giving us a quick introduction? How did you get into this space? What are you doing these days? What would and what was your first design/slash product gig?

SPEAKER_00 1:39

Yeah, so I'm Blaine. I'm currently a designer at OpenAI. And um, I didn't like study design or engineering formally. I actually have a degree in music composition. But um my grandfather was a technologist. He worked at uh NASA on all the Apollo missions. So we had like computers around a lot. I was really lucky that I had a lot of uh technology at my disposal as a kid. So in high school, I would just like make websites and posters for my bands mostly. And then in college, I made websites for spare change um around campus. And then I moved to San Francisco and got a job working data entry at a startup, and then just kind of learned on the job from there from engineers and designers. So since then, I've um I've worked at uh you know, Gmail, Airbnb, worked at a couple of startups, been through two acquisitions, worked at YouTube, Slack, and now here at OpenAI. Fantastic.

SPEAKER_01 2:33

And we're gonna talk a bit about uh a new feature that you've helped develop. And but before we get into that, I'm just curious, you coming from a design background, you've worked at all these jobs in places where you know design theoretically is a more traditional pixel and interface-based thing. But open AI and you know, well, all of the the the LLMs that we're working with, the interface is mostly a text box. So what is the role of the designer when you're moving from from Slack and Gmail and things like that to OpenAI? What what is the what is the job that you're doing?

SPEAKER_00 3:10

Yeah, totally. I know a lot of my my family is like, well, what do you do? Isn't it like finished? You know, there's not that much to it. Um, to some degree that there's truth to that. Um, there's a lot of stuff that's similar, you know, like Slack is an open-ended text box uh with some non-deterministic inputs and outputs, also just comes from humans and Gmail, same thing. So there's a lot of familiarity with conversational experiences that I've had in the past. I think the biggest thing that changes is like a lot of my job um is more about what the output of the content should be. What um evals are we looking at to grade success in the first place, actually kind of deciding how we measure ourselves and then doing that heuristic evaluation of like, do we actually editorially think you know, this is the format or structure that we think is a good output? That's a huge part of my job that's uh kind of categorically different than it's been in the past. But a lot of design has always gone beyond just like the interface and the pixels and figuring out what features we need and how to how big should they be, how small they should be, how should they feel. And all that still, you know, that all still totally applies.

SPEAKER_01 4:20

So let's let's talk about the new thing that you've just launched. So I'll ask you to introduce it, and then we let's talk about what it was like to work on it and what does a team look like? What is it? What did you actually do on a day-to-day basis? And not just you, but the whole team, how did you all work together? But let's let's set the scene first. Tell us about the the new launch.

SPEAKER_00 4:39

Originally I was hired at OpenAI to work on a different product that actually just got announced called Presence. Um, but the last couple months, I got moved to work on how to make this new ChatGPT work work well for designers and make it a daily driver for designers. So the whole premise behind this merge is that Codex is really great at writing code, but it's that actually means it's really good at doing kind of anything with a computer. And ChatGPT, with that new harness and model underpinning it, can do a lot for workers, whether you're an engineer or a designer or in marketing or or in any in any profession. And so I was the uh quote unquote subject matter expert to sort of figure out how to fill the gaps to make it really useful uh on a day-to-day basis for for designers or for non-designers who need design.

SPEAKER_01 5:29

Okay, so uh Chat GPT being really good at doing anything on a computer, that is, given the news of the last week and with the what happened with Hugging Face, it's kind of a kind of uh short selling it. So, but the reality is obviously that was all safety measures turned off. It was not the normal the consumer product by any stretch of the imagination, but it just goes to show there's a huge it's infinitely malleable the the way that these things work, but and also hugely unexpected. So, what's it like when you're sitting there and you're trying to determine, you know, create an experience for people of this is good content, this is going to be a good experience, this is useful and and safe and good. How do you approach that job?

SPEAKER_00 6:12

Yeah, it was really interesting. I mean, I I had the luxury of working on the job that I've had for you know 15 or 16 years. So I've had this, you know, just background of what it's like being a designer at small companies, what it's like being a freelancer, what it's like being at a huge company. I've been a manager, I've been an IC. So, you know, at first we were just kind of relying on my experience of what it's like working at companies as a designer, what's hard, what are the things every designer wishes they could do. And we talked about it. It was it was just me and one other guy, an engineer and myself working on this. So we were just like chatting about, you know, what is what do designers do every day? You know, we're in Figma 90% of the time making mock-ups, you know, we're trying to make prototypes all the time, and we're gathering requirements, we're listening to research, sometimes we're running research, and we just started kind of thinking about like, well, where do we want to, where do we want to focus? Do we want to help Codex get a lot better at using Figma? You know, do we want to focus on what it's already kind of like great at, which is writing code? Um, do we want to focus on all the minutia and day-to-day around being a designer, like gathering context and writing briefs or clarifying briefs? And um, we explored a lot of stuff, like we tried a little of that and some more um over the first few weeks, but um, we ended up kind of always coming back to leaning into what Codex is already really great at and what was coming to Chat GPT in the in the near future at the time, which was computer use. You can kind of use your computer on your desktop on your behalf. It's really great at making images. The new image models are absolutely incredible at interface design. It's like a uh a zero-to-one shift uh from where it was when I first joined. And if you kind of lean into that stuff, and of course writing code, so if you lean into those three things, you know, you're taking advantage of what it's already been trained on and what it's what it's naturally going to be really good at. And it turns out there's a ton of stuff that you can do with those uh capabilities at its core. So from there, we you know, thinking about it through that lens, where we ended up landing after a lot of exploring a bunch of different um different kinds of workflows that, you know, some were really hard and some were really easy. We landed on first and foremost, just helping designers do what kind of codex traditionally was already good at, which is just make code prototypes and how to do that faster, make more prototypey decisions, and give it a little bit of bootstrapping so that um it can do stuff really, really um nicer than it would just one-shotting it, how to help it kind of find your design system, et cetera. So we explored that for a long time. I'll show you a little later. We we spent a lot of time trying to build a like a prototype starter kit builder, which um you can kind of do in Codex on your own. We kind of realized we don't need to do too much extra, and you can kind of do that. And then uh we did spend a lot, uh, that kind of brought out a lot of different um smaller workflows, like being able to send Chat GPT off to take a bunch of screenshots of your app and audit the entire product surface was kind of an experiment that came out of that process of building these baby apps. That actually has a ton of uses. If you're a you know design system manager, you're trying to audit uh a certain component, or if you don't know how a certain flow works and you want to have it recorded. So, yeah, the like a lot of little mechanics sort of came out of just like how do we get people building code prototypes easier and faster? And where we're focusing now is like going beyond just um building what you already want and helping you actually like think through and ideate new ideas and and make better ideas, hopefully. So that's where we're focused now.

SPEAKER_01 9:52

Let me ask you about those three stages, uh, prototyping, auditing, and production. And I'm not sure if auditing and production go in that order or the other way around, or you know, auditing is kind of a continuing thing. Well, let's start with prototyping, Miz. So traditionally, when we are doing a new product and we want to we want to go really wide, you know, we've got the double diamond approach, we want to go as wide as possible and explore. And LOMs that you know they're all trained on prior art. And granted, so are we, but we're trying to break out of it at the at the early stage. So do you find that when you're doing this that it is more of a constraint or is it an enabler? How do you how do you make activity and when you're uh working with an LM, which I find generally I find that the it's trying to conform uh by much?

SPEAKER_00 10:44

Yeah, so yeah, I know what you mean. Um, you know, I I hope it's okay to go on this like personal tangent for a minute, but you know, when I first joined the the world of software design, I was so um like uh I just felt like I I didn't have the skills, you know. I came from a music background and I was just kind of learning on the job, and I had this like I don't feel like I'm naturally a creative person. And there were some people who would just like dream up like the most amazing stuff. So I spent a lot of my early career working on like really structured ways to be creative. Actually, even in music, that was something that I worked on was you know, if you've got a commission and you have three weeks to do it and you don't feel the inspiration, you still got to write the music. And um, and there are techniques to just get material out of you. And um, I I've always kind of relied on that. I I like this insight um that I've cobbled together from a few sources over the years of like creativity is you touched on one piece of it, remembering what worked and what didn't and why in the past, just sort of like what you've sponged up over the years. It's also kind of um forgetting your principles sometimes or forgetting your requirements and sort of saying, like, I know we've been saying this can't change, but what if it did change? Like, what would we get? And then the other piece of it is just kind of arbitrarily throwing new directions at you, like um take out two pieces. You know, one of my favorite design leaders would always tell us, make it 30% simpler, and it didn't really matter what you did, you just had to take a third of it out. And there's all these different techniques and and methods that that people do to sort of like bring that out. What I think is really cool is all of the ideas on Earth are available to an LLM. And all it really takes is a little bit of patience. I think a lot of times, even in our demos, we try to show these like one-shot approaches of like make me this thing and it'll be perfect. But that's actually not how you find the best thing. You sort of like poke and well, what if we do this? And what if we say this thing that was important isn't anymore and play around? And what makes Chat GPT work really useful is you can have it cover a much wider range of techniques while you're at lunch, you know. So a lot of times I'll do a lot of my own ideation, and then I've got to go to a meeting and I'll I'll ask it to do some structured um exercises for me, and it'll generate a bunch of things on my behalf. And I can come back and be like, oh wow, that was a total, totally not useful, and that one or two things is really cool and inspiring. And and so I think just having a little bit of um taking that stage seriously of like giving it the space to think about how it wants to be creative first, and then relying on like generating lots of things goes a really long way.

SPEAKER_01 13:32

So, one of the things I would normally do like at the in it during a design sprint or in a kickoff might be the crazy eights exercise where you're generating uh eight things in eight minutes, eight designs in eight minutes, and you've got you know, it's a huge construction, you've got no time, and they each one's got to be different, so you're getting really funky by the end of it, and you're getting a room for people to do it, and it's just opening things up. What kind of thing might you do in this case where you've got, you know, I mean, you you're in the the privilege case where you've got infinite tokens. So true. What kind of what kind of thing are you doing to say, hey, I want to explore and not get locked down at an early stage?

SPEAKER_00 14:12

Totally. Yeah, I'm I'm glad you actually brought up the like the sprint techniques because you know, those are sort of designed for like a whole team of people on a really short time frame to cover as much ground as possible. And those are I think that's like a good framing for this stuff. Like the Crazy Eights exercise, it's really hard to come up with eight ideas that are coherent in eight minutes. Um, it's really hard to come up with like 80 ideas in eight minutes, but an LLM can come up with 80 ideas in eight minutes. And you know, the the whole concept of that crazy eights exercise is that it's all about volume. It's like it's not really about finding like the perfect idea and being really precious. It's just like out of the insane volume that you'll get, like there'll be some nugget in there that's gonna stick. And so I don't know if you've ever heard that like um, you know, the best way to make a pot is you can either try to make the best pot or make a thousand pots, and the thousand pots approach always wins. That's how I see that's like the superpower of being able to use uh Chat GPT or an LLM for creative exercises, it can just cover way more ground, and there's gonna be like terrible ideas. There's gonna be a few that you're like, okay, that's a little silly, but it sparks something in you, and you can you can iterate from there. And um, that's really useful. The other thing I always like to talk about is like reimagining these processes, or like at the end of that exercise of sprinting, everybody has to vote. You know, you get three votes and you put them on your favorite idea because it's really hard to make those ideas, even just to mock them up, can take a long time, you know. And with an LLM, you don't have to vote at least as much. You can skip that or radically speed up that sticky note to low fidelity to high-fi prototype process. You can get like really rich representations of your sticky notes, um, yeah, while you're while you're getting a bite. And uh that's pretty cool to come back from lunch and have like 80 ideas mocked up for you.

SPEAKER_01 16:12

It's great, but one of the other key things you said in there was everyone gets a vote. And one of the interesting things is you know, we're now getting to a point where uh I'm seeing in startups and scale-ups a lot more one or two-person teams, and everyone is just me kind of thing. Yeah, uh you said earlier it was this working on this, it was mostly you and an engineer, at least in the beginning. So, what is that like? What does that do to you when you're in the development stage? Do you I mean it's the two of you and any number of agents, but it's not the same as a room full of people, is it? What do what's totally?

SPEAKER_00 16:51

Yeah, I mean, it's like you know, having like the most helpful coworker as an agent is awesome, but the other key ingredient in those, like, you know, we'll keep on the sprint uh train for a minute. The other key ingredient is like all the friction, right? Of like, oh, I don't like that. I like this other idea, and why do you like that other idea? And that's really helpful to have. And you can kind of, you know, you can get that to some degree with an LLM, but there's nothing like, you know, real humans with real passions and real experience um talking things out. And you are right that there like we're so much more empowered than ever with with these tools to do stuff on our own. Like, I I submit way bigger pull requests than I've ever been able to directly to production. I don't have to ask anybody. You know, there's um I'm I'm in I'm empowered to do a lot more than I ever have. Other people are empowered to mock up all their own ideas, and it it does run the risk of you operating in your own little world, and especially in the design sense, like the the fun part is like is spitballing with people and like, oh, what do you think about, you know, like for me, I like just seeing somebody else be like, oh yeah, that's it, that's what I was talking about, you know. And you can lose that. Um, but I think you know, the other side of that coin is everyone else is coming to those group meetings with a lot more fleshed out concepts, or they've done a lot more sort of background side quests. Um, it's actually scientifically proven that like the best group creative thinking is going away on your own and then coming back and reflecting. You know, it avoids group think, especially if you have a good process for sharing. Once you do come back, it really levels the playing field. So you do have to kind of make sure you don't lose those rituals that bring you together to share. In our case, since we were so small, it was really easy because we're just two people chit-chatting every day and sharing each other's stuff. But in a way, it's you know, for me at least, I can't speak for anybody else. It's fun to be like, oh, look at all this stuff I made, you know, like I tried this and that didn't work, and I thought this was cool. And then, you know, my my coworker will say, like, oh, I think I can get that going. Like, let me, let me, let me, uh, let me try if you can get it to work. And I just feel like we're coming with a lot more like material to the the talk instead of like doing planning artifacts. You know, there's I I haven't made like a planning deck since I joined OpenAI. I feel like we we focus a lot more on the present moment because the kind of fear of time has has gone away a little bit. Like we know we can whip stuff up fast. And we're just talking about what we're making actually um in in more real time. So I think as long, yeah, you do have to watch out for it though. I mean, I think it is easy to like um be less collaborative if you don't keep those rituals to to share and show and tell.

SPEAKER_01 19:50

Yeah, one of my favorite uh things, images I've seen recently is you know, we've got the classic iterative development thing of you know, that there was the waterfall of you do each part of a car and you assemble the car at the end. There's the agile of doing the skateboard, the scooter, the tricycle, whatever. And then there was uh the the LLM version of starting with Homer Simpson's supercar and then going through an active refinement. Um you can do totally wacky at the beginning, and it's the the cost of imagining is is so low compared to compared. But uh, so if it's you and an engineer on this team, uh the two of you, and you're coming together, and as you said, you're submitting poll requests, they're able to uh iterate on design with their comments. What is what is your job anymore? What's the difference between your job and their job and and someone else on the team's job potentially?

SPEAKER_00 20:44

Oh yeah. Um this is like the we talk, this is like the lunch talk every day of like, what are we doing? Like, what is this? And like, is this what it's is this what it's supposed to be like? Like, is this good? I I uh candidly, I mean, I often I'm like, I'm either like, you know, a few months ahead of the curve, or I'm just like way off in left field doing a totally different job than I'm supposed to. And um, yeah, we we talk about this a lot internally of like I'm either like the worst engineer on the team, or you know, we're we're all kind of project managers, and it's a very strange thing. What I like about it as um like as a manager, I always liked kind of um an equalized ritual. So like it wasn't like everyone rotated taking notes, everyone rotated uh being the lead on a meeting. And I kind of like the the idea that um, you know, we're we're we all have some specialty maybe in our experience, and we can help in one place or another more or less. You know, like I can't get a pull request to uh pass if it's failing a bunch of CI tests as quickly as my engineer friend. And also, if we really need something dialed in, I can do it in about 10 minutes and it would take him a long, longer time to do it. So we we still have specialties, but I like that it's a lot. Less specialized. I think a lot of times at jobs, kind of at their worst, the fact that I know like how to physically do something in Figma means no one else that if you don't know how to do that, then you can't share your idea in the same fidelity. And I and I, you know, I've seen that at comp at companies all the time. That's used as a gatekeeping mechanism of like, well, I'm doing the design, you know, or or you know, no matter how cool I make the prototype, if the engineer doesn't want to build it and doesn't want to do, you know, the transition, they don't have to because they're the only one with access to the code base. So I kind of like that it's like this slightly more even ground. Although right now I will say I don't really know, like my specialization feels so much more vague than it ever has. And the one thing that's new for all of us, at least for many of us, is how much just evaluations of like run this thing a bunch and see what comes out and and evaluate either heuristically or through some other rubric, like, are we getting what we think is a good result? That is a huge part of my job that has never been a part of my job in the in the past, and is a part of everybody's job. But it is very strange. I mean, it's a it's a the a little bit of just like uh feels like a a band of just people uh trying things out, a lot less like, you know, I I have this function, you have this function. It's a very different experience.

SPEAKER_01 23:33

Well, let's stay micro on it for a minute, then go macro. But uh you just brought up evals. And you know, uh it's not something I've done a ton with yet. Um, but every time I talk about them with people, it philosophically, it's test-driven design, it's behavior-driven design. Evals is essentially the same thing of how do we describe the what good is. So talk a little bit about how you've learned to use evals, how you use them in this product and then and in this development process. What does that look like day-to-day for you?

SPEAKER_00 24:03

Yeah, totally. It's a it actually was really a confounding thing for me because I'd worked on AI features in the in the past. At Slack, I was the director of AI, and so we there's a ton of AI features there, and we had an eval process and everything, but I I wasn't the one doing it. You know, it was a little bit of the more traditional breakdown of specialization there. And like Gmail, I think we we launched uh Smart Replies in Gmail, which was like one of the first Google brain-powered features, and we ran through the spreadsheet every day of training data and said what was good and bad. What's different about this is um it almost feels like um it's like synthetic user research in a way. It's like you you get, depending on what you posit, people are gonna do. Here's what, here's what the experience is gonna be. And is that good or bad? The hard part is positing what people are gonna do the right way. And I found myself getting really hung up at the very beginning of just, well, what uh what are the things, what are the prompts people are gonna put in? And early on, you know, we I would write out some stuff we would try and we would hill climb, as they say here at OpenAI on that and sort of get that working really well. And then we'd share it internally. Okay, we think this is this is working really nicely. The very first thing somebody tried was like some marketing website style we had never thought about and it did really bad. And they're like, okay, what have you guys been doing for the last week? Um, and so real I think it really is um eye-opening how limiting our understanding of like in my mind's eye, I think I know what this is going to be used for, but literally even just sharing to like a third person inevitably opens that aperture up to be a lot wider. And so the big lesson for me in learning with evals was not being, you know, getting a set, trying to continually make other sets, kind of like throwaway sets, widen up that aperture as you can. And um, I see online when I go on YouTube a lot of talk about like having a golden set and using that for regression testing, uh, you know, where you have a big long list of prompts and you make sure that they kind of don't break as you iterate. That's super important. But actually, the the you know, once you get that, you'll build that over time. The more useful thing is like what am I working on now? And like what are all the different things that it could be used for? And what do I want failure to look like? What do I want success to look like? And how can I share this as quickly as possible with somebody else to see what they're gonna ask it to do so I can check my assumptions? I think that's where we got tripped up early on. Just kind of it's it's so hard to imagine beforehand like all the different things people are gonna try to to use this this uh plugin for or anything for.

SPEAKER_01 26:47

Yeah, you I mean, Gmail and Slack, you've got experience with this with going out to an incredibly wide user base. And traditionally with product development, when we're trying to do something, we start with let's thin slice this, let's do this for a narrow group, let's do prove end to end that it works. You can't really do that with something that's this wide open. Totally. And you talked about synthetic users a moment ago. What's how are you handling, you know, with this kind of volume, this kind of infinite variety, how are you using synthetic user testing versus real user testing?

SPEAKER_00 27:20

Yeah, good, good question. You know what this is actually really interesting because when I my first like big kid job was on Gmail, and we at the time we were redesigning it from scratch. It was this product called Inbox, and um, it was like redone from from the beginning, and I was like just over the moon to be there, and but I was also terrified. I had not, I had I was a really young designer at the time, and I had only worked on like little little stuff, and none of it was that good, to be honest. So um I remember asking my manager at the time, like, how do you how do you do this? There's like you know, over a billion users of Gmail, and like I don't know how to think about this. This is too hard, you know. And he made this really good point of like when you go study, you know, an HCI program, you learn about like researching your users, understanding their workflows, building features for them. And it act it feels a lot more scientific in a way, because if you're building software to do invoicing or software for a very particular kind of person, you can really dial it in. It's like part of the joy of HCI, is you can really understand what those people need. When you're building something like Gmail, or I worked on YouTube on TV, which you know, half the world's internet population goes to every day. You can't think in those traditional terms of HCI because the scale is so big, and you have to think about it more of like general mechanics and how those mechanics fail and succeed at their extremes, and and just kind of bake in these like simple mechanics of capabilities into the product. And that really it took me like years to really kind of like let that bake into my brain and and understand that. But um, I feel that so much more here, where there's no way you're gonna cover the ground of what people do. So it's not worth trying. The trick is to try enough different stuff that you're sampling gives you some perspective and to focus just as much as you focus on what good looks like. How does failure look? Like what's the worst it will do? And do you feel okay with that? You know, we used to call it on inbox like where we declare bankruptcy. You know, we want declaring bankruptcy, we want to know what happens when the app declares bankruptcy and is just totally bored. We don't want that to surprise us. So that's kind of my take on it is you there's no way to cover everything people are going to do at a scale this large. Uh, the best you can do is pay attention to the failures and try to sample you know as wide as you can and start to see with these few mechanics that we offer, like what does it do good, what does it do bad, and how can we sort of uh cushion cushion the blow on the bad stuff.

SPEAKER_01 30:04

So you are you are you actively doing the research on this with synthetics, with real people, with with uh production data? What's what does that actually look like?

SPEAKER_00 30:14

Yeah, good question. You know, we our this particular um work working on this product design plugin for Chat GPT. Um I feel actually a little embarrassed talking about how we did because it's not how the rest of OpenAI works. And it's not like I wouldn't uh put this out normally as like a job interview of how we did it, because it was really like experience and vibes based early on. Like I said, it was like we don't have a ton of time, we don't have a ton of access, we don't have like we don't have a bunch of companies we can get to sign an MDA and use it. We don't have anything to use. And so it was really like based on my experience talking with you know the design team here about how it would work for them now, what was it like at their previous companies, and just doing like the best research we can with the people closest to us, which I think is really valuable. I think it's really easy to um kind of get in your head about like, oh, I can't have access to like real users, or it's hard for me to see what real users are doing, or it's hard to get that information. Um, but like any information is good information. And um, and so we, you know, we looked at that, we we read on the community forums, you know, on Twitter and Reddit, and we just sort of like soaked up like what are people talking about and and tried stuff out. And if we liked it, if we thought it was good, then we would show a couple more people at work and they would, if they liked it, we'd show a few more people, and then we'd build like eval cases to cover all the stuff that it did really bad on, and then we'd start, you know, poke around at why it was bad at that and explore some more. And um, for example, like icons, it was like so bad at getting icons. It was like it could do so much, it was like magical, it would build an entire app for you, but like we we got stuck on like the weirdest little stuff sometimes. Um yeah, so we would it we did it really like kind of guerrilla style, like if you know, we were just two people, we didn't have a lot of access to to other companies. At the very end, right before we launched ChatGPT work and released the plugins, we got a couple of companies signed up to just like try it, like literally the a couple of days before, and just like did it, you know, did it turn on basically? Did it work? And did you get something? And then after that, it was just lots of post-launch, lots of reviewing, like classifications of prompts. So we we don't have access, except if people have opted in in certain cases, to the actual uh prompts themselves, but we can get classifications of them, and so we get these like breakdowns of like what types of things people were trying to build, and then we would just sort of start making up our you know examples and testing and seeing how it went from there.

SPEAKER_01 33:01

One of the things I've noticed in playing around with with various tools and and hearing from other people is you know, that 40 first 80 percent of product development getting to prototype stage is so much faster and so much more efficient. But unfortunately, that second 80 of going to production is is still really hard. So, what was that like for you as you're uh playing around with this and trying to work in a new way? What was that like? We talked about prototyping initial stage. What's what's like production 100% true?

SPEAKER_00 33:33

I actually think um, you know, it's really I I love that that uh saying of like the second 80% because it's it's just so spot on. Um, like in a day or a weekend, you can be like, oh my gosh, I got this. This is great. And then the next few weeks of your life are just like absolutely grueling. We certainly had that experience. Um, you know, one of the things about OpenAI that's really fascinating, I touched on it a little bit earlier, that kind of blew my mind was like there's not a lot of like roadmap planning, you know, like at my other companies, we'd have these like huge cross-team request processes, and every quarter we'd like try to align priorities for the next four months. And so one of the things we had really optimized for uh ChatGPT desktop app, which is local on your machine, it can do a lot of magic because it can control your computer, it can store things on your local device. And then we found out we're merging these two products together and bringing this harness into Chat GPT on the web, totally different thing. And it was, I they did it in like three weeks or something. I mean, it was the most insane timeline for like such an insane project. And so then it was just a scramble of like, oh, we were testing all of this in one environment, now we got to test it all in another environment and rewrite it and make sure it works in both places. And yeah, you know, it's uh that's I'm almost like normalized to it because that's actually just every day of like you're just trying to get to like making it work based on what the priorities of the day or the week are, because things move really, really fast now.

SPEAKER_01 35:06

So a lot of what I do as a consultant is working with organizations as they're scaling up or or they're large and they have the issues around communication and collaboration and prioritization at scale. And you just touched on that. Um, when things are moving at such a rapid pace, and anyone in the company can make changes and is encouraged to be working on making changes. How do you get that alignment? How do you uh maintain an organization to be focused on on, you know, it may not be OKRs, it may not be roadmaps, but what are you doing to try and create some sense of alignment so that the there is a cohesive whole?

SPEAKER_00 35:44

Yeah, it's a it's an interesting question. And I don't I've seen I've seen so many different approaches. You know, I've worked at like 10-person companies where we're all friends, and it's like we just have to kind of look at each other and we're kind of like, nah, okay, we'll do that. We'll do what you say, you know. And uh, you know, I've worked at places like YouTube where they have a huge responsibility to a massive scale of people. The process of alignment is very formalized. And then here at you know, at Slack, it was very different. Slack was maybe a little more actually how it is uh at OpenAI in some ways, where something would become a priority and like everybody would swarm on it. Like it, you know, every from every team, everybody kind of played their part. Um, but I like the feeling of like, okay, we know what the company's goal is, and everybody, no matter where you're connected to that goal, it might be really uh far away from that goal. You're all contributing to that. And um, I I find that to be a good rallying cry, especially when the pace picks up of just like knowing really clearly, being able to say really clearly, and having anyone at the company being able to say really clearly what the one or two most important things the company's doing are. And having everybody working on that together is a little chaotic, it's a little rough at times, but it's like all that alignment pressure falls away and it's suddenly everyone's helping each other. And I I'm a little idealistic in that sense, but I like it when it's like, okay, we don't have to argue about is this important or not. We're just figuring out how we all fit together. And I think, especially with the speed and momentum moving, that's the only way to do it is say, like, we know what the rallying cry is, and now everybody just play your part, you know. To me, I think where companies get hung up is when there's like cross-team competing priorities, and then you have to, you know, that that kind of stuff ends up becoming like an internal struggle as opposed to like, well, what is the company, what does the company need to do right now? And maybe what I, my team needs to do isn't the most important thing right now. So how do I contribute to the bigger, the bigger thing? I think just developing that is is the way to get uh rid of a lot of unnecessary strife.

SPEAKER_01 37:57

Yeah, it's one of those things that for in my experience culture-wise, it works really well when you're in the exponential growth phase. But once things start to plateau, it's a lot harder to maintain that attitude and and uh things things that turn different. But we've talked a lot about how you've built this. Uh, we're gonna try something new on the podcast, but we haven't done it very much. We are recording video, we are in YouTube as well. So uh, you're gonna show us a bit about what show us a bit of what you built as well, I think.

SPEAKER_00 38:25

Yeah, oh I'd love to. I hope it uh goes well. Um let's see here. So um I did this a little like uh cooking show style, just so you don't have to sit here and watch my uh Chat GPT think uh for the whole time. But um I wanted to give you a little bit of a breakdown. So I'm using the desktop app of Chat GPT, which um, as I've mentioned a few times recently merged between ChatGPT and Codex. Um it's all the same harness, kind of under the hood. And the desktop experience, especially for workers day-to-day, I think has a few really cool um abilities because it's actually on your computer and can do things on your machine. And the thing I've been working on is this plugin, which I'll just show you quickly so you can kind of get your brain around it. This product design plugin, which you can install once you download ChatGPT. And I'm just gonna show you a couple of the ways that I like to use it and the ways that it'll work best, um, just to give you some inspiration, I guess. So this is probably the most boring one, but it can save your preferences for you. So at any time you can tell it like, remember where I like to get inspiration, or remember where my design system is, or remember where my project, you know, source of truth is, and it'll remember that. So it'll make things quicker if you're doing the same thing over and over again. Here for this set of demos, I just had it remember where I like to get my inspiration, like Twitter or a couple of other websites. Um, so that's a really nice thing about it, just will make your job your life easier once you're using it. You can you can ask it to do that at any time. Um, but you know, one place that I always start is just kind of like looking around at what's out there. And um, so I'm starting a new project. I want to make a chore app for my wife and I. I just ask ChatGPT to go browse the web and uh find some cool ideas. And it actually opens up Chrome, it opens up like 10 tabs and it goes hunting and pecking for stuff that might be useful. And um, it got a few things. I asked it to get a little more from Twitter, it just random design because it it focused a little more on chores specifically. Um, I told it to make sure to check out the people I'm following. And in about 10 minutes, you know, it pulled a bunch of ideas, a bunch of references, and it made this fig jam for me. Another thing I should call out is this is uh a browser, you know, I can go to any website I want here, including tools like Digma or Linear, and you can actually just use it right next to your thread, and um, it can um it can uh kind of do stuff on your behalf. Um and there's ways to connect those even deeper if you want, but even if you don't do that, you can literally just have it like click around for you, which is really nice. Um now from here, I might want to, yeah.

SPEAKER_01 41:16

Let's see, I'll I'll show you really quickly if you um just organize it to organize the the fig jam mood board for you.

SPEAKER_00 41:24

Yeah, exactly. And so you see it's kind of taken over this tab. We've got a new player here, this little cursor, and it'll start moving stuff around for me and um and doing stuff. So it added all the sticky notes, it added all the stuff. I didn't touch the Figmo once. And um, you can do that. It's um taking a little longer than I'd like. Yeah, so yeah, you can see it's it's making a section for me, and um, this is amazing. Like in my personal life, I have it go like fill out forms for me and stuff, and you know, it's just browses the web on my behalf and does all the boring stuff for me. It's really cool. So I highly recommend uh playing around with just what you can get it to do. Um, just a little preview of that. So then from there, um, you know, the next step is like, hey, we got some inspiration, come up with some ideas for me. So this goes back to your earlier question. Um, I just forked this conversation from the mood board and um and then asked it to mock up some vibes. I told it to make 10 different styles. To your point about how to make it be creative, I gave it a thing. Yeah, I gave it a few different things, but it was like go crazy and make 10 versions. And uh I got 10 versions here that I can play around with that are all very, very different. You know, I've got this kind of like nice the AI khaki vibe. Um, very popular. Um, we've got I liked uh this this kind of web 2.0 one a lot too. This one's really cool. Windows 95 is a mobile app. That's something we probably don't need. Um, but you know, you can get it to do whatever you want and uh you could keep going forever. I like to think about the next phase of like you can either go deep on an idea or you can keep riffing and stay wide. Um, I'll show you kind of what that looks like across the board. So if I liked this web 2.0 approach, another one thing I like to do is like, well, hey, mock out the flow. And this is where I actually find Chat GPT starts to think about like, well, what is this thing gonna do? You know, some of these, you're just getting a vibe, you know, like the done button here makes makes very little sense on the home screen, and it has it on a lot of them, but you're like getting a feel, like, okay, I like that feel. Now actually mock up what the flow might feel like to do a feature. Um, I think that's what I I asked specifically. I said, yeah, I love the web 2.01. Mock up what the next step uh what should happen when I do the next step feature, and then mock up all the key screens uh of the bottom nav. And so it did that for me. I mean, this is like mind blowing. In 10 minutes, there's no way I could get this much going of like this uh home screen. I could get the whole flow of what it's like to mark something complete, all the other key screens.

SPEAKER_01 44:13

So something like this, what I found is there are times when I've uh uh been working with tools and it does something like this, and it just makes some weird assumptions. Oh, totally things that are that are definitely not right, so it's really helpful as a starting point, and then I want to mark it up and I want to refine and I want to work with it uh and with other people, but the temptation is well, it's done, isn't it? You know, it's just go ahead and build this and ship it. How do you totally when you're working with other people? How do you remind them that this is not production ready?

SPEAKER_00 44:50

You know, I think the best I love that question. One of the best ways of doing that, so like that whole idea of like, you know, right now we're sketching, basically, right? It's just like our much higher fidelity sketch, and like you said, like there's a lot of stuff in there that doesn't make a ton of sense, and that's okay. Like, that's actually part of the process. The whole point of that like rapid cycle of of ideating and prototyping and rinse lathering and repeat is actually not to get stuck in one part of that cycle. So I I have a little bit of a hot take, maybe, but I like the process of yeah, build it, make a prototype for me, and it will not make sense when I use it, and I will know immediately. And if I make a lot, I mean, my whole career is littered with me making prototypes that I did think make sense, and nobody didn't make sense to other people.

SPEAKER_01 45:41

It's very easy to argue about screens and it's making and have a definitive opinion, and then you try using your oh no, no, that that's not that's just not right.

SPEAKER_00 45:51

Exactly. So to me, the fun part, like I actually think the right way to go about it is like, you know, if it doesn't make sense to you, if you if you see it and it's gobbledygook, well, you can tell it, you know. I actually think this next example where instead of going through flows, I was iterating. And um, this is such a great failure example where it it put mark complete like in the middle of nowhere, and it started making this a card, and then it decided it was gonna be full bleed. And it's like, you know, a rather avant-garde styling choice here. Um, and so like I know that's not right, and I should I can just um you know pull pull this up and uh and I can tell it, I can comment on it and say, like uh you messed this up and uh send it and it'll fix itself. And so like it's good to like work out all the stuff that's just literally gobbledygook. Um the other feature I love is the remove feature where you can literally just like blot out things that you don't want to be there, which is really helpful. Yes, exactly, exactly. This is basically uh my my favorite feature is being able to like remove stuff really quickly. Um, but from there, like let it, you know, put it, put it into a prototype and uh and see what you get. Um, it's really easy to get a working prototype rather quickly. This is another example. If you just ask it to prototype a mobile app, you know, it'll make it here in React for you, but it puts it in a nice screen for you. It'll do this is actually not a great example, but it'll do like carousels, it does all the dragging for you. And um, you can actually play with it, and then you can share this through sites, which is another great ChatGPT feature, and share the link to your coworkers and say, what do you think? Does it make sense? Is it ugly? What do you like? What do you not like? And um to me, the whole point is like make a ton of throwaway stuff. And so I think it's good to just go all the way through it, let it not make sense. And and I I find that people get that more when there's more volume, you know, when you only get one prototype in a monthly review, you're like you you can be precious about it on either end, or you can hate it and not like any of it. But when you know you're getting something every 10 or 15 minutes, you know, it's a lot easier to be like, yeah, that's pretty. I like this part. I'm a lot more relaxed about it because I know it's easy to make another iteration.

SPEAKER_01 48:10

So, you know, I don't think well, the way we work on a day-to-day basis, obviously, it's changed a huge amount. The the tooling is uh fundamentally different, but from a uh philosophical perspective, this feels like you're just doing a lot of design sprints. Is that essentially what it is?

SPEAKER_00 48:31

I love putting it that way, and I think you're actually very right. I mean, there's a lot out there in the internet community about how like the you know, the role of design is changing forever and it's all different. I completely agree with you. Like the the principles underneath our process have always been like get the closest um analogy of the working idea as quickly as you can, you know, whether it's paper or sketches or sticky notes, like anything. How do you get there quick so that you can feel it and you can share it? And it's the same idea. It's the same idea of like getting a lot of ideas, sharing them, hearing what people have to say about them, not being precious with your ideas. It's all the same stuff we've been doing for a long time. We just have new ways of doing it.

SPEAKER_01 49:19

So I think we were kind of running short on time, unfortunately. But I want to ask you, you you said we're talking about the team that built this, which was essentially you and an engineer working together on this. Um, we're part of my the product. I'd be remiss in not noticing there was no product manager mentioned part of this, but frankly, you know, at a at a senior level, the difference between a product lead, a design lead, a dev lead, you're all just have slightly different biases about and experiences, but you're all trying to do is pretty much the same job. And it sounds like where you guys were on this. So I'm curious. So if I was to join OpenAI tomorrow as a member of the product slash design team, or if somebody else was to come in tomorrow as a designer, what kind of advice would you give us on day one about how do we work in this world? What what should what do we need to unlearn and what do we need to learn?

SPEAKER_00 50:13

Yeah, I love I love that question. And um, you know, I'm a jazz musician, so uh pardon the jazz analogy, but in jazz, like everyone plays a different instrument, and it doesn't really matter what your instrument is. What matters is like where are you at that level and what role does it play in the band? Are you a soloist? Are you an accompanist? You know, you need an accompanist, you need a soloist, you need a low end, but that could be a bassoon, it could be a bass, it could be whatever. And I like thinking about teams in that way, instead of thinking, oh, like that's your job, that's not my job. I like the feeling of being on a team where we're all responsible for an output and we all have different specialties. We all play maybe slightly different instruments, but we all have to know the tunes, we all have to be able to count. You know, there's like basic aspects of working together that we need. And it's actually just about finding the people and strengths that uh fit together well. Like I'm really bad at uh time management, and I'm really bad at like um like uh you know, updating, updating other stakeholders. If I get kind of in my zone, I just need somebody who's good at that. And you know, sometimes that's the product manager, sometimes it's the engineer, sometimes it's important for me to have to do it and I have to like figure out how to make it happen. But I think the best thing to do is like don't worry so much about the title or the role and figure out what where you fit into the the spectrum on your team and how do you contribute to that team? And good things are gonna happen, and and that's the best way I can I can put it. It's just uh the the the titles don't make sense anymore, or that things are shifting, we're in a moment of transition, and uh just focus on who you're working with and what you're doing and how you can contribute, and and yeah, good things will happen for you.

SPEAKER_01 52:09

Let me follow up on the jazz analogy because I'm I'm a terrible musician, uh, but I am a recovering music journalist from many, many years ago.

SPEAKER_00 52:16

Oh, very cool.

SPEAKER_01 52:17

One of the things about jazz is you know, to get good at doing that avant-guard, the free form, the collaborative thing, you have to be really solid at the basics. You can't just do it, and it's not actually all that different with other types of music. I uh remember talking to a friend uh about the BC Boys when they came back uh after many years as a hip-hop band and started playing instruments again and going out as a punk band. And the the observation I made was you know, you can get away with a lot by being enthusiastic on a guitar and bass, but if you don't have a good drummer, that's gonna sound terrible.

SPEAKER_00 52:54

So true.

SPEAKER_01 52:54

So so in terms of you know coming to these tools, which allow you to a lot of people going to vanguard and free form because you can just do so much so quickly, but how important is it to have all the experience and and uh I don't want to say it's judgment or taste, but there's something there that you have to have gone through and know how to use these tools, they're just tools, you still have to know how to use them all.

SPEAKER_00 53:18

Totally, totally. Yeah, I think it is. Um, I mean, I hope that experience buys you something, you know, because that's all I got at this point. But um I uh, you know, also there's I mean, the the young designers that I work with today are just like on a whole other ball game of like they don't have any of you with all that uh experience, also comes like a bunch of baggage and presumptions, and it's really refreshing. Like some of the best work I've seen in the last few years is from like really junior designers, so they don't have the experience, but they've got some juice that's like a whole new thing, and um I love that. And so to me, it's another part of like just don't be uh you're not missing anything. You just need to do stuff. I think you just need to go make stuff and go get your hands dirty to make to start making a thousand pots and like find what is exciting to you. And if something's not exciting to you, don't do that stuff, you know. Like the stuff that gets me excited about little interaction patterns, like just go make I'll I just go make that stuff on the side because it's fun, but that's you're building that muscle of what you find cool and what you think is is good taste or whatever. So, you know, yeah, it does take it takes a lot to get there, but it it it's not like years and years. Uh it's a forever journey, but you can start playing with the band really fast, you know.

SPEAKER_01 54:49

Yeah, and and just to go back to jazz one more time, you and R and B, uh Miles Davis and James Brown would bring in people like Bootsy Collins when they're teenagers and exactly a combination of the two things that worked really, really well.

SPEAKER_00 55:03

Exactly. I mean, all the the heavies, they were young, young, young. And that, you know, uh it's a make it's you do need dedication, but it's not like uh you need decades of time. Um there's a there's a lot of it's an exciting time because it's it's very easy to explore now, and that's really what it all comes down to.

SPEAKER_01 55:21

Blaine, this has been fantastic. Thank you so much for taking the time out. We really appreciate it.

SPEAKER_00 55:25

My pleasure. Randy, it was absolutely a blast, and uh hope we get a talk soon. Thank you so much.

SPEAKER_01 55:31

Fantastic.

SPEAKER_00 55:32

Thank you.

SPEAKER_01 55:33

The product experience hosts are me, Lily Smith, host by night and chief product officer by day. And me, Randy Silver, also host by night. And I spend my days working with product and leadership teams, helping their teams to do amazing work.

SPEAKER_00 55:49

Lou Ran Pratt is our producer, and Luke Smith is our editor.