How this PM streamlines 60k-page FDA submissions and saves millions with Claude, Streamlit, and clever AI workflows | Prerna Kaul

[00:00] So you are working in the life sciences on really health impacting vaccines and treatments. And with all that amazing investment in scientific work also comes a lot of paperwork. We had to develop a nearly 60,000 page document. Would have taken about four to six months of effort and nearly 20 specialists, not to mention the millions of dollars spent. Where did you start? I gave Claude [00:30] and a demo. The thinking I had in mind is that Claude is a software engineer and I'm talking to them and trying to tell them why it matters like any good PM would and trying to tell them what end product we want to produce as a result. And not only did it fully understand what I was asking it to develop, it created a little setup instructions markdown file for me and it gives me all of the setup instructions and all of the tasks that it does and its capabilities, which is super handy. Were you really able to shift [00:58] the cost there? Did you see an impact here? We did. We did. And any kind of cost savings you can generate have a direct impact [01:06] to the bottom line. And in addition, if you're saving on time, you're bringing life-saving vaccines in the hands of people who actually need it. Welcome back to How I AI. I'm Claire Vo, product leader and AI obsessive here on a mission to help you build better with these new tools. Today, we have Prairna Kahl, who's taken years of machine learning experience

[01:30] and help develop some of the products you know and rely on every day [01:34] at companies like Amazon Alexa, Moderna, and Panasonic. She's gonna show us how she uses AI to accelerate drug discovery [01:42] and vaccine approval, something that takes [01:44] tens of thousands of pages of documents, [01:47] dozens of people and months and months and months of work. [01:51] She'll also show us how you can use Jane Austen, [01:54] Dale Carnegie and some AI to manage your stakeholders just a little bit easier. Let's get to it. [01:59] This episode is brought to you by CodeRabbit, the AI code review platform, transforming how engineering teams ship faster with AI without sacrificing code quality. Quality code reviews are critical but time-consuming. CodeRabbit acts as your AI copilot, providing instant code review comments and potential impacts of every pull request, beyond just flagging issues. CodeRabbit provides one-click-fix suggestions and lets you define custom code quality rules using AST [02:29] patterns, catching subtle issues that traditional static analysis tools might miss. [02:35] CodeRabbit brings AI-powered code reviews directly into VS Code, Cursor, and Windsor. CodeRabbit has so far reviewed more than 10 million PRs, been installed on 1 million repositories, and has been used by 70,000 open source projects. Get CodeRabbit free for an entire year at coderabbit.ai and use the code HOWIAI.

[03:00] Prerna, it's so nice to have you. I'm really excited about your workflows. Thank you so much for having me. Very excited to be here. So you are working in the life sciences on really health-impacting health. [03:14] vaccines and treatments. [03:17] And with all that amazing investment and scientific work also comes a lot of [03:23] paperwork, if I am correct. And so in addition to doing all this amazing research, [03:31] You also have to document and submit a lot of stuff to the FDA and other regulating industries. And I am sure... [03:38] That is a total pain. So tell me a little bit about the problem that led you to building some of these AI workflows. Well, yeah, firstly, thank you so much for having me. And you're totally right. With life sciences, I think the challenge across all parts of the organization is [03:54] A lot of paperwork, a lot of regulation, and going back and forth with these regulated industries could take months of time. Not to mention all of the people resources involved and delays in getting, in some cases, life-saving drugs and vaccines to market that people need. For example, when I was working at Moderna, we were developing the INT cancer vaccine, which is necessary for folks with skin cancer. [04:24] out. And we had to develop a nearly 60,000 page document, which is known as the, it's a technical jargon, but biological license agreement, which took around four to six, it would have taken about four to six months of effort and nearly 20 specialists, not to mention the millions of dollars spent across the organization and getting this document together and providing it to

[04:54] very time consuming. [04:56] And that's where I kind of came in. I was a Gen AI product manager there and really came in to think through how can we reimagine this using generative AI? This was back in 2023 where I came up with the solution. So a couple of years back. And I kind of evolved it over time to kind of specialize in certain areas as well. I'd love to show what we came up with. So it sounds like you created a really high impact tool and 60,000 pages is pretty intense. [05:26] What I love about this workflow is you're going to show us actually how you use Gen.AI to build this Gen.AI solution. So like all good product people, you started with requirements. Can you just show us your process from you've identified this problem, which is we've got to create this massive problem? [05:42] document. And it's going to take millions and millions of dollars and tons of people [05:45] to actually having this production thing that that your team and colleagues could use. So where did you start? Like every good PM started with the requirements. And I actually didn't, hadn't fully formed all of the ideas initially because I think that [06:02] With the complexity in the domain knowledge required, it took a few iterations too. But I'd love to showcase sort of where we started with very simple pitch and then go through [06:14] how we actually evolve the idea as well. So I gave Claude the problem statement, the pitch and a demo. The thinking I had in mind is that Claude is a software engineer and I'm talking to them and trying to tell them why it matters like any good PM would and trying to tell them what end sort of product we want to produce as a result. So I've included statements such as, you know, this is Stephan Barcels, who is Badan SCU's passion project. He's very excited about it.

[06:44] Here is the cost savings you can get. Here's why it matters in the impact. And then also spoken about the demo that I want to showcase and how we could sort of iterate and build on it. And Claude came back to me with, not only did it fully understand what I was asking it to develop, it created like a little setup instructions markdown file for me. And it gives me all of the setup instructions and all of the tasks that it does and its capabilities, which is super handy. [07:14] Not only did it do that, it also gave me all of the code and all of the different aspects to the code as well. So I kind of started here and started thinking through what do I really need to do to make this a production ready system. And there were two things that were extremely important in the process. A first is because the BLA is a structured document, it is XML based, it has certain formats, I needed Claude to successfully [07:43] in a very strict and structured way, take data [07:46] and structure it into a certain format. So that was a very strict requirement and that's not super easy to get with some of these models as well. [07:55] The second was that because it is clinical data, it's highly regulated and being able to detect PHI, redacted information was extremely important. So that it could correctly redact that and then not have that included in the actual BLA that we would be sending to any authority because it's patient data essentially.

[08:15] So the reason why we chose Anthropic CLAWD is because of the reputation around safety and alignment with human values as well. And we formed this partnership with CLAWD and really focused on some of the PHI and the safety aspects to make sure that not just from a compliance standpoint, but also as we think about alignment, we were taking care of those use cases. [08:40] So this is what it came up with, did a really good job, and I was able to get some code running as well. [08:48] And I'd love to show you what it came up with. [08:50] If we can go back to the requirements really quickly, there's just a couple of things I want to call out for folks that I think is really interesting. If you go to the step-by-step instructions, what I love about it is if you go to the top, it not only gives you sort of how to technically write the code, but something I wanted to call out at the bottom is it actually gives you a pitch narrative to how to run your demo. So what I think is so interesting about this specific prompt is you gave it [09:17] kind of pretty loose requirements. It's only a couple paragraphs up top. [09:21] And it gave you a whole set of things, I'm sure many of which you weren't anticipating. I wouldn't have anticipated getting a demo script. [09:28] from Claude when I'm asking it to build Python, but it gave you that. I thought that was really interesting. And then if you scroll down in the left... [09:36] side, something else I'd like to call out for folks that I think is really interesting is [09:42] It gave you these kind of like component names of different things.

[09:47] technical pieces of of the product so like coming up with mock data [09:53] doing PHI reduction, all that kind of stuff. You know, you as a software engineer and a product person, [09:59] How long do you think it would have taken you to come up with this full set of just this first one-shot prompt? [10:05] requirements before? At least a couple of days, at least about a week to really think through everything. [10:11] And then in addition to knowing you wanted to work with the anthropic models because of safety and alignment, [10:17] Were there other libraries or other models that you knew you wanted to use? Did it pick the right ones? Like how close to the technical implementation did it get? In particular, with the with the redaction piece, [10:30] It would have taken a couple of data scientists to figure out the right solution and work with us. So data scientists as well as software engineers to work with us and figure out what's the right solve for this. How do we implement it? How do we test it to get a certain synthetic data? So I think where it really excelled is in picking the right models. [10:50] And I have some amount of machine learning knowledge, but it would have taken me a couple of days of study to figure out, OK, this is the right model for medical named entity recognition. And this is how you go about it. But it did such a great job with very little information up front on the domain itself. It was able to absolutely pick the right models and give me the code and then implement it and show me the confidence score across that as well. [11:14] Yeah. And if we look at the code, I just have to laugh because, you know, many, many years ago, I think I have lost.

[11:22] hours and hours and days and days of my life to writing regex. Just I used to spend all my time writing regex. Terrible. That didn't work. It was just awful. And so one of the kind of miracles of AI I see is it reduces this toil in [11:41] Technical tools you know you need to use that have documented patterns, but... [11:45] They're just so hard to research and get perfectly right out of out of the gate. So this was my favorite part of seeing seeing your code up is seeing all that code. [11:54] just pre-populated for you. [11:56] Yeah, that's such a common and shared pain point. Regex has been a bane of any software engineer's existence. Yeah. So, OK, so it created this Python [12:07] script for you. And you said you scrolled through it and it looked... [12:11] You know, pretty correct. And then you decided to deploy this on Streamlit. Can you tell us a little bit about that choice? Why you picked that? [12:18] Yeah, so I love using AI to solve for some and automating my own work and solve for my own sort of day to day problems, either be it related to work or something I'm trying to work on in my side projects or personally. [12:48] it locally has proven to be a really, really big unlock, both career-wise and also, you know, working with different colleagues, they find that to be really valuable because they might have something in mind and I have all these skills and I can sort of offer that. So that's actually something I've been doing in my current organization as well, where I recently created a Google add-on to convert PRDs into Jira format. And that was a lot of fun. And, you know, the product

[13:18] because they don't necessarily like having to write requirements twice, one in a PRD and then also in Jira. And we found a good way to kind of circumvent that and do it in a matter of minutes. So I did that. And then the Streamlit idea, I also did in a couple of my previous roles and my current role as well, to give non-technical users access to all of these tools. Yeah. And just for folks that don't know, Streamlit is a really easy way to put a pretty simple UI approach. [13:48] on front of like a Python kind of file instead of APIs. Exactly. So Claude created your PRD. It gave you code. You were pretty happy with the code. [13:57] How do you get from us looking at this code right now to a product that your team could actually use to create these documents? [14:04] I'm gonna run for you. [14:06] the actual Streamlit app. [14:08] So I'll take you to terminal. [14:10] I essentially run it from the command line, and it opens up a web app. [14:14] It takes me here. [14:16] It has preloaded an API key from Claude, which I've provided in the code. So I've hidden some of those properties. And I'm going to generate synthetic data for us, and I'll talk through [14:28] And I'll go through the step as well because it takes a bit of time. And I'll talk through what it's done. So as I mentioned, there's this format called a common technical document as part of the BLE. There are several models, about five or six different modules. And it has a structure around it. So there is a study report that has a tabular listing. It's supposed to have very specific points across the data from the clinical trials and then different literature references as well.

[14:58] know about this format. [15:00] kind of from just general knowledge or did you have to provide the format to it [15:05] So it searched up a [15:08] the common technical document format for me, and then it figured out the right module to apply to my use case. [15:15] Great. That's awesome. [15:16] Yeah. [15:17] It's doing pretty well. [15:18] So I'm running here what is, I'm going through the different, we have different participants here, and there are some clinical notes per participant. This is all clinical trial data, very, very sort of common that you would see in any trial that runs. And by the way, the trials itself, they could run in the order of billions. So redoing a clinical trial is very, very expensive. And so as much as possible, we want to [15:46] give very accurate information to the FDA so they don't ask us to redo anything. And what I've done here is I've de-identified some of this patient data to remove any sort of names, date of birth, any information that could be associated with the person. And one of the challenges I'd love to call out for folks that maybe aren't seeing this is you generated some fake data so you could validate it works. [16:10] And it's not only this tabular data that has patient ID, ethnicity, whether they were in the [16:16] control or otherwise. But you also have these free-form clinical notes where there may be embedded PHI written in by a clinician or a [16:27] study administrator,

[16:29] So it's not as simple as saying just find all the data birth columns and redact them. You actually have to go through this unstructured data and [16:35] and find potential sources of PHI. Is that part of the challenge? [16:39] That's exactly it. It's a complex problem usually, you know, machine learning [16:45] doesn't always do a perfect job there, but Claude immediately identified the right model to do this. [16:52] Great. So, okay, so you have this clinical daily preview, and then you're going to show us how [16:56] you de-identify it. So you're not sharing that with folks that don't have access to that health data. Exactly. So with the synthetic data, I clicked on this button to detect and redact PHI information. And it scanned through all of the rows and it found for me all of the PHI. So [17:16] Here's what it came up with. [17:18] So seeing the clinical notes, it found that there was some dates, names associated. There was some birth date information as well. And the remaining, it was able to keep sort of in a de-identified way so that I can include it in my reports. [17:34] - Great, and then how would this have been done in the past? I think you said there are software engineerings or ML engineers that would have built these models to redact the data. Was there any human in the loop at any point in this as well? Like how was this done before? [17:49] For the past couple of years, we've been, most pharma companies have been using machine learning for this. But these projects, they end up, because it is PHI and there's very high bar, they end up taking months of effort to get right.

[18:06] And especially when clinical trial data might not look the same for every drug, [18:13] As a company like Moderna was trying to scale, we were trying to launch nearly 50 drugs, bring 50 drugs to market [18:19] Um, [18:20] in clinical trials and then 15 to market in the next three to five years. [18:26] problem was exacerbated. So thinking about coming up with new machine learning models to solve different problems across each of these vaccines. It's a matter of scale and time, really. And we really needed to find a better way to do this. [18:42] Great. Okay, so we've got redacted health data. You have all your great, you know, billions of patient data points. What's next in this flow? So the next thing is to generate this common technical document. And I'll show you a small preview. [19:00] It's just for the purpose of this demo. So what it did is it took all of that and it gave me a summary of the different data. It was able to give me a synopsis, which is what the FDA looks for, the study participants, the counts, the age, all of the statistical data points, as well as some of the specific medical terminology that they're looking for. [19:21] So it kind of summarized everything. It gave me this. And for context for folks, this is typically done by a medical writer and their [19:32] their whole, I mean, I've worked very closely with them. It's so time consuming because they not only, they can do a pretty good job of generating a like a first draft summary, but they have to go back and forth with their stakeholders for weeks sometimes on a single module to come up with the right set of criteria and getting everyone to align all of their stakeholders to align is very, very time consuming.

[19:57] 20 specialists, as I mentioned, are working on it at any given point in time. And as part of what some of these... [20:06] the chat GPTs of the world and Anthropic can provide. There are also common projects and common environments where all stakeholders can kind of collaborate and work together on these summaries, which is so much easier and so much better than working, let's say, in a sort of a shared document and just providing comments. Yeah, great. OK, so it generated from that data... [20:28] This very, I mean, it looks official to me. I'm not part of the FDA. But, you know, it's a summary of statistics. It's a summary of the methodologies. It's a summary of the impact. [20:37] It's pretty long and detailed. It looks like you can download it both to... [20:43] a TXT form, like a doc form, as well as this, as you mentioned, proprietary XML form. [20:49] format that's required. And I'm presuming the Claude code just gave you two buttons to make that happen. That's exactly it. And I know it's a little bit proprietary, but I cannot tell you how excited I was to see this button. [21:04] This episode is brought to you by Lovable. If you've ever had an idea for an app but didn't know where to start, Lovable is for you. Lovable lets you build working apps and websites by simply chatting with AI. Then you can customize it, add automations, and deploy it to a live domain. It's perfect for marketers spinning up tools, product managers prototyping new ideas, or founders launching their next business. Unlike no-code tools, Lovable isn't about static pages. It builds

[21:34] real functionality. And it's fast. What used to take weeks, months, or even years, you can now do over the weekend. So if you've been sitting on an idea, now's the time to bring it to life. Get started for free at lovable.dev. That's lovable.dev. [21:52] Okay, so if we could just take a halfway point of the podcast, the two things you were most excited about and I was most excited about, I was excited about the regex, you were excited about the XML. This is a very nerdy and very specific How I AI podcast. Okay, so you've generated the doc and then I love this last part. So let's talk about what's this last piece of the product. [22:14] What's super interesting is I have worked in a couple of different organizations where, as we are trying to scale different AI use cases, cost comes to be a factor. Because not just token cost, but also acquiring licenses for these models and then running it. And as the number of consumers or the users of the actual use case scale, the dollar cost also scales proportionally. [22:40] So one of the ways in which we could potentially continue to launch and continue to build products around this is by actually being transparent about the cost associated with the operations. And, you know, generally one of the stakeholder arguments that I've previously received pushback is this is going to be high cost. Can we find a simpler way that's not AI based?

[23:10] about the cost associated with it, how long it would take. And as you can see, this trace and cost analysis is super useful because it says cost per operation. It gives a duration against it. And then it actually breaks down by operation what was the token cost and the total cost. So extremely useful. You can continue to track and monitor this in production as well. Yeah. So then you can say, for example, you know, PHI redaction per patient costs 15 cents. And you [23:40] organization that [23:41] Is that more or less expensive than [23:43] alternative models uh that's i mean i actually haven't heard anybody give this uh [23:49] this suggestion, so I want everybody to hear it, which is if you're getting internal resistance to cost, in fact, I feel this even as a... [23:57] um, [23:58] Even as a solo founder who's vibe coding things and I see Claude Code give me like this cost you $5.13. And I send that to people. They're like, that's so much money. I'm like, it's $5. And so I love this idea of bringing true transparency. And I'm sure these numbers are much smaller than your production numbers on your, you know, million person studies. [24:18] But it does bring a true sense of ROI and investment here that I think can get you through internal hurdles. [24:25] Yeah, it really has helped. And I really love Streamlit for this because it's tracking everything by per operation so we can even plug and play very easily. [24:36] This is awesome. And so... [24:38] You built this and are getting it in the hands of folks that need to submit this. And what do you think the actual.

[24:46] You know, you were telling me millions of dollars, months and months and months, tons of people... [24:51] Were you really able to shift the cost there? Did you see an impact here? [24:56] We did. We did. And we actually applied some of this thinking to our cancer vaccine, the INT vaccine that's coming out and our RSV, COVID-RSV combined vaccine as well. Both are past the clinical trials. I think we're looking at commercialization there. So we found that cost savings was definitely one thing, but the stakeholders themselves, [25:21] were super engaged in the process and they built a system that scales pretty well as the company scales as well. So that was the most useful thing is we're saving on repeated work. [25:32] weird. [25:33] If one thinks about, I'll just give some numbers out there. So billions of dollars in costs per trial, three clinical trials per vaccine, want to bring 15 vaccines to market. That's probably in the order of a couple of hundred billion. I mean, that's a pretty significant number. And usually most companies don't have that sort of cash. And any kind of cost savings you can generate have a direct impact to the bottom line. [26:03] bringing life-saving vaccines in the hands of people who actually need it. So I was really motivated to kind of help here. [26:11] Well, this is such a great use case. I love that you showed going from quad to streamlit app and then from streamlit app to real impact on something that I think, you know, as a mom, RSV and COVID vaccines, I thank you. So the sooner you can bring those to market, the happier I'm going to be.

[26:26] So this is amazing. I know you have one more workflow for us on the product management side and you tease this like, [26:34] PRD to JIRA thing. So I know you're thinking a lot about just product management generally and [26:39] AI. So you want to show us your next workflow? Absolutely. So the problem statement here is that with product managers, we in our day to day rules end up working with a lot of different stakeholders who have a [26:56] various priorities as when it comes to the product and [27:00] I personally, I really want to make sure my stakeholders have heard that we're working very, very collaboratively and [27:08] While we do that, at the same time, I want to make sure we're making the right decisions and also kind of bringing everyone along and moving things forward. And with influence and communication, which is a super important skill as a PM, it's sort of custom where it's hard to say which communication framework is. [27:28] to apply when and [27:30] what type of communication would land best with a certain stakeholder because it's so personal. [27:36] and you really have to empathize and understand folks. So, a very important skill to work on. I've been starting to lean in on Claude a little bit and training it to kind of give me ideas and use it as sort of a brainstorming buddy in the process. And this has actually really helped me save time in meetings, ahead of meetings in preparation.

[28:00] So I'm going to go through... [28:02] a prompt that I created. I want to show you this really, really cool thing that Claude developed. It's called a prompt generator. So if you go to the console, console.anthropic.com, you'd be able to create [28:16] a net new prompt or generate prompts as well. And I created one for influence and communication, where essentially I was I didn't create this, by the way, generally it did for me. I gave it some instructions and then it gave me very, very precise information on what what the instructions are. And then it gave me structured values that I can include as part of the prompt. [28:40] and how it would break down the problem, how it would think and reason through it, and then what outputs it would give me in return as well. [28:47] So a couple of things I see here is one, you use this tool that Claude has called Prompt Generator. It generated the prompt. For those that aren't looking at the screen, the other thing I want, XML is the name of the game today. So it has these... [29:01] kind of like code block sections for the situation, your knowledge base, how you do strategic thinking. [29:07] Like all good prompts, it starts with the URA blank and gives it a roll. And then if we scroll down a little bit, [29:15] You can see that it also lets you put in different knowledge and then it tells you to provide your response in [29:23] a very specific format that has a combination of analysis, your communication strategy, [29:28] play-by-play, and then, like, an anticipated question. So this is a very...

[29:33] structured prompt. Have you started to reuse this sort of like kind of XML style [29:40] prompt or have you played with different things? What's your approach? Or do you just use what Claude gives you? Yeah. So I usually start with generating a prompt and then you see this button here, use Claude to optimize your prompt. This is extremely useful because what it does, you can give it feedback and in real time it will improve your prompt and then add it back in here. So now let's put this prompt in [30:07] in the actual Claude project. [30:10] So here we go into projects. [30:13] And this is our influence and communication coach. [30:16] I'm going to update this. You can see here that you can set project instructions. So I'm going to update this prompt here. [30:24] and to Claude. [30:25] and save these instructions. And I've also given it a project knowledge. So I've actually trained it on free books from Project Gutenberg that consists of all sorts of things from literature like Dale Carnegie and tactics on persuasion and how to win friends and influence people. So all of these cool books that are all free and in the public domain. [30:50] So I was not expecting Jane Austen to be in your product manager strategic communication, a quad anthropic project, but I am excited to see it. So you... [31:02] not only gave it its prompt, but you kind of loaded it up with classic

[31:07] Classic literature, classic writing on influence, persuasion, communication. Very, very interesting. [31:14] Yeah, Jane Austen is a favorite of mine. I'm not sure how many folks would use that for persuasion, but I found it very enlightening. [31:22] So now I'm going to walk you through... [31:25] A prompt that I have been working through, and this is just a test case, but we can try something live. How about we try that, Claire? Let's do that. [31:35] Amazing. [31:36] So do you have a situation in mind, or I can ask it to generate a couple of situations? Oh, let's ask it to generate a couple of situations. [31:44] I'm curious what it will come up with. [31:46] So I'm using Claude Sonnet 4, which is one of the reasoning models. And as a reminder, [31:52] the goal of [31:54] this particular project is to essentially take a curveball problem at work and then come up with sort of a structured strategy. So [32:02] hear what it's done. [32:04] It's made the assumption that I'm Alex Kim. I'm an AI product manager working at a healthcare AI startup. And these are all of my stakeholders. [32:13] very, very, you know, sort of experts in the domain and have very strong opinions about various things. And it has... [32:22] It's broken down a problem statement that I'm trying to solve, where two weeks before a major presentation, [32:29] Um, [32:30] These are all of the concerns that my stakeholders mentioned. So it's actually created that for me.

[32:38] It is also described the core challenge here, navigating ethics, compliance, competitive pressure, stakeholder expectations. And it's also going to personalize it, which is cool. So for people that aren't seeing what it generated, which is a very realistic, unfortunately, challenge, which is it generated this scenario, which we're just going to pretend is a scenario we're facing at work. [32:59] which is I'm a PM and my head of clinical data just identified a bunch of issues, both privacy and accuracy data related. [33:09] two weeks before we have a major presentation with a huge, huge prospect. And then it lists out [33:15] everyone's different opinions. And I think PMs can really empathize with maybe not this particular situation, but a situation like this. So you've created an AI project that says, give me a sticky situation. [33:27] And I will help you navigate how to navigate the situation given all the stakeholders. [33:34] It's listed to the stakeholders. It's given you the core challenge. So what do we do? What do we get next? All right. So I'm going to ask it to use this and [33:43] Proceed. Next. Oops. And then this is what it's going to use your strategy framework that you came up with [33:51] in the prompt to sort of break down the problem and give you an approach you might be able to use. [33:57] Exactly. So it's now sort of analyzing [34:01] Oh, [34:01] based on the project information that I'd shared with it. And as we can see, it's picked out [34:06] the most relevant sections from the knowledge as well. So I had given it around 10, 15 different books and documents, but it found the top two and then picked out the right sections from those. So all of this, it was able to search that information and point to that and analyze the whole workflow.

[34:26] So it gave me sort of a situation analysis, which I found particularly helpful because it's sort of validating that it understands the core challenge and the organizational context and some of the cultural factors as well. And here's my favorite part of the prompt where it actually goes in and it looks at some of the influential tech leaders that we have all heard and know of. [34:56] would actually approach the problem. [34:59] This is amazing. And so you're giving really [35:02] real life examples or quotes that someone can say, okay, if I were to approach it like this other person, which I see, I hear is a very common example [35:10] AI prompting technique, act like this person or act like you work at this place, can really bring a set of unstated principles to a problem that I think is really important. [35:21] effective. Okay, and so it's come up with [35:24] Document examples, tech leader examples, communication principles, we all love. [35:29] "How to Make Friends and Influence People." So some Dale Carnegie principles. [35:33] And then it's giving you some strategies you might be able to take. [35:37] here. Yeah. So out of all of that vast array of information that I gave it, it picked out the three strategies that would apply. So it talks about the ethics of it, which I personally agree with quite a bit. So it's talked about positioning as privacy first as a competitive advantage.

[35:56] And then it's talked about compliance. And then now it's going to give me sort of a collaboration strategy as well. So after analyzing the situation, it kind of summarized for me [36:09] Here are the three things that you should do in terms of strategically approaching the situation. First, make sure you're acknowledging and validating each stakeholder's expertise. [36:19] Second, collaborate to discover an approach. [36:22] That demonstrates responsible AI leadership and then talk about the ethics aspect of it. So it's really kind of honed in there. So it's very helpful as, of course, like a good PM product leader in the space would have this info, but it's helpful to see that as a, [36:39] sort of a thought partner as you're brainstorming over this as well. And then it's actually broken down my pre-work [36:47] for me on a day-to-day basis. So what are the individual conversations I can have with each of my stakeholders and what question to focus on and ask in that meeting? [36:58] And then it's given me a minute by minute structure for the final leadership meeting as well. And here is my favorite part. Not only the play by play in the meeting, it has given me back pocket questions that could be thrown as curveballs that I should probably have an answer prepared for. [37:17] My mind is blown right now because I can just think of so many PMs. [37:22] that stress out so much when they're asked to [37:26] take on a big problem that has executive kind of like leadership eyes on it. And so many just don't know how to approach those meetings like

[37:37] Who do I talk to? Who do I not need to talk to? What do I show up in this meeting and do? And how do I show up to this meeting really, really well? [37:45] prepared even when it's a challenging problem. And you've created this kind of like project that just says, here's my problem. Here are all the people. [37:53] And it's given you a schedule and an agenda to follow in these conversations, which is [38:00] Super helpful and would help PMs not only save a lot of time and prep for these meetings, but actually show up just more prepared. [38:06] and more confident. I agree completely. It has been extremely helpful. And yeah, I'm actually also even happy to share this publicly if it's helpful to folks. [38:16] People are definitely going to want it. So post in the comments if you would like this publicly shared so you too can prepare. [38:22] for your meeting where all your stakeholders disagree. Well, this has been [38:27] So amazing. I love that you both showed us building something very technical that solved a really complex and expensive problem. [38:34] problem, but also something really [38:35] practical that brings a lot of like heart and experience and strategy. [38:40] to a human problem that feels like it doesn't have a structured solution to it, but in a lot of times it does have a structured solution. [38:47] solution to it. So we're gonna [38:49] hit you up with some lightning round questions, and then I will get you back to whatever you are building next or your really tough meetings that you're very prepared for. So I think the first thing I would say is so many of our listeners are in [39:02] software and technology, but what you're showing is that the life sciences

[39:07] biotech, hard sciences have so much to benefit from [39:12] embracing AI. So what would you say to your peers in life sciences, [39:18] who have maybe been doing something a very specific way for a long time about how they can adopt AI and how it can transform the impact that can have on the world. [39:27] The thing that I have been doing quite a bit with my stakeholders and I have been encouraging folks as well, my peers to work on is figuring out where they spend majority of their time. [39:42] and which things [39:44] which tasks do they actually want to focus on and which tasks do they actually want to dedicate. So what are energy... [39:51] sort of maximizing tasks and what are actually a drain on their energy and time. And [39:57] now you have access to AI and [40:01] You can take a plot project and create one for any of the use cases and tasks that you don't really want to spend time on. So I think the best thing about this is it's not just for individuals, but... [40:14] you can make GPT's and projects available to all of your stakeholders and all of them can benefit from it and really enjoy the perks of your great thinking. So I would encourage people in life sciences as well as any folks who are interested in to play around with these tools and not be afraid of actually trying to optimize their own work because they might find it's a common thing for people.

[40:37] all of their peers in the process. [40:40] So speaking of being a little bit afraid, I mean, you work in a highly regulated industry with [40:47] you know, patient data, you have to have the highest ethics. You really the burden of accuracy is so important. The burden of privacy is so important. How do you think about safety, privacy, alignment? It's not a problem that I think a lot of the kind of like vibe coders or PMs and software companies really have to grapple with in their day to day. But you do. And I'm just curious. [41:10] how you approach it, what's your framework, where does it stack rank in your priorities? Absolutely. I think with most regulated industries, I've worked in finance before and in healthcare, [41:23] I also currently work with family wellness, where there are teenagers and kids involved. Compliance and privacy comes first. [41:32] But it's a bit more than privacy and compliance. It's the ethics overall that one must prioritize. [41:40] As it relates to LLMs, I think looking at the benchmark data of which LLMs are performing higher on ethics and what edge cases have been observed in terms of safety is... [41:53] sort of the priority zero of a product leader in the process because [41:59] Of course, technical leaders, engineers can make decisions on which models to use when. But I think the product leaders can provide that lens of

[42:10] hey, this model has shown for these kind of populations, has shown not so promising results in terms of ethics and alignment with human values. How might we shift the process and how might we avoid using some solutions that might not benefit the end consumer? [42:27] The other aspect of this is also in particular with safety alignment, it's important to identify how you would evaluate your solutions both [42:38] offline as you're developing it but also in production. So the online performance monitoring of it. And that's also an area where product leaders are [42:47] are particularly helpful because they can provide both and share like here, here are my like hundred things I want to know that the model does well on when it comes to safety. If it's not doing well, let's do a rollback. Let's find no quick solution. And oftentimes as, as you know, in tech, things move extremely fast. We're also rolling very quickly. And when we're operating one click faster than than usually we do. So [43:12] I think just like taking that strategic view is extremely important. And those are some practical ways that I would suggest perhaps folks who are using LLMs could apply. [43:23] Well, you're setting the bar, and I'm really glad that you are one of the product leaders out there that are building these kinds of products because it's clear you take it seriously. [43:32] Okay, last question. Have to ask everybody when AI is not listening, when it gives you terrible strategic communication advice,

[43:42] What's your tactic? How do you get the LLM to listen? [43:45] Many emojis. It does really well with emojis. And it also seems to adjust when you [43:56] put in emotion, I literally tell Claude, "Claude, I'm sad, I have a bug" and it immediately fixes everything. [44:03] I do love the reasoning models that expose their reasoning because I just love reading like in the sidebar of cursor like, [44:11] user is extremely frustrated and disappointed in my response. I understand why they're so sad. So, okay, emojis, emotions and XML, that's going to be the tagline of [44:24] of this episode. Prerna, it was so great to talk to you. Where can we find you? How can we be helpful? [44:29] You can find me on LinkedIn and I'm happy to connect with folks. I'm also happy to share some of the [44:34] tools I've developed and how you can help me just stay in touch and continue to engage and talk about all the cool things you're building. Amazing. Well, thank you so much. Thank you. Thanks so much for watching. If you enjoyed the show, please like and subscribe here on YouTube or even better, leave us a comment with your thoughts. [44:52] You can also find this podcast on Apple Podcasts, Spotify, or your favorite podcast app. Please consider leaving us a rating and review, which will help others find the show. You can see all our episodes and learn more about the show at howiaipod.com. [45:09] See you next time.