Building AI That Builds Itself - Ep. 35 with Yohei Nakajima

[00:00] Building baby AGI does feel like a self-reflection process, trying to get it to figure out how to learn to do new things. And then simultaneously have three kids, seven, five, and two. I'm trying to do that with bio-agents at the same time. That's one way to put it. They're better prepared. They don't need me as much. Which models give us this new metaphor by which to understand how our brains work? I already hear people saying like, that's not in my training data. Oh, sorry. I just hallucinated that. Yeah, yeah, yeah, yeah. [00:30] run and test an autonomous robot society. Did you see there's this thing I ran into this like random Twitter thread the other day where there's some discord apparently for AI researchers where they've like let loose a bunch of bots. Marc Andreessen like staked them like 50k to try to escape or something like that. [00:59] Yohei, welcome to the show. Thank you. [01:02] Good to see you again. [01:03] Good to see you too. For people who don't know you, you are the general partner of Untappd Capital. But maybe more importantly, you're like one of the coolest online AI tinkerers in the whole like AI tinkering space. I just feel like every day I go on X and I see you releasing something new. You famously built the first open source autonomous agent, Baby AGI, I think about a year ago. Yeah, last March. [01:27] In the last March, I did an interview with you on every, I think around that time too. And you just have this, an incredible array of tools that you've built yourself to make your work and your life better using AI. And I'm just like really excited to have you on the show to talk about tinkering.

[01:41] Thank you. You're too kind. [01:44] I'm just really lazy and whenever I'm working on something that I don't like doing, I'm always asking myself, like, how can I cut this out of my work? [01:52] So even before AI, I was a pretty heavy no-code Zapier user, but then LLM is just unlocked so much. It's been so fun to just tackle one task at a time, try to remove it. [02:04] Yeah, it's really awesome. I feel like the arc that you've been on is you started off with this baby AGI autonomous agent thing, and that honestly kicked off this hype wave about agents. So talk to us about baby AGI and then tell us about the arc that you've been on since then and sort of what you're building and what you're thinking about and where you think the future of this kind of thing is going. [02:25] Yeah, so Baby AGI introduced essentially the idea of looping through an LLM, having an LLM generate a task list and parsing that by code and then tackling the tasks one by one. At that point, just using an LLM. But I think the 100 lines of code, the simple pattern inspired a lot of people. I think the reason it was so popular is that everybody who saw it could think of ways that they would make it better. [02:46] And I think the simplicity is what kicked it off. Since then, you know, from our fund, I've been able to invest in a handful of companies like E2B, AutoGrid, and a few more in the agent space. So it's been incredible working with those founders on thinking through how to build more reliable agents. [03:03] Were those people explicitly inspired by Baby AGI? I think Autogrid Cognosis, I want to say was. I'm pretty sure they built something right after Baby AGI launched. Some of them were. That's so cool. It's like incepting. It's like a manifesting portfolio. It is. It's kind of like an incubator, but more like a public open source incubator where you're just like incubating ideas publicly. And simultaneously to investing in the space, meeting founders, I've been building my own Baby AGI, kind of iterating on it.

[03:32] I took the first original Baby AGI and did about seven iterations last year called Baby Bee AGI, Cat AGI, Baby Deer AGI with animal names, all the way to Baby Fox AGI. And each time I was introducing a new design pattern, it was really a way for me to share ideas on what I think could make autonomous agents better. [03:50] And then this year I started from scratch with a new idea and was kind of mulling around, playing around for the first six months. And then something clicked last month and then I built out Baby AGI 2. [04:03] which is a framework for self-building autonomous agents, which has been my theme for this year. [04:08] which is the idea of an autonomous agent that can build its own capabilities to improve itself. But yeah, that's kind of where we're at. This week I released another kind of small script called DEMO. [04:17] Ditto, which is, I feel like [04:19] More similar to BabyAsia because it's a super simple 500 line script that can build multi-file apps, kind of like a little poor man's Devon. [04:29] And then last night I figured out how to incorporate that design pattern into baby AGI, so I'm pretty excited about [04:35] I guess I'm going to call it baby AGI 2.0 because [04:38] Sully on Twitter suggested I name it that. I feel like open AI's naming convention is just like polluting the whole AI world. [04:50] Okay, this is really cool. I really want to do a demo of baby AGI 2, but before we get into that, I'm going to go ahead and talk about this. [04:59] How has the advent of 01 changed?

[05:03] and like sort of better reasoning, like how has that, how have you incorporated that? How has that changed what you think is possible with these kinds of things? Yeah. Tell me about that. [05:12] So I've been coding with AI since DaVinci 002, which was very poor. It went to 003. [05:19] and so on. [05:21] every time the model gets better, [05:23] my projects get more complex. [05:25] Mm. [05:26] because they can handle more. [05:28] With O1 Preview, I find that it's incredible handling [05:32] multi-file edits. [05:34] Until 01, I never really worked on any multi-file projects. So when I released BBAJ2, which was a full framework with a front end as well, I don't think I could have done that without 01 preview, at least not in the time I allocate to building. And what about the capabilities of these agents? [05:51] The agents themselves are also getting much better. Like the coding agent I mentioned, Ditto, [05:56] When I build, I do this thing where I often build with 3.5 turbo. [06:01] to get the framework working and then when the framework [06:04] doesn't error, I'll upgrade the model. But yeah, when I use Sonnet 3.5 on it, I mean, it does so much better at coding than-- [06:12] than any of the prior models. And have you tried putting it in 01 or are you afraid it'll take over the world? So 01 preview specifically, the way I've designed the most recent projects, I'm using tool calling, [06:24] And O1 Preview doesn't support that, so I haven't played with it. So I've only used 4.0 or Sonnet 3.0. [06:30] But given what I know about the strength of a win preview, once I...

[06:34] can integrate it, I suspect it will be even that much better. [06:38] Yeah, they're going to do tool calling, they said, by the end of the year. So hopefully we'll get to see that drop. Yeah. Cool. Can we see a demo? I want to see baby AGI too. [06:47] Cool. So baby AGI 2 is a framework. Actually, you know what? Honestly, I actually think it would make more sense to start with Ditto. So let's start with Ditto because it's... [06:55] All right, let's pull up a... [06:58] Ditto is named after the Pokemon that changes into whatever Pokemon it's facing. [07:03] Oh, sweet. And just to remind everyone, so Ditto is the self-building coding agent, right? [07:09] Yes. So think of it like the Replit agent or... [07:13] girl doesn't [07:15] I'm going to pull up Replit, which is where I build everything. As an amateur developer building autonomous agents, I find that playing in a sandbox with a stop button feels very, very safe for me. [07:27] as opposed to running it on my computer. Like the big stop button. Yeah, there's just a stop button at the top. Like if something goes wrong, if I see a whole bunch of red, I can just push stop and like Replay will stop it for me. And like that feels very safe. So I build and run everything in Replay. [07:41] That's good. I love Replit. I use it all the time, especially when we do courses and stuff. And just for building these little one-off things, especially, and sharing them with people, collaborating, it's really cool. [07:53] Yeah, it's great. So again, you see this main.py, it's a Python 500. [07:59] rows of Python, is all this, 500 rows of Python. When I run it, it'll ask me,

[08:05] um [08:06] what I want. [08:08] it just builds up a form. So I can say, you know, [08:10] So for people who are watching, basically like you've got Replit open, you pressed run. Now we have a little website that says Flask app builder and it says describe the Flask app you want to create and you're typing in a game of snake. [08:23] And then I'll just click Submit. [08:25] And I think it's amazing that you can do this with a single... [08:28] single file, right? So what it did was just created the routes, [08:32] static and template folder. It's actually going through [08:36] And building out how do we make it easier to read? [08:39] It's using some tools to create routes. [08:43] create directories, [08:45] Huh. [08:46] And then it went in and started [08:48] building out the file. So if we go into main.py, we'll see the routes. [08:54] Couple routes here. [08:56] So it's trying to... Oh, okay. So this means it's finished. So what I need to do here is I actually need to stop it and I'm going to just run this again. [09:04] I haven't done anything except ask for a game of Snake. [09:07] And when I open it up, it should be a game of Snake. [09:12] Whoa. [09:14] I'll just open it up in a new window to see if it takes keyboard controls. It does not take keyboard controls, but [09:21] I mean, it's a front-end, back-end, and it has worked previously before. [09:27] It must be in the JavaScript somewhere. [09:29] That's really interesting. Okay, so basically like you said, give me a game of snake. [09:34] And now we have like, I assume it's sort of some sort of, I don't know, HTML5. We don't really know what it's actually doing, right? But it built the JavaScript based

[09:47] Snake game. It gave me an index, it gave me a JavaScript, it gave me a styles, and it gave me a main.py, which serves the route, I guess. [09:55] And then it kind of stored all the history of how it built it right here. [10:00] And all this was done by the single file Python script. Can we make something else? Yeah. [10:06] So now to run it again, I have to go and delete everything. [10:10] it's a one-time use script you can't just like do you make copies of the yeah the way I usually use it actually I think I'm in [10:21] The the the replica one that I shared publicly, I should actually be forking this anyways. All right, let's do that. That's the right way to do it. What should we try to do? It should be simple, though, because this is not as good as replication. I've done like to do list up. [10:32] It's a pretty simple one. [10:34] It's a classic, but if you have a question. Well, you tell me if this is too complicated, but this is the app that I have in my head that I kind of want to build, but I don't have time to build. [10:42] Let's try it. [10:44] I often have people that want to do a meeting with me [10:49] Um, but I, I just like, [10:51] scheduling that meeting all the time is like really hard. And what I just want is a list of people who want to talk to me with their phone number, where when I have like 15 minutes during the day, I can just like, [11:03] Uh, I can just pick them off the list, call them and then check it off. [11:06] um i don't know how you want to express that but yeah i just want to be able to input names with phone numbers and then check off names [11:13] on a list. [11:15] for you to call. [11:17] Yeah.

[11:18] Okay. [11:19] Um... [11:25] I'll just try that. So I said, what did I say? I said, uh, [11:27] An Optotrack name and phone number of friends with a checkbox to track when I called them last, which isn't exactly, but it's close. It's about right. Yeah. [11:36] And so first created the routes, the static, the templates. This is how Python Flask. So this is a Python Flask app builder. [11:42] - Mm-hmm. [11:44] It says it created the directories. Now it's going to implement the HTML. So it has an index.html that it created. [11:52] It created the routes for it. [11:56] I guess the back end and then it's creating a CSS for it. [12:01] and then it's creating a script.js, which is right here. [12:09] And this is all powered with, it's called Sonnet? So I just updated to Sonnet 3.5, yeah. [12:15] OK. [12:16] And it's creating a friend's route. This is the backend route for tracking [12:20] friends are, so it's using just a [12:22] Dictionary is the database and it says done. [12:26] So if we close it and open it back up, [12:32] Whoa! [12:35] Add friend, there. That's so cool. So basically, so it's a track your friends app. It has, you put in your name, you put in a phone number, you can add it, and then you can press called. What happens if you, okay, so if you press call, it just changes off. So it doesn't do anything aside from that. Yeah, yeah, yeah. But I mean, it's more or less what we described in a generated one go. It's a multiple file app. And again, I think what's amazing, I mean, you know, Replit and Devin can know what's amazing is that like you can do this with, it's a single loop through an LLM with five tools.

[13:03] Well, actually, like help me understand conceptually what's happening. Like I noticed, for example, that it's basically doing different iterations. So it takes the prompt and then it goes through an iteration of trying to turn that prompt into code. But it does that like five times or six times or whatever. Like how does it know, like what are the iterations doing? So when the app starts, it sets up the Flask app. It checks to see if an index.html exists. This is what triggers the form. If it exists, it'll just serve the app. [13:33] which was the app afterwards. But if it's not, it goes to the user input form. [13:38] Hmm. [13:39] Once we get a user input, [13:41] We send it to an LLM call. I'm using light LLM, which allows me to route between OpenAI and Anthropic. [13:48] and then it decides if it wants to use a tool. And it's actually the LLM at that point deciding if it wants to use a tool. If it says yes, the tools it has is... [13:57] create directory, which is creating a folder, [14:00] create file, [14:01] And the create file tool asks for the code to go in it too. [14:06] So in using this tool, it actually just generates the code to put into the create file. If it errors out, [14:12] It can use update file to update a file. Or if it wants to make sure everything looks right, it can use fetch code to fetch a code file to review it. And then if it deems that it's complete, it'll call a task completed tool. [14:24] which actually just exits the loop. [14:26] Okay. [14:27] And then if it uses a tool that gets sent into a second LLM prompt, essentially, to come up with the response based on.

[14:35] based on the tool call. [14:37] When you use a tool call, there's just kind of a second kind of back and forth, which includes a tool call. And then I update a history. And then the history I store as an array that I feed back into the prompt. [14:47] so that it constantly knows what it's done historically, and then it kind of just loops through until it deems the task is complete. [14:54] Hmm. And is there like, I assume there's like a planning step, like the first LLM prompt that you've got up there, like it creates a plan of here's what I'm going to do. And then it checks off items on the plan or how does that work? [15:06] Yeah, so previously with BabyAsia, I had a separate planning LLM call. [15:12] But now that the token links are long enough, I kind of just include in the same prompt that loops through, like first plan out everything. I think it's in the, oh, we can pull up the prompt here. [15:20] um [15:21] Again, it's a single LLM loop, so we can just look at one prompt to basically understand the whole thing. [15:27] Before coding carefully planned out all the files, roots, templates, and static assets needed. And this is really the only line that... [15:34] But it says understand the requirements, plan the application structure, implement step by step, review and refine, ensure completeness. Do not modify main.py. That's important because then it'll stop working. And then finalize. [15:46] And then there's the kind of describes that application files must be in templates, static and routes. [15:51] root should be modular, index.html, [15:55] will be automatically served to make sure to create it, don't use placeholders. [15:58] Don't ask user for additional input. [16:02] And then it kind of describes the tools. So it's pretty simple. It's easy to improve. You can add a couple things and it'll make it work better.

[16:08] Hmm. [16:08] That's really interesting. Okay, so you made the original one, you made the next one. For you, what is driving this experimentation? This isn't your day job. I guess it's related to your day job. You could have maybe made it your day job if you wanted to. What's driving it and what's keeping you from just doing just this? [16:26] Um, [16:28] That's a good question. [16:30] I love being a VC. Early on in my career, when I got introduced to the startup ecosystem, [16:36] um, [16:37] I realized that [16:39] you know founders are my favorite people in the world and my north star has always been whatever role i can find where i can engage [16:45] often and most with founders. [16:47] And VC, I found, is a great role for that. So that's definitely what I enjoy the most as a job. [16:52] Um, [16:53] This is more like an incredibly fun hobby... [16:56] It just like, you know, the books I've read historically have been around the brain mostly. And this feels like a weird kind of... [17:04] mix between how I've been thinking about and understanding myself and the brain, combined with the current state of technology. And it's just this great mix where [17:14] where it feels like a hobby, but I also know that it helps my day job. So to some extent, you know, some VCs will learn things and write blog posts about it. [17:25] This is my version of that. [17:27] but instead of block boost, it's code. [17:30] honestly like that's differentiated i like it um anyone can write a blog post um and i guess like with uh with what you're building anyone will be able to build a tool pretty soon but i think you're you're like a little bit ahead of the curve um and also i feel like i'm good at it so i can't not work on it but yes it's uh but but but i like being a vc too much not not not be a vc i feel

[18:00] reference. [18:00] Tell me about that. You know, I mean, that was probably an accident, right? That BabyAsia, when I built BabyAsia, I wasn't actually trying to build an autonomous agent, per se. [18:10] but I was challenging myself to prototype an autonomous founder. [18:16] And that was inspired by Hustle GPT, where people were using ChatGPT as a co-founder and doing whatever ChatGPT told them to do. And I was thinking to myself, I mean, if they're doing whatever ChatGPT says, like, why can't that be an AI as well? And so that brainstorming led to, and I have the original chat request with that built baby AGI too, which I sent you. I can cue that up in the original prompt. [18:39] But that led to... [18:42] Other people noticed, when I shared a demo, other people noticed that it could do more than be an autonomous founder. [18:47] And then that led to Baby Asia. And when Baby Asia blew up, I think, you know, the loop happened to be in there and that was [18:53] that's what I became known for. So that's going to become my brand to some extent. [18:59] Should you rename your firm Loop Capital or something? [19:04] And Topps Capital is a great name. [19:07] It is a very good name. So I guess the other interesting thing to me here, which I imagine would be on people's minds, is this is not your day job. You have kids. [19:24] Like, how are you finding time to like, [19:27] go do the tinkering? Like when are you doing it and how are you able to like do it so consistently given all the other things you have going on?

[19:34] I only build at night. [19:37] Mostly weekends, sometimes weekdays after I get my kids down. [19:41] Which means it's like [19:43] 10 to 12 or 11 to 1 is like my short period. [19:47] I code in bursts. I kind of do little quick bursts and I might even do it while I'm doing dishes where I ask a one preview question and I'll go do something for five minutes, come back, copy paste it into replet, run. And if there's an error, I'll just copy paste that error into replet and just go do something else for five minutes. [20:05] And so I code like that a lot, especially combined with mobile, where I have ChatGPT and Replit on my phone. I also copy paste things back and forth through ChatGPT and mobile. And so that expands my ability. If I'm picking up the kids and I get to school five minutes early, I can sit there five minutes iterating on a project I've been building. I love that. I've actually found that too. The place where I found it is with Devin. [20:26] where, like, I don't really use Devin for, like, [20:30] big projects because it's not like ready for it yet. But if you have like a little idea that you want to test out or you have like a little feature you want to build, you can just like spin up like four different Devins and have them working on all these different things. And they're going to get [20:43] stuck or they're not going to like they're going to need your input every once in a while. [20:49] But it's the kind of attention, it needs the kind of attention where it's like, you know, like when a coworker like pings you on Slack and they're like, hey, can you like look at this for a second? And you can just like pop in and be like, here's what to do. And then you like pop out. [21:01] it turns coding into a task that only requires that level of attention, which is like a totally different thing than it used to need to be, which is like you're like in a deep flow and you're like there's no distractions for like several hours or whatever. And yeah, it sounds like that's how you've been able at least to some degree to like incorporate this into your life is like you can build with fragmented attention now.

[21:24] You can be a manager, right? Like I choose when my code needs me. [21:28] I can be gone for 20 minutes and I'll come back and you'll still be exactly where I thought you'd be. I'm going to ask you to do something else and then you're going to do it while I go do something else. Like it's exactly how a manager operates. It's fun and weird. [21:41] You know, one thing I wanted to bring up, I think relevant to kind of the whole podcast and how I view these tools is I often think about. [21:51] like tools as an extension of oneself. But I think there's been like studies on how, you know, if you if you take somebody who's been, you know, who's been using a hammer for their whole life and you scan their brain while they're using it, they almost treat it as part of their extension. And I think we do that with many tools we use regularly. [22:08] And I think when we're applying for jobs and we say we can do something, we're talking about what we can do, assuming we have access to the technologies that we're used to using. [22:16] And so I often do think of these AI tools that I'm building as an extension of myself. [22:22] And so when I'm working on these tools, to some extent, I am also thinking of it as like working on myself because the tools I use are an extension of me. [22:30] That's really interesting. And how have these tools changed yourself? Well, I can do more in parallel, right? I have an AI due diligence tool I use called Wakello that will generate 20 to 40 page industry reports for me in 30 minutes. [22:44] So if there's an industry I want to learn about or a company I want to dig in on, I can ask it to go do something. [22:49] and then go work on something else. And then 30 minutes later, I can pull up this report and scan through it.

[22:55] And yes, I am using an external tool, but if I think about, you know, I just think about how much can I personally get done, then having AI do things in parallel to you is just an extension of what I'm capable of doing. [23:07] of doing. [23:08] Right. That makes total sense. I think what I'm asking about is like, okay, so Yohei today, [23:15] because you can parallelize all these tasks and you can get work done with fragmented detention, is capable [23:22] is like, you're a more powerful person [23:24] yo-hey, to get done what you need to do in the world versus who you were maybe three years ago, even though maybe neurologically or biologically, not that much has changed. [23:35] The only thing that's changed is your ability to interface with these tools that are now [23:40] around and obviously like interfacing with tools changes your brain, like all that kind of stuff. What I'm trying to get at is if I told you like four years ago before any of this stuff happened, like, [23:50] you're going to be able to like... [23:52] have these powers where you're like parallelizing all your tasks, you can get way more done. I think you would probably have a, um, [23:59] an idea of what that would feel like and what it would be to be that kind of person. And I'm wondering, like, [24:04] how that feels to you now that you're now that you're like in it and it's you it's like part of your daily day-to-day reality uh it feels on one hand it feels extremely empowering [24:16] Because I'm just getting more done. [24:18] On the other hand, it feels overwhelming. [24:21] because my brain is not used to the throughput of information and tasks that I'm getting done.

[24:28] So I do spend a lot of time like slowing down and rethinking or organizing just because like I can't I can't just impeller to do too many things if I can't follow up correctly. So there's a lot of balancing there. [24:39] So those are, I think, the two kind of like opposing ends of how it feels. [24:43] You said that this is like for you working on this stuff is like working on yourself. Yes. What like what are you trying to change about yourself? Well. [24:54] I guess in this case, specific to baby Asia and this autonomous agent, I feel like we're close to being able to build something that can truly be helpful, remember everything we're working on, be able to handle a lot of my new tasks and truly be like an assistant that can help me with everything I need to work on. If I can get that working, I feel like I can really increase my throughput. [25:15] significant. And then all the patterns related to making it work, I think are relevant to places, spaces I should invest. So there's also that benefit on the work side. Right now, it's not going to directly answer your question, but a lot of the ways I think about building is looking at [25:28] when it can't do something [25:29] thinking about how I would have solved it. [25:32] trying to abstract the highest level pattern so it's not too specific and seeing if I can [25:36] if I can provide that into the prompt or system architecture so that it won't run into the same issue in the future. And so a lot of it is like watching something do something. And then if it doesn't work, really thinking through how I would do it. [25:49] and then trying to extract that pattern and then feeding it back into it. But through that, I'm also just understanding myself better because I have to reflect on, like, why do I solve it this way when this doesn't? Yeah. Do you have an example of...

[26:01] Like an interesting pattern. These are sort of like meta skills, like an interesting meta skill that you've learned about yourself from trying to do a task that, you know, an AI was failing on. [26:12] Okay. [26:12] This seems kind of obvious, but... [26:15] In earlier versions of Baby Asia, I started giving it tools and web search using, you know, SERP API, which is a Google search tool, and then web scrape, which was, you know, go to the website and grab all the data were separate tools. [26:28] And I found it sometimes... [26:30] not working well together for whatever reason it would do a search and then it would do a search again afterwards without scraping the sites or not [26:38] What I realized was that anytime I use Google search, I will always click on a site and read it. Like the search and scrape were not two separate tools when I use it. There was actually one, they're two separate kind of tools, but they're wrapped into a larger tool, which is a... [26:54] search and scrape until I find the information I need. [26:57] So I realized that the actual tool to provide the agent was this wrapper tool around two smaller tools. [27:03] that the agent had figured out how to combine, you know, that in my case, I figured out how to combine the two together and I use it the same way. [27:09] So that was an interesting realization about how we have these kind of [27:14] Core skills that then we combine into bigger skills, which combine into bigger skills. And if you want to break down Google search into more skills, I mean, typing on keys and like clicking the moving the mouse is also skills that are... [27:29] part of, in our case, a Google search skill.

[27:32] Yeah, that's an interesting one. And I'm trying to like, I'm trying to like unpack the like meta of that, which is sort of like how you got from the problem to the solution, which is like the problem is that [27:42] the AI is trying to use two separate tools that are, that need to be connected and normally are connected, but like, [27:49] uh that's different from my experience with them where they where they're wrapped together and so what i need to do is wrap them together [27:56] I think I know where you're going. And I think, in this example, I didn't do it, but... [28:01] When we figure out how two tools work together, it might take a couple of iterations, but once we figure out how to get it working together, [28:09] we can repeat that process easily. Like we learn from our earlier iterations and that wasn't happening properly. [28:16] um, [28:17] And I think that was really the issue, right? Ideally, I wouldn't have to wrap it myself. [28:21] But it would figure out when it like it would figure out on its own. Hey, actually, when we do a search, we need to scrape afterwards. And when we do that, we complete the task. Therefore, in the future, we will do it that way. [28:32] Yeah. [28:32] yeah that's that's the thing as i'm as i'm trying to get at is like a skill like that it's actually really hard to put it into words like what that is because any way that you put it into words is too [28:45] Does that make sense? Like it's a... [28:47] it's almost like some sub symbolic active like intuition or reason that can't be like [28:54] It's hard to logic it. [28:56] Yeah. And it and therefore it's like kind of hard to like put it directly into a model, but seems to maybe like arise as the model scales. Do you know what I'm getting at? Am I being crazy? Yeah, I mean...

[29:10] If you're talking about the model, I do think there's going to be much more... [29:15] kind of constant fine tuning of models as we use them and eventually i feel like that's going to get personalized where whatever ai system you're using your usage of it is going to fine tune the underlying models um that that the ai system is using but but when i use it it's going to be fine tuning a separate model so that [29:33] I feel like that seems to me like a pattern that would come out. [29:39] Yeah. [29:39] Interesting. No, I like that. I mean, this is exactly why I like AI stuff, or it's one of the reasons why I really like it is... [29:50] I think they're very good mirrors for yourself. And that happens on different levels. On one level, it's just really good at being like, here's what I see you saying. And here's what your personality is like. And all that stuff. It actually reflects back facts to you in this really nice way or perspectives. But then the other way that I think it's really interesting [30:20] metaphors for like who we are and how our how our minds work so like one of my favorite examples is like you know plato sort of likened the mind to a wax tablet where memories were like things that were inscribed on the wax and that's because wax tablets were like all the rage back in like 400 bc another really good one is a lot of early psychoanalysis like freud time stuff when you talk about like repressed emotions the model is like if you repress your emotions enough that

[30:50] there's like pressure and the pressure like, [30:52] kind of comes out. And that's based on like pneumatics. It's steam engine was like the metaphor. And I think even today, like for pre AI computing, we have a lot of metaphors that we sort of liken our minds to like those kinds of computers. So it's like, I don't know, like I ran out, I didn't have enough bandwidth. I crashed, you know, I had a little bug. And I think that language models give us this new metaphor by which to understand how our brains work. [31:22] oh, that's not in my training data, you know, or like, I, I, I'm just like, sorry, I just hallucinated that. Yeah. Yeah. Yeah. Yeah. Like all that kind of stuff. And, and the reason I love that, the reason I think that's such a cool, um, [31:34] cool thing is that, um, [31:38] previous iterations of those metaphors, which were like really scientific, like computers or the steam engine or whatever, they're, they're typically like really, uh, [31:50] uh, [31:51] really uh rational like really like the reason we like computers is because they're very like step by step um they do exactly what you tell them to for better and i think like language models are the first technology we've ever created that's not like that and that um by by the its very nature operates in this like way that's almost like it's sort of like human intuition where you can't really totally explain it or totally predict it um [32:19] And but for the first time, having a metaphor for that that's like out there, that's a tool, I think might help us understand more about like how our own intuition works and like why our own intuition is like a really valuable partner to our rational minds. So that's like my hope. And that's that's how I see it has sort of changed my own perspective. I think so. I agree with that. I feel like, you know, building Baby Asia does feel like a self-reflection process.

[32:47] I'm trying to get it to figure out how to learn to do new things. And then simultaneously have three kids, seven, five, and two. I'm trying to do that with bio agents at the same time. [32:58] That's one way to put it. They're better prepared. They don't need me as much, but there's interesting parallels. Yeah. [33:09] Are they using this stuff yet? [33:11] Not agents, but we play with AI together. Early on, we started doing Dolly sessions. [33:20] where specifically with the kids, we would all come up with a theme or something we'd want to see in the picture. And we'd combine that into one image. [33:29] Mm. [33:31] So it could be like unicorn, ninja, rainbow would be the three different words that the three kids throw out. And then we would get one image that combines this way. So that was fun. And then we started doing that with Suno as well. [33:41] So for music, so they'll request music and I'll have them each come up with some topics and then we'll generate a song that combines them. [33:48] What do they tend to gravitate to? [33:50] unicorn princesses [33:52] Zombies. Ninjas. [33:57] I love it. Your kids sound awesome. [34:01] I think like, because one of my questions, I don't have kids, but I have a nephew. He's two. One of my questions is like, [34:10] And if we're on the topic of changing self, we're mostly baked as people. Obviously, there's neuroplasticity. You change to some degree or whatever. But kids are way different.

[34:21] And I'm always kind of interested in... [34:25] what it will be like to grow up in a world that's like this, like where any question can get answered like immediately in the way that you want it. You know, it's just it seems very different. And I feel like parents are kind of at the bleeding edge of like watching that happen. Like, what have you observed? [34:41] I, you know, I don't hand it off to them. So it is very like when we play with AI, especially, I mean, it's usually me holding it. [34:48] and like guiding the experience. [34:51] So I don't know if I have too much to say to that. But I do have... [34:57] I remember talking to some parents who were talking about like that they took the Alexa away. And I've heard this, I'll read this elsewhere too, just because the kids using Alexa just got, we're so used to just making commands. And it was just like a bad habit to pick up where like they don't say please, obviously, you know, they don't say please. They just ask for things and get it. [35:14] And they felt like it was not the greatest thing. [35:16] That's so interesting. They got to make an Alexa that only responds if you say, please. Yeah. For a kid's AI, you should do that, right? The kids have to be polite. They should be, you know. And you said that there's like, to you, there's parallels between, you know, building kind of baby AGIs and children. And I'm kind of, I'm curious, like what the, yeah, what the parallels are, like what have, where are the overlaps or what have you learned that's sort of similar or maybe different? [35:45] That's an interesting question. I don't know if I've actually thought about it that much. [35:49] Kids feel much more unpredictable, as unpredictable as LLMs are. Kids just feel more unpredictable. We have three kids, but they're all completely different. So there's some base that's very, very different.

[36:02] But there is the... [36:04] I mean, it's less parallels, but I guess it's more me trying to learn from my kids and seeing how that would apply to baby AGI. So I'm seeing information repetitively becomes something that, you know, if you see two things that often together, those things are more strongly correlated in your mind. Right. And we see that all the time with kids. [36:23] With the current baby Asia, that's not necessarily the case. [36:26] So how do we bake that in? So that leads me, that kind of thinking led me to thinking of graph databases as a good data structure for storing knowledge, because then you can start adding weights to edges or relationship between objects. So it's not like a direct parallel, but it's an example of like something I noticed that then might turn into an idea to try within Baby AGI. [36:46] That is interesting. I mean, but it also reminds me like in the training process, like in order to train one of these things, like it has to see the same stuff over and over and over again. And that's, that's like one of the knocks on Transformers is that they're only good for things they've seen over and over again. But if you look at kids, you're like, [37:04] kids repeat stuff like [37:05] all the time, like constantly, which is kind of an interesting thing. And I think your point about like the chaos of kids is an interesting one too. Like it makes me wonder if we're not going to get [37:20] AGI until like AIs are like allowed to just like try lots of dumb stuff and just like they don't have they're not allowed to be like curious in the way that kids are curious where they'll just like do something completely different.

[37:33] ridiculous. You give them an object and they'll do something that you'd never thought could be done with that object. [37:38] Um, and, and that seems like it, it's such an, um, like, uh, it's such an important part of, [37:45] becoming intelligent that i don't think ais are like doing right now i don't know um what do you totally random but related idea i would love to see an island somewhere where we can like run and test autonomous like an autonomous robot society [37:59] Did you see there's this thing? I ran into this random Twitter thread the other day where there's some discord apparently for AI researchers. [38:10] where they've like let loose a bunch of bots like ai bots on the discord and you can just watch them all interacting yeah [38:18] Is that the truth terminal goat nonsense? I think so, yeah. I'm not fully following it, but it's... Yeah, yeah, yeah. They started mimicking GoatSea or whatever that guy from the 2000s was. [38:32] And then Marc Andreessen... There's a meme coin with some large value where the AI holds a big wallet. I mean, I'm sure it's orchestrated to some extent on the back end, but it's pretty fascinating. But Marc Andreessen staked them 50K to try to escape or something like that. [38:48] I don't know. We'll put it in the show notes. But like that kind of thing is, I mean, it's wild, but it's also, I think that's really interesting. And I do wonder like, right now, for example, baby GI, like it has a max number of iterations on the loop. And it has a specific goal that it's trying to like achieve. And I do wonder, like, to the extent that you just start allowing these things to just loop forever, if they start doing things that in on the 10,000th iteration that you didn't think were really necessarily possible, you know,

[39:15] Yeah, actually the first baby UGI did not have a max iterations. I mean, that was probably the biggest criticism of it is like I had to press stop and replicate to get it to stop because it would just keep trying to think of things to do. [39:25] um [39:26] But it wasn't ready because it couldn't take actions. But this new one, you want to see BabyAJ20? Might as well do another demo. I definitely do. [39:33] This is... [39:35] This was inspired by Ditto, which I shared two days ago, and then people commented on it, asking if the tools were dynamic. And I was like, no, but that's a great idea. [39:45] And then last night I was in bed about to go to sleep and I was like, I got to try this. And I like snuck downstairs and tried it. And it worked. [39:54] So, [39:55] This is a single LLM loop with, I think, three tools. [40:02] create or update tool, install package, and task completed. [40:06] And so I think what I tested was scrape, tech meme, [40:13] And [40:14] Tell me what you find. We'll start with that, and then I'll let you... [40:18] Thank you. [40:19] Okay, so basically like, so let me just set the stage here. So you've got this new version of Baby AGI, Baby AGI 2.0. [40:26] And, and baby AGI 2.0 has three tools, creator update tool. So it has a tool that allows it to make a tool. [40:34] So that's the sort of recursive part of it. And if you just scroll up for a second, [40:39] Yep. [40:41] Um... [40:43] And then it has install package, so it can get software packages, and then task completed, so it can check off tasks that it's given itself. So task completed is to say when the entire task is completed.

[40:54] Okay, got it. Okay, that makes sense. And so you gave it a task, describe the task you want to complete, scrape tech meme and tell me what you find. [41:02] And then it basically ran. - The first thing it did was, yes, was calling tool create or update tool, [41:07] is the tool, CreateUpDetail is what it's called, and it tried to create a tool called Scrape Tech Meme. [41:13] Hmm. [41:13] and it registered the tool and it created it. [41:16] And then in iteration two, it tried using the tool, but the result was empty. [41:20] So it said it didn't go well. Let me try to adjust the tool. [41:23] So use creator update tool to update Scrape Tech Meme. [41:28] and it still didn't work the second time. And then iteration five, [41:35] It did it again, but this time it seemed like it worked. [41:37] So scrape technique with ours. So this one actually, there's a result of scripts, scrape technique iteration six. It did it. [41:45] fix itself. [41:48] And then in iteration seven, [41:50] It summarized it for me. [41:53] That's really cool. And what's fascinating is that scraping tool, I did not give it to it. It wrote its own scraping tool and then rewrote it twice until it worked. [42:04] That's really cool. [42:06] That's the sort of like meta thing that I think is so interesting is like, um, [42:11] It seems like the first version had a set of tools it could use, and this one you're just like... [42:17] The tool is you can make a tool. How meta can you make it, right? And the more meta you make it, the simpler it becomes. You know what's fascinating? Actually, earlier, it can actually build floscopes because it actually...

[42:29] initiates the flask up and accidentally runs it, but [42:33] Create a [42:35] folder, [42:37] and build a to-do list. [42:40] with HTML, CSS, JavaScript. [42:44] So this is a similar... [42:46] request to the earlier ditto. Again, that was twice as many lines of code because I had to give it specific tools to create a directory, create a file. [42:53] What I believe this will do is it'll create its own created directory, create a file skills. [42:59] Hmm. [43:00] Interesting. [43:01] It actually didn't, it didn't generate a folder, but it is creating, creating the different content, but it does, it does similar stuff. So this can give itself. It didn't work it, but yes, it does. It can give itself its own tools. Okay. What about a thing that can make its own AGI, like make its own agent? [43:18] Like, isn't that sort of the next level of meta? Or are we already there? [43:24] No, I think so. I think like how do you get it, getting it to self-improve itself is another one, I think. [43:31] You know, right now, when I ask it to do something, it creates tools, it uses it, but then it throws those tools away. [43:38] um which isn't the right way to do it the right way to do it is if it creates a tool and it works like in the case of techme and by the third time it worked we should be storing the one that worked right and then reusing it next time there's a similar request and not only that there should be like a you know a public ai tool library that any other version of this can can scrape from and get get tools that have worked for anyone ever

[44:01] - Exactly. [44:02] which actually does take me to, I think it's personal baby AGI, baby AGI 2. This is not 2.0, but 2, which is a full framework. [44:12] which I had mentioned earlier. So 2.0 was the little side project inspired by Ditto. [44:17] But because of what you just said, it makes sense to show 2.0 at this point. [44:21] So 2.0 is a framework for storing and executing functions from a database. [44:28] So it comes with a dashboard, which is more for management. You don't need to actually use it, but it comes with a whole bunch of functions. [44:34] to actually interact with the actual database, such as kind of add function, create function, display functions. [44:40] And so these are all functions that can be used as tools by an LLM. And it comes with a description, input parameters, output parameters, [44:49] um [44:49] Functions can call other functions, so we have dependencies. This also has imports and installation baked in, and you can have functions triggered by other functions, which are called triggers. [45:00] So the actual, this BBAJ framework is more about just storing the functions [45:05] um [45:07] themselves with an admin dashboard to be able to look at the code look at logs [45:13] But going back to what we said, assuming, you know, look, if you look at baby AGI 2.0, if it actually stored the function in this database, [45:21] then it would automatically, you know, [45:23] have have logs for every single function call. It'll track errors. [45:28] It could reuse code that I wrote previously. So I think my next step is going to be taking BBAGI 2.0, which I think is the right loop structure,

[45:38] but then leveraging this framework for storing and executing code functions. [45:42] I think that's really cool. The thing that I like about this is, like, if you look at the success of AI on different coding tasks, I think pre-01, it's like, you know, on coding benchmarks, it's like 20 or 30% or something like that. [46:00] Maybe it's getting better and better, but it's like, it's definitely not. [46:04] For most coding tasks, for a lot of coding tasks, it's definitely not a one-shot deal. [46:10] And in general, I think when you run them in a loop, because it's stochastic, the further down the loop you get, the more and more likely it is for just to go off the rails. [46:21] And what I think is really interesting about this is it has the property of allowing AIs to learn from experience where they only ever have to get it right once. And then once it's right once, it can do it again. [46:36] And for something like this, like, yeah, if you just let the agent run long enough in parallel enough, it's going to figure out what to do probably. And once it figures it out once, like it'll just be able to further back to it. Like that seems exciting. [46:52] Yeah. [46:53] I think so. I think that's where we can get. And just to layer on, earlier we talked about how, you know, web search and web skill are skills, but then there's another skill that wraps it. [47:02] So you'll have a lot of our skills actually depend on a whole bunch of other skills. [47:07] My Google search skill depends on my typing on a keyboard skill. Yeah. So there's a lot of skill dependencies. And so I actually realized that graphs are a fantastic way to track this. So what you're seeing here,

[47:18] is all of the functions you saw earlier [47:21] but with all the dependencies mapped out. [47:23] Right. Do you think this is the right level of the stack to solve this on where it's like all this stuff is like explicit versus like just make a smarter underlying foundation model? I don't know. I mean, ideally, I feel like the coding models are getting so good that I can see a future where I don't need this framework. And 2.0 is enough where if I ask it to generate skills on the fly, it always just generate the right skill. [47:53] run inference to generate code every single time when you have the option of storing existing code. That works. [48:00] Right. [48:01] That's interesting. [48:03] So from a cost perspective too, I do think a framework like this is still helpful. [48:08] Right, right, right, right. And it'll be faster, right? Because you can skip the time it takes to write the code, you can skip the potential error that it might happen, [48:15] And it should make it faster and cheaper if you can give it good memory. What it reminds me of is in the 50s through the 70s, in the sort of first wave of AI, in the symbolic AI, a lot of those approaches were about looking at the heuristics that people used to solve problems and then equipping a basically a reason or like a program that had the ability to like manipulate logical symbols with a set of tools that mapped to heuristics that people would use to solve problems.

[48:45] end up becoming [48:46] AIs, basically. And what they found was it could work in limited domains, but when the domain wasn't limited, the number of possibilities that it had to reason through would get so big that like [49:00] it. [49:00] It couldn't. [49:01] there wouldn't be enough computing power in the world to like ever solve the problem and that's why deep learning is really interesting because you kind of like you you sort of solve that problem because instead of going through all explicitly going through every single logical possibility you just like generate a probabilistic like guess pattern matched guess based on previous contexts that you've seen and so what i'm thinking what i'm thinking through is part of the thing that i think made symbolic ai brittle is like [49:26] having to have everything be like explicit and [49:30] When things are things are really good when they're explicit in, I think, particular circumstances, but they're but you have to like narrow in the circumstances in a way that makes being explicit possible. [49:42] I think I know what you're going for. I think I think internally you want the architecture to be as flexible as possible. [49:49] Again, at a very high level. The internal architecture has to be flexible enough that it's not designed for anything in particular, but [49:56] I think that where it makes sense to be more deterministic to some extent, right? Or deterministic is when the agent interacts with the external world, [50:05] because we have defined ways [50:08] We've agreed as a society to communicate in certain ways, right? Whether it be an API call or the size of a nut.

[50:17] that you need for a wrench. The reason we need a specific size bolt is because, not a wrench, is because we as a society has agreed that it makes sense to have standards. So we should be deterministic about the size of bolts. So I think that's where the deterministic part comes in. And I think most of the tools that I save tend to be external facing. [50:39] right like using an api and i think for those [50:43] it does make sense to be more deterministic. Again, it's so that we're not randomly coming up with ways to try to talk to each other. That doesn't work. Yeah, yeah, yeah. No, I love that. I think that makes a lot of sense. It's interesting. [50:54] Yeah. [50:59] You're making my brain go in a lot of different directions. [51:03] Yeah. Is there anything else that you wanted to cover? I think for me, like the thing that I'm interested in, [51:09] in before we go is like where you see things going over the next like, I don't know, six months to a year, like obviously very, very hard to predict. [51:17] over the next like further than that um but like what are you interested in what are like what are you what i think you have a really good sense of um what is interesting now what is going to be interested so like tell me like about where you're exploring right now and where you think that'll lead in the next you know six months to a year yeah i mean i think today um [51:37] Um, [51:39] From a business standpoint, [51:41] Right. There's so many low hanging fruits on things that can be automated.

[51:47] that just we have it. [51:49] I mean, there's so many businesses out there. Most businesses could probably lower costs by figuring out how to use Zapier. Well, [51:54] but they don't. [51:56] So I think there's inherent slowness in terms of like till full adoption. [52:03] I think in the meantime, there's a lot of opportunities to... [52:09] solve problems for specific people. And that tends to be more deterministic, more focused on common workflows. [52:17] And I think businesses building those are going to be able to generate revenue faster. [52:23] But I like founders who are thinking about that at the same time, thinking about how do we solve every problem for this customer in the future? So as you're building workflows, using AI to build a vertical app to solve a specific problem that maybe leverages AI, [52:38] I like founders who are thinking about building those in modular ways and constantly thinking about, hey, actually, these two parts of this workflow, if we combine it with another tool, can create another workflow that solves another big problem for this customer. And I think that feels like the right way to build businesses right now, which is, again, customer focus, problem focus, but building it in a modular way that you can eventually see treating all these tools as... [53:03] being dynamically put together by an AI when kind of agentic architecture gets more flushed out. [53:08] Do you have a specific example of someone that's doing that well right now? [53:12] I mean, well, Kelo AI, the one I mentioned, AI due diligence tool, I think they do a great job of generating reports. Most of it is deterministic in the flow, but they are building those modularly and building out tools where I can ask a specific question about a company or just maybe combine just two of the tools.

[53:30] I think Augie is another portfolio company of ours that generates kind of videos from scratch. I can give it a prompt. [53:38] It'll generate the transcript for the video. [53:40] And if I pick a voice, it'll generate the voice for it. It'll chunk the transcript into sections. And then for each section, we'll either go find a video clip or generate a video clip and then stitch it all together and basically provide a video for you. I mean, that's an incredible workflow, but it is very much like a deterministic right now. [53:59] But I think it's another example, like they're very aware that, you know, if you're generating, that you might jump into the workflow, maybe you already have a transcript or maybe you already have a media library that's your own. And so building them modularly allows them to kind of dynamically create more and more tools and functionalities. [54:17] uh that ai can put together for them that makes sense when you're evaluating companies like this in this space [54:23] Like, how are you thinking about investing in companies [54:27] I assume they're either at the application layer or they're sort of like at the application layer and sort of in the like infrastructure layer. How do you think about who's going to win in markets where like, [54:38] the underlying technology is moving like so quickly and like whether or not the, the foundation model, for example, should be doing that job versus like the, you know, end user application or whatever. Like, yeah. How do you, how do you think about, about the competitive landscape? [54:52] in that way. I mean, the space is moving really fast, so... [54:55] My thought around this is constantly fluctuating. One of the ways I've approached it is...

[55:02] And just for context, we're pre-seed investors. We tend to invest very early. I tend to even invest sub 10 million in valuation. [55:10] One of the things I've figured out for us as a pre-seed investor... [55:15] and thinking about where the right pre-seed bets are, I have found that the most obvious ideas, [55:21] aren't don't seem too attractive for me [55:24] And the reason is, if it's an obvious idea, especially now with all this attention on AI, I mean, a coding agent is a great example. It's hard for me to invest in that space. It's almost too obvious an idea that I expect there's going to be a lot of great teams building it. As a matter of fact, there's going to be some experienced founders who are going to go and raise a $5-10 million seed round to start companies, and there's going to be more than one of those. [55:44] And on top of that, you have the risk of a big model companies also tackling the space if it's too obvious enough idea. So for us. [55:53] We tend to invest a little bit outside what I would say the obvious areas, maybe a little bit more forward thinking where not everyone's tackled it yet or not really thinking about it quite yet or slightly niche ideas. [56:05] Yeah. Is there anything else that you wanted to talk about before we before we end? [56:10] I mean, we can do this again. There's plenty of always fun stuff to talk about. But no, this was, I think, this is great. I had a great time. I really appreciate you coming on to show us all this stuff. Where can people find you if they want to see more of your projects? Thank you. Probably the best place to find me is on Twitter or X, Yohei Nakajima. [56:30] I do have a build in public log, which is like my blog of stuff I build at yohei.me. That's M-E. And then if you're actually very specifically interested in...

[56:41] So, autonomous agents, I will soon be announcing an autonomous agent-specific rolling fund. So, that'll be a fun one to track. So, keep an eye out for that one. [56:53] That's awesome. Well, you know, I will be keeping an eye on myself. Awesome. [56:59] Thank you so much for doing this with me. It's always a pleasure to talk to you. And I can't wait to we should do another one of these when 01 gets tool calling. I would love to see you. I would love to see where baby AGI goes at that point. [57:12] Awesome. [57:13] Let's do it. Cool. [57:14] Have a good one. [57:15] You too. [57:24] Oh my gosh, folks. You absolutely, positively have to smash that like button and subscribe to AI&I. Why? Because this show is the epitome of awesomeness. It's like finding a treasure chest in your backyard. But instead of gold, it's filled with pure, unadulterated knowledge bombs about chat GPT. [57:47] on the edge of your seat. [57:48] craving for more. It's not just a show. It's a journey into the future with Dan Shipper as the captain of the spaceship. [57:56] So do yourself a favor, hit like, smash subscribe, and strap in for the ride of your life. [58:01] And now, without any further ado, let me just say, Dan, I'm absolutely hopelessly in love with you.