# Generated at: 2025-07-24T15:45:09.773Z
# File: /content/learn/dev/ai_show_tell/arjun.mdx
---
title: Arjun, CEO Of ResiDesk
description: Hacking legacy systems with AI, saving 30+ man-hours a week with clever automations, and why overthinking AI can be your biggest downfall
date: 2024-11-12
---
[Arjun](https://x.com/arj_shiv), CEO of [Residesk](https://hello.theresidesk.com/), shares his insights on implementing AI workflows in a business context. Residesk powers over 1,000,000 AI-driven chats per month, with AI integrated seamlessly into various aspects of their product.
## Insights
### Key Takeaways
* **Model Strategy:** Residesk uses a best-of-breed approach, leveraging different models for specific tasks. They favor Gemini 1.5 Flash for its cost-effectiveness and latency advantages. OpenAI models excel at analysis, Anthropic models are preferred for writing, and Gemini models are best for information retrieval ("detectives"). They use Claude to generate code for easy model switching and testing.
* **Prompt Strategy:** Residesk relies on Anthropic's prompt generator (Claude) and structures prompts using XML for cross-model compatibility. They prioritize stuffing context into prompts, anticipating future improvements in prompt caching and batching. Prompt caching yields approximately 50% savings, with OpenAI's implementation proving more reliable than Anthropic's.
* **Human-in-the-Loop Approach:** Residesk views AI as a tool to enhance workflows rather than fully automate them. They aim to shorten the time it takes to complete tasks, not eliminate human involvement entirely. This allows for faster iteration and a higher margin of error.
* **Retool + Commandbar:** Residest utilizes Command Bar (now Command AI) for in-app shortcuts and automations, Retool for embedded applications and integrations, Front as their email client, and Make (formerly Integromat) for workflow automation and connecting AI systems. Make is highlighted as an underrated tool.
* **Customer-Facing AI:** AI is used in various behind-the-scenes processes, including lease document processing, payment reconciliation, and generating summaries of resident conversations for escalation to site teams.
* **Multiple Options Approach:** Presenting multiple AI-generated options to human agents, particularly in nuanced resident conversations, proves more effective than relying on the AI to select the single best response.
* **Risk Level Assessment:** AI is used to determine the risk level of conversations, using a five-tier framework developed through iterative refinement with an AI "sparring partner" (Claude). They prioritize avoiding false negatives over false positives in risk assessment.
* **Call Transcript Analysis:** Claude is used to analyze call transcripts and generate concise summaries and emails in a desired writing style. The "sparring partner" approach helps refine and polish these outputs.
* **Mermaid Diagrams for Workflows:** Claude is used to generate Mermaid diagrams from textual descriptions of workflows. These diagrams are then used for documentation, process analysis, and QA purposes, ensuring adherence to established procedures.
* **Unstructured to Structured Data:** A Retool application allows the team to upload unstructured data like lease documents and PDFs and convert them into structured FAQs, improving information retrieval and response generation. This has significantly increased their first-day response rate to resident inquiries.
* **Payment Processing Automation:** Make is used to automate the complex process of reconciling payments made through one-time-use credit cards sent by a third-party clearing house.
### Short Lessons Learned
* **Focus on specific use cases:** Don't try to "throw AI" at your business; identify concrete problems where AI can provide value.
* **Embrace a human-in-the-loop mindset:** AI is a tool to enhance human capabilities, not replace them entirely.
* **Iterate and refine:** AI workflows require continuous improvement and adjustment. Use feedback and data to optimize your prompts and processes.
* **Start small and scale up:** Pilot AI projects with a few key individuals and then expand based on their success.
* **Empower your team:** Give your employees the tools and resources they need to experiment with and utilize AI effectively.
## Frameworks
### Best-of-Breed Model Strategy
* **Choose horses for courses:** Select different AI models optimized for specific tasks (analysis, writing, information retrieval, etc.).
* **Evaluate and monitor:** Continuously assess model performance and adjust your strategy accordingly.
* **Flexibility and optimization:** Gain advantages over a one-size-fits-all model approach.
### Human-in-the-Loop Workflow Enhancement
* **Enhance, don't fully automate:** Use AI to improve speed and efficiency, but retain human oversight.
* **Shorten the commute:** Focus on reducing the time it takes to complete tasks, not eliminating human involvement.
* **Faster iteration:** Allows for more rapid experimentation and improvement of AI workflows.
### Bottom-Up AI Adoption Playbook
* **Empower individuals:** Give employees access to AI tools and encourage them to find relevant use cases.
* **Start small, solve concrete problems:** Focus on practical applications that resonate with individual employees.
* **Build internal champions:** Showcase early successes to inspire wider adoption and generate organic enthusiasm.
## Transcript
00:00:00 Arjun: We favor Gemini 1.5 Flash over everything else. We found that the OpenAI models tend to be the best analysts. The anthropic models tend to be the best writers and Gemini models broadly tend to be the best detectives. We trust the anthropic prompt generator by a lot, and we're making the bet that it's gonna get easier and easier with prompt caching, batching inside all of this. So everything below escalate is actually a retool app that is embedded within my actual code. So this is happening 100 and 100 of times a day for every one of the customers that we support. When you wanna ask an LLM to do something, you better have a pretty damn good idea what good looks like for yourself.
00:00:40 Arjun: Talking about how to put AI in my business is about as broad and silly as talking about how do I use Excel in my business.
00:00:58 Greg: That's Arjun, CEO of Residesk. They power over 1,000,000 AI chats per month. Their clients interact with AI in almost every single part of their products, but they don't know it. In this interview, he shows me how to create business grade prompts using Claude's workbench, how he saves 100 of hours on development using retool and command bar, and why he favors Gemini 1.5 Flash over any other model. Let's get into show and tell. So, Arjun, sent out this Twitter thread asking who are the scrappiest LLM operators I knew, and you were mentioned more than anybody else in the entire thing. So I was super excited to get on the phone with you.
00:01:34 Greg: You told me about a wild stat about ResiDesk, about how many chats you're processing with AI per month. And so I'd love to hear about that as well.
00:01:42 Arjun: Yeah. For sure. So I'm Arjun. I'm the cofounder of ResidDesk. On average today, we process about a couple of 1,000,000 conversations per month with residents over text across the US and Canada. And so each of those is being processed by our systems, LLMs that are looking at everything we know about the property to create the appropriate responses and then helping our human team in the loop to have better conversations with residents. If you ever if you've ever rented before, you know it kinda sucks because your landlord doesn't care about you. That's what we're trying to change.
00:02:16 Greg: That is so wild. So when I think of other businesses doing a couple million conversations per month, I'm thinking of the huge people that literally do chatbots as a service, like the intercoms. What's your model strategy? Like, who are you paying a ton of money to, or are you running your own open source, or how do you think about that?
00:02:33 Arjun: We're not paying a ton of money to anybody, but our models model strategy is sort of just best of breed. Is so we use a mix of models across all the closed source providers. We have some stuff that we're testing with Llama on the side. But broadly, it's a mix of, you know, OpenAI, Anthropic, and Google are the big ones. And the what we have learned right now, I would say we favor Gemini 1.5 Flash over everything else, which is how we also manage our costs and latency by by a pretty good margin. But, like, what we found is that we have a work we don't think of it as, like, one model for everything. We think about it as there's a ton of different workflows.
00:03:17 Arjun: We'll get into some of those as we talk through. But, like, we found that the OpenAI models tend to be the best analysts. Mhmm. The anthropic models tend to be the best writers, and the Gemini models broadly tend to be the best detectives, if you wanna think about it that way. Right? Like, they're great at finding needles in haystacks when looking at large contexts. And so that's our general framework. But then from a tactical standpoint, if you ask me how I did this, it is literally I had Claude spit out a bunch of code that lets me switch models on the fly by just tweaking settings in my database. And so we're always testing new models on some subset of our conversations Mhmm.
00:03:58 Arjun: And then trying to figure out what's best for what without trying to dive too much into the rabbit hole of constantly tweaking and testing what is ultimately a nondeterministic thing.
00:04:08 Greg: Sure. Well okay. And so that makes sense on the model side. The other part that's gonna get complicated very quickly is the prompt strategy. So, yes, there's the actual prompts that you do, but how do you do prompt versioning or, like, prompt testing or orchestration of your prompts? What what do you what do you all do there?
00:04:24 Arjun: The the simple answer is that we just we trust the anthropic prompt generator by by a lot. Right? So we basically have our prompt strategy is dictated by whatever Claude thinks is a good prompt. Sure. We have tons of structure in our prompt too. And there's a couple of maybe deeper learnings that we've that are fairly hard earned. I think what what we found is that if you if you can manage your token budget decently well, the anthropic strategy of everything is XML works quite well across any model. Mhmm. And then it's it's gotten much easier now because everybody offers some some form of prompt caching as well.
00:05:08 Arjun: But even before that, we were all in on XML and structured inputs and outputs. And then more broadly, I think we think about we stuff a lot of context into the prompts is, I think, the easiest way to think about it. And we're making the bet that it's gonna get easier and easier with prompt caching, batching, etcetera, to manage stuffing the prompt on the input side than to try and break things into multiple workflows. Right? Like, I I like I always like the Sam Altman thing of just try and just keep betting that the models and the interfaces are gonna get better.
00:05:42 Greg: Sure. Sure. That makes sense. Out of curiosity, have you all done the math on how much you're actually saving on prompt caching? I'm always curious to hear, like, because anthropic, they market it's up to 90%, but then in actuality, it's not actually 90%. It's more like in the 30 or 40 range I've heard from other folks. What what what are you all at?
00:05:58 Arjun: About half.
00:06:00 Greg: I mean, still, 50% is, like, pretty insane.
00:06:03 Arjun: Yeah. I would say we we see more savings, funny enough, on on GPT than we do on Claude. I think whatever OpenAI launched in the last few weeks, whatever they're doing out of the box on their back end is working much better. Yeah. Claude is we find a little bit finicky with the prompt caching, and we've we've actually seen sort of mixed results. So it's 50 on a good day, but it can be 30 on a bad day.
00:06:31 Greg: Yeah. Oh, 15. Okay. Yeah. That makes sense. Maybe it's because OpenAI, you don't have to do any extra work. You just literally send them the same prompt and they take care of it for you. But on Claude, you have to do the breakpoint, which is kind of an not annoying per se. Well, it's extra overhead, so it's annoying, but it's just more management you have to think about. Yeah. Yeah.
00:06:47 Arjun: That's right. So I think it's like it it is variable, but it's certainly not 90%. Not by a long shot.
00:06:53 Greg: Sure. Sure. Sure. Yeah. Totally. Okay.
00:06:55 Arjun: At least not yet.
00:06:56 Greg: Yeah. For sure. And so I was gonna say, one of the other things from our previous conversations that attracted me to your background a ton was that you have a unique mindset as a business owner to kind of go Sherlock Holmes AI style into your business, find different workflows that need automating and being like, actually executing against them. So before we dive into specifics, and I know we'll get into that in a second, but, like, what's the mindset that you have as you're looking at your team and you're looking at AI, and they're like, okay. I know that there's something here. How do you go find out what they need?
00:07:28 Arjun: Maybe one thing that I've noticed about me and my cofounders and just broadly at ResidDesk that I think is slightly different from other founders working with AI I've seen is we don't air about AI doing the whole thing. Right? Like, our business is built with the model of a human in the loop. Mhmm. And so we think of AI almost as shortening the commute between starting a task and finishing a task, not so much as automating something. So I am okay with, you know, if I have a 5 task 5 step workflow, I'm okay with AI doing 80% that you know, going from 10% to 90% on each of the 5 steps
00:08:06 Greg: Sure.
00:08:06 Arjun: As opposed to trying to figure out how can I automate the whole thing? And I think that frees you up. It makes you look at AI in a whole different way. It's just if this can it's just about saving me time and not so much about automating the task. And I think that's maybe a little bit different than than people looking to just automate things with AI.
00:08:28 Greg: Totally. And to take the self driving car metaphor, it's like you're cool if the FSD wants to drive on the freeways, but you'll still drive it on the cities to in in the certain parts.
00:08:37 Arjun: That's exactly right. Yeah.
00:08:39 Greg: Yeah. Yeah. Okay. Well, I love that. So let's get a little tactical here. So let's before we dive into some demos, because I know you have some really cool demos to show us, I would love to hear about what do your customers not see that's AI enabled?
00:08:54 Arjun: So everything from when we sell into a property. Right? Like, how they set us up, sending us their lease documents, and turning that into a questionnaire. What somebody else would do as a big, like, training step on the AI, They just email us, and then we take care of the rest in terms of turning it into structured data. When they pay us with we live in an industry that still pays with paper checks. Right? Like, I have AI that's I I scan the receipts in the power scanner, and then I have AI that reconciles those payments. Because if I sign a contract with Greg's property management company, I'm actually getting paid by all 40 of Greg's properties.
00:09:30 Arjun: And so I have a diffusion problem that I'm then solving on the other end. When we have conversations that we are escalating back to our customers about a something that needs to get fixed in somebody's apartment or a ticket that we create in their system. Right? They get a diagnosis, but all of that stuff is happening through AI powered workflows in the background. There's always a human component to it, which is why we have so much margin for error. That's the other thing that I think is unique about us is I can kinda go ham with AI stuff, because I know there is always a human in the loop to backstop anything I'm not doing well.
00:10:08 Arjun: And because of that, I can cycle through a bunch of iterations much faster than anybody else can.
00:10:12 Greg: Sure. Sure. And what sort of tool stack do you have that makes it so easy to do human in the loop stuff? Because that's not easy to, like, orchestrate and integrate new existing tools. And so what do y'all do?
00:10:22 Arjun: We make heavy use of the drop in workflows from tools like Command Bar, now Command AI that was recently bought by Amplitude and a bunch of retool embeddings. And then we have a lot of back end workflows that are specifically enabled by Front, which is our email client, which is then super pluggable into into make, which then plugs into all of the AI systems. That's why. So, again, for our internal teams, they're never really flipping back and forth and going to chat GPT to use something. I mean, they have the tools. They are they're encouraged too if they need it. But broadly speaking, we're baking AI into those workflows as a kind of automatic thing.
00:11:02 Greg: Yeah. Yeah. Yeah. So of those tools that you did mention between front make Mhmm. And the other 2, I'm forgetting what they were. But which one impresses you?
00:11:09 Arjun: Retool and Command Bar.
00:11:11 Greg: That's right. Retool and Command Bar. Which ones do you look at and are just like, holy cow. I'm pretty impressed with the product that they're doing, and I think that it's underrated. More people should be look looking at them.
00:11:21 Arjun: So it's very hard to say that retool is underrated even though that would be my natural answer. Like, retool is the one that I think I just wouldn't be able to ship code without. Mhmm. And command bar is not underrated. Front is not underrated, so I'm gonna go with Make. Right? Like, I think Make is Make lives way too far in Zapier's shadow. I think they're actually a far better product. It's it's insanely good at the in the ways that it plugs into things, and you can actually debug workflows that would otherwise be a nightmare to build with any other tool.
00:11:56 Greg: That's interesting. Cool.
00:11:58 Arjun: And it's like it break breaches the low code, no code barrier enough that I think a person with skills on the engineering side can use it phenomenally well, as can people who are maybe systems minded but don't have the engineering skills.
00:12:12 Greg: Sure.
00:12:12 Arjun: And I think that's that's kind of a special thing for a tool to achieve.
00:12:16 Greg: Yeah. Yeah. Yeah. Absolutely. Well, I tell you what. You mentioned some of those tools. I would love to dive in to some of those tools. And so prior to the interview here, you shared a a couple of really cool examples. What do you think about starting with the command bar retool create summary emails one? I'd love to see how you do that.
00:12:31 Arjun: Yeah. Absolutely. So I'm gonna go ahead and share my screen here.
00:12:34 Greg: Beautiful.
00:12:35 Arjun: In a prior world, before we had these integrations, what would happen is we would basically copy this conversation, put it in a transcript Mhmm. Write a summary email, open our email client, and then send it to the team that actually works on-site. Right? And the idea is that, as I said before, the site team interacts with us as if we were a virtual extension of their team. That means we don't give them new tools. They just live in their inbox, and we we send things to their inbox when they need to look at it. Right? As you can probably imagine, bunch of context switching, bunch of lookups, lots of room for manual error.
00:13:12 Greg: Sure.
00:13:12 Arjun: And so that's we wanted to bring all of that into the context of Residash. So the first investment that I'm gonna talk about here is command bar. So command bar is for anybody who's unfamiliar. Effectively a way to replicate the power user command k functionality inside your app. So think of it as opening up a menu of functions that you can actually trigger. Right? And so I press command k here. It shows me all of the different shortcuts in my application. It can even be context dependent. And the cool thing about it is that you can create these commands both via JavaScript API as well as a GUI. So I basically just dropped you can see all of my actions here because I'm in the admin view.
00:14:01 Arjun: But I just dropped a couple lines of code into my application, and I was already able to set up command bar. You can do a lot of cool things with it that we're not gonna get into right now, but you can even create, you know, automations based on, hey. If I hit the shortcut, click this button then this button then this button. Super powerful. Right? Cool. And so we have a workflow here that just tells me I have here's how I escalate a conversation. Again, our team would actually use the shortcut. I use it much less. And one of the cool things that it does is we can then have our team make judgment calls on a couple of steps that they need to fill in before we can generate the summary.
00:14:45 Arjun: Right? So I now see this is actually a pretty high risk conversation because the resident is really upset. And if we don't fix this, they're not gonna pay their rent on time, which is then a big issue. Right? So I'm gonna say charges on rent. And I'm gonna highlight a couple messages that they talked about. And
00:15:07 Greg: Oh, cool. So it's actually going from the chat and it's it's actually grabbing all that information so you can pick what's what's relevant and what's Correct.
00:15:15 Arjun: Correct. So because command bar is super flexible, right, I had a multistep workflow. My team is picking sort of what they wanna talk about. And then, again, because I can embed retool apps into anywhere in my code, command bar ran a bunch of steps, hit an API on my back end that goes through read through all of this, read the things that I wanted to highlight, and then give me a summary of the conversation. This opens a model. And now what you're looking at inside all of this, so everything below escalate, is actually a retool app that is embedded within my actual code. Mhmm. Right? And the cool thing about this is retool embed, I I have now passed in through React a a couple of parameters that give it context about what account this is, what customer this is, and what property this belongs to.
00:16:09 Arjun: Retool is then interfacing directly with Front, which has our contact lists and our user lists, which is how we sort of keep them in sync. Picking out the right people that they should go to, in fact, for some of our customers, we even customize this based on what the topic of the message is. So, you know, payment conversations go to payment teams. Electricity conversations go to the utility teams, whatever that might be. It is auto generating a a subject line. It's auto generating a conversation summary. And then all my team needs to do there's a picture of a cat because we're including the entire thing.
00:16:47 Arjun: Nice. But it's looking at the images. It's gonna figure out what I am trying to say and tell us, okay. Incorrect charges on the resident account blah blah blah blah blah. All of this is being generated automatically with AI. And then if I hit escalate, which I'm not gonna do now Uh-huh. This message goes straight to the site team. Again, this is a workflow that before sort of shipping all of this stuff would be copy the conversation, summarize it, put it into a new email, look up who it should go to, put that in, fill out the subject line, send it, and then log in the conversation itself that this happened.
00:17:24 Arjun: So this is happening 100 and 100 of times a day for every one of the customers that we support. Right?
00:17:31 Greg: Uh-huh.
00:17:32 Arjun: And by keeping our team in in the context of the conversation at all times, I am saving I think we estimated it on average something like 20 to 30 man hours a week.
00:17:45 Greg: Wow. Yeah. That's that's wild. That's absolutely that's that's wonderful. Yeah. What are the other top Yeah. Used command bar shortcuts on there?
00:17:57 Arjun: So we have a bunch of things. I would say the most important thing is just translating messages. We use that quite a bit as you can probably imagine. There's also quite a few other things as well. So we can do things like I'd say escalating conversations and looking at looking at common themes is a big one. And so you can sort of navigate through the inbox. So we've basically built this to be a power user app. Okay. But, again, because we have command bar and retool, you can use this with key with just your mouse and be okay. But if you're a power user, you can use it well beyond that. The other thing that we have, just like any other prompt engine, would be we have a bunch of AI use cases.
00:18:47 Arjun: Again, this is all just set up with command bar. I've set up a bunch of different templates.
00:18:51 Greg: So
00:18:52 Arjun: you can see it can do all the things that Grammarly can do for you. So, again, I have Grammarly on my machine, but our team can use it without needing to again set up a new tool on their machines.
00:19:02 Greg: That's very cool. Do you have stats on what percentage of your company is using, like, AI features within Command Bar?
00:19:09 Arjun: Yeah. I mean, it's about well, all everybody who use actually, it's a 100%. And I'll I'll explain why is all of our team has to use the Resides gap to to actually talk to residents. And so we don't think about it so much as AI features, more just that it's baked into it. Sure. And I'll explain a couple more things in a second, but, like, actually, no. I'll I'll do that right now because this is this is also a fun thing. Right? Is if you go back, and we'll get into this, I think, as we talk about, like, thinking about unstructured data and turning it into structured. Sure. I would say our team doesn't need to think too much about AI because what's happening is k.
00:19:53 Arjun: Another conversation. Here's somebody who's asking if they can make, you know, partial payments. So they don't wanna pay their rent up front. They wanna pay it in a couple installments. Right? We had, at some point, a a guideline that we scraped from the property or from one of the responses from the site team about how you can pay your rent through alternative methods. That then comes is auto populated, as I'm sure you can see here Mhmm. As a response. So just, again, we draft responses to every message as it comes in based on property context and conversation history. And, like, the way that our customers that our that our team thinks about it is we don't really even talk about things like saying AI, but it's like, okay.
00:20:40 Arjun: Let me think about a couple of different ways to respond to this conversation. I click view all responses. It's like, okay. Do you wanna be a bit more supportive? Do you wanna be a little bit more factual? Different residents require sort of different touches.
00:20:53 Greg: Sure.
00:20:53 Arjun: We try to make it basically a native part of how they think about it more so than trying to get our team to adopt new actions and responses, if that makes sense.
00:21:04 Greg: Yeah. It does make sense. And what's interesting about this is the theme of presenting a few options to the user and then having the user pick which one they want keeps on coming up, like, just in in my around me here. So, like, instead of making the AI go and guess what is the exact response that the person wants to do, just let the human take it the last mile and figure out which one. So, like, how do you how does does that make its way into other parts of your flow as well with regards to multiple options presented?
00:21:31 Arjun: Not quite. I think the you saw the 2 main ones,
00:21:34 Greg: which
00:21:34 Arjun: is the like, having a conversation with the resident is actually the place where you need the most number of judgment calls by a human. So that's where we let sort of to keep going with your FSD analogy, I think that's where we would let the driver take the wheel Sure. Just because there's so much nuance to be applied here. And while we trust our trust our models to come up with with good answers, they don't always come up with great answers unless you force them to bring in options. Yeah. So it makes the system a bit more fault tolerant in both ways. There are other places where this is not quite true. So you can imagine so you can see here.
00:22:13 Arjun: Right? Like, there is a you can see that we assign a risk level, for example, to every conversation.
00:22:19 Greg: Mhmm.
00:22:19 Arjun: This is fully determined by AI Mhmm. Where it's like, okay. This is a high risk response because it's a rent payment financial concern that requires a timely response. This is all fully generated. So I think you picked up on the theme because I'm also showing you the 2 most external facing workflows that need a human to sign off
00:22:39 Greg: Yeah.
00:22:39 Arjun: Before they go out into the world.
00:22:42 Greg: Totally.
00:22:42 Arjun: But outside of that, we do have it generate options and reasons.
00:22:46 Greg: Mhmm.
00:22:47 Arjun: And I we find that that actually works quite well.
00:22:49 Greg: Yeah. Yeah. Totally. Well, on that one specifically, using LLM as a judge more or less to grade the risk level of these things, Sometimes that's not as easy as it looks because you need to, like, encode different criteria. And if it meets criteria, then give it a certain level. What, like, what was that process like to implement that? Did was it straightforward, or did it give some hard time?
00:23:09 Arjun: It took us, I would say, a few months to actually figure it out.
00:23:14 Greg: Mhmm.
00:23:15 Arjun: I think there are a few things that that we were that we were struggling with. I think one, it's funny. Right? Like, the I I the one universal lesson slash callus is when you wanna ask an LLM to do something, you need to you better have a pretty damn good idea of what good looks like for yourself.
00:23:36 Greg: Uh-huh.
00:23:37 Arjun: You're all like, as we're like, k. What is the risk level of the conversation? And you there you can go really deep down that rabbit rabbit hole and realize that there's a different answer for everybody. And so we did a lot of we have a we have a custom GPT called sparring partner, which is basically just built to challenge every idea you throw at it.
00:23:59 Greg: Could we could we take a look at that one?
00:24:01 Arjun: Yeah.
00:24:01 Greg: That'd be fun to see. You're like
00:24:03 Arjun: Actually, I think we have the latest duration on Claude now. So let's just go
00:24:07 Greg: Well, so that that's interesting. So why move over to Claude instead of the GPT on Chow GPT?
00:24:15 Arjun: Right now, it's just because we have chosen Claude as the as the blessed tool for the team, but we move around quite a bit.
00:24:25 Greg: I'm curious. How did you tactically make the sparring partner? If it's a project, is it just like the custom instructions that come on into it?
00:24:31 Arjun: Yeah. Pretty much. So you can see here I'm trying to make a risk framework for conversations. I want it to be binary. Tell me how you think about this. And I'm oversimplifying this. Like, ours were pages and pages. Right? And it's a lot of, like, okay. Clarifying questions. What do you actually mean when you say binary? Blah blah blah. And so we went back and forth through this a bunch of times to then come up with effectively, we came up with a 5 tier framework. Right? Which was okay. Like, broadly speaking, and it's funny because you can always intuit your way back into this, but it's like, okay. There is a really urgent fire, flood, blood category, and then there's a really not urgent okay, thank you category.
00:25:24 Greg: Mhmm.
00:25:25 Arjun: And so even if you went by it's just sort of standard survey design, you would come up with 5 tiers. Then we had effectively went back and forth with the anthropic prompt generator to say, here's what a tier 1, 2, 3, 4, 5 conversation looks like.
00:25:39 Greg: Mhmm.
00:25:39 Arjun: And then it was a bunch of iterations on different models. We looked at a sample of about about 1,000 conversations to look at the distribution of scores and why. Mhmm. And then this answer, I don't think anybody's gonna like, but we eyeballed it to see if it intuitively made sense to us. And then you just gotta pick where you're willing to be wrong. Right? Like so we were okay with overfitting the risk and saying conversations were we were okay with tons of false positives. We just didn't want any false negatives. Mhmm. Because, again, if you're complaining about a leak in your kitchen sink, I wanna know about it.
00:26:17 Arjun: I'd and I'd rather be a little bit more alert
00:26:20 Greg: Mhmm.
00:26:21 Arjun: To problems that aren't actually problems.
00:26:23 Greg: Sure.
00:26:24 Arjun: So once we had that, that gave us sort of the first iteration of our risk framework.
00:26:30 Greg: Cool.
00:26:31 Arjun: And then after that, the process is pretty straightforward. Right? Like, so we generate we we look at a sample of maybe a few 1,000 conversations every week and have, well, previously, 4 o, now o one, go through it and highlight anything that doesn't feel right. And then we also have a a human on the on the engagement team who's responsible for finding conversations that are an NA on the risk scale and then giving us an opinion on what it should be.
00:27:05 Greg: Sure.
00:27:05 Arjun: And that's how we're going through it. Eventually, we see this as, and we're partly in the process of building this, a much more automated eval framework that will sort of self correct as it goes.
00:27:16 Greg: Yeah. Yeah. Yeah. Yeah. Totally.
00:27:18 Arjun: And look at the last 5 correctly and badly marked conversations.
00:27:22 Greg: Totally. Tell tell me about the shift to o one and why you chose it for that one. Because I think a lot of people are still asking, like, I know Owen's good. I know that it thinks a lot before it gives me an answer, but, like, what types of tasks do you think it's better at versus 4 o would we get at?
00:27:38 Arjun: The rule of thumb for me is is this worth overthinking? Right? Like, it's it's a lot of, like, unstructured stuff like, hey. I just had this really rough conversation with a customer today. They meet they went on about a lot of things they don't like about our process. Here's the transcript of the conversation. Here is the PDF of my website so you have context on what I do. Here's what I am worried they're actually saying. Talk to me about this. That's the kind of stuff that it's fantastic at. Right? Like, it's trying to find a little meaning in fuzziness, But I would we don't use it for anything in production right now.
00:28:27 Arjun: We find it to be a little bit too much of an overthinker for the types of process we have. And I the bias here is that we try to build processes as you've seen even with that little workflow. We try to build processes that are many small steps that you can compose together.
00:28:43 Greg: And
00:28:44 Arjun: so we get a lot of bang for the buck out of just breaking down prompts to the simplest possible thing and then giving it to the dumbest possible model.
00:28:52 Greg: Sure. Yeah. Yeah. Yeah. I love that.
00:28:54 Arjun: So it doesn't meander.
00:28:55 Greg: Yeah. That's a workflow I think a lot of people could do. Well, I tell you what. I wanna jump on to just at least a few more of these cool suggestions you had. So let's stay on the clog let's stay on the clog projects 1. So you're uploading call transcripts and voice notes and yeah. Tell me more about that.
00:29:10 Arjun: We have a project that's basically set up to our writing style. It's it's not that fancy. It's literally just a Harvard Business Review writing style of bottom line up front, and here's what you say. Mhmm. And then we upload our calls, transcripts, and then go back and forth with it. I will say mostly a lot of it is just for me to say candidly what I think after
00:29:31 Greg: a call Sure.
00:29:32 Arjun: And then have it sort of sort of polish the the thing. Right? Like, everything from, hey. You need to pay for the product or we're gonna shut off service, which I promise you I did not word as nicely as that Sure. To, hey. Like, you might have a bunch of in house things that you think could replace Residuesk, but here's why our solution is actually better.
00:29:55 Greg: That's interesting. Do do you have that project? Like, could you just show us, like, going into the project and let's see let's see that that like, even a preview of the system prompt would be really cool to see.
00:30:06 Arjun: So this is something that we did as a one time exercise and then set up as a system prompt. Awesome. So it's pretty straightforward. Right? So this is a conversation that we had with a tough prospect who was trying to build their own solution in house, which often happens. Right? Like, sometimes people are evaluating buy versus build.
00:30:27 Greg: Mhmm.
00:30:27 Arjun: And so the prompt is literally, like, write me an email, somebody who was skeptical about why our tech would beat something they did in house. Here's my draft so far. Ask me clarifying questions. I'm aiming to get a short, sweet draft that encapsulates our tech technical depth and data mode. This is literally the system prompt.
00:30:45 Greg: Uh-huh.
00:30:46 Arjun: And then at the end, I also have this, which which is I'm attaching the transcript of our call so you get a deeper handle on the issue. And then the only thing that I add to everything is ask me clarifying questions. By doing this and then throwing what I want in there, it is okay. Can you provide more details on 1, 2, 3, 4, 5, 6? And then I'm just going back and forth on it conversationally. I'm just throwing everything I have at it, which then gives me a rough email that I can use. Right? And then I'm gonna, like, how can I make these ideas hit harder? And then eventually going back and forth and, like, I don't have enough data to hit harder.
00:31:24 Arjun: Okay? Ask me clarifying questions on each point. I'll provide these insights. Go back and forth. Go back and forth. Go back and forth. And then eventually ended up with a fairly if I scroll down, a fairly quick email. And then I was like also use my tone, and then I can be like I can say something like this. Right? Pretty straightforward thing. I wouldn't do this for everything, but, like, the fact that you can just drop this in there and then think through the LLM interface is insanely good. Because then I can take the same thinking and share it with my team and have them debate it, and so you just get better off for every turn.
00:32:06 Greg: Yeah. Yeah. Yeah. That makes sense. I mean and even just having, like, a brainstorming partner and buddy with that to give you, like, a different perspective is super valuable as well.
00:32:14 Arjun: Yeah. So here's another fun piece, which is I was working on and I'm just gonna show you the output of this. But one of the things that we need to put together for some of our due diligence workflows is trying to help people understand how we integrate and when we interact with residents. Right? And so I actually worked with Claude to basically say, here's our, like, general workflows, and we have that outlined as a piece of text and then had it turn each of them into a mermaid diagram that I could then and let's find out. So this is like okay. Here's what happens. When somebody opens a ticket, we start with, is the issue reported, get the details, does it already exist, create it, update it.
00:33:06 Arjun: Then after that, we start processing it. If it happens, escalate blah blah blah, and then come back. And then when it's closed, we close it. Right? These are all fairly straightforward things that I could describe as as code by actually writing the code. Mhmm. But the ability to have a conversational interface that generates these diagrams for me and I can edit in real time was insanely helpful. Right? Like, because we went from having a due diligence request on on Monday to having a full presentation on everything on Tuesday, which is not not how things usually go.
00:33:43 Greg: No. That's not usually how it goes. With those flows and diagrams, are you training your team on how to do those and so there's, like, no ambiguity? Or what do you use those for?
00:33:51 Arjun: We can feed the mermaid diagram back into prompts to check after after the fact on a conversation if we actually follow the process. Right? And that's actually really fun because I can be like, here's the transcript, and we do this less often than we really should, but it's like, here's the process we should have followed. Here's the process as it played out in the conversation between Greg and us from day 1 to day 30. How close were we? And it's actually something that we're starting to bake into our QA process. So something we've learned, and this comes back to one of those little things that you learn about token efficiency, is if you can describe the process as mermaid code, then you get a lot of bang for the buck inside the prompt, especially when you're building like a QA workflow of how closely did we stick to what we were supposed to do.
00:34:41 Greg: Sure. Yeah. That's actually fascinating because it's traditionally very hard and very fuzzy to, like, look at a conversation of unstructured text and say, did this agent or did this person or did this employee actually follow the flow they're going to? So to have 4 o Mini or 4 o look at it and be like, yes. It yes. They did. Or Yeah. No. They didn't. Like, that seems like a really cool tool to make sure your team is staying on track.
00:35:01 Arjun: Yeah. So something we're still building out, but the fact that it it was actually fairly trivial to get a prompt to tell us if we were, you know, 60, 80, 90% on track.
00:35:13 Greg: Yeah.
00:35:13 Arjun: And we're still iterating through again, much like the risk level calibration, I think we'll end up with a 1 to 5 scale on this too.
00:35:21 Greg: Yeah.
00:35:21 Arjun: But, like, being able to look at that for a 1,000,000 conversations at scale is super interesting.
00:35:26 Greg: That's for that's that's that's absolutely wild. What about in, like, in terms of observability tools do you use? And so what are you using to, like, track your calls and costs and everything?
00:35:37 Arjun: So we do a couple of things. So, obviously, we have everything that we do with AI goes through we I just have a light middleware that is logging every request and results back into our database. I think it's kind of absolutely mandatory that the first thing that you build is just observability infrastructure with all of the AI stuff. And so I literally just have a postgres table that logs every request I've made to every provider through every pipeline. The the other thing that we use that we are just testing out is I don't know if you've heard of Gentrace.
00:36:18 Greg: Oh, yeah. Big time.
00:36:19 Arjun: So we use Gentrace quite a bit, and that's who we use to monitor costs broadly. Mhmm. So we have all of our calls going through them too. These guys are great. Know the founders love them. Nice. They also offer testing, which we don't quite use. But for costs, we use GenTrace. And then for internal monitoring, just because so much of our workflow is just conversation QA Mhmm. We just use we just use our database.
00:36:47 Greg: Yeah.
00:36:48 Arjun: So the idea behind it is that once you have, you know, basically an entire history of every AI call that you've ever made
00:36:55 Greg: Uh-huh.
00:36:56 Arjun: Then you can just look at subset of that data to feed your QA workflows or at least test them.
00:37:02 Greg: Yeah. That's wild. I love that. That's super cool.
00:37:04 Arjun: Let's maybe start with thinking about how we turn unstructured data into structured data that we can actually use. Mhmm. Right? So one of the one of the other cool things that we do with Retool is the hard part about understanding and answering resident questions at any property is just knowing what the property is about. Right? Do they have a pool? When is it open? What is their pet policy? What is their parking policy, etcetera? If you ever lived at any rental property before, you know that these things don't live in, like, neat structured online documents. They live in random PDFs that nobody's ever seen before.
00:37:42 Arjun: And so it is really hard save for save for actually like going in and writing things down to get the institutional knowledge that you have on a property. So one of the fun things that we built was a retool tool so that our teams can use internally. So the interface for this is we will typically ask our customers when they set up a property on Residuesk. Hey. Like, send us any documents that you have, our way. Right? Like, it's a lease document. It's a flyer for an event, whatever it might be. And so by doing that, we take on the sort of work of turning that unstructured data. It could be a PDF. It could be a painting.
00:38:25 Arjun: Who the hell knows? And turning it into guidelines that we can actually use. So our team has this retool app that you can browse. I'm just gonna take here's like a pet policy that I have on hand, and I can take that and say, okay. Here's the guideline. And by saving it, what it's doing is taking a document that let me just quickly show you what it looks like. But this is what that document will typically look like. Right? Like and this is actually more structured than a a good a a typical document would be. But here's all the responsibilities. Here's all the agreements. Here's the liability, etcetera, etcetera.
00:39:12 Arjun: We have a tool that is then taking that and actually turning it into an FAQ. So it creates a blurb. It uploads a document, and then it actually uses this is using Gemini behind the scenes because it's so good at finding needles in haystacks. The most common questions that you have that a resident could have about these things. Right? But then what happens on the back end is even more interesting, which is all of this gets vectorized. The raw text of the dark of the document, the blurb, and all of the individual questions that we picked up from it. And so when that happens, what that lets us do is bring up do rag in a really efficient way where we look at conversations that people are asking about.
00:39:57 Arjun: So, like, this resident asking about partial payments and then bring up the related guideline and and tell you what the answer is. This then feeds into the doc into the answers that we suggest, which then makes our rag workflow a lot easier. By doing this, we were able to sort of see an uptick in, in the number of questions we answer on day 1. So when we started Resideo SA, I think we were answering something like 60 to 65% of resident questions on day 1. After we started just taking in any sort of unstructured data, we're now answering closer to, like, 85% of resident questions on day 1. And it only happens because we are able to abstract away this really dumb workflow of send me whatever the hell you have that looks like a property document and we will make it useful for ourselves without having people go through, like, a big questionnaire that they need to fill out.
00:40:45 Arjun: So Yeah. Anybody building an AI AI tool out there that needs to be trained on data, this is an easy hack.
00:40:53 Greg: And what's crazy is that this easy hack, you just did it yourself with some retool and some, like, slight prompts and then turning into structured data.
00:40:59 Arjun: Yeah. So, here's another internal workflow if you're ready for some insanity of just dealing with payments and property management. So, again, context is we sell ResidDesk to Greg. Greg has a set of, let's call it, 15 properties. Right? If I bill Greg for 15 bucks and I'm making the math so funny here, but, basically, like, it's a dollar per property. Let's just call it that. The way that real estate has worked for a long time, I said, was first, each of your properties would just pay me a dollar. So I'm left stuck with a $15 invoice that I then have to itemize and then reconcile. Customers pay us in checks.
00:41:43 Arjun: We have a workflow that scans and then reconciles for us in Stripe. But that's not the crazy end of it. In an effort to digitize, what's happened in real estate is now those 15 properties will typically pay push that check to a third party clearing house that then turns that check into a one time used credit card that they send us along with the invoice number. So Wow. What I end up with, I'm gonna show I'm gonna share my screen because this this is literally the nuttiest thing that we have seen in our time here. What I end up with is something like this. So you can see this email here. It comes from a third party clearing house.
00:42:25 Arjun: It's, you know, a payment for $5 for this and a payment for $208 for this for these invoices. Right? Behind the scenes on this is you can see here this little thing called click here to view card information. You open that. Our customers actually send us a one time use credit card that is only valid for 30 days, and we have to pick up the card number, figure out what invoice this belongs to, and then pay it within those 30 days or that payment expires. Right? And then we have to go through the whole process all over again. As you can probably imagine, this is a complete pain in the neck for us to do manually.
00:43:07 Arjun: Because as I said, if Greg has 15 properties, we get 15 of these every month. Right? And so this is where make has been our best friend, and Front has been our best friend. So what we've been able to do is say, okay. You can automate things on tags in Front. So when I assign a tag that just says, you know, billing payment reconciliation
00:43:32 Greg: Mhmm.
00:43:32 Arjun: You can create a webhook that goes to front and or to make. Sorry. And then what happens inside of make is it then finds the sort of content of the message. It opens the link, takes a screenshot of it after a small delay, and then OCRs it to get the card details, finds the link of the conversation in Stripe sorry. Finds the links of the associated invoice in Stripe and then adds them back as comments into the conversation, which we then, if the payment amounts actually fully line up, we will automatically pay, and then the whole thing is automated. If the payment amounts don't line up, then it it alerts our billing team to then manually pay the invoice in Stripe.
00:44:21 Arjun: So, again, by doing this, we are processing as I said, we serve about a 1000000 residents across the US. So we are now processing something on the order of probably like 6 to 800 one time credit card payments a month. And like nobody has the time to do that. The other side of the automation, which I'm incredibly bullish on, will happen with things like cloud computer use but isn't quite there yet Mhmm. Is we're interacting with software that was built in the mid to late nineties. Right? We live in an industry where nobody actually employs software engineers. They just employ contractors to build a thing and then that's the thing that's that stays forever.
00:45:07 Arjun: So we're dealing with a lot of like kind of like the, you know, what Plaid used to deal with back in the day. We're dealing with a lot of legacy software that isn't is actually fairly RPA resistant too. So because that's the one thing that people actually will invest time and money into into combating. Mhmm. And so we there are systems that our customers use that have APIs, but by and large, they don't have APIs. And so we're slowly starting to automate some of the some of the data intake, but then I'm I'm excited to get to a future where, you know, Claude or GPT can just have access to a VM where it logs in and gets the data that we need automatically.
00:45:49 Greg: Sure.
00:45:49 Arjun: So we TLDR is I think we wanna in we wanna automate all of our data intake problems. Mhmm. But from a from a conversation output to the residents piece, I think that will always have a human in the loop, and that's mostly driven by the product that we're building. We're we're in the trust business, and residents trust us because we have a human team.
00:46:10 Greg: Sure. Yeah. That makes total sense. Alright. Last question I wanna end up with with you. So there's a lot of viewers out there who may be thinking about, hey. I wanna implement AI in my business, but I have no idea what to do. So let's say that you're CEO of a new business. Legacy, the business has already been going for a long time, profitable, it's going well, but you're the new CEO. You drop in on day 1, and they say, we need you to evaluate our workflows and what we're doing, and we need you to make us AI enabled or get us on AI. What's the playbook that you're gonna run to look at the whole business and figure out what to do?
00:46:47 Arjun: It's a great question. We think about this a lot because, again, our industry is really focused on talking about things like what is AI in real estate as an example. Right? To me, the funny part is I think talking about AI in real estate or talking about how to put AI in my business is about as broad and silly as talking about how do I use Excel in my business. Right? I think what people fail to understand is that this is really about specific use cases, not about, you know, throwing Excel or AI magic into your business. So what I would do on day 1 would be to look at much like what we did with Residesk, look at the most manual workflows that people have and what you need to do.
00:47:31 Arjun: And so what we do internally is, first of all, give people broad access to LLM tools in house, you know, whatever. Like, on prem, think about data security. All that stuff is great, but, like, give everybody broad access. I think the playbook for getting people to use AI to help themselves is not a centralized top down instruction to do certain workflows the AI way, but instead to give people the tools. And then the second part, which is much more important, although the first part never seems to get past IT teams. Mhmm. 2nd part that's really important is go find a couple folks whose lives would be a lot better.
00:48:09 Arjun: Ideally, somebody who's doing a really repetitive tedious task, something super manual, and then work with them to build a workflow tailored to them using AI. Right? So example for us is we actually scan paper checks. Our customers pay us with checks. Right? And so reconciling invoices becomes a huge deal. I literally sat down with our billing person and said, okay. How do we make this easier? Build and make workflow that then talks to GPT to look at all the receipts that we get and then reconciles it and pays it on Stripe so they don't have to spend time on it. There's probably a 100 examples of this in any company that you go to.
00:48:48 Arjun: Find a couple of people who would be great to sit down with and build a really small, cheap, fast process with them. Like, none of this stuff should take you more than Dataship.
00:48:57 Greg: Mhmm.
00:48:57 Arjun: And then make case studies out of that and then have them spread the word to other people in the company about how this was useful to them and how you can be creatively thinking about AI for your own workflow. I can be told as an employee to use AI as much as the executives want.
00:49:18 Greg: Uh-huh.
00:49:18 Arjun: But, ultimately, I'm only gonna care when it helps me solve something I don't like to do. And it ends up being a very personal workflow to each person. And so what you really need are just these sort of champions of AI inside your organization. If you can go find 2, 3, 5 people across different departments that you can spread the word to and then put them in charge of sharing with everybody else how they do things better, that's when you really start getting the magic. Right? It has to be almost like a it's almost like a viral b to c adoption curve. You have to make it useful to a few super users and then have them spread it out.
00:50:00 Greg: That's so cool. I love that playbook, and it makes that makes sense to me. Well, I tell you what, Arjun. This has been absolutely fabulous. Thank you very much for just showing like, just screen sharing and just showing the workflows that you have going on. It's really cool to see what, like, you're actually actually doing. Yeah. So looking at the prompts, looking at the projects. So I really appreciate that and thanks for chatting today.
00:50:19 Arjun: Absolutely. Thanks so much for having me.
# File: /content/learn/dev/ai_show_tell/deedy.mdx
---
title: Deedy Das, Engineer-Turned-Investor
description: Deedy Das, a unique blend of engineer-turned-investor, brings deep technical expertise to the venture capital world.
date: 2025-01-09
---
[Deedy Das](https://x.com/deedydas) is a engineer-turned-investor who brings deep technical expertise to the venture capital world. After a 10-year career in engineering and product, including roles at Google, Glean, and other tech companies, he transitioned to investing because he felt the industry needed more investors who could "nerd out on tech."
He's known for his viral technical content on Twitter, where he frequently shares hands-on demonstrations of AI tools, coding projects, and technical insights.
### Insights
* **Technical Revolution & Historical Parallel:** Deedy compares today’s AI revolution to past industrial revolutions. Just as factories transformed clothing manufacturing, AI is transforming knowledge work, automating previously manual or tedious tasks.
* **Investor Approach:** Tech Nerds Needed: Coming from a 10-year engineering background, Deedy believes more “tech nerd” investors are essential. Rather than focusing purely on financial metrics, these investors bring deep product and technical insight to discover genuine breakthroughs (and call out snake oil).
* **AGI Perspectives & Model Scaling:** Deedy points out that if you asked someone 4 years ago what AGI looked like, many would say today’s large language models already qualify. The real question he’s interested in is: “How much further can these models go?”
* **Full Self-Driving Analogy:** He likens the current stage of LLMs to “full self-driving” in cars—impressive demos exist, but hitting 100% reliability remains an unsolved frontier. This gap is evident in tasks like coding, where LLMs get you far but still stumble.
* **Meta-Prompting for Better Results:** A practical trick Deedy uses is to break down big requests into smaller instructions—often called “meta-prompting.” He’ll ask the model to output a series of sub-prompts and handle each one individually, greatly improving reliability and debuggability.
* **Three AI Startup Categories:** In his view, current AI startups often fall into:
* Foundational Model Companies (trying to build the next big model, e.g., Descartes, Carticia),
* Workflow Automation (replacing back-office tasks, vertical or horizontal),
* Advanced Agents (e.g., AI Site Reliability Engineers, AI coding agents).
* **Call BS on AI Snake Oil:** In a hot market like AI, many startups over-promise. Deedy emphasizes looking for real customer value, renewals, or working demos. If claims are all buzzwords (like “RAG but better!”) without a real product or proof, it’s probably snake oil.
* **Coding Agents & Tool Use:** Deedy frequently uses Claude’s “computer use” (and custom forks) to generate code, automate tasks, or even create waveforms in audio generation. Though it’s powerful, it’s also prone to getting stuck if not carefully prompted or monitored.
* **Diagramming for Better Comprehension:** A favorite tactic is generating diagrams (using Graphviz/DOT files) to summarize transcripts or codebases. Seeing relationships and flow visually reduces overwhelm and offers quicker insights than reading raw text.
* **Building & Sharing:** Deedy’s prolific online presence stems from building interesting demos (like automated scrapers or music generators), then sharing them via short videos on Twitter. This cycle—idea > build > post—keeps him motivated to experiment.
* **Virality & Thick Skin:** Going viral on social media (via Elon Musk or Paul Graham retweets) can open new doors but also invites criticism. Deedy advises staying open to feedback, yet remembering that negative reactions often blow over—keep creating and learning.
## Transcript
00:00:01 Deedy: We are in the middle of a technical revolution.
00:00:01 Deedy: There was a time when people made machines to make manufacturing clothes better. We're in that revolution for knowledge work. I felt like there needed to be more investors who just nerded out on tech.
00:00:13 Deedy: I think the biggest unlock for me personally
00:00:13 Deedy: is it's really simple. I I use I personally is it's really simple. I I use for like half of the stuff and it buzzes and it says something something you want to tweet it some. But then my phone just starts blowing up like notification I've been on. I'm like what is going on? When I do play with Claude, I noticed, like, this one small, like, one weird trick that makes it so much better.
00:00:48 Greg: Hey crew. What is going on today? We have DD Doss with us today. Very excited to have him on board today. We are gonna be talking not only about his builder hat but also his investor hat, how he's evaluating different companies. But the coolest part, which I love and I'm excited to share with you, is he's actually gonna screen share, and he's gonna show us how he builds with Claude Computer Use and MCP. So let's jump into it. So, Didi, you're a builder and you're an investor. What are the cool people or what are the smart people in AI talking about right now?
00:01:18 Deedy: I think the one question on everybody's mind is how far does this go? Like scaling or just Whether it's I think scaling is more of a tactical thing. It's just how good can these models become?
00:01:30 Deedy: Mhmm.
00:01:31 Deedy: Can it do we get to a world where this solves problems we could never solve ourselves? Yeah. I think that's the frontier. Because the when people talk about AGI, it's quite it's it's this amorphous thing I I love to tell. Exactly. You know
00:01:44 Greg: what blows my mind is that us as an industry, we don't have a solid definition of intelligence that we formally rely on. It's wild that we're still arguing about what is intelligence. It's like a weird side product of this whole, kind of revolution.
00:01:54 Deedy: I was telling a buddy yesterday. I was like, if we ask somebody 4 years ago what AGI was, by all practical purposes, we're there.
00:02:01 Greg: We'd probably we'd probably agree that you open up literally GPT 4 0, start chatting, and you're like, well, yep. We're here.
00:02:07 Deedy: Yeah. Yeah.
00:02:08 Greg: Yeah. Okay. So, how far does this go? So you're it sounds like you're talking about the application side, like, value providing. Right?
00:02:15 Deedy: Oh, well, there is the value side. I'm less concerned about that. Obviously, as an investor, I concern myself about that. As more of a technical nerd, I just think about how good can these models get. Like, I still have flaws today. Right? Like, there's still limits. You have you have the ARC prize and everything. There's limits to what it can do. Sure. But what are those limits? Where how far does it go?
00:02:35 Deedy: Yeah.
00:02:36 Deedy: And I love the alpha zero, alpha go analogy. We'll see a world, hopefully, 2025 Uh-huh. Where these models give it a general task. Yeah. It'll figure out the rules. It'll figure out what it needs to do. It'll take some time. It'll take as much time as it wants Sure. And it just gets superhuman level intelligence Yeah.
00:02:54 Deedy: Yeah. On arbitrary tasks.
00:02:54 Deedy: And in tasks. And in some ways, depending on how far you want to take that, I think that that is reasonably close to AGI.
00:03:00 Greg: Yeah. It's pretty wild. So that sounds like that's your builder hat definition of AGI which is pretty cool. So while we're on this topic, I gotta ask about your investor hat of AGI. So how do you approach what the next 5 years are gonna look like from an investor side?
00:03:14 Deedy: I've only been investing for a year. The the reason I wanted to get into investing after a 10 year career in in engineering and and and product Yeah.
00:03:23 Deedy: Yeah. Yeah.
00:03:23 Deedy: Is because I felt like there needed to be more investors who just nerded out on tech. Sure. There's a lot of investors who are really good at what they do, analyzing businesses, but I wanted to be that voice of, like, just show them what's cool and show me what's possible.
00:03:39 Greg: Yohei comes to mind. Yeah. Fantastic. Builder. Like, it's so cool to be, like, here's my pipeline, here are the tools I built for myself, and here's how it's actually providing value.
00:03:47 Deedy: Value. Exactly. I love that. I love that. Exactly. So as an investor, I also take, like, at least 50% is that view
00:03:53 Deedy: Nice.
00:03:53 Deedy: Which is just show me what's possible.
00:03:55 Deedy: Yeah.
00:03:56 Deedy: And we'll figure out a way to make it work. Yeah. We'll figure out a way to make money.
00:03:59 Deedy: Sure.
00:04:00 Deedy: Sure. That's the that's much easier than Yeah. Groundbreaking technology. And then the other 50% is, yeah, how do you take what exists today and apply it in in useful and meaningful
00:04:11 Greg: Yeah. It's also nice because you can call bullshit on a lot of the tech when you need to. Oh, man. You could just, like, call them out and, like, ask real questions.
00:04:17 Deedy: I was literally I don't wanna name the technology. Maybe I can't name the technology. I was talking to a couple of, other investors the other day, and they were like, you know, I had this pitch and these guys said, you know, they do, like, rag but better, and they threw all these fancy terms at me. Like, what do you think? And I'm like, look. I feel like that's every pitch, almost every every other pitch I've had for the last 3 months.
00:04:42 Deedy: Yeah. Yeah.
00:04:43 Deedy: And when it comes to things, especially that are hot like AI, there's gonna be a lot of snake oil sales. Sure.
00:04:48 Greg: Yeah.
00:04:49 Deedy: And it is pretty important to call BS on a lot of these things.
00:04:53 Greg: Well, so I'm I get a lot of pitches myself too, and one of the easiest ways to go through it is don't tell me what your tech does. Show me the value or show the customer who's paying for it or, like, just show me something that's an output that comes from it because it's hard to bullshit value. Like, that's proof of work almost right there.
00:05:08 Deedy: True. I would argue that it is still it's it's the reason investing is especially hard with AI is if you can bullshit value. Like, if I've seen so many start ups, 1,000,000 ARR because, you know, CIOs and CTOs are saying go buy a gen AI tool.
00:05:22 Deedy: Sure.
00:05:22 Deedy: Does that mean they created any value yet?
00:05:24 Greg: The the revenue's there in the beginning year 1, but the renewals may not be there. And even further than that, you dig down to the user metrics, the intention of engagement and all inactivation. It's like, well, if that's not even there either, then you're gonna get yourself in trouble. Okay. So I love the question, how far does this go? What are the obvious gaps that are limiting us going further from a, tech, AGI perspective?
00:05:46 Deedy: There's a couple of different ways to to break that down. I think let's talk just about text models. Yeah. I I look at text models today, and I'm like, okay. Well, this is good for a lot. Uh-huh. I use Claude almost all the time. Nice. And what I and I played with all the other models too, like all like, o o one and and llamas and everything. Yeah.
00:06:08 Deedy: Yeah. Yeah.
00:06:09 Deedy: And what I really wanna know is, okay, I I've seen people have all had this comment is when you use it for a very specific task, like a a task that has a correct outcome, an objective task, it goes pretty far and it can amaze people, but it doesn't go a 100% of the way. And I I love to call this, like, the full self driving problem where it's kinda easy to get a demo of full self driving to kinda work in Sure. Most situations. It's really, really hard to get an actual full self driving
00:06:40 Deedy: Yeah.
00:06:40 Deedy: Vehicle that can do everything in real roads everywhere. Yeah. And let's just take, say, coding for example. That's one example of a very objective thing. You can also take math. You can also take any other form of logical reasoning, biology, which combines memory and reasoning. Sure. And I think about those things and I and I wonder, like, okay. We're clearly at a gap. People love Cursor, but there's also some people who are like, okay. Cursor is awesome, except I hit them all. Yeah. I can't do beyond a certain thing, and I don't know what went what went wrong.
00:07:12 Greg: Imagine how how entitled or, like, selfish those people must be to think that, oh, it's this AI is not good enough to do my job for me. You know what I mean? It's just funny that they're critical of AI when it's, like, literally so helpful for so many things. But no because it's not throwing you a touchdown at the very end of the end zone that you're giving it a hard time.
00:07:29 Deedy: Oh, yeah. We're all humans are infamous. I think Louis CK of all people has this great joke about how people complain in flights and they're like, guys
00:07:36 Deedy: Yeah.
00:07:37 Deedy: Flying in the middle of the sky. Yeah. Yeah. Yeah. This is the most incredible thing that could ever happen. Like, have some have some you know, be thankful. Be grateful.
00:07:44 Deedy: Yes. Yes.
00:07:44 Deedy: And so I think about that too, obviously, but, you know, we also wanna think about the frontier, what's possible. And, you know, when I've been coding with Cursor and things like this, there does come a time where I'm like, damn. It did a lot for me. But now I kinda wonder there's clearly something it can't do no matter how much I prompt it, and it goes, like, 5 cycles, and it's just not working.
00:08:03 Deedy: Yeah.
00:08:03 Deedy: And then I'm in this weird spot where I don't even know where to look. It's generated, like, 10 files, 500 lines each. Like if I was writing it myself I would know how to debug this. Now I don't.
00:08:16 Greg: Like a vocabulary word for a shared feeling of I just had the AI generate a ton of stuff and now there's a bug, and I have no idea where it is. Exactly. I don't know what that word is, but I I hear people talk about that all the time. We need to come up with a word for it. I have a hypothesis that GPT 4 can do almost any task as long as it's sufficiently scoped down.
00:08:39 Deedy: Absolutely. And we'll maybe get to this later on, but I love this like that that anecdote because I play with computer use all the time and I wish I could do it with with GPT 4 and all the o one stuff as well, but I just don't. Like, they don't have the computer use version yet. But when I do play with Claude, I noticed, like, this one small it's like one weird trick that makes it so much better. And that really it was a really simple trick, and it was exactly what you said, which is I just went to Claude first and said, here's my high level 3 prompts. Uh-huh. You, Claude Yeah. Have to now give instructions to a computer
00:09:11 Deedy: Yeah.
00:09:11 Deedy: To break this down into tasks Yes. Yes. And then I feed that huge prompt
00:09:16 Deedy: Yes.
00:09:16 Deedy: Into computer use and say, now you do it. Huge improvement.
00:09:19 Greg: One of my when I work with clients and we talk about prompting, one of the most common problems I see is that they stuff too much freaking stuff inside the same prompt. They want 5 tasks. I wanted to make a mini little tool and it sounds like we have the same thought. It's like feed them the prompts and output what are the individual prompts you should actually be getting from your big one. Just scope it down even further because then that makes evals a lot easier too. It does. You know where it's messing up. It does.
00:09:42 Deedy: And they
00:09:42 Deedy: call it meta prompting and I think Yeah. It it it kinda alludes to this, like, fundamental CS thing which is everything is solved by one additional layer of abstraction. And in this case, it's just on you to to build that layer of abstraction and then and then, you know, give a prompt that gives you more prompts and then Yeah.
00:09:57 Greg: That's exactly it. On the investor side, you're seeing a lot of startup pitches come through. What are the trends in the past, let's call it 3 months? Not like the beginning stop, not the 2023, but what's the last 3 months look like from an investment side?
00:10:08 Deedy: Well, so so a lot of people are talking about agents. Right? And agents are one of those other words that people are like, what does that really mean when you say agent? And and so when when it comes to AI companies, let me let me sort of break break down the field.
00:10:21 Greg: Sure.
00:10:21 Deedy: There's foundational companies. Yeah. Foundational companies can be tough because sometimes you're really asking the question, are you really trying to compete with people with Sure. 1,000 of the money?
00:10:29 Deedy: What are
00:10:29 Greg: some examples of these?
00:10:31 Deedy: Well, a good example is actually this company Descartes. Okay. They bold, brilliant founder. It's just one of those guys and as an investor now, I've learned this. Like, just trust your instinct. You meet this guy. Dean, I think his name is. Just whiz. Like, you hear, like, oh my god. Like, this guy's so smart.
00:10:49 Greg: It's almost like you're back in the you're back in the person.
00:10:51 Deedy: Go get the idea. Yeah. You're almost always actually back in the person. But some people just have this energy where they're like, this guy is a genius. Yeah. And he was one of those Descartes taking a problem way out of their league, which is this idea of generative gaming. So Mhmm. You know, it's hard enough to generate good images. It's hard to do it real time and and then consistently. So what the question to ask is, you have a game where people which involves thousands and thousands of game designers, engineering different parts of the graphics, how things behave in that game. What if you could just throw a gen AI at it and just have each pixel just show up and travel in the game?
00:11:26 Deedy: Yeah. He has no business doing this. Okay. He's like, kind of a kid. Uh-huh. And it's with a small team, and these are things that, like, Google can't do in OpenAI. Now Google has a thing that does this OpenAI, I can't do. Anthropic can't do. But that was a beautiful one. That's foundational, and that's great. You have other companies. There's a company called Carticia. Mhmm. Carticia is a company doing a foundational model for audio. They came definitely after 11 Labs, and, you know, they had a different approach to it. And so good. Man, I know. Reasonable outcome
00:11:56 Greg: so far. Labs is crushing it right now too. They're freaking shipping and building right now. A ton
00:12:00 Deedy: of stuff.
00:12:01 Greg: Yeah. Okay. So foundational companies. Alright. What are the other trends we're seeing here too?
00:12:04 Deedy: Then you have the companies that are are very clearly on the application side that are doing workflow automation, I like to call it, but that's a really boring word. The way I would think about it is there we are in the middle of a technical revolution. There was a time when people made machines to make, like, manufacturing clothes better and other things better. Sure. We're in that revolution for knowledge work. And the question is, what machines can we make? Yeah. And and we're calling those machines agents, but this is everything and and people love to, like, take the extreme side and go like, woah.
00:12:38 Deedy: Well, my job is not automatable because it's so smart and my but, like, there's clearly jobs on the other side which are dumb and boring, and no one wants to do them.
00:12:48 Deedy: And a
00:12:48 Deedy: lot of them really involve look at this Google Sheet, look at this Excel, look at this PDF, get this data, write this data here, send this data to this person. If you would look at a response this person sends you and do this, that's most jobs
00:13:00 Greg: Yeah.
00:13:01 Deedy: And moving stuff around.
00:13:03 Greg: Sure.
00:13:03 Deedy: And if you have a job that's regular enough, you could kind of automate that today. Sure. So a lot of people doing automations there and that can mean voice stuff, that can mean pure, you know, text and
00:13:15 Deedy: Are we
00:13:15 Greg: talking about vertical AI? Are you talking about, like, the Lindy's and the Zapier's, like, the horizontal?
00:13:20 Deedy: I'm in this case, I think most of the ones in this category are are pretty vertical Sure.
00:13:24 Deedy: Yeah.
00:13:24 Deedy: Because at least at first. Because you have to start with a way to say, hey, company. I know you have a 100 people doing this job. Yeah. We think you don't need that and we can do it for you. Yeah. That's kind of the Yeah. The high
00:13:36 Deedy: level pitch.
00:13:37 Greg: It feels like there's an arbitrage opportunity right now for just going and selling AI plus your industry. Like, people will get over that eventually, and that's gonna get diluted and already is starting to get diluted. But if you go to a law firm and you say, hey. I'm I'm gonna get an AI lawyer. They're like, great. I need one of those without even knowing what the heck it is.
00:13:52 Deedy: Yeah. There is the that there is this idea of or this phenomena of people buy tools that overpromise and then underdeliver. Totally. Totally. And it doesn't work a lot of times. But on the other side, you know, Maxine, insurance, health care, finance, there are a lot of pretty tedious industries, a lot of industry that had back offices especially where you it's just labor that you don't wanna pay for basically and so you're like automate that away and let's move forward as basically to be grandiose as a species and not have to do Yeah. Dumb work. Yeah.
00:14:26 Greg: We can move on we can move on to other dumb work.
00:14:28 Deedy: Yeah. We want other dumb work. Every generation has dumb work.
00:14:31 Greg: One of my favorite consulting books, I freak I forget what it's called, but basically one of the main lessons is there's always a number one problem. Once number 1 is solved, well number 2 gets a promotion to number 1. Right? So there's always gonna be dumb work. There's always gonna be interesting work. It's just a it's a big long cycle that comes from there.
00:14:46 Deedy: True. But that yeah. The the to your point, like, this is this is there there's a whole thing of a workflow automation. Like, how do I automate people's jobs? Assist or automate whatever, people's jobs and make it a little bit easier to do. The 3rd sort of category I will touch on is it's very hard because there's so there's a big long tail of AI AI startups, but especially you asked last 3 months what I'm seeing more of. I'm seeing a lot more companies really talk about this idea of let me let me I'll I'll take 2.
00:15:19 Greg: Yeah. Totally.
00:15:20 Deedy: 1 is the idea of what they call AI SREs is a common bucket of things that people wanna solve, and I'll I'll get to what that is. And then the second one is a is is also in in a some semirelated space. It's just programming agents. Sure. And and that's not like that's a 3 month thing. It's been around for a while. But in the last 3 months, the the the evolution has
00:15:40 Greg: been pretty interesting. Yeah. Well, and there's a cursor competitor coming around now too. Yeah. I mean, I love competition. So I I'm glad to see it.
00:15:47 Deedy: Yeah. It's, well, cursor's doing really well. Well, the who's the competitor you're
00:15:51 Greg: Oh, Windsurf. That's yeah. Correct.
00:15:54 Deedy: So it's it's awesome to see. Yeah. It'll be interesting to see how this all goes out.
00:15:58 Greg: Alright. So 2 categories. So first one, SREs.
00:16:00 Deedy: SREs. You know, tons of companies doing various versions of of this. And I say SRE, but it's an also like, support is kind of included in that technical and nontechnical support. The idea of being look. I have a problem in the company. Code goes breaks in production. Things go wrong. That is money on the line, especially from a big company. Like, every you can do the math. Every hour can be, like, tens of 1,000,000 of dollars, and it's kinda crazy that right now, the best thing we have is we send a guy. Like, that's the the the the resolution is a guy goes and looks at it and tries to, like, figure this issue out.
00:16:37 Deedy: And it and I've done that at at Clean and at Google. It is one of the most nerve wracking things you can do as a software engineer, which is, like, thumb things down in production. There is time on the line and there are literally petabytes of data you can look into. Like, which dashboard do I look into? What do I check? What do I check next? What do I query for? How do I fix it? When does the release go out? Like, your brain's going bonkers. And this could be 3 AM in the in the in the Sure. And I understand
00:17:01 Greg: ping and it's like, oh, boy. It's crazy. Uh-oh. Yeah. It's crazy.
00:17:05 Deedy: So it's a really bad hard thing to do and there's ways to make make this easier. Sure. And there's like a couple of companies that are that are doing this pretty well. There's there's there's resolve. Sure. And there's like a couple of companies that are that are doing this pretty well. There's there's there's resolve. There's company Traversal. There's a bunch bunch of these other companies trying to do this. Super interesting sort of sort of category of problems and just to me. And then the last one, like, the the one I was talking problems and just to me. And then the last one, like, the the one I was talking about is coding agents.
00:17:31 Greg: Nice. Yeah.
00:17:33 Deedy: There's just so much more stuff we can build here.
00:17:35 Greg: Just so much more, and it's so cool. It's a it's a it's a place that's right too because everyone talks about, you know, code is easily verifiable. Does it work? It does not. Right? The hardest part is still just with the planning piece, and do you get the intention right? And can the human adequately express their intent to the coding agent in the first place?
00:17:50 Deedy: Exactly.
00:17:51 Greg: It's amazing. One of the side thoughts I've had is I bet the outside of just AI in general, like, epistemology as a study, like, the study of thought and the study of how knowledge works must get such tailwind from AI the AI move move that's happening. So people are thinking so much more about how do I express language? How do I formulate these thoughts? How do I think about planning? How do I think about communication? All that wouldn't have happened unless we had these text based LLMs that we now have a financial incentive to really dig into how we're dealing with these things.
00:18:22 Deedy: Absolutely. I mean I think it's incredible where we are in the world.
00:18:26 Greg: So let's move on to the builder side. Yeah. So what technologies are you personally building with right now that you're you're excited about?
00:18:34 Deedy: Right now, I think the biggest unlock for me personally is it's really simple. It it it I I use cloud computer use for, like, half of the stuff.
00:18:43 Greg: Oh, true.
00:18:44 Deedy: And I think there's with the caveat. Alright. There's 2 caveats, actually. There's a version of cloud computer use that came out. They wanna do it safely. Anthropic is all about safety Yep. And as they should be. Like, I don't think this is something that you should just let everyone willy nilly go and and run a mock in their computer. Everyone willy nilly go and and then run a mock in their computer. But, oh, I'd like to take some risk. And so I cloned the version that they have for Linux, and you have to do a couple of edits to that and go into as much detail as you want. Yeah. And and I'll show it to you later if you want to.
00:19:15 Deedy: You can get but you can make it work on your actual Mac, and it makes things a lot better. That's caveat number 1. Second thing that you can do is I realized over time that as cool as this is, it's cloud computer use did this really cool thing where it can can control your mouse and view your screen and then take an action. Yeah. But as cool as that is, it's kinda the most lame part in the sense this is something that takes a human a second, and now it takes this 20 seconds.
00:19:41 Deedy: Yeah. Yeah.
00:19:42 Deedy: And it's really, really frustrating when it does all this cool stuff and then it gets stuck trying to find a button and click it Sure. On your computer. So when I prompt it and I do a couple of alterations, I just say, can you not do that? Like, just try to stick to the command line.
00:19:59 Greg: Sure. Okay.
00:19:59 Deedy: You can pretty much run anything on the command line, pretty much.
00:20:04 Deedy: Just stick
00:20:04 Deedy: to that.
00:20:04 Greg: Okay.
00:20:05 Deedy: And I felt like the progress has been incredible.
00:20:08 Greg: That's cool. That's very cool. So is cloud computing use is that gonna be the product or is that gonna be the platform that other people are gonna build really cool products on top of?
00:20:16 Deedy: It's unclear. Like, now they have MCP, which is I like, I don't I don't wanna speak for anthropic but I feel like MCP is the version of safe cloud computer use. Sure.
00:20:25 Greg: Well, so can you define what what is the MCP? Could you define it in in layman's terms too?
00:20:31 Deedy: The way I see it, and this is probably not doing justice to everything it can do, is MCP is a way for you to safely say, hey, Claude. You can use these tools. I can give you some memory. You can go use these tools
00:20:42 Deedy: Uh-huh.
00:20:42 Deedy: And customize the way you can you you can do things with Claude, but it's very dev dev centric. It's not really, you know, consumer facing already.
00:20:50 Greg: There's this whole spec that comes with it. Right?
00:20:51 Deedy: Thing like that. Yeah.
00:20:52 Greg: Yeah. Yeah.
00:20:53 Deedy: So I don't know I don't know where Anthropic goes with this, but I do know coming back to the startup thing, it's a big thing that startups are thinking about. What's that? Computer use. How to use things like computer use to automate more kinds of tasks. Yeah. Like, the first era of automating tasks was all browser based.
00:21:13 Deedy: Yeah.
00:21:13 Deedy: And the general way it would work
00:21:15 Deedy: is
00:21:15 Deedy: scrape the DOM, everything on the HTML on on a web page, and then try to figure out various hacks on what I can click on.
00:21:23 Deedy: Yeah.
00:21:24 Deedy: And, you know, like, you could think about that for 2 seconds and go, like, that's a pretty shitty strategy because that's not how humans work. Uh-huh. It would be insane to read 1,000 and thousands of lines of, like, HTML code and then figure out what to do on a page. Yeah. The whole point of the page is you see something and you do something Yeah.
00:21:39 Deedy: Yeah. Yeah.
00:21:40 Deedy: And computer use makes that a lot better because you have the vision angle.
00:21:43 Deedy: Yeah.
00:21:43 Deedy: But it's slow.
00:21:44 Greg: It is.
00:21:45 Deedy: So but there is going to be a time where the vision latency comes down and you can start doing actual actions on pages
00:21:52 Deedy: Yeah.
00:21:52 Deedy: That get to automating a whole other boatload of tasks both on web and on your desktop.
00:21:58 Greg: So it's
00:21:58 Deedy: an exciting
00:21:59 Deedy: Yeah.
00:21:59 Deedy: Sort of high level technical thing that startups can do now.
00:22:02 Greg: One last question before we dig into computers. I'll pull all computers out here. So there's an open question on if the future of automation what's the interface for an AI type of thing with, call it, 3rd party software? Will it be more like cloud computer use where you're actually interacting through the UI, or are the API is the API layer gonna get built out? And that's where the automation's gonna happen. The I think the jury's still out. What what's your opinion on which way it's gonna go?
00:22:25 Deedy: There have been couple of startups I've seen trying to sort of have various approaches to how that layer should look. So there's one called Composio. Right. Composio is a way of saying,
00:22:37 Deedy: there are a bunch of tools that are out there for you to use. Mhmm. I'm going to toolify it for you so that Oh, yeah.
00:22:38 Deedy: I'm to use. I'm going to toolify it for you so that you can go and and the LLM can see a prompt and understand an API spec of, say, GitHub
00:22:49 Deedy: and
00:22:50 Deedy: then figure out what to do. So you can think about it
00:22:52 Deedy: almost like documentation, technical
00:22:52 Deedy: documentation for LLMs. Yeah. And then I will
00:22:52 Deedy: teach an LLM how
00:22:52 Deedy: to do element how to use the tools that it has. I will keep it up to date. Mhmm. And if you just feed it to the LLM, it can use my tool. That's one angle. There's another company called Anon. And what Anon does is it's more like, almost think of it as the plaid for APIs Okay. Which is, like, the same way you do bank authorizations via protocol or with plaid. This is, like, trying to do to extract authorization out to the user, but let an agent act on the behalf on behalf of you.
00:23:26 Greg: And the funny part about Plaid is it used to be just via scraping, and now it's, like, done proper. And it's almost like the metaphor for how this is probably gonna go too.
00:23:33 Deedy: Maybe. Yeah. I I I still think tool use has not got mainstream adoption yet. Yeah. So it's it's TBD where that direction goes. Obviously, a lot of start ups looking after it
00:23:44 Deedy: Yeah.
00:23:45 Deedy: Looking out for it. A lot of enterprise companies that do a good job there, but end users still kinda kinda haven't done it. Most end users are still just kinda talking to Claw, talking to Chat JBTs like hey.
00:23:55 Greg: What the heck's going on right now, right? They're just waiting for more builders to go build stuff to go use it from there.
00:24:00 Deedy: To me it's crazy, maybe maybe as good a sign there but you step out of Silicon Valley bubble Yeah. And nobody knows what's going
00:24:06 Greg: on. It's it's it's just it's almost baffling sometimes because when you spend so much time on AI Twitter and you're literally in San Francisco or the heart of the city right now, it's hard to get away from it. You know, when you go to these meetups and it's just like, oh my goodness, I'm so far behind because there's so many other people that are so far ahead of me. But like you said, you step outside of it and it's almost like you're speaking different languages when it comes to all of it.
00:24:24 Deedy: Yeah. Yeah. And and there's like a socio cultural element to this too. I don't know if you've been seeing this, but like a lot of the youth of of the world today like Gen z, I don't know what Gen and Gen Alpha. Yeah. There's, like, this hatred of AI Interesting. In in a in a huge way
00:24:43 Deedy: Wow.
00:24:43 Deedy: Where it's, like, it's not cool to talk about it, to use it. Interesting. It's it's the most weird like like, I feel like an uncle now even saying this, but I'm like, dude, when I was a kid, all new tech stuff was cool. It was really cool. Nerdy, but cool. Yeah. What is what is that reaction? And and maybe it came from the world of art where I know AI has had a really, really negative reaction from
00:25:06 Deedy: all artists.
00:25:06 Deedy: For sure. But now it's it's it's it's seeped into just AI at large. People are just like, ugh.
00:25:11 Greg: Yeah. Alright. Well, whatever. They're they're lost.
00:25:14 Deedy: They're lost.
00:25:15 Greg: Cool. Well, I tell you what, let's do some building. So, Didi, one of the cool things is that a lot of the projects you have surround or revolve around diagrams, and that's cool because it's not only very pretty for humans to look at, but it also actually helps AI understand the flow of something a little bit better that language may not express in the first place. So why do you love diagrams so much?
00:25:34 Deedy: I like diagrams are just way more expressive. It's it's bringing structure to unstructured data.
00:25:39 Deedy: Mhmm. And I
00:25:40 Deedy: think the caveat is, look, it's not gonna be perfect. You're gonna get things wrong, but just the fact that I'm a huge book nerd Yeah. And I love reading books and it was always just, like, thing in the back of my mind where I'm like, I wish I had a character diagram because my memory is terrible. Oh. Like, again, I watch Game of Thrones and every episode, I'm like, well, who's this? Yeah. Who's that? Why do they know them? Yes. And I just I don't wanna read a Wikipedia again. I want to see the relationships. That's cool.
00:26:03 Greg: And that
00:26:04 Deedy: kind of inspired that, you know, viral Harry Potter treat back in the day Sure. Where I was like, let me just throw all Harry Potter at, in that case it was Gemini for the context of this and just get a graph out of it. That's perfect. A diagram out of it. So maybe you can do that. That's one and I'll another one I'll talk about and we'll do it Yeah. Is getting diagram from code. He said, hey, the biggest problem in the world when you look at a new code base is it's not the time you spend reading it, it's the time you spend not doing anything because you're overwhelmed.
00:26:35 Deedy: Sure.
00:26:35 Deedy: You're looking at it, you're like, nah. I don't know.
00:26:37 Greg: That's because it's just a bunch of text files. It's like it's where's the insertion point? Where do I go start? Where
00:26:41 Deedy: do I start?
00:26:42 Greg: Right? Yeah.
00:26:42 Deedy: It's kinda crazy
00:26:44 Deedy: if you
00:26:44 Deedy: think about it. Where do I start? And so I oh, we'll do a demo. But what I did was I, like, like, dumped the entire repository into a plain text file Nice. Put it into an LLM and go like, hey, tell me tell me what the code looks like.
00:26:57 Greg: Yeah. Love it. Beautiful.
00:26:58 Deedy: So let's start with let's
00:27:00 Greg: start with number 1. Beautiful.
00:27:02 Deedy: We will what do
00:27:05 Deedy: you wanna summarize? What'd be cool? What's something relevant? Okay. I have a good one. I have a good one. Let's let's start with this. Can you you can see my screen. Right? Yes.
00:27:12 Greg: Yes. I can.
00:27:14 Deedy: It's funny that you mentioned the the work Oh, good.
00:27:16 Deedy: The the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the
00:27:17 Deedy: the the the the
00:27:22 Greg: that would be nice. Okay.
00:27:23 Deedy: Cool. Let's see what we can do here. This is all live. I have not done anything before. I have not prepped for this and I think that is going to be the beauty of it. So where are we?
00:27:34 Greg: So you like term like working in the terminal almost?
00:27:36 Deedy: Yep. I am a purist I guess they would call me at some point. So I'm going to use, I like to use Zed so That's cool. Sorry cursor guys but this is awesome. So this is a VTT file and what this means is I use this tool called YTDLP. YTDLP not only helps you download YouTube videos but it helps you download transcripts. So I'll actually show you a It's
00:28:00 Greg: just the transcript that comes straight from YouTube itself. Right? Yeah.
00:28:03 Deedy: Okay. So we got it. So what that did was I generated this previously with Claude. If you look at this code, it is basically trying to take out the timestamp artifacts which take a bunch of tokens and it gives me some plain text. And this plain text is not great and there's some repetition Uh-huh. But it's it'll get
00:28:21 Greg: you there.
00:28:23 Deedy: It'll serve our purpose. So let let's see if this one fits. We're we're gonna go we're gonna put it all in. Oh, great. It fit. And I'm gonna say draw a graph. I know mermaid is the default, but I don't like using mermaids. So draw a visually aesthetic graph.
00:28:41 Deedy: This is
00:28:42 Deedy: what I just like to do.
00:28:43 Greg: There you go.
00:28:43 Deedy: In as a dot file.
00:28:46 Greg: Why don't you of mermaid out of curiosity?
00:28:48 Deedy: I think mermaid is too simplistic and I'm just like a a more old school and mermaid comes from something called dot and dot or graph is is what I used to use in college and so I'm just more used to that.
00:28:57 Greg: Yeah. There you
00:28:58 Deedy: go. As a dot file
00:29:01 Greg: For those that don't know, Didi was a hardcore engineer before investing.
00:29:05 Deedy: I know.
00:29:05 Greg: We actually had our first conversation when you're still at Glean a a a long time back.
00:29:09 Deedy: Yeah. I I was an engineer. I was an engineer with with subgraphs demonstrating their relationship between various things that are talked are spoken about in this podcast.
00:29:28 Greg: What what about prompting techniques? Do you do you like to think out loud? Do you like tips? Or what what's your take on those?
00:29:33 Deedy: I wouldn't say, like, I'm a pro prompter. Sure. Like, my my advice is don't think. Do. Yeah. Just go
00:29:41 Greg: try something out.
00:29:42 Deedy: Just go try something.
00:29:42 Deedy: Yeah.
00:29:43 Deedy: You'll figure out in the end. Like, I a couple of these things that I've done, I've just figured out over time.
00:29:47 Greg: Sure.
00:29:47 Deedy: So here, it generated a bunch of stuff. Now what I do is I will go and take this to online graphviz, Paste it in and Oh, look at that. There's
00:30:02 Greg: one shot too. That's
00:30:03 Deedy: nice. Yeah. It it's just so cool. So we have some personal background, hearing impairment, social isolation, reading focus, talks about rabbit holes, patterns of theses. And it gets it gets it gets you some somewhere
00:30:19 Deedy: Yeah.
00:30:20 Deedy: And it looks pretty nice.
00:30:21 Greg: This is just such a nice, like, better way to look at a long body of text. Yeah. It just makes more so much more sense.
00:30:28 Deedy: Makes it so much easier to do. And Yeah. And we'll do a little bit more on this because I think this is a question I get asked a lot, which is how do I get out of this context window limitation a little bit? Correct. So I like to do this is one of my techniques, my prompting hacks. Assume all of these nodes and edges exist. Keep going. Add new ones and make it exhaustive.
00:30:52 Greg: And so you wanted to go you wanted to double click here?
00:30:54 Deedy: I just want it to go deeper. I think this graph was a good start.
00:30:57 Greg: Yeah.
00:30:57 Deedy: And let's see let's see where it can get me. And so it does regenerate some of the boilerplate here. Yeah. But it it's going on and on and on. Oh, look. We have some GLP drugs. Some China. Some great stuff. Great stuff. So now what I'm gonna do is I'm actually not gonna copy the whole thing. I'm gonna copy just the subgraph thing and let's see if it worked. So go back to this and oh, look at that. And it got even more complicated.
00:31:36 Greg: That's nice. So That's cool.
00:31:38 Deedy: So now it's like, okay. You're, like, if if you're a hyper productivity guy, you don't have that much time, you wanna listen to this podcast, but you don't have 3 hours.
00:31:47 Greg: Yeah.
00:31:47 Deedy: Download it, and now you just know.
00:31:50 Greg: It'd be so nice to have time stamps for each one of these and just click on it and then
00:31:52 Deedy: just listen
00:31:53 Deedy: to it.
00:31:53 Deedy: It's funny that you ask. It's, like, let's say I'm interested in AI and intelligence. Can you time well, it can't do that. It doesn't have the time stamps
00:32:04 Deedy: right now.
00:32:04 Greg: But it would be really cool.
00:32:05 Deedy: It would be really cool.
00:32:06 Greg: Yeah.
00:32:07 Deedy: Look, in a world where we could fit the timestamps and I've done that before too which is like just give me quotes with timestamps
00:32:13 Deedy: Yeah.
00:32:13 Deedy: For for these areas that I like.
00:32:15 Greg: That's cool. So then other than just looking at the diagram, will you use that diagram schema anywhere else for it? Like any other problems? Like, does that make its way to an input to a prompt for you?
00:32:25 Deedy: I haven't tried that yet.
00:32:27 Greg: Yeah. One example I talked to, I talked to another person on this interview series called Arjan, and he wanted to judge how well his support people were following the call script that they were instructed to do. So we first started with it with the diagram flow of the call script and what it should go here go here, if this then go here blah blah blah. You give that, pair it up with a transcript, and then the AI will LLM as a judge will judge how well he followed that script.
00:32:50 Deedy: I've seen some voices coming up to that and that's
00:32:53 Greg: very cool.
00:32:53 Deedy: That's pretty cool.
00:32:53 Greg: Yeah. Okay. So that's on the diagram side. Let's go into computer use.
00:32:58 Deedy: Let's do it. Yeah. So we're going to and again this is my fork of of of anthropic computer use. Totally unsafe. Do not use it.
00:33:06 Greg: Fork. Fork. Okay.
00:33:07 Deedy: Yeah. Do not use it. It can do all sorts of weird stuff. One thing that went wrong when I when I was trying to make this do something was they started a rogue process and just created, like, a 1,000,000 files in a directory which then stopped loading because the computer just went bonkers. So let's try to do something here. What I'm gonna do first is we're gonna go to Claude and we're gonna say instruct a computer to and, you know, there are some people who like to save prompts and they're, like, very meticulous. I'm I'm not that guy. I kinda I I instruct a computer in as much detail. Do not write the code now.
00:33:47 Deedy: If you tell Cloud to do that because it loves to write code. To do the following. Create a directory called dev slash we'll call it greg demo. Nice. And in that directory, create an you tell me, Greg. Let's we could interact.
00:34:13 Greg: Create for it app wise.
00:34:13 Deedy: We're gonna create an app with a front end and a back end. Give you some ideas, like, what kind of stuff do you think and I can push back. I think
00:34:19 Greg: it's a complicated
00:34:20 Deedy: but
00:34:20 Greg: For the front end and the back end. Let's I mean, let's just do it on brand topical. No. It's kinda lame. I was gonna do, like, VC inbound blah blah blah. Blah. How about, like, simple restaurants in San Francisco?
00:34:35 Deedy: Simple. Thinking about that one. What's the name? APIs APIs will be tough because you have to
00:34:41 Greg: Oh, you wanna pull some of my
00:34:43 Deedy: real data too? Oh, we could. I'm trying to think. You know what? I'll do one now. I wanna to get us started.
00:34:51 Greg: I don't know how complicated you wanna go. I wanna scrape OpenTable to know when reservations are opening.
00:34:56 Deedy: Oh, interesting. Because there's this one spot
00:34:58 Greg: in San Francisco that every time I look, it's always booked, and I'm never there early enough.
00:35:03 Deedy: You know what? It's not gonna be as visual, but let's let's see how far we can go. Alright. Let's see how we can do that. In that directory, create an app to scrape OpenTable. OpenTable.
00:35:19 Greg: And I have the restaurant
00:35:22 Deedy: reservations in San Francisco to find do you have a restaurant in mind?
00:35:30 Greg: I do. I I just put it in the Riverside Studio chat. It's called 4 Kings San Francisco.
00:35:39 Deedy: How do you spell that? I was just Oh,
00:35:40 Greg: just 4, like the the the number 4.
00:35:43 Deedy: To find 4 Kings availability, and let me no. I'll I'll I'll make it interesting and but tell I'll just keep it simple for now. Tell me when it opens up. Nice. And so let's see how this goes. And it might it might say something about
00:36:12 Greg: So are you making the prompt right now that you're gonna go then give to the computer use? Yeah.
00:36:16 Deedy: And then okay. Yeah. Man, this is if you could I mean Oh my goodness. This is a lot. I don't even know. This is a lot. This is a lot. So, yeah, this is gonna be fun because I have no idea if this is gonna work. We'll see how far it goes. Would you like me to proceed? And then let you know I don't want you to proceed. Try to use this is another one. Try to use command line only Nice. As much as possible and build an MVP. Yes. Boom. So now okay. Let me open up what's going on here. This is how I typically look at it. Okay. It is trying to it's creating all these files. It's trying to it's just trying to create a config. Mhmm.
00:37:07 Deedy: It's trying to download all the requirements for Python. This is where it's running at. And then real quick, let me just go into that demo. But I think for a lot of people, if you do prompt it the right way, absolutely.
00:37:20 Deedy: Yeah.
00:37:20 Deedy: Like, I have a friend, one of the one of the people who work at one of the companies that I I I named
00:37:25 Deedy: before Let's let's pause
00:37:27 Greg: there and just bring it up for a second. You had a tweet that said everyone thinks this is an exaggeration, but there are so many software engineers, not just Spang, who I know that literally make 2 code changes a month and they're getting paid 200 2 to $300,000 a year. It went viral. It popped off. You listed the name of companies. So who who's this friend at the company?
00:37:44 Deedy: There's many friends at many companies. Wow. Andre Karpathy tweeted back at that and I love like using other people to validate that this is a real thing. And he's like, people brag about this all the time.
00:37:55 Greg: Yeah. There's entire subreddits dedicated to
00:37:57 Deedy: it. Yeah. Overemployed is an entire subreddit dedicated to it. So I think like for me as somebody in the software engineering community, I don't think this was like a de novo type thought. Everyone knew this happened.
00:38:08 Deedy: Sure.
00:38:09 Deedy: I think outside that community, maybe people didn't know as much. Maybe the executives didn't know as much. And now maybe with the street, some people do. Without question, there are people who don't do anything. Okay. And is it their fault? Is it their fault? No. I mean, they're not sometimes Well,
00:38:23 Greg: it's just it's where incentives are.
00:38:25 Deedy: Right? Incentives and the incentives go down up the chain. Right? The managers aren't giving them work. They don't have anything to do because the managers don't expect them to do anything. Managers at any big company never have incentive to get rid of you. Yeah. They always have I have an I have a follow-up tweet about that. They always want to increase headcount because that's the only way that gets them promoted.
00:38:45 Greg: Sure.
00:38:46 Deedy: So no matter how bad you do, they have no way. They have no reason to get rid of you.
00:38:49 Greg: Sure.
00:38:49 Deedy: And and look, the the the nature of work is you're not gonna have work for 5 people. Let's say you're a team of 5. You're not gonna have consistent work for 5 people consistently. The problem happens is when you have basically no work for no people. Uh-huh. And then you're like, I can't like, I'm a manager. I can't tell my manager that I don't have any work for these 5 guys. So I have to then pretend that I have work so that he like, I keep my job.
00:39:10 Deedy: Mhmm.
00:39:10 Deedy: And then it's just this thing goes up the chain and then nobody does anything. Mhmm. But to get back to it, there's a follow-up. I'll I'll mention in that thread where somebody said something something. What does this have to do with like, does AI have anything
00:39:22 Deedy: to do
00:39:22 Greg: with it? Sure.
00:39:23 Deedy: And the answer was no. Like Yeah. The people who are doing these jobs have no idea what computer use or Claude or attach a p t. They're practically not even they don't care. They're in Mexico. They're in Hawaii. They are vacationing all the time. Yeah. So That's so crazy. But when you think about and I can can give one example of a couple of friends that work in a job like this. They were telling me I'm like, dude, what when you do work, what do you do? And it was like, we maintain the service. It's a big Java service and, like, I just had to do, like, one dependency upgrade once a
00:39:59 Deedy: month. That's
00:40:00 Greg: so crazy.
00:40:01 Deedy: A, that's ridiculous. Uh-huh. And b, let's talk let's talk about the dependency upgrade. Yeah. How much time do you spend on it? And he's like, oh, I spent a lot, dude. It was a hard one. I spent, like, 10 hours. And I'm looking at like, okay. Can you can you show it to me? Show me show me what you're doing. It was a buddy of mine, and we we met, and he pulls up his computer. He was showing me a little bit. I'm like, dude, why are you doing this manually? That's insane. You're really trying to resolve Java depths, and I know that this is insane.
00:40:26 Deedy: It's like
00:40:26 Greg: go to Stack Overflow or something.
00:40:28 Deedy: Yeah. He's basically googling a bunch of things to try to figure out how to use this dependency system that he does not know how to use. I'm like, dude, you can run this through, like, Claude and it will will fix it for you. Yeah. And and his response was, well, they don't let us use AI at work. Mhmm. And I'm like, okay. Well, that's a pretty shitty response because, like, you're the person who's gonna have to do it now.
00:40:50 Greg: Yeah. Exactly.
00:40:51 Deedy: Not not me. And so that's kinda the that's kinda where it is.
00:40:55 Greg: Alright. And so where we at now with this thing?
00:40:58 Deedy: Let's see. There are this is a problem that happens where here's what happened. It ran the scraper. The scraper ran and it blocked the thread.
00:41:09 Greg: Oh, okay.
00:41:09 Deedy: It waited 4 minutes and then it timed out.
00:41:12 Greg: Okay. Well, there you go. That's the work in progress with it.
00:41:15 Deedy: So we'll see whether and and there's a log. So there's a little bit of a log, and it said
00:41:23 Greg: Nice. At least we're at the right URL, which is good.
00:41:25 Deedy: Started a
00:41:25 Greg: reservation. Loading reservation bridge. Yeah.
00:41:27 Deedy: Oh, look at that.
00:41:28 Greg: Pretty good about scraping.
00:41:30 Deedy: Yeah. Yeah.
00:41:31 Greg: If any OpenTable engineers are listening to this.
00:41:35 Deedy: So it looked like the error was there was a time out, and then it schedules to check every 5 minutes and then, therefore, computer used timed out.
00:41:41 Greg: Schedules to check every 5 minutes. There you go. Nice, man.
00:41:46 Deedy: But, look, it wrote a lot of code. It somehow wrote wrote this, like, notification sending code, which
00:41:53 Greg: That's crazy. It just
00:41:54 Deedy: has to be emailed
00:41:55 Greg: from it.
00:41:55 Deedy: And emailed to you.
00:41:57 Greg: It's like sometimes when I look at AI code that's like this, I just get overwhelmed. It's like, man, I don't have any mental bearing or anchor points to to lock myself into this and it makes me a little nervous sometimes. But I'll tell you what, when you click run and it works, it's like, well, alright.
00:42:10 Deedy: It's true. I I feel a little bit better because I, like, sometimes understand what's going on. Mhmm. I'll give you one example now that we're that while we're at it. And I know I know we're
00:42:18 Greg: Yeah. Yeah. Yeah.
00:42:19 Deedy: At the time but I can I can go to to 1:30 flats? Don't worry. Beautiful. Here. I will do another one of these and actually, you know what? Let me do a simpler one. K.
00:42:34 Greg: And while you're doing that, I would love to hear about what's the stack you're using right now? Where do you like to deploy? What what's your DB? What's all that stuff?
00:42:40 Deedy: I keep it really simple. Yeah. My this was true of when I did this at companies which is do what you know. Yeah. Don't do what you don't know.
00:42:47 Greg: Sure.
00:42:47 Deedy: Sometimes I I wanna learn something new. I'll learn something new. I know Google Cloud really well. Nice. So I do Google Cloud auth Yep. That's one one command after I run on my command line and then I say go deploy it with Google Cloud. I ask computer use, tell me what to do. Yeah.
00:43:00 Greg: And it just does it. Nice. So GCP.
00:43:03 Deedy: GCP and then for what else? What what other
00:43:09 Greg: Let's see. I mean, GCP, you you're using zed? You're using a lot of clog?
00:43:12 Deedy: Oh, I use I use zed. I use cursor too. I I I use a lot of clog and computer use. Sometimes when claw doesn't work, I won't go to o one, and I'll try to, like, go back and forth and get different answers. I use Gemini when I need long context. Gemini's pretty useful for that. Yeah. That's that's kind of my stack. When it comes to languages, there's another really cool project that I that I was working on where in in Rust. And so sometimes I'll use Rust when I view it to be super fast. But I'm mostly doing stuff in Python and then if I do front ends, it'll be in React. Cool. That's mostly it.
00:43:48 Greg: Beautiful. Love it, man.
00:43:49 Deedy: Real quick, I wanna tell you about, like, the coolest example Yeah. Of something I learned when usually, it's one of the hard mental constructs with AI is you don't know what it can do. So there's, like, this unknown unknown problem where I know what I can do. Therefore, I ask it to do things that I know that I can do. Yeah. I don't know what I can't do. Therefore, I can't ask it to do that. But there was this one example where I I asked Claude again, meta prompting, to our point. I'm like, give me some random ideas. And I just got out their ideas. And then I took some of those ideas, and one of those ideas was get it to make a song.
00:44:22 Deedy: Yeah.
00:44:22 Deedy: And in my head, I'm like, well, that's stupid because obviously computers can't do that yet. And but I'm like, okay. Well Yeah. What would happen if I asked it to do that? And this one blew my mind because I didn't know it was possible, and I'll show you this example Sure. That you could create waveforms from scratch
00:44:40 Greg: Just like with code.
00:44:42 Deedy: Like with code. Yeah. And
00:44:47 Greg: Yeah. Looks like
00:44:48 Deedy: And I just built that from absolute scratch using a thing called socks, which is a library I had never heard of and I did not know. And it downloaded it and it used it. And this is a really simple one, but I've gotten to get some really, really
00:45:04 Greg: crazy That's wild. Okay. So say we were gonna take the, OpenTable example we just did. And it's like, okay. It works. It's cool. It's there. You're about to package it up and you're about to put it on Twitter.
00:45:15 Deedy: Yeah.
00:45:15 Greg: You're very good at this. What is the mental process you're gonna think, okay, here's how I package it up and everything up until you click send or a post.
00:45:23 Deedy: Okay. So step number 1 is GitHub repo. Usually you can get computers to take care of most of that for you. So put it on GitHub. Couple of general sanitary things such as get get some documentation going so people would, like, know what it's Yeah.
00:45:36 Deedy: Yeah. Yeah.
00:45:36 Deedy: So you can just spit out all the code and say, now create a read me for it. Yeah. Get a nice read me, whatever. That's step 1. Step 2 is I think about, like, videos and there's 2 ways to do this. There's, like, the video of the process which has a certain audience and there's a video of the output which has a certain other audience. Sometimes, just a picture works better than video. It's pretty laborious because what I have to do is QuickTime, screen record. I'm not super fancy with it. Yeah. Do a screen recording, run the entire process back, literally, like, load it into Imovie. Yeah. But sometimes I use some other video software and, like, clip it up to see like, make it faster.
00:46:11 Greg: Cut the ums. Cut the pauses.
00:46:13 Deedy: Yeah. And I I usually do no audio for this because that's a whole other
00:46:16 Deedy: Oh, okay.
00:46:16 Deedy: Thing, but, I just wanna, like, show the the demo Yeah. Because it's usually for Twitter or not YouTube. Mhmm. Cut it up. Make it 4 x faster if it's a computer used thing and export. Boom. I use an app called Type Fully to schedule tweets. So I'll put the video on of you never put a link on your first tweet. You put the link in the thread. So I link to my repo in a follow-up tweet and then I just compose. Like, what do I and how do I wanna talk about this? So one one example of that was say I I built a thing on on front of my friend's computer which is I always thought this would go which would be a really nice concept for people to understand which is everyone talks in SF especially is like make me a bot that swipes on Hinge.
00:46:54 Deedy: And especially is, like, make me a bot that swipes on Hinge. And iPhone screen mirroring came out, computer use, and, like, put 2 and 2 together. Like, let's make this happen. And so that was a cool one. And so take the whole video, get computer used to do it, and then package up the demo, get rid of PII, boom, tweet, and then go.
00:47:15 Greg: So you have an idea. You go and build it. You share it, and then it seemed like it goes viral every other post. In fact, when you were doing the pre call for this one, I said, I wanna talk about your viral post. Then you said, which one? And so it's like, yeah. You have so much good content out there. So how how do you build so much? Like, what what's the secret behind it?
00:47:32 Deedy: Firstly, I just wanna say well, I'll I'll tell you the story of how I got into Twitter because I think I think people misconstrue a lot of the the online stuff and I hate, like, the word influencer because that's not what I try to do at all. The reason why I'm able to output so much is because I I think I just genuinely love building stuff. And in the past, one of the biggest constraints I think mentally now that I introspect about it was what's the point of me building this? Yeah. No one's gonna see it and no one's gonna care.
00:48:01 Deedy: Yeah.
00:48:01 Deedy: It's just gonna be a little project that sits on GitHub with no views and I can tell one buddy of mine who cares and nobody else Yeah. Gives a shit. Uh-huh. And now that I have an audience, I'm so I I feel really, like, genuinely grateful that it lets me go, like, I'm gonna go build it because at least some people will see it. Yeah. And And and and it kinda brings me and that's kinda all I wanted to do and That's so cool. The reason I got into writing online in the first place was because I mean, the story is me and a friend were out at drinks. We were having a great time, and I gave him a like like, an idea. And he looks at me, and he goes like, dude, that's a terrible idea.
00:48:40 Deedy: And I'm not I'm arguing, and I'm like, what are you talking about? Like, no. It's a great idea. And he kinda does this flip on me, and he goes like, look. And this is before I really tweeted, but I've been right online a little bit and he's like, I trust your judgment. If you, sober, can go online and write about this publicly, I'll believe it's a good idea. And I'm like, oh, well, that's a that's an interesting thought right there.
00:49:05 Greg: It's literally just framing the idea.
00:49:07 Deedy: Exactly. And then I realized that gap between you having an idea and thinking you're awesome Yeah. Versus pressing the send button Yeah. So big for so many people.
00:49:17 Greg: That's so true.
00:49:18 Deedy: And I realized that that was stopping me from doing a lot. Yeah. I was I I thought I had great ideas, didn't put them into production or put them out there.
00:49:28 Deedy: Sure.
00:49:28 Deedy: And the second thing is everyone does this on the Internet. You go into rabbit holes. We learn things, but you don't remember much.
00:49:35 Greg: Yeah.
00:49:35 Deedy: And this was a good way I found. I promised myself I would go online. I would write one thing every day. Nice. One thing that's somewhat useful to other people that I learned. Sure. And that's how it all started. And I just promised myself to do that and it gave me a reason to go and learn stuff.
00:49:49 Greg: Yeah. That's very cool.
00:49:49 Deedy: Aimlessly exploring, I would say I learned a thing, here's a thing.
00:49:54 Deedy: That's right.
00:49:54 Deedy: And that's kind of how it evolved but then obviously, you know, there
00:49:57 Greg: It goes from there. I was just listening to the Darksh Goran interview, and I think Goran said that one of his north star metric for himself is to maximize the number of rabbit holes he goes down. So it's literally just getting in flow state into an esoteric topic and just going and having fun with it. So when you said Raval, it made me think of that. One thing I would love to do to inspire people to share more here. So going from the journey of having no audience all the way up to 100 of thousands of followers on Twitter. I'm sure there's a lot of, like, cool serendipitous moments that happen along the way.
00:50:27 Greg: Maybe somebody famous reaches out or an opportunity prevents itself or maybe a new career thing. What are some of the cool things that happened along the way of your journey that are due to having a bit more exposure?
00:50:38 Deedy: Oh, this is my favorite one which is I was on the first date with the the girl I've been seeing now for a while and I remember we were we're driving in my car together and across across the Bay Bridge going to Oakland from SF, and it's the first time we ever met. Yeah. And my phone starts buzzing, and it's like, I see a notification. Like, you know, I'm driving and the phone's up there on the on the dash. And it buzzes, and it says something something. Elon tweeted something. And and but it wasn't the normal Elon tweeted something. It was I couldn't tell what it was. I wasn't really looking at my phone, but then my phone just starts blowing up.
00:51:17 Deedy: It's like notification after notification. And I'm like, what is going on? And probably shouldn't publicly admit this, but then I put my car in self driving, and
00:51:24 Deedy: I'm like, I don't know
00:51:25 Deedy: what's going on.
00:51:25 Greg: You're like, babe, I gotta look at this.
00:51:27 Deedy: No. It's because my phone's blowing up. And I saw an Elon thing, and then it blew up. And I'm I looked at it, and it's like, Elon had retweeted and written something to me. And in my mind, I'm like, oh my god. What is going on? And to her, this is, like, meaningless. She's like, I don't know what you're so obsessed about. But I have friends reaching out to me going like, congratulations. Oh my goodness. And and I'm obviously, like, I'm a huge fanboy of of a lot of things Elon does. And so that was that was one moment where I'm like, damn.
00:51:56 Deedy: Like, you
00:51:57 Deedy: the richest guy in the world know I exist. Yeah. That's kinda, like, nice. So those are the positive ones. Like, yeah, the the Elon is I think we've had 10 replies and stuff now and then That's cool. Paul Graham, Mark Andreessen, a bunch of these, like, pretty esteemed
00:52:11 Greg: Yep.
00:52:12 Deedy: Tech people. Yeah. But I think it's worth, like, to take the positives with the negatives and I think the hardest part about writing online is that there's gonna be times people tear you apart and that's happened a ton of times too and
00:52:24 Greg: What what percentage of the energy when you when you when it ends up blowing up like that, what percentage of the energy is negative that comes up?
00:52:30 Deedy: It depends on the post. And I think, you know, there's some there's some threads where you're like, okay. 90% positive. There's a couple of people who are kinda dumb and they're saying, like, nonsense, and that's fine.
00:52:41 Deedy: You you
00:52:41 Deedy: you build thick skin to those things. Yeah. The hardest part is when you get a full blown cancel
00:52:47 Deedy: Yeah.
00:52:48 Deedy: Thing. Sure. And I don't think there's anybody I've met who has some scale online who hasn't had at least one of those. Course. Yeah. Yeah. But when it happens for the first time in a big way, it messes with you, man. Like, as much as you wanna say I have thick skin, you're just like I don't know. You're, like, looking at people on the street and going, like, do they know? Do they know? Do they know that I'm, like, I'm a villain online to millions of people? And and you get over it, but that's happened a couple of times too.
00:53:15 Deedy: So
00:53:16 Deedy: and I I will share that there's so many younger people especially that I've met who literally have had one of those happen and have stopped posting online for years. Wow. It kills people Yeah. Energy and motivation.
00:53:28 Greg: And what's the advice to them?
00:53:30 Deedy: For me, it's like it's the same advice and and, you know, in real life when people say you care too much about what other people think of you, they don't really care.
00:53:37 Deedy: Yeah.
00:53:37 Deedy: It's kind of that. It's like people aren't gonna say shit.
00:53:40 Deedy: Yeah.
00:53:41 Deedy: Keep doing you.
00:53:42 Greg: Just keep on doing it. And that's the other thing too is like the algorithm wants to promote good stuff. And if you post something that's not good, don't worry. Nobody's gonna see it. They're only gonna see the good stuff.
00:53:49 Deedy: There's only one caveat which is like take feedback. So if there is something you're actually doing wrong Yeah. Well, like, okay. Well, what am I actually doing here? But in many cases, that feedback is, you know, just keep doing what you're doing.
00:53:59 Greg: Didi, this is fabulous. Awesome. Thank you
00:54:01 Deedy: very much
00:54:01 Greg: for chatting.
00:54:01 Deedy: Thank you so much, Greg.
# File: /content/learn/dev/ai_show_tell/nicole.mdx
---
title: Nicole, CEO Of Headstart
description: AI-Native Development, Cloud Project Power User, Six-Figure AI Projects, Scaling with AI Agents
date: 2024-12-05
---
[Nicole](https://www.linkedin.com/in/nicole-hedley-a7848a44/), CEO of [Headstart](https://www.headstart.nyc/), an applied AI services company, shares her journey of building a successful AI-native business in just two years. Starting solo and leveraging AI for coding, she quickly scaled to a team of four, charging high six-figure project fees. The interview dives deep into her unique approach to AI implementation, leveraging Cloud Projects, and building internal tools for efficiency. Nicole emphasizes the importance of communication, product thinking, and a "hard problems first" mentality as key drivers of success. She also outlines ambitious growth plans, aiming for a tenfold revenue increase and significant team expansion. The discussion reveals how Headstart uses AI not just as a tool, but as a core element of its business strategy, enabling rapid growth and high profitability.
## Insights
* **AI-Native Approach**: Headstart uses AI to write code for AI implementations, creating a highly efficient and scalable workflow. This allows for faster project delivery and the ability to handle multiple projects concurrently.
* **Proprietary Data Structures**: Nicole highlights that structuring data is as crucial as the data itself. A well-defined data structure enables seamless integration and maximizes the value of the data.
* **Focus on Product Thinking**: Headstart goes beyond just coding; they act as thought partners, helping clients refine their product vision and architecture for scalability.
* **Prioritize Hard Problems**: Tackling challenging projects is seen as a strategic advantage, leading to valuable learning experiences and higher client satisfaction. This approach also commands premium pricing.
* **Internal Tooling for Efficiency**: Headstart invests in developing internal tools, like an AI-powered code review agent and a prompt optimizer, to continuously improve efficiency and quality.
* **Communication is Key**: Strong communication skills, both written and verbal, are paramount for effective collaboration with clients and leveraging AI coding tools.
* **Network Effect of Knowledge**: Headstart maintains a GitHub Wiki to document best practices and code snippets, which are then fed back into the AI, creating a continuous learning loop.
* **Cloud Projects for Contextual Coding**: Nicole strongly advocates for using Cloud Projects over other AI coding platforms due to its explicit context management and superior code output quality.
* **Trusting the AI**: Nicole emphasizes the importance of trusting the AI's capabilities, allowing for faster development cycles and greater reliance on generated code.
* **AI for Prompt Optimization**: Headstart has developed an AI tool to refine prompts for AI implementations, ensuring higher accuracy and effectiveness.
* **AI-Powered Code Review Agent**: An AI agent built using Claude Computer Use assists with code review and automates tasks like creating pull requests. This functions like a junior engineer on the team.
* **Focus on Client NPS**: Client satisfaction is a top priority. Headstart prioritizes delivering high-quality work and building strong relationships.
* **AI-Driven Growth Strategy**: Headstart's growth strategy relies heavily on leveraging AI for efficiency, enabling them to scale revenue without proportionally increasing headcount.
## Transcript
00:00:00 Nicole: We'll deliver like an entire application build in like from scratch in 4 weeks, like fully functional. We're not afraid of anything. We try to take on the hardest projects. We believe that access to hard problems is like a proprietary business value for us. I'm taking like swaths of code from cloud projects when I'm working on client projects. What I love, love, love about cloud projects is that you can share them among your team. One engineer told me he's like you trust it much much more than I would ever trust it.
00:00:42 Greg: That's Nicole, founder and CEO of Headstart, an applied AI services business. She started the company right at the launch of ChatGPT, and now she's charging a minimum of 6 figures per project with only 4 employees. In this interview, Nicole shares the homegrown automations that keep their revenue per employee high, how she uses Cloud Projects to run her entire business, and the playbook that she's using to scale her business from 3 to 14 employees. Let's jump into it. So, Nicole, you and I first got introduced by a mutual friend, and he says, Greg, I met this person. They just started their own agency, and they're absolutely crushing it.
00:01:21 Greg: Can you tell me more about what you have going on at Head Start and what y'all are doing?
00:01:25 Nicole: Absolutely. So I started the company just over 2 years now. I left my job in October of 2022. I actually created the business in December, and between that, chat gbt launched to everyone, November 1, 2022. So it was quite lucky timing in all of it. I had actually had early access to the OpenAI models. And so I had been using GPT 3 before it was a thing, and I knew I wanted to start my own business, but I didn't know how to go about it. And so I started consulting coding in line with the chat gpt launch. So I was using chat gpt 3.5 to write code for me before everyone was doing it. And I remember to this day people being like, oh, chat gpt 3.5, it's not good at writing code.
00:02:11 Nicole: And I had a consulting project that I was on that was Ruby on Rails. I had never done Ruby before, and so it was writing all my Ruby code. And I'm like, well, it's good at writing Ruby code, so I don't know if it's something about the way that I'm using it or what's different. And then by the time GPD 4 launched in April of 2023, everything had changed. And so, obviously, since then, we've gotten almost weekly updates from Anthropic and OpenAI dropping things that we use to code. And so my business, Headstart, is an AI native applied AI services firm. And so it's very meta in terms of, like, how we use AI. So we use AI to write code that implements the AI for our clients.
00:02:56 Nicole: So very much just AI. Yeah. Yeah. Yeah.
00:03:00 Greg: Yeah. And so did you start head start with AI in mind, or was it no, we're we're gonna do coding services and then AI came around?
00:03:09 Nicole: I didn't I kind of started the business more as just I was consulting myself, and what I knew how to do was code. And so it was more around, like, what can I sell in general that I can just bootstrap a business? And so that was really important to me was not necessarily raising capital, but figuring out how to do it on my own. The AI piece came in line with it. I started using the AI to write the code before I started implementing the AI. At that time, every company was like, I want a chatbot. I want a brag implementation. I was like, I need to learn this, and I was very motivated by the money incentivized to learn it if I were to get paid to learn it.
00:03:45 Nicole: And so I basically started implementing it for people knowing that they would pay for it and then building the business up that way.
00:03:52 Greg: For sure. So tell me about what what are the bread and butter projects that y'all take on right now? And, like, what are your clients asking you for?
00:03:58 Nicole: Yeah. So it's really interesting. It shifted over time. The 1st year we were in business, it was a lot of chat bots and rag implementations. We actually still do a lot of those, but there's a bit of a different flavor. And then the more interesting projects, in my opinion, are when the client has actually tried to implement the AI themselves, and they're not seeing as strong results out of it, or they just want it to do more than it's able to do with the implementation that they have. And so we'll actually come in and either redesign an implementation or do something new and get very, very strong results with the new implementation.
00:04:31 Nicole: So we do a lot of unstructured data to structured data, which is obviously an incredibly large bucket. Sure. But document processing, a lot of web scraping, anything that can be kind of data mapped, so one structure to another. We think in this world, proprietary data structures are as valuable as the proprietary data itself when you think about, like, how you put things together. We do a lot of that. Just honestly, the technology is so far ahead of everyone using the technology that I think it's gonna be years years before people are actually, like, caught up to the usage of it, which is
00:05:08 Greg: I get the proprietary data side. But what do you mean by proprietary data structure? And why do you think that is so important?
00:05:15 Nicole: So that's interesting. I think there's so much data out there and there's so much data available. And the advent of all the LLMs, like, all the data has been now processed by the LLMs, and the LLMs can generate data too as much as you need it. And so the structuring is interesting because, like, as you think about software products and as you think about value within it, like, something being structured either relationally or however else you wanna structure has inherent value to it. And so if you put the thought process behind what should that structure be, and then you can access the data from wherever it is, that becomes very, very valuable, if that makes sense.
00:05:53 Greg: Sort of. I would I would love an example of those. Like, what what do you mean by that?
00:05:56 Nicole: Like, if you think about so every API, right, every company that possibly has an API is a certain structure, and you think about, like, lately, a lot of software that's built is integration, so you're integrating one thing to another. And that kind of API wraps the proprietary data structure in the first place. If you're then trying to, like, plug into multiple integrations, there's a mapping that you would maybe do that's like, this is a common data structure between these integrations. That becomes valuable too because then you can kind of, like, pipe things to other things much easier.
00:06:31 Greg: Sure. Sure. Sure.
00:06:31 Nicole: Yeah. And so, like, as you think about a company that's, like, leveraging data to make money in whatever sort of way, and maybe they're taking they're inputting the data from all these third party sources, doing that mapping into their structure is almost as important as the data.
00:06:47 Greg: Sure. Yeah. I mean, it's the normalization process to, like, actually make Exactly.
00:06:50 Nicole: It's normalization as a yes. But normalization is so much easier with the LMS.
00:06:55 Greg: Yeah. Yeah. Yeah. Yeah. For sure. Okay. So you started Head Start 2 years ago. What did the team look like now and or what did the team look like then and what's it look like now?
00:07:04 Nicole: Yeah. That's a great question. So it was just me for a year and a half. I was able to How
00:07:09 Greg: did you find how did you find clients? Like, how did you get your book?
00:07:12 Nicole: All inbound referrals. So it was all people that were like, I either need the software help or AI help. Part of the reason that the book was so broad was because I had no restrictions on technology because I could do anything with AI. So it wasn't like, oh, I'm just a Python engineer or I'm just a JavaScript engineer, I can only take those projects. I could take anything because I could literally do anything because the AI was coding anyway. So, you know, that helped make it, like, very, very broad, in terms of what projects I could take on, and then I was able to scale just me without hiring anyone because of the AI.
00:07:46 Nicole: Because I wasn't writing the code, the AI is writing all the code, so all of a sudden instead of doing, like, one project, you can do 5 projects at the same time. And Yeah. Yeah. You're like, okay, I don't really need employees if I have AI. Obviously, we've shifted away from that, and we have employees now, but
00:08:03 Greg: Sure.
00:08:03 Nicole: That's still, like, the general way that we want to scale the company is in a very, very AI native way, which kinda goes against the way that you would think about, like, a traditional company of just grow headcount to grow the size.
00:08:18 Greg: Totally. Well, Nicole, I don't think you're giving yourself as much credit as you should be because there are bajillions of people sitting in the rooms messing around on cursor, like, trying to build stuff. But, yeah, you've actually built a company and you actually have real clients. And so what is your unique advantage? Or, like, how would you assess your skill set on top of just AI coding for you? Like, what else do you bring to the table for that?
00:08:38 Nicole: Yeah. That's really interesting. So I I don't think that what we do is, like what we're doing with the AI is not what's proprietary about the business. Part of the reason I'm happy to, like, hop on the phone and show everyone exactly how we're doing things is I don't think that that is exactly the mode that we have in the business. It's incredibly powerful. Everyone should be doing it. It's incredibly valuable for our clients, and you get a huge return on it. But I don't think it's, like, the most important thing for the business. Ultimately, it's, like, communication, how you get things done. The whole thing's hard work, and I think that's probably the biggest blocker is, like, even coding with AI to get projects done.
00:09:19 Nicole: It's hard work. It's communication. It's figuring out how the product should work. It's getting into the nitty gritty details. Like, the AI does a lot, but, ultimately, so far, it only does the coding, which can reduce your time significantly, but you still have to do everything else.
00:09:35 Greg: Well, so what what's in that everything else bucket that you think is most important?
00:09:39 Nicole: I'd say product is a huge part of it. Right? Like, how should it work and being a thought partner on that. Part of the reason clients like to work with us is because we actually help them figure out how the product should work versus just implementing it exactly how they want it, like, to work. So a lot of clients, you know, if you just know what you want, you can go to any developer and get them to build it for you. But we do more of the thought partnership and, like, well, how should it actually work? What is the best thing to do? And, like, what different things can we build around that? And then the architecting of it for scalability.
00:10:09 Nicole: Like, that's the other piece is, like Sure. With AI doing all the coding, we can focus on the architecture of the solution and making sure it's the right solution that scales and everything's, like, just well done and in components with the design system and everything that you possibly need because the AI is gonna do it all that way anyway.
00:10:29 Greg: For sure. So For sure. I
00:10:30 Nicole: think that's the other benefit is you actually get all these other pieces. But, ultimately, it's kinda communication and delivering the end result, and a lot of that is working with a client to figure out what they actually want.
00:10:42 Greg: Yeah. Yeah. Yeah. Which is the hard part a lot of the time. So you have 3 people on the team right now. Right?
00:10:47 Nicole: Yes. Three people, a cofounder, an engineer, and a second engineer starting week after Thanksgiving.
00:10:53 Greg: Nice. Congratulations.
00:10:54 Nicole: Thank you.
00:10:55 Greg: So soon to be 4 people. And so, 2 engineers or I guess what what are their skill sets look like? So what what skill set are you doing versus your cofounder versus the 2 engineers?
00:11:04 Nicole: Yeah. That's a really good question because we're also hiring and we're looking for a very different skill set than is like a traditional engineer skill set that you would look for, Between me and my cofounder, who's also my brother, I'm the technical one, he's the nontechnical, and so we honestly, like, have very complimentary skill sets, which is amazing for the business. I do all the technical and a little bit of the sales and client comms. He also does sales, and he does all the operational. So he does all the internal operations. He does a lot of product work too. So product scoping is a huge part of our business because we charge flat fee, and so we have to get all the scope up front in order to do that.
00:11:41 Nicole: He does that. On the engineering front, we look for product engineers. So we look for product builders. We look communication is such a massive thing. I think communication is always important as an engineer, but lately when we're looking to hire, we're actually looking for written communication skills as well as verbal communication skills because when you're using the AI to build, you actually have to use language incredibly well
00:12:05 Greg: to do the
00:12:06 Nicole: code that you're looking for, and so we need people that can like look at code and understand whether or not it's good, but they actually don't need the skill set to write the good code.
00:12:18 Greg: Sure. Sure. Yeah. It totally does. Alright. So 4 people, how many clients? Right? Or, like, what is the what is the portfolio look like?
00:12:26 Nicole: Yeah. So we typically take on, like, 4 to 5 at once is our kind of typical workload, but that's growing as we grow engineers and the size and kind of timeline of the clients differs dramatically. So we'll typically do, like, a pilot project with a client that's anywhere from 2 weeks to 2 months, and then the client takes the results of that product. They go and user test it. They use it for whatever they would like to if it's an internal product. Like, their entire team's using it, and they figure out what more that they want In that off time, we're working on other clients, and then they'll come back for another project.
00:13:01 Nicole: So we kind of have clients coming in and out. So it's not contractually recurring revenue, but it's reoccurring revenue because they're typically having a good experience, and they wanna then build more. Our best client is someone that's like, I wanna build everything. And
00:13:15 Greg: so you
00:13:17 Nicole: slowly break off chunks and build it all for them.
00:13:20 Greg: Yeah. Yeah.
00:13:21 Nicole: Yeah. Which is really fun, but we're figuring out exactly what is kind of a client per engineer thing look like and how to scale that. Typically, I'm still doing all the client comms, which is really interesting, and we are keeping the engineers more on product work and building out internal tooling to support us handling more work at once.
00:13:40 Greg: Sure. Sure. Sure. How long is the average client engagement that you have?
00:13:46 Nicole: Average, maybe, like, 4 to 6 weeks. They're typically short. We deliver quickly, and so we kinda charge a premium in order to guarantee a fast delivery. And so we work with people that want something done yesterday, essentially, and they know exactly what they want and they're ready.
00:14:05 Greg: That's awesome. That's so nice.
00:14:07 Nicole: Built as quickly as possible.
00:14:08 Greg: Okay.
00:14:09 Nicole: We are a bootstrap business, and it's been really, really incredible to hire out a team of US based, like, highly paid engineers.
00:14:18 Greg: And you're all local to New York. Right?
00:14:20 Nicole: Yeah. Local to New York. It's been really incredible to sometimes I don't even believe it. You're like, I'm running a real business now. Every step is that I'm running a real business from, honestly, like, creating the LLC all the way to now we're looking at a bigger office and more employees. So
00:14:39 Greg: That's why I'm thinking more in real time. How many people are you looking to hire?
00:14:44 Nicole: So we are looking to hire 5 more for January and then five additional for May. We have a lot of cloud demand. So
00:14:53 Greg: No joke. So you're gonna go from 3 right now, almost 4, all the way up to 14 in 7 months.
00:15:00 Nicole: Yeah. So Yeah. Really, really crazy. We're being very, like, particular about hiring just in the skill set that we're looking for because we're doing such AI native things. It's a little bit different. So we are honestly still figuring out and defining what that exact hiring profile looks like as Sure. We get into it. We're very, very interested for May in new grads. So people with computer science backgrounds who are actually not trained in the typical way of coding so that we can train them in our way of coding.
00:15:33 Greg: Nice. Yeah. That's very cool. Random question, but out of curiosity, how do you change your language when you're talking about AI with clients versus, like, technical folks? Like, what is the client tongue you put on to make it make sense to them?
00:15:46 Nicole: I talk very technical. Sometimes my brother has to, like, pull me out of it because it gets a little bit too much in the weeds if I'm getting, like, really into it. But we we're very technical with our clients. I think, ultimately, everything does have an explanation, and, like, people are curious about it, and we're we're kind of, like, willing to explain as much as anyone's willing to listen and hear about. We find that our most engaged clients actually really appreciate it because they even if they're not engineers themselves are, like, interested in learning about how the technology can work and how you can use it in order to really understand that, understanding some of, like, the baseline of it is helpful.
00:16:26 Greg: Totally. And where do you like to play with regards to minimum contract size? Like what what what what starts the conversation with you?
00:16:33 Nicole: Yeah. So it's
00:16:34 Greg: Hey. So after the interview, Nicole said, hey, maybe we don't actually wanna share the exact number of their minimum contract value because it changes all the time and no no reason to stay get in the ground. But what they did say is that they're charging mid 6 figures on short term contracts. That's 100 of 1,000 of dollars, which is very cool to hear. And they're also planning on 10 x ing their revenue in 2025 as they get more enterprise contracts, which I just think is absolutely amazing. So let's go back to Nicole and hear more about Head Start.
00:17:01 Nicole: We've had to raise it due to demand. When I first started the business, the first flat fee I did, I think, was 10 k. And so we just started raising it according to demand. And what's cool about that too is, like, we get a lot done for that amount. Like, we're getting so much more done for that amount than I think anyone else would.
00:17:20 Greg: What do you mean so what do you mean so much more done?
00:17:23 Nicole: Like, we'll deliver, like, an entire application build in, like, you know, from scratch in 4 weeks, like, fully functional, completely ready production ready on prem.
00:17:32 Greg: And so one thing that really stood out to me when we were meeting in New York, you're like, Greg, the more projects we do, the the quicker each next project gets because we learn from the last one and we have templates. And so how does that work with regards to it's almost like your or, institutional knowledge that you're building up that makes the next one quicker. So what does that look
00:17:50 Nicole: like? Yeah. So, we have a Wiki in GitHub, and we're trying to build that out. I'm sure you've heard this all with LLMs is, like, garbage in garbage out. And so if you're putting, like, shitty stuff into it, you're not gonna get good output. But if you put really good input into it, you're gonna get great output. And so we're doing similar things over and over again. I'm usually using using the LLM to generate the first version of what we're doing anyway. So I have it generate a wiki alongside that. As I go through its instructions, I'm actually fixing it. So you're almost, like, kinda fine tuning it yourself.
00:18:19 Nicole: What do you think about, like, okay. That wasn't the exact output I wanted, but this is save it into our internal Wiki, and then we have it next time that we need to do it to either use from our Wiki, but better yet, put it into the l m. And so this is how we know to do it, and we need to do this thing on top of it. So it's very network effecty of, like, the more work that we get, the more we can build out the internal knowledge, the more the entire team has access to the internal knowledge, the more that we can feed that internal knowledge into the OLM and get even better results out.
00:18:50 Greg: Okay. Wait. So this sounds super interesting. I I gotta dig into this one. So it sounds like and I wanna speak super tactically here so that we can get, like, to the to the crux of it. So you have a a GitHub Wiki and we're literally talking about a text file that says, here are the conventions and the technologies that we like to use and then the style and order of them. And as you do more projects, that gets more and more refined because like as you said, it's almost like fine tuning, but it's not because it's more prompting more than anything. Then for the next project, you'll copy that Wiki and basically give that as context to the l m.
00:19:20 Greg: Right?
00:19:21 Nicole: Yeah. Yeah. So it's in all markdown files, essentially, and then we use cloud projects for everything. So, basically, whatever relevant markdown is relevant to the new project, you just pop that in alongside the code, and then it has that context to go off.
00:19:36 Greg: I wanna keep on going down this, but why cloud projects over cursor?
00:19:39 Nicole: Oh, I so I love cloud projects. I love anthropic and cloud in general. Like, I they should pay me, they don't. I think it's, like, the greatest thing in the world. I did a case study with them, and they have a quote from me in the article saying, like, anyone that doesn't use it is dumb or something like that.
00:19:54 Greg: Nice. Nice.
00:19:54 Nicole: Which is, like, so great. I I don't know exactly what Cursor is doing on the back end and stuff. I don't know what their system prompts are. I don't know enough about it to know exactly when they're including the context into the prompts. I know you do the command k from the file. We do use cursor. I have very niche use cases that I use cursor for that, like, my engineers laugh at me. Cloud projects, what I like about it is it's so explicitly contained. So you create a new project.
00:20:23 Greg: Sure.
00:20:24 Nicole: You write the description, you write you put in whatever you want into the content, and so I know for every prompt that's going into the project exactly what context is available. And I think part of all with cursor is, like, I actually don't know depending on where I'm prompting in the tool, what context it's pulling from or if it's the right context. I was talking about this with engineers the other day because people always say, oh, well, like, if it's a really big code base, like, how do you fit it in? And we work in a lot of really big code bases. We work with our enterprise clients, and the thing about it is, like, as an engineer, you're not looking through every file in
00:21:03 Greg: a really big code base when you're going to change code.
00:21:05 Nicole: Like, you couldn't. Like, you never would. And so whatever files that you go look for in the file structure to then figure out what you're gonna code next, whether it's a new feature or you're going to edit something, those same files you include in the cloud project, and then the l l m can do everything.
00:21:22 Greg: So it's Basically, you're doing a lot of copying and pasting too?
00:21:25 Nicole: I do a lot of copying and pasting. Literally, like, my entire workflow is just like copy paste, copy paste, copy paste.
00:21:31 Greg: Yeah. Yeah. Yeah. Yeah. For sure. Okay. So that's on that side. You don't have a sample of those system instructions, do you? Can we check those out?
00:21:40 Nicole: Yeah. So we can. I have a test project pulled up that we can kinda, like, look at how we're doing it, and I'll talk more about a couple other things with Cloud projects. So yeah. So this is our projects at head start. You can see there's your projects and then there's all projects. What I love, love, love about cloud projects is that you can share that amongst your team. So we pay obviously for an enterprise account. You get full enterprise security baked in, so you don't really have to worry about, like, the fact that we do upload code into here. I have this AI interview with Greg Project. So what I've done here is I have pre uploaded an entire code base that is a test code base that we're we're working on on.
00:22:18 Greg: And is that the file tree structure in the name itself?
00:22:21 Nicole: Yes. So this I do so that the AI understands where all the files files are coming from. A friend actually used the AI to write this script for me. It's a bash script that I just run-in the root of my repository that says, like, flatten the repository. I can show you. I think I had just done it in here. Yeah. So it's literally like a flattened repo script, and Uh-huh. Run it, And then you can see, lsdot. That I have this flattened one. And so that just allows us like, if you go to Claude and you actually upload here, you can see how if it's not flattened, you can't, like, actually upload everything.
00:23:04 Greg: Yeah. Yeah. Yeah.
00:23:05 Nicole: You go into here to flatten it. And then the other nice thing is when it's flattened with the file path, if you then ask Claude, like, hey. I need to create a new view. Can you a new view, let's say, for settings for the user, can you tell me where I should put the new files and then give me the content of the files? It should look like a standard settings screen. I also typo, like, nonstop dot ai. It doesn't matter at all. It's, like, my favorite.
00:23:45 Greg: It's so nice. It's so nice that you can
00:23:46 Nicole: You don't have to have anything be perfect. And so this is kind of essentially what we do. So we work with clients that have a ton of designs. We work with clients that have no designs. And so if they have no designs, like, it'll literally just, like, come up with everything itself, and you can see here it gives you the file path. So then what I do is I just go back into the repo.
00:24:08 Greg: You can
00:24:08 Nicole: see in chat. Where did it want it? Chat settings page dot tsx, and so we have ID, and instead, we literally do settings. And maybe, like, if people know how to use Cursor to better create your files, I would love to do that. I don't know how to do that. So I'm always gonna
00:24:32 Greg: Well, you you can do it. It's just kinda still the wild west because it takes so many liberties and assumptions and just blast it all out that you can go deep down the rabbit hole. It's tough to get yourself out of it sometimes.
00:24:41 Nicole: Really? Okay. So, yeah, what I do then is I literally just copy this and I paste it there, and I don't even really read the code all the time. I do when we're reviewing it for clients, but when I'm just kind of, like, writing it like this. I think this Sure. Told me to rename it. Settings, nav, and then you put that in there. And I do like, I'll check to make sure, generally, it looks right, like, no TypeScript error or something like that. Okay. So it says create the files. I already did that. And then it says that in the app sidebar, I need to add this. So we gotta go into the app sidebar and see. So I just, like, do this.
00:25:26 Nicole: And so you don't have to write any of the code, but you do have to figure out let's see. Sure. Bar user nav. This one, this here, and then you just have to make sure it gets import. And then you should be good. And let's see if that even worked.
00:25:47 Greg: Here it is. Settings.
00:25:50 Nicole: And then we just have to figure out what Yeah.
00:25:52 Greg: You gotta go figure out, like, put the proper page in there. Yeah.
00:25:55 Nicole: Oh, yeah. Because it's under it's just the routing that it actually got wrong because it said chat dot settings, and then we didn't actually put it in the chat place correctly. So let's
00:26:14 Greg: see. If we do this, it should work, and then we just have to hook up correctly. Yeah. So this is
00:26:19 Nicole: the settings page it came up with.
00:26:20 Greg: Look at that. It's so wild.
00:26:22 Nicole: Yeah. It's crazy. And then what's great is when you have Figma designs, essentially what I do is you just dump, like, here are the 5 Figma screens that I need. 1 by 1 do the screens, and then I'll get, like, 1st screen over, second screen over, 3rd screen over. And then what I typically do is I'll take all of the code that it gave me. So I'll go through the 5 screens here, and I'll copy paste them back, and I'll be like, here's one screen. I'm just gonna give it these random screens. Here's a second screen. Let's see.
00:26:56 Greg: It's we can be so sloppy with our code in LLMs. It's wild. And by sloppy, I just mean, like, you're not even giving context. You're just giving the code. Yeah. I just give it like this, and
00:27:04 Nicole: I'll be like, between all of these screens, is there anything I should generalize out into components? Like, something like that.
00:27:14 Greg: Uh-huh.
00:27:15 Nicole: And the screens I gave are kind of random, so
00:27:18 Greg: I don't know if you
00:27:18 Nicole: can just say, but, like, if I gave it the right screens. Part of I'm trying to describe this right. Without a good client code example, which we can't share, it's hard to explain this, but, essentially, like, when you're making new screens, you as an engineer know generally how the components should break out of those screens. So, like, let's just say a front end example of some sort of form flow. Like, very standard, you have next buttons, back buttons, maybe you have a little dot navigation thing. You have your various forms. You have headers. You have descriptions. You kinda generally know how those should break out.
00:27:55 Nicole: You have Figma designs that are attached to that. So you put in all the Figma designs, you get all the code, and then you just tell the l m, I need this component, this component, this component, this component, and it just pops them out for you in artifacts, which are so nice because you just get the full file copy paste. Like, part of the reason I don't like cursor is when, like, if I highlight something here and then I'm editing here, it does it in line, but it doesn't always, like, get everything you need, and not everything you need is always, like, in the right place. And then Yeah.
00:28:24 Greg: Yeah. Yeah.
00:28:25 Nicole: The one thing I'll say on cursor too and you can see cursor when I scroll over or no?
00:28:30 Greg: I can see your cursor right now. Yeah.
00:28:31 Nicole: Okay. Cool. The code output I find in cursor, and my engineer finds this too, so it's not just me, but it is just the 2 of us. I've been asking around to see if anyone else knows. The code output is better from Claude. And I don't know if that is due to the system prompt in Claude, which they do publish. So I don't know if you guys you've looked into that at all, but it's fair I
00:28:55 Greg: haven't seen that.
00:28:57 Nicole: They have anthropic system prompt. It's in the API docs. Oh god. No. It thinks I'm a robot. These are really interesting. And so I imagine that cursor is maybe using a different system prompt.
00:29:15 Greg: That's the
00:29:15 Nicole: same thing.
00:29:17 Greg: I'm surprised that cursor wouldn't be able to increase the performance, like, if it's literally just for code.
00:29:22 Nicole: Yeah. And Claude I find that the code back from Claude is much better. I also, like, very rarely use chat GPT decode anymore. Mhmm. But occasionally, if I'm running into something very gnarly, I will go into chatgpt and get it. I find that clogged artifacts, like, these like, just being able to copy paste on the right is much easier. What I do end up doing a ton is and this cursor has this too is, like, if it's a really long file, like, 500 lines or something, it'll cut it down and be, like, you know, same code as before here.
00:29:55 Greg: Sure. And
00:29:55 Nicole: I just have to say, like, hey. Can you give me the entire file the entire file?
00:29:58 Greg: Yeah. Yeah. Yeah. Yeah.
00:30:00 Nicole: What I also find so the long, long files, basically, like, there's certain things with coding that I think are gonna stand the test of time with AI as it gets better, and there's certain things that I think will go away. I think the IDE as a concept, I think I'm, like, questionable on. Like, I personally do not like working out of the ID a IDE in an AI driven world. I go back and forth between that. I'm still in it. I get all the TypeScript stuff and like everything like that, that's what I use cursor for, it like fixes all my type errors because I'm like just a baby about typescript and like don't understand how it works.
00:30:36 Nicole: Little things like that, but the IDE is a pattern I'm not confident on. Small code files and things broken down into like proper component structures and like file structures being correct, I am actually bullish on. I think that will persist. I'm sure you've seen stuff with people where they're like I think you and I had spoken about this once where it's like, does it need to be an English language anymore? Can it just cut down to characters, like, short enough? Some of those things like that. You can imagine different ways of coding. What I think is interesting is I think English language is incredibly important or any language, language, just spoken language, human language, not
00:31:12 Greg: Sure.
00:31:12 Nicole: Computer language. File structure is incredibly like, structure organization is very, very important. Small components are important, like building blocks, puzzle pieces. So it's interesting.
00:31:25 Greg: Very cool. You know, it's funny you say that because I've heard an opposite not an opposite opinion, but a counter opinion, which is that file structures are just human conventions and that machines don't really care about final construct file conventions. So, yeah, I don't I don't know which one it's gonna be.
00:31:40 Nicole: No. I could not agree more that file structures are just human conventions. LLMs are human conventions. They operate on human language and human convention. And so what's fascinating about them is the better you are at human language and not machine language, I think the better you are at using the LM. And that's a very different paradigm than is traditional in engineering. Totally. Which is what I think is very, very interesting. It's really interesting because I do think one of the skill sets that I have, which has made me able to do what I'm able to do is, like, the human kind of communication and writing ability to get what you want out of the LLM.
00:32:22 Nicole: And it's, like, very, very interesting if that can evolve away because it is evolving more towards it for right now, which is fascinating.
00:32:32 Greg: Yeah. That's very cool. And another awesome thing too is you had a case study published by Anthropic on behalf of some of the work you did with one of your clients. What what what what was the story behind that one?
00:32:44 Nicole: Yeah. So that was really cool. We're just top users of cloud and cloud projects. We love it. I think it's the greatest thing that has ever been developed. If you took it away from me,
00:32:54 Greg: I would really struggle to run
00:32:55 Nicole: my business, which is probably like the highest NPS you can get for a product. And so they wanted to hear about how we were using it and potentially share that with more people. And so it was really cool because I think we're at least using cloud projects in a pretty different way than a lot of their other case studies. And so that was kind of cool to get, written up for that usage.
00:33:17 Greg: Yeah. They must have a data analyst looking at clog project usage and being like, dang, Nicole at head start. She's freaking crushing at least her team is.
00:33:25 Nicole: Yeah. No. It's it was a really exciting feature for us for sure.
00:33:29 Greg: Yeah. That's very cool. The question I wanted to ask, what other pro moves are you doing besides the random of course, we're using cloud projects, but, like, what other pro moves with AI encoding are you doing?
00:33:40 Nicole: I've been trying to explain it to other engineers and figure out what is different about how I'm using it. One engineer told me, he's like, you trust it much much more than I would ever trust it. And so we're I'm taking, like, swaths of code from cloud projects when I'm working on client projects and I'm just like copy and pasting over and everything's broken down in your typical engineer way, like we still have PRs, we still review the code, it still goes in in chunks, It's just happening that much quicker because we're really relying on a lot of the generation to write it. So I do think the trust level is there when we we use it for everything.
00:34:22 Nicole: So, like, I'm sure, like, when we write prompts for the AI implementations for our clients and then we're evaluating those prompts, we have our own evals product that we built, we basically, like, take the prompt, take the evals result. So, basically, if it's not passing a certain test case, we take all the reasoning behind that, and we feed it back into Claude to regenerate the prompt. So we have, like, a prompt loop in terms of generating all of that.
00:34:48 Greg: So you have not only your own eval tool, which you built yourself, but you also have your own, like, Claude, like, prompt optimizer.
00:34:56 Nicole: Yeah. We built that. We do have that. And then we also have, like, an AI computer agent that we build via cloud computer use. So, you know, the new functionality.
00:35:07 Greg: Oh, yeah. Yeah. Yeah. That's cool. You already built something on it.
00:35:09 Nicole: Yeah. Yeah. Our engineer, Tiff, she built it in, like, 2 days, and so we've been using that. It basically it'll create PRs for you for changes to the code. It's really good. It's kinda like a junior engineer, and we've been running that on all our projects, which is really, really cool. So it's been we've been able to build products in parallel with running the services and then leverage those products to be able to then, you know Yeah.
00:35:32 Greg: Work in the services. The computer use, are you doing that on a VM, or is that literally on her, like, laptop that she's
00:35:37 Nicole: We run it on our own laptop, so it's a script that runs right now. We're working on Dockerizing it and put it on the server, but there's actually a couple of nice things about running it on our laptops. It, like, uses the GitHub CLI, so it'll actually create a branch and do it from your own terminal, which then what I end up doing with it right now is I tell it the change. It creates the change. It creates the PR, and then you're already on the branch in your project, like in cursor. And so from Cursor, then I test it myself, like, locally. And so I can still do the testing. We don't have the AI agent doing the testing yet because that's obviously, like, a little bit more complex.
00:36:13 Greg: Yeah. Yeah.
00:36:14 Nicole: And then I can make the change either personally or I'll have the AI agent do it. And so we have that kind of tight feedback loop within it. And what's so cool is, like, we're dogfighting our own product. Yeah. So we can immediately, like, just you know, this thing didn't work. Like, it created an empty file and it got caught in a loop or whatever else. But computer use is, like, very powerful.
00:36:39 Greg: Yeah. That's wild. I wanna ask about tool stack. It's always cool to hear which tools people have in case there's one that I'm, like, isn't part of my routine yet. So what's in your AI development stack, and what are you using besides Cloud Projects and besides Cursor?
00:36:50 Nicole: Cloud Projects, Cursor, Chat GPT, GitHub, obviously, huge fans of GitHub. We use we build a lot of, like, different products for people. So we use React Native for mobile apps. Gotcha. We use any kind of full stack application. We use Vercel a lot for new projects. I like you using Vercel to spin things up. A lot of Node on the back end or Python on the back end is typically what we use. We work within any client's existing code base, so we'll use whatever they're using across the board. But when we're doing net new, typically, it's like React or React Native front ends, TypeScript, Node. Js, Python back ends, depending on what we're building microservices.
00:37:34 Nicole: We're typically using Python for different databases, stuff like that. Right. Yeah. It's kind of we we can like, we're just paying for Claude, Cursor, OpenAI, and then, like, the usual business stuff, Slack Yeah. Etcetera.
00:37:50 Greg: What that doesn't sound like a lot. What are your what are your margins for your business?
00:37:55 Nicole: Oh, we just I mean, we pay engineer salaries plus, like, $60 a month per employee for, like, all the AI. Like, nothing, basically. All our clients pay for the for their AI costs and everything. So
00:38:09 Greg: Yeah.
00:38:10 Nicole: And we have a little office in New York. So those are basically the costs for the business. I still think it's insane that through OpenAI and Anthropic, you can get this tech for $20 a month or $30 a month, whatever it is.
00:38:22 Greg: Absolutely wild. And that yeah. That's crazy because, like, you're not even paying the API costs for your cloud projects. Right? You're just paying the business enterprise. What? Sorry.
00:38:29 Nicole: We pay the API cost for our own test projects.
00:38:31 Greg: So that's
00:38:32 Nicole: the but, honestly, it's pretty much driven to 0. Like, we do a lot of quoting for our clients on how much the AI will cost them, and it's so like, it's, like, 0.00002 then multiply it out. Like, it's hard to get numbers up there.
00:38:49 Greg: Yeah. Yeah. Yeah. For sure. For sure. So one thing that's come across in this interview is you have a lot of technical confidence, which is very cool. It's like any project, just bring it on and we'll go do it. What types of technical projects were you steer away from? Like, which one are you saying no to that clients bring to you?
00:39:06 Nicole: Like, below the minimum fee, basically. We're we're not afraid of anything. We try to take on the hardest projects that we can. We believe and it's written down on, like, our strategy board that access to hard problems is like a proprietary business value for us. So we think that having that access is incredibly powerful, and we definitely don't stray away from it. Sometimes it is scary because we
00:39:30 Greg: have to do really hard projects that we don't really know
00:39:33 Nicole: how to do on the onset, but we are able to figure it out and have that confidence. But, yeah, typically, like, we will do anything that people think is impossible, and we like to do that. One, because we can charge for it, which, you know, as we should if people think it's impossible, but also because we learn from it, and we think it makes our business more powerful.
00:39:54 Greg: That's wild.
00:39:55 Nicole: And the clients are happy because we're doing essentially the impossible for them.
00:39:58 Greg: Yeah. What else is on that whiteboard of values that your company has?
00:40:03 Nicole: Oh, yeah. So we're very values driven. Company values are simplicity, patience, and compassion. I pulled up from the Dao De Jing, the Steven Mitchell translation. It's one of my favorite books, and so that's kinda like the core values. But in terms of, like, access to, like, the hard problems being proprietary, like, that's where we think about, like, our modes. Like, what is more of, like, the power for the business, and it's not how I use Claude even though how I use Claude is, like, driving the business forward. It's very much kind of, like, you know, doing good work for our clients is number 1.
00:40:35 Nicole: Client NPS is the thing that we care about the most. Our clients understanding the type of work that we can do, and the quality of work we can deliver for them is number 1, most importantly. Efficiency of the business, so we track revenue per per employee. That's really important as we grow the business.
00:40:51 Greg: Which it sounds like that is just insane right now.
00:40:54 Nicole: It's high right now. Yeah. And we have to figure out how to continue to scale it. Basically, because we're a services business and we're also investing in the product, that it doesn't scale like every other business. It's more jumpy because as we take time to build product, we'd have less time for services. So we're figuring out how to, like, even that out a little bit as we grow. But, yeah, we think communication client NPS, access to hard problems, being able to solve hard problems, being able to do things in a repeatable way, creating network effects within the business, and, like, good data, even, like, good data in terms of, like, how we use the LLMs.
00:41:31 Nicole: Sure. Good code is kind of proprietary. And so, yeah, we think about it a little bit differently than, like, how we use LLMs as prior proprietary.
00:41:41 Greg: Sure. What about businesses that are 2 to 3 years ahead of where you are right now or that you wanna be or 2 to 3 steps ahead? What what do those businesses look like?
00:41:51 Nicole: I mean, I think this is just gonna make me sound like a crazy fangirl, but I think Infronix is the greatest business out there right now. I think what they're doing is incredible, and, like, the product itself that Claude is is really, really powerful. Sure.
00:42:03 Greg: No. I mean, like, where do where do you wanna go, though? Like, where do you wanna take your business, like, in 2 to 3 years?
00:42:08 Nicole: Oh, that's a great question. We're trying to grow in an AI native way. So we want to continue to hire, but not hire to the scale of the client work. We wanna hire and then train up a team that can then do that work exponentially. We think that the progress of AI is inevitable, and we're building into that inevitability as a company. And so whether or not the AI will be able to do this work completely next week or whether that will happen in 2 years. To me, it's inevitable even if it's 10 years. And so whatever that timeline is, we're kind of just making sure that we are the best at using the AI, and we're best at implementing the AI in the business.
00:42:50 Nicole: And if we do those things, I think the business will continue to grow.
00:42:54 Greg: Yeah. Well, what about products or SaaS? Is that in your future?
00:42:58 Nicole: So the way we're thinking about products right now is we have them, internal products that we've built. We are gonna continue to build those, and we want to be the users of our products first and foremost. We're not trying to build products for other people right now. We're trying to build products that make ours us more efficient across the board. If we can do that and we've productized that in a really powerful way and we feel like maybe selling it, it's an option, but it's not the priority.
00:43:26 Greg: Cool. Beautiful. Nicole, that was fabulous. Thank you very much for joining us today.
00:43:31 Nicole: Thank you, Carrie. It's always great chatting with you.
# File: /content/learn/dev/ai_show_tell/sully.mdx
---
title: Sully, CEO Of Otto AI
description: Hacking together GPT-4 and Claude, using AI to write tests and why talking to your computer like a human gets better results
date: 2024-11-07
---
Sully Omar ([/in/](https://www.linkedin.com/in/sulaimanmarouf/), [X](https://x.com/SullyOmarr)) is the CEO of [Cognosys](https://www.cognosys.ai/), the company behind [Otto](https://ottogrid.ai/). He's one of the best LLM practitioners I've met, and you can tell he has a really deep feeling for how these models are actually working. He speaks from experience. In this interview, we go through his 3 tier system of actually ranking language models. He shows us how he uses meta prompts to develop his real prompts that he uses in production. He also shows us his cursor development flow where he actually has the language model write the test first and then write the actual code. And finally, he walks us through distilling performance from large language models to small language models without losing performance.
## Insights
* **Treat LLMs like humans**: Speaking to the model naturally, as if it were a human, improves performance. **Voice input** facilitates this natural conversation style.
* **Embrace the "vibe"**: Developing an intuitive understanding of each model's **"personality"** and **nuances** comes from consistent use and experimentation. This **"vibe"** helps predict how a model might respond to different prompts and tasks.
* **Min-maxing is key**: Constantly seek ways to **optimize workflows** by combining different models' **strengths**. Don't be afraid to **experiment** with unusual combinations and orchestration strategies.
* **The last 5-10% is hard**: Getting an AI product from **90-95% accuracy to near-perfect** is exceptionally challenging, even with robust evaluations. Be prepared for this final hurdle.
* **Model routing is coming**: The future likely involves **automated model selection (routing)** based on task characteristics, but current limitations make **manual routing** more effective.
* **Distillation requires vigilance**: While **distilling knowledge from larger to smaller models** offers efficiency, it requires **meticulous data pipelines** and **evaluations** to avoid performance regression.
* **Context is king (especially for Tier 1)**: **Thinking models** excel when given ample **context**. Use Tier 2 models to **gather and structure information** before feeding it to Tier 1 for deeper analysis.
* **Structured output struggles**: Some models (like Claude) struggle with complex **structured outputs**. Consider alternative models or workarounds (like using a different model for JSON generation) in these cases.
* **Needle in a haystack vs. Reasoning**: Some models are better at **finding specific information within large datasets** (Gemini), while others excel at **reasoning over that data** (GPT-4-0-mini). Choose the right tool for the job.
* **Don't be afraid to "hack"**: Early-stage LLMs require **creative workarounds ("hacks")** to achieve desired results. These hacks evolve as the technology matures, but they will likely remain a part of the development process.
## Model Tier System & Usage
* **Tier 3 (Workhorses):** GPT-4-0-mini and Gemini Flash. Cheap and fast, used for high-volume tasks like document processing and podcast analysis.
* **Tier 2 (Balanced):** GPT-4, Claude 3.5, Gemini Pro. Good balance of price and performance, ideal for common tasks like coding, writing, and email editing.
* **Tier 1 (Thinking Models):** Google's Gemini Ultra, other "thinking" models. Used for complex reasoning and deep dives. Sully's workflow involves building context in Tier 2, then feeding that context to Tier 1 for improved results. Deduplication is another strong use case for Tier 1.
* **Multi-Model Approach:** Sully leverages different providers because each model has unique strengths and weaknesses. He even orchestrates models to work together, like Claude managing GPT-4-0-mini for structured output.
## Prompt Engineering
* **Meta-Prompting:** Begin with a general problem statement and ask an LLM (like Claude or GPT-4) to generate a prompt. Refine this prompt further with a thinking model (like Gemini Ultra) for optimization.
* **Voice Input:** Sully uses voice input for faster, more natural prompting and context building.
* **Prompt Management:** Uses LangSmith for prompt evaluation and dataset management. Prompts themselves are version-controlled in GitHub alongside code.
## Development Workflow
* **LLM-Driven Test-Driven Development (TDD):** Have the LLM generate tests before writing the code. This improves code quality and allows the LLM to self-correct by analyzing test failures. This approach is particularly helpful for complex, multi-file projects.
## What're the smartest people in AI talking about?
* **Test-time compute:** Adding more compute at test time to increase performance.
* **Thinking models (like o1):** Using step by step planning at inference time to increase performance.
* **Agentic tasks:** Increasing more autonomy in the AI to get things done.
* **Built-in tool usage:** Like Anthropic's Constitutional AI.
* **Model distillation:** Training a smaller model based off the output of a bigger model.
* **Rigorous evaluation:** Pressure testing with evals.
* **Performance plateau:** Wondering if AI has hit a peak.
## What's in Sully's Toolkit?
* **LLM Platforms:** Gemini Studio, ChatGPT, Claude, OpenAI Playground, Anthropic Workbench.
* **Coding Tools:** Cursor, Replit, VS Code.
* **Other:** Excalidraw, Whisperflow (transcription), LangSmith (evals), v0.
## Transcript
00:00:00 Sully: It lets you use AI in basically every nook and cranny of your day to day. When that model came out, it actually opened up a lot of things that you could do. We use a lot of different providers, and that's because what we've seen with our internal evals is that they're all so nuanced and different in like a variety of different ways. But you also start to see where they lack. You'll get to an AI product, you'll get into 90%, even 95%. But that last 5, 10% is nearly impossible.
00:00:27 Greg: How do you think about model distillation?
00:00:29 Sully: It's very powerful, but you have to be very careful.
00:00:43 Greg: I just had an amazing conversation with Sully Omar, the CEO of Cognizys, the company behind auto dotai. Not only is he one of the best LLM practitioners that I've met, but you can tell he has a really deep feeling for how these models are actually working. He speaks from experience. In this interview, we go through his 3 tier system of actually ranking language models. He shows us how he uses meta prompts to develop his real prompts that he uses in production. He also shows us his cursor development flow where he actually has the language model write the test first and then write the actual code. And finally, he walks us through distilling performance from large language models to small language models without losing performance.
00:01:23 Greg: Let's jump into it and let's see what wisdom our friend Sully has to share. The reason why we're doing this interview here is because I see all the cool stuff you're sharing on Twitter. And I'm like, this guy clearly has not only, like, a checklist learned, ability to manipulate these models, but I could tell you feel them. Like, you really feel how these things are actually going in the personalities and the nuances. And so I wanna dig in dig into that today.
00:01:48 Sully: Yeah. Well, thank you. And and I think it just comes from playing with these things every day, day in, day out, and using them and pushing them to their limit. And, like, as cliche as it is, it's just, like, sometimes you just gotta use them to vibe with them. You know? Like like right? So
00:02:04 Greg: Yeah. Yeah. Yeah. It's so true. Well, I tell you what. I wanna start off with one framework that I saw you document recently, which was your 3 tier model of language models. So tier 1 through tier 3. So could you tell me, like, starting at tier 3, what are those and how do you work your way up?
00:02:20 Sully: Yeah. So that's a that's a framework that I I mean, I don't even know if you wanna call it a framework, but it's I like to categorize it and it's, like, based on intelligence and price, which is correlated. Right? Like, the less intelligent models are gonna be your tier 3 models, and then your more expensive, slower are gonna be your more intelligent model. So the reason I I thought of it in 3 tiers was because of the application purposes. So the way that you use something like, let's say, o one, so that would be like a tier 1, is different than the way they would use something like Gemini Flash, which is tier 3.
00:02:53 Sully: And that's because they all provide different purposes. One is super cheap, super fast. The other one's, like, really smart and really slow. So I I broke it down to those 3 tiers. And the 3rd tier is basically what I like to call just like the, you know, the workhorse, the the ones that you're just constantly using 247. And within that category, I think there is 3 main models, but it's kinda come down to 2 for me personally. So the first one is the one that I think people are probably more familiar with, which is GPT 4 o Mini. Now that model and is actually like, I I really, really like it because it lets you use AI in a way that previously you couldn't.
00:03:35 Sully: Like, if you were to go back, let's say, 6 months ago when we had no cheap models, you had, let's say, GPT 4 and maybe even Claude 3.5. There was a lot of scenarios where you couldn't just be, like, throwing that at, like, random problems. Like, you couldn't just be like, hey. I have this, you know, 20 page document. I want you to go paragraph by paragraph and, like, extract the details because, realistically, like, you know, you're gonna be paying a lot of money. So when that model came out, it actually opened up a lot of, like, things that you could do. So that was the the the the first one was with GPT 4 Mini.
00:04:08 Sully: And then the other one that I'm starting to really like is Flash. So Gemini Flash is actually half the price of GPT 4 0 Mini. And those are the tier 3 because, like I said, they they give you a lot of optionality in the different things that you could do that you couldn't do before. It lets you use AI in basically every nook and cranny of your day to day. Right? If whether it's your coding and you wanted to look at, like, you know, 50 different files to summarize to help another model. For example, if you wanted to take a podcast and, you know, look at, you know, when did someone say a specific word in that podcast.
00:04:43 Sully: Right? You're not gonna go to a bigger model. So that was that's what I call the tier 3. And then the second tier that I have is sort of like the the middle, obviously, is the middle tier. And this is where I like to slot in the actual GPT 4, cloud 3.5, Gemini Pro. This is where I think the majority of people use these models and and kinda get the maximum usage out of them. And then the last year is obviously like the o one o one preview, and then what I like to classify as thinking models.
00:05:09 Greg: Yeah. That's so cool. So I wanna dig in more into the use case side. So which use case tasks are you doing tier 2 with? And then I know that o o one and tier 1 is gonna be it's not just, oh, I need it smarter. It's almost like a different type of task you're gonna ask it to do. So how do you differentiate between those 2?
00:05:25 Sully: Right. So the way that I like to differentiate is I like like I pair them. So I will use o one, and I do use this in my day to day. It's like, I'll go to chat gpt. And if you just go and say, hey. Like, 2 0 1, can you do this task for me? 1, it's gonna take a little bit of time. You're probably gonna hit some rate limits because it's highly limited. And, realistically, you're not going to use the model the way that I think it was intended somewhat to be used. So if you say like, hey. How's it going? Like, okay. Sure. You could use it like that. But, realistically, you're better off using, you know, the the tier 2. So how I use the tier 2 is actually the most I use it the most.
00:06:01 Sully: Obviously, everyone uses it for coding, whether it's cloud 3.5, GPT 4. Using it for, like, function calling or tune call tool calling. Like, it it is obviously, like, a good balance between intelligence and price. Mhmm. And and that's kinda, like, what I use it the most, whether I'm writing, whether I'm asking it to, like, hey. Help me edit an email or things like that. I'm using those, like, middle tier ones. Now how I actually use that in tandem with o one is I'll sort of one of the use cases I have is I'll come to Chad GPT or or Claude, and I'll sit there and I'll just create a giant conversation about a specific topic.
00:06:37 Sully: So let's say, for example, you know, I'm deep diving into a research topic, and I wanna learn more about. Now I'm not gonna actually go straight into o one because I feel like, one, it's a bit slow. What I'll what I'll do is I'll start the topic with GPT 4 or Claude, and I'll, like, add files because, obviously, I think right now, o one doesn't support, like, files and web search. So there's a lot of capabilities that o one doesn't support. And what I like to call is the context building. So I will just go and build as much context in this chat as I possibly can or or it could be, you know, in any platform. And and I'll sit there and iterate.
00:07:10 Sully: I'll actually use voice mode as well to sort of give context just a lot quicker, and and that's another workflow. And as soon as I have, like, you know, let's say, like, 2 to 3 pages worth of documents, I'll actually take that and paste it into a chat with o one or o one preview. And I'll say, hey. You know, do this gigantic task for me. So for example, I'll I'll give you one thing Uh-huh. To use it for is, like, I was using it to generate use cases for my product. And I was like, okay. I want to generate use cases, and I want to understand, you know, what are some potential customer segments and ICPs. It's just like a pretty technical question.
00:07:46 Sully: And if I were to just go to o one and ask it that, it would have no context. It doesn't know what my product is. It has no clue what my product does, who my customers are. And if I were to sit there and chat with it, well, I'm gonna hit that limit. But if I go to Claude or Chad GPT, I can upload documents. I can create this basically a PDF and copy paste it into o one. And then I can say, generate me, you know, personas, ICPs. It does a lot better. So that's sort of the the workflow and use case that I have currently running with, like, the the tier 2 and the tier 1 models.
00:08:14 Greg: Yeah. Yeah. Yeah. One of the ways that I found o one works for me really well is around actually deduplication. So if I have a long list of items, that say I've processed 5 different chunks with the same type of workflow for each chunk, well, I'm gonna have a list of duplicated items. I give that whole thing to o one. It's actually really good at deduplicating, and then I'll use one of the tier 2 models to do the structured output after that since o one doesn't yet support structured output and go from there.
00:08:37 Sully: Yeah. That actually, that's a good one. That's another thing I do as well is I'll take o one and give me, like, a long verbose output and then take that and turn it into structured datasets with the the tier 2. And even sometimes, you could even get away with using it with a tier 3 because it's you don't even need to worry about the output. You're just like, hey. I want this nicely formatted in in whatever shape.
00:08:57 Greg: Yeah. Yeah. Yeah. For sure. So it sounds like you're using different models across different providers too for different use cases, or do you stick with 1 all the time?
00:09:06 Sully: Yes. So we use a lot of different providers, and that's because what we've seen with our internal evals is that they're all so nuanced and different in, like, a variety of different ways. So, obviously, the big one, Gemini, multimodal right off the bat. Like, anything to do with videos or audios, I'll go, you know, dive in straight into that and and kind of use Gemini. But you also start to see where they lack. So for example, a really interesting one is Gemini models are really good at needle in the haystack. And so if you say, hey. I want you to find 1 or 2 pieces of information in this, you know, giant long piece of text or video, it's actually really good.
00:09:51 Sully: But then I started to notice that something like GPT 4 0 Mini is a little bit of a a little bit better reasoning over that. So if I give it a long piece of context and I say, hey. I want you to sort of understand the context of it. I saw I found that GPT 4 o Mini is a little bit better. So you start to see where one model does better than the other model in specific area. So, like, another example is claud 3.5 and GPT 4 o. Now clog is obviously everyone loves the model. It's a really good model, but one thing it's absolutely horrible at is tool use with structured outputs. And you'll start to see this if you the very complex tool, like, I want you to create the very deep like, a a nested JSON, a a very, you know, long structured output.
00:10:35 Sully: Like, a very large amount of the time, it fails, and it gives you XML, and it just breaks all your parsers. Whereas GPT 4 0 Mini does a lot better job. But then the caveat is that g p t 4 o Mini is not as good at actually, like, thinking through the problem and acting as an assistant. So there's always these, like, tiny trade offs that you don't really, like, notice. One of the that we did was we set up a like, one of the use cases was to get around that was we set up Claude and GPT 4 0 Mini to work together where the tool use for Claude would be to call GPT 4 0 Mini. And we basically system where Claude could orchestrate GPT 4 0 Mini to create the structured output.
00:11:17 Sully: So it would say, please do this. So the user would say, I want this task. All g p all Claude would do was relay that information to GPT 4 0 Mini. 4 0 Mini creates a structured output, and then that gets returned. So that was like another use case of, like, how we mix and match so many models across different use cases.
00:11:34 Greg: Yeah. Isn't it wild how all these little mini Vibe tricks we have to kind of like hack together in the early days of elements here? And then I think back to how far we've already come. Like, because even like, you know, like January 23, we're dealing with, like, 4,000 token context limits and GPT 3.5. And all the hacks that we had then, we've upgraded from them now, but we still have a bunch of hacks like the ones you're talking about. And so it just makes me think we're never gonna get rid of the hacks, and they're always gonna be there for for a long time.
00:12:01 Sully: I would say so too because, yeah, like, you're right. It's funny looking back at it. The hacks that you used in 2023 were so different. You were hacking around context window, and now you're hacking around, well, tool use, which didn't even exist a year ago, right, or, like, you know, a year and a half ago. So I I agree with you that we're always going to be min maxing. As a user of multiple models, you're gonna be min maxing, trying to figure out for your use case, for your product, for your company, where can I, you know, match these together so that I get the best possible outcome for my users? And I I know a lot of people have, and I'm curious what you think.
00:12:35 Sully: A lot of people have spoken about, like, model routers and how Mhmm. You know, at the end of the day, like, a model is just gonna pick it. But my my personal opinion is I I think that it's gonna cause a lot of unintended side like, you know, side effects. But I'm curious what you think on, like, this whole idea of, like, model routing because, you know, we're talking what we're basically doing we're internally with code model routing, but I'm I'm curious what you think.
00:12:56 Greg: So whenever I get asked a question like this, I think, is there any behavior in practice that tells me what the prediction should be? And you just described, basically, you're doing model routing on your own, like, in and and in and of itself. So that tells me, yes, model routing will be a thing. And I do still think that fine tuning models and having bespoke small models is still too much overhead. Like, it's really hard to do that and manage them and do them all right now. All that is gonna get so much easier. So I would imagine that not only will we have model routing for task specific things against like some of the big ones where you have Vibe based fields with regards to structured output or tool use or whatever it may be.
00:13:32 Greg: But then also for task specific things, I will absolutely do model routing. So I'm a fan. I think it's hard. I think it will be the future. We're not quite there yet, though. That's for sure.
00:13:43 Sully: Got you. Yeah. Like, my my my sentiment there was that there and I it could be just because the model's just where we're at right now. What I've noticed is and I'm sure you've seen the same is where you'll get to an AI product. You'll get it to 90%, even 95%. But that last 5% is last 10 5, 10% is nearly impossible, I find. Like, it even you can run all the evals you want. You can run all the benchmarks. Getting that last 10%, and I my thought process there is that if you have the model sort of choosing other models, that adds to the variance so it causes a lot more potential. Like, you know, that that's kinda where my thinking is.
00:14:25 Sully: And that could just be because, like, we're early. Like, realistically, we're so early. Models have, you know, multiple generations to get better. So that was my thought was that maybe in the future, but right now, probably not because it's it's so hard to get a product in specifically, like, LLMs into production where you're handling every potential edge case in a manner that gives you as high of an accuracy as you can and adding models that you might not have an eval for could give you an output that you didn't expect.
00:14:57 Greg: Yeah. Yeah. Totally. Well, I tell you what. One of the other interesting things that came up during research was your opinion on what is kind of becoming known as model distillation. So you have a really, really good model. You perfect the output from there, but then you realize, wow. I can actually come up with a little bit of a better prompt here and give it to a smaller model so that you have it's faster and it's cheaper. So can you talk me or walk me through how do you think about model distillation
00:15:22 Sully: in your own workflow? Yeah. So that's something I think about a lot, and it's one of those things where you need to be very careful because it's very it's very powerful, but you have to be very careful because it requires a lot of work. And the reason it needs a lot of work is because you need to have a a good data pipeline and understand what you're distilling. So one of the things and mistakes I made previously with the product was that we went we had GPT 40. And this was actually before GPT 40. It was GPT 4 turbo. And we used it, and it was slow. And we're like, hey. Let's distill that to 3.5. OpenAI has a has a really nice way to do it, so we did that.
00:16:01 Sully: And then the problem was that we didn't have good enough evals. We didn't have a good enough dataset. So as the potential you know, the various areas grew that people could use the product, we would notice, okay. We have to revert back to g fifty four because 3.5 was, at that time, not good enough. Now where I do see distillation in our workflow is when you have a defined eval set, you have, like, all your benchmarks, and you have a very good data pipeline where you can say, okay. In this 500 example set, I'm using cloud 3.5 SONET or your, you know, o one, for example. I have my dataset, and you can use a bunch of different there's a lot of different companies that provide you with, like, ways to manage your and prompts and evals, whether it's BrainTrust or Lanxnite.
00:16:46 Sully: And then you can very accurately detect and determine the accuracy of the distilled model, then 10 out of 10 times I would use it. Yes. And the ease and it's it's actually really easy. Like, to actually distill the model down, it's like it's like an it's a single API call. The challenging part is making sure that you don't regress your product when you do the distillation. But I I think it's one of those things that it's gonna become more and more apparent as the tooling around distillation becomes, like, better. I know there's a couple companies working on it. Like, OpenPipe is one of them. Mhmm. And I know OpenAI straight up offers you that.
00:17:24 Sully: So I think as the tooling gets better, you're gonna see this pattern in production of companies launching with the biggest, best model. They collect a bunch of data. They have a good email set and engineering team to support that. Then they go and they distill it to whether open, you know, G50 4 o Mini or an open source model.
00:17:41 Greg: Yeah. That's beautiful. My favorite line with that is the whole make it work, make it right, make it fast. And so it's like, look, you're gonna use the biggest one to start us off, but then you're gonna make it fast eventually and go from there. This is awesome. I tell you what, though. So I know you're a practical person. I would love to jump into, like, you actually showing us some of the ways that you use these tools. And I think a really cool starting off point would be I know that you're a fan of prompt optimizers or, like, meta prompt writing. And so Yes. Because you had a you had a tweet and literally said, pretty good chance you won't be prompting from scratch in 2 to 3 months.
00:18:16 Greg: So I would love to see the way you kind of prompt engineer your way from, like, an idea to, like, I'm gonna go use this thing.
00:18:24 Sully: Okay. Yeah. Hopefully, my prediction ages well because I feel like it's been a month since I said that, and I don't know if we're 2 to 3 months away from it. But Yeah. Yeah. Yeah. Okay. Let me yeah. I so just to add some context, I do a lot of this sort of meta prompting where I'll come in with a problem.
00:18:39 Greg: What is what is meta prompting? Let's start there.
00:18:42 Sully: You come in with a general idea of what you're trying to do. You have a problem that you're trying to solve. Like, realistically, if you're coming in, you don't know what problem you have that you're trying to solve with AI. It's it's sort of useless. So an example would be the other day, I was trying to get one of the models to write like me, which to to this day, I I cannot for whatever reason. Yeah. Yeah. Yeah. I was like I came into it and I came into chat GPT and I had all my examples. And I was like, okay. What do I write? And I normally, I would write something like you know, you you write like a basic prompt structure, and the reality is that prompt's probably not that good.
00:19:17 Sully: So what meta prompting or what I like to think about this work this idea is that you come in with an idea. Hey. I want to have an AI right like me. I have examples. And then I just give that to 0 1 or Claude, and I say, please create the prompt for me. And that's sort of what I like to think of like this. I come in with a a rough idea of what I'm trying to do. I don't really know specifically how to optimize it. I'll go to these models and say, hey. Like, actually, give me this prompt structure, and it does a pretty good job. So that's kind of the the rough idea of how it works. But let's That's Let me should we just hop into, like
00:19:48 Greg: Yeah. I would love to jump into it if you could share your screen. And then are you using just a regular chat interface, or are you going to Anthropic's Workbench and doing their prompt dot optimizer?
00:19:57 Sully: I I just use the chat interface because Cool. I feel like the prompt I mean, people some people do use it. I and I think you can start with it. But I just find it easier because I can iterate a lot better. I can say, hey. Start like this and and do that. So let's actually do it. But I I wanna start and say, do you have some sort of task that, like we should we start we should start with, like, a rough idea. Because I like, do you have any like, what what's the task we could demo?
00:20:22 Greg: Let's do a straightforward one. Let's do I I guess I'll give you a few options. You tell me what you think is best. We could do the classification 1, which is very standard. Hey. I have some data sources, or can you please label them for me? We could do either, like, unstructured to structured extraction, so, like, extracting insights from a piece of text, or we could do idea generation. That's always a fun one too.
00:20:45 Sully: Okay. Let's do the let's do the extracting text 1, and I think that's a good one. So let's say we I like to always preface it with, like, the problem or what we're trying to do. So, again, what I like to come into is, like, alright. I have a problem. I'm trying to do a specific task. And usually, this is like my blank state slate starting point. So let's say the task that I'm trying to do is I have a large piece of text, and I want to, you know, turn that piece of text into something else, some sort of structured output. And it's it's funny because a lot of people say, like, oh, is it complicated? It's really like I just come to chat GPT and I or or Claude, and I basically say that.
00:21:21 Sully: So the way that I go is I'll say, you know, you could use Claude or the or chat GPT. I haven't found which one is really better. Again, I'll and then this is kind of going back to my original workflow Sure. Is what I'll do is I'll actually start with GPT 4 or Claude, and I'll get, like, a rough idea for a prompt. And I'll copy that, and I'll give it to o one. And then I'll start to compare across all three to see which one, like, makes the most sense. So let's say, for example, in this one, I am grabbing transcripts from podcasts, and I want to know, like you know, I I want a nice, like, structured output for Mhmm. All of the key exciting moments.
00:21:57 Sully: Let's say that that's, like, the problem space. So now you could come in and you could create a prompt and says, okay. Given this video, I want you to do this. Or I come to Cloud and say, look. Like and actually, the other workflow that I I wish I could demo is I use voice a lot. So I don't know if if you use voice a lot, but I've noticed that with voice here
00:22:17 Greg: I don't use it a ton. Yeah. It hasn't entered my workflow yet, but I'm I'm voice curious. So I I wanna try it now.
00:22:22 Sully: See this. Let's see this. Okay. So I have I have something here. I wanna show you the whole workflow that I use so that I so Nice. Let's see here.
00:22:31 Greg: And let me know if you need a transcript. I have one handy for us.
00:22:34 Sully: Actually, yeah. Could you could you toss me it there? No. And then I will use it. I will copy paste it. So okay. Let me know when you have that transcript, and then let me see if it's okay.
00:22:43 Greg: I'll plug this is MFM Vault, a website I put together that does insight extraction from my first mailing. There we go.
00:22:49 Sully: Okay. Cool. So let's say our goal is to extract insights. Now my workflow is I have a tool that transcribes this, so I think it works. So let's say I'll just exactly show you how to do it. Okay. Hey. I need a bit of help creating a prompt for a use case. So what we're doing right now is taking podcast transcripts and trying to extract all of the key moments slash key insights. So I need you to create a a nice prompt that will, you know, help us do that, and I'll I'll give I'm gonna put in the prompt as well later on the actual transcript, but I need you to create the prompt slash system prompt. So boom. So that's that's actually sort of how I do it.
00:23:27 Sully: I it's Nice. There's no real science to it. And I and I'll sit there, and I kinda like here, and I'll copy this. And I'll actually do this. So I'll go into chat GPT. I'll paste it. And I'll actually also place it into Claude. And it's gonna go, and it's gonna give me, like, a starting point. And so right off the bat, like, if you're maybe not as good at prompting or you're new to prompting, like, you can read this. Like, obviously, if you're more experienced and you kinda know, like, what you're doing, these kind of prompts are, like, pretty obvious. But for a lot of people, they'll come in and and be like, okay.
00:23:59 Sully: Cool. I have a a good starting point. So then I'll look at it and say, k. The following is a pocket transcript. Identify some so and I'll compare it to here. So right off the bat, I don't know if you which one you think is better, but I'm looking at this, and I like the Claude output better. Beautiful. A little bit more what's it called? Clear direction. So I'll actually copy this, and I'll be like, okay. We have a rough outline. I liked the first pass. I liked the one from cloth. I'll take that, and I'll go back to chat gpt, and I'll open up a new tab. And then I'll say, let's go to o one preview. So then I'll actually do the same thing.
00:24:39 Sully: I'll say and I'll actually give it more context. So I'll say something along the lines of and, again, I'll I'll go back to the voice mode here. I'll say, hey. You're gonna help me optimize a prompt. So I already got another AI model to give me a rough idea for this prompt. I want you to look at it and tell me if there's any areas in the prompt that we could improve. So I'll give you the prompt, and I'll actually give you the prompt that I gave to the I AI that generated this prompt. So it's gonna go, and then I'm gonna go like this. So this is sort of here, you know, original prompt to AI. Mhmm. Paste that in a sec.
00:25:15 Greg: It's amazing just how you speak to it just like a human. Like, it's not complicated. It's literally just being clear in your directions.
00:25:23 Sully: It's something that I recently started to do, and I think it's a very a lot of people talk to the AI as if it's not a human, but they perform the best when you just speak to it naturally. And I found that voice is the best modality to do that in because it's very hard to sound robotic when you're talking to, like, the the chat. It's like you have to just talk naturally. And then I found that it's it's also a lot faster. Like, if I were to sit here and type that, it would take me a lot. So here, I'll go here. I'll I'll paste this original prompt. You know? And then I'll say, okay. Cool. So I like that one. And now this is the second pass.
00:26:02 Sully: And now this is where, again, kinda going back to the workflow that I use, right, is I'll come in here and iterate with voice on this specific subset of a problem, which is generating this kind of, like, mega prompt. We sat there with g p two four o. We sat there with Claude, iterated a bit. And then I'm I'm like, okay. I have a rough idea. This prompt looks somewhat good, and then I'll come back to o one preview. And I'll say, okay. Cool. I want you to optimize this. And I haven't found like, I don't have a real scientific method to which one is best because I just kind of sit here. And and this is kinda where I have, like, a good first generation of the prompt.
00:26:36 Sully: Realistically, I'll put this into production. I'll write a couple of, like, you know, evals. I'll say, okay. How does this actually perform? And then kinda iterate back. But this is sort of my starting point. So we'll let this go. Okay. So here and then it gives me some things. Can you please generate the new prompt now? Alright. Cool. It gives me the revised prompt.
00:27:05 Greg: So it it did take
00:27:06 Sully: me out a
00:27:06 Greg: minute because you finally the answer.
00:27:09 Sully: Yeah. And and sort of you can see here and you can always say here this is just for the sake of this. And now what I'll do is I will take this, and then I will actually go to and this is my full workflow. We can use any model, but let's say we're gonna use you have a preference of which model you wanna test out the actual transcription? We can actually do
00:27:31 Greg: I'd love to hear which one you think and why, and let's just test it out.
00:27:35 Sully: Let's let's test it out. So now we go to Studio. So and you see what I mean? It's like there's all these different models. I'll go to Studio, which is Gemini. Now we're gonna go to Gemini, which I found. So specifically Gemini Pro, better at sorts of these these sort of tasks. And now I'm here with Gemini Pro, which I'm gonna take and grab the prompt that I crafted with o one, put it into the system prompt of, what's it called, Gemini Pro, paste in the the transcript, and we'll see how it goes.
00:28:07 Greg: Alright. Beautiful. Yeah. That sounds great.
00:28:11 Sully: Alright. Just copy this here. Okay.
00:28:15 Greg: This is how the sausage is made.
00:28:18 Sully: Yeah. It's it's this is how I like to think of, like, the first generation of a prompt where I'm not really sure where I'm starting off with. Obviously, like, is this something that I would use in production? Probably not because you wanna test it out and and have a lot of back and forth. But okay. Cool. Can I is there a way to copy paste the transcript?
00:28:36 Greg: You're just gonna have to select all down at the bottom there. That would be nice to just copy the transcript. Actually, I think I might add that feature in there.
00:28:45 Sully: Yeah. So let me see if I can just copy this. Alright. Cool. Now we go grab this. K. Let me and then I'll obviously, like, do a second pass to make sure that this actually makes sense. Key moments. Yeah. This looks pretty good. Time stamp, 3 to make takeaways, extract one sentence, discussion themes, theme name. Yeah. Like okay. Cool. So here, I'll paste this in, and we'll let it we'll let it run here. So I'm using Gemini Pro. Alright. 17,000 tokens. And and for for people who are curious, like Gemini Pro, I I talked about this recently is that a lot of models can't actually reason over a large context. Like but for something like Gemini Pro, anything under a 100 k tokens, it's it's pretty good at, like, being able to synthesize a relatively intelligent answer.
00:29:44 Sully: So here okay. Cool.
00:29:47 Greg: That's really cool.
00:29:50 Sully: And how, yeah, key moments, how you leverage CrossFit. I I'm actually curious to just, like, see how it just would do against, like, you know, other benchmarks because we don't really know if this is a good output or not, and that's where the the whole point of eval is. But there you go. You have how I went from an idea to generating, like, a full, I guess, optima air quote here, optimized prompt. And the reason for that is just like, for me to sit here and write this probably would have taken, like, an hour, hour and a half maybe, like, give or take depending on how good you are. But, you know, we just did it live in whatever 10 minutes.
00:30:27 Sully: So
00:30:28 Greg: Yeah. That's super super cool. I love that. So then out of curiosity, what are you using for prompt management? So I saw a a tweet by the CEO of Prompt Blair, Jared, and he's like, yeah. I see everybody that go through the same they go through the same world. 1st, their prompts are just hard coded in their code. And then second, their prompts are hard coded in text files, but they're still in their code base. And then third, you actually go to a prompt manager. What what are you using for prompt management?
00:30:54 Sully: So for that's an interesting one. We obviously, we use GitHub for our our our prompts. Yeah. So we use a lot of a couple of different things. And maybe maybe we're not, like, we're not prompt managing correctly, but Uh-huh. We just have our prompts that we store in Lang Smith and Mhmm. Sort of I'll just have datasets, and I'll compare that prompt to that dataset. So for example, we have a giant dataset of, like, a 1000 examples that I I run our test against different models, different prompts, and that prompt is just, like, stored, you know, in in the dataset. And then whenever I wanna change the prompt, I'll actually change it and and duplicate the dataset, paste in the new prompt, and like, my version control, so to speak.
00:31:41 Sully: So the actual prompt stays in my code base with the latest version of, like, this is the the source of truth. And all previous other versions are different datasets where I can see how they perform. So for example, if I wanna go back to a prompt that was, like, you know, let's say from a week ago, I just look at the dataset that was from a week ago and I can see the prompt is there, and I can also see how it performed. So that's how I manage, like, in inversion it. I'm not sure if it's that right approach, but that's how I do it.
00:32:08 Greg: Sir, so in your code, is the prompt that's being called, is it actually in your code, or are you calling out to Langhub and Langhub every single time?
00:32:16 Sully: It's in the code. So the the code our code, it's in GitHub. And the nice part is because it's just all version control. Like, I could look at the Git history and I can actually see, okay, this person changed this line as well, which is nice. So I have the line by line version of control from Git. And then if I wanna see the full prompt, I can look back at, like, you know, the the data management tool.
00:32:38 Greg: Yeah. That's very cool. I tell you what, I had one more demo on here that I was like, this would be so cool if solely solely would show us how we use this. It's a cursor one actually. So I saw that you tweet. You you said, well, I actually have the LLM write the test first, then the code. It helps a ton, which that's a framework I don't see too many people doing. Of course, there's test driven development, but, like, not in practice, not usually. I'm not seeing a lot of people do that. Could you walk us through, like, how do you write that test first and then how do you ask it to write code right after that?
00:33:08 Sully: Yeah. Okay. This is one that I the reason I started to do was because the problem I was facing, the model just kept messing up. Like, every single time, it was within our code base, and I was like, this is this is a waste of my time. The model can't figure it out. How about I just get it to generate the test first? And then if the test works, then I can maybe look at the code and say where the issues are because models guess what? If a test fails, you can grab the error output, give it back to the model, and say, hey. Like, please decipher that. So let let's actually see if I can, like, I can spin up
00:33:41 Greg: Like a little mini project or something. Or
00:33:43 Sully: Yeah. Yeah. Let's see here. If I can spin up something new.
00:33:46 Greg: I actually think this is really cool. And this is like something like, really, truly, not enough people are doing this and if it legit helps you write better code because it makes sense. You have the test that's supposed to run successfully, and it can use that as instructions, and it can use that to, like, test to make sure it's actually working.
00:34:01 Sully: I'm surprised not a lot of people not more people are doing this where it's like right? That's like it's just a lot easier for the LLM to, like, do that, and then your code is safe. I guess, like, you know, less spaghetti because you're not you don't you're not worried about, you know, if something changes, like, the model like, you start with the test, and it's really easy for the model to generate it. Okay. So I got a that took a little time. I got a, say, cursor here. So this is just a super quick let me just grab the screen here. Super quick here. So I have this, you know, super basic thing. We can just terminal.
00:34:35 Sully: We can run it, and I can go, you know, fun index. That's yes. Hello, world. Now I actually like to start with cursor, and I'll just say something along the lines of, like, literally and, again, I actually don't know how to write tests and fun, so I can just go to cursor. I open up command I, and those who know this is like the composer. It lets you coordinate and create files. So I'm gonna say, you know, I'm using fun for now. Create a test file for a method, and then make the method that, let's say, for now, reverses a string. Super simple. And, oh, I guess I'm at a slow request, unfortunately. K.
00:35:19 Greg: Wow.
00:35:19 Sully: So it what it'll first do is it'll create the test. Right? And this is obviously a really simple example. And so here, I'll I'm happy with this. Alright. I'll I'll just accept this. Sure. And now right off the bat, like, there's, you know, how many, whatever, 5 tests here. So, obviously, I have the actual function. So here in this example, it's just reversing a string. Now the nice part is I can go here. I can say, you know, bun I guess it's reverse test dot ts. And I can, again, debug with composer. This is a nice part. I can just go debug with AI. Gotta up I gotta up
00:36:00 Greg: my press.
00:36:02 Sully: Man, I'm I'm out of the free that that's how much I use cursor. I I just Yeah. Always blow to the device. But okay. Cool. So here, it, like, you know, passes the test. But let's actually say that, like, we are using something a little bit more complicated than reversing a string. Now I can go into here, and I can say let's just not reverse it. Let's just say, like let's just break the code. Let's just say here we'll split it like this. Okay. Return dot. K. So now if I go here, I go test, I go test files, so all these tests fail. Right? Now, obviously, it's like a pretty simple example. And it's almost as simple as just clicking this button that says add to composer, and then I say, you know, please fix the reverse method due to errors.
00:36:54 Sully: And now the nice part is here, Cursor will pull in that terminal that'll throw, you know, the errors where it happened. And what Cursor will do is they'll look at that, and they'll say, hey. Look. I see what the issue is, and it'll just fix it. So this is kinda what I like to call it. Like, I I don't actually have a name for it yet. Maybe LLM test driven development, whatever you wanna call it. But it's like you come in and you describe what you're trying to do here. The LLM writes the tests for it, and then it's gonna write the method. And then what you can do is have it run. And now if the method itself, like this function, which is reversing a string, is is complex or confusing, it will be able to sort of, like, essentially agentically, air quote here, fix itself, if that makes sense.
00:37:34 Sully: It'll test the code, see if it passes the tests. If not, it'll update the code and then sort of do that until it can, you know, pass the test. And all you have to do is make sure that the tests you're writing are correct. And I and I use this a lot for obviously, for simple functions, it's not that useful. But when you have code that is across a couple different files, you know, in a in a modern code base, it's not just a single function. It's like you have, like, you know, a bunch of different files and and stuff connecting. And ones that require a lot of, like, conditionals or, like, they're not as simple as this.
00:38:09 Sully: It's like that's where I found that whenever I would try to get, like, cursor or sorry. I'll get, like, Sonnet to one shot it, it would fail every single time. But then a second that I was like, okay. Please let's write the test for it, and then I would sit there and kinda help it the test. It was able to debug itself a lot better and go through these, like, bigger, maybe meatier functions that normally wouldn't be be able to even, like, o one and o one mini couldn't solve, but a second that I would apply this, like, test driven development, whatever you wanna call it, The model is able to look at the output, see where it messes up, adjust the code, and kind of iterate on itself like that.
00:38:41 Greg: That's cool. So not only does this test first mindset it's kinda like a prompt engineering technique. It's almost like think out loud, but it's almost like like write the goal first and then tell me what you think we should do for it. But you also get tests out the other end, and so you get a little bit of extra utility as a byproduct.
00:38:59 Sully: Exactly. It's it's a it's a win win. You get a little bit of both. And to me, that was the one thing I never understood why people haven't done more of because you would think, well, if it passed all the tests, the the code is, like you know, you're happy that it passed the test, but it's something that I haven't seen a lot of people do.
00:39:14 Greg: Yeah. Yeah. For sure. Well, that's awesome. Well, that's fabulous. Thank you for showing me the cursor example. One of the questions I love asking is I wanna know what the smart people are talking about right now, like an AI. So, like, as you observe on Twitter in your circles, what are the smart people talking about?
00:39:31 Sully: That's a good question. Oh, man. I think what I see a lot of people talking about is sort of the you know, what's it called? Like, test time compute, like, o one thinking. I see a lot of people talking about those. I see a lot of people talking about having think those thinking models do more agentic sorts of tasks and basically bringing this what I like to think of as an agent as a 4 loop inside to the model thinking process, having and training the the model to just innately be able to call tools, like and we saw that. I think a good example of that is computer use, right, from Anthropic. Right? They they obviously fine tuned it on that.
00:40:12 Sully: So I see a lot of people talking about that. I do see what I started to notice is people starting to talk about whether we've hit some variation of a wall. I don't know if you've seen it too. And I'm hearing the little rumors that, you know, cloud 3.5 Opus is not up to par and, like, the the new Gemini model is not as good. So I I'm hearing that as well. And what else are people really talking about? And I think I think we spoke a lot about the other things, model distillation. And the other thing I've started to see more of is people being a little bit not I guess, talking more about evals. Like, I I think a lot of people didn't really talk about it, and people are saying, hey.
00:40:50 Sully: Like, from a product perspective, if you want your product to be good, you need to write evals, which are just a way of writing tests. So that's kind of what I've been seeing, and I don't know if you see anything different, but just from what I've heard from people talking.
00:41:01 Greg: Yeah. Let me think. Is there any anything else I would add to that list? The one thing people aren't talking about it, but I think it will be a big deal when it actually comes out, is the whole feature engineering weight manipulation like the Golden Gate Claude Anthropic. Yeah. I'm still waiting for access to that because that is going to be an alternative to prompt engineering. And I have no idea, like, how easy it's gonna be to work with, what kind of results we're gonna get, but I'm excited to test that whenever it comes out.
00:41:28 Sully: Yeah. I I I remember seeing that, and I was, like, I was blown away, and I kinda forgot about it. So that that I'm actually interested to see if they ever will ever let you have that much interoperability with those models. Like, maybe there's, like, no. No. We're good. Sorry. We're shelving it. Like, you know how to touch it. Right? But For sure. That would be really interesting.
00:41:44 Greg: Yeah. For sure. For sure. Awesome. Two more questions here. Last one. I love hearing about what is in people's toolkit. So I've seen you use Accely draw on ex Accely draw on your YouTube videos. I've seen you use Replit. I've heard rumblings about v zero. What else is in your toolkit that is in your kinda day to day workflows?
00:42:03 Sully: Okay. Okay. So there's a lot, I guess. Yeah. You got you got a couple v zero. Obviously, there's cursor. Excalidraw, I like it for drawing the little diagrams. The other one, I guess, that I use a lot is the playground from Anthropic and from OpenAI, which is, like, different than ChatGPT. I use that to iterate on prompts. I use this yeah. The the one that I use for transcribing the actual audio is called Whisperflow. It's the one where I, like, I have a hot key that I press, and it takes the voice and transcribes it into the inputs that you saw me use. The other tooling that I use I mean, we can go in do you wanna go into the technical side, or are we just gonna leave it at, like, the high level tools?
00:42:43 Greg: I let's let's not go, like I don't wanna know your entire tech stack, but, like, what is in, like, the cool AI stuff that, like, you're you're you're grabbing for?
00:42:52 Sully: I think that's pretty much it. I think I think you got it there. I I there's not many other tools that I honestly use. I I just I like a lot of it's yeah. Like, just writing the code. LangSmith is one actually. I will say that we we we use LangSmith a lot for evals. That's, like, the other one. But, yes, that's pretty much it from from me. I think you nailed it. V 0, cursor, Excalidraw, OBS if you're recording videos. Yeah.
00:43:18 Greg: Yeah. Yeah. For sure. Alright. Last question. And this is kind of off topic from the AI side, but I know people would be interested in it. So you've had a few bangers on Twitter, like, just some things that just absolutely pop. And as somebody who does a little bit of Twitter himself too, I can look at a tweet and be like, that person thought about it, and they did a really good job as to how they architected it and and constructed it. And I noticed that with yourself. So what hits on Twitter? And what would what's your advice for people who, like, wanna do better on it?
00:43:44 Sully: Oh, man. Okay. So Twitter is just this hilarious platform that the algorithm changes a lot. So it's you kinda gotta get a feel for what works and what doesn't. And luckily, the cost so if anyone's looking to grow, the cost to post on x slash Twitter is 0. Like, you don't pay anything. If it doesn't do well, no one cares. So it's the one platform where the cost is literally 0 because you're just typing. So type things away. How I craft a banger, it's like a mixture of what I see trending. So what I see, what people are talking about. And there's 2 ways to craft a banger. 1 is you have to be controversial. You are not gonna craft a banger if you're not controversial.
00:44:24 Sully: Now there's pros and cons is if you're posting that kind of stuff all the time, people will be like, hey. You're just posting clickbait. So you gotta be careful with it. You can't be like, this is insane and every single tweet starts with that. Like, no one and no one's gonna believe you. But saying something controversial. And the most important part of crafting a banger is your hook. It I can tell like, honestly, I will post something, and I can tell within 20 minutes if it's gonna be a banger or not. And it's basically how natural does it come. That's one. It's like, how natural did this thought come to me, and how well did I craft that hook?
00:44:58 Sully: Everything in between, like, you could you can kinda sit there in min max, but the the that's how I sit there. And sometimes I'll sit on something, and I'll be like, oh, man. Like, I just don't know the right way to say it. So I won't post it. But then it'll just come to me, and I'll be like, alright. I got this. I all the words, I'm using the right structure. It's like the the right timing, and and that's kinda what goes into crafting it. So the one piece of advice that I will give from my personal experience is don't spend too much time on a tweet. Because I unless you're doing it educational, there's there should be a a diagram where the more time you spend thinking about a tweet, the worse it does.
00:45:36 Sully: Because I swear, the majority of my bangers, I spend, like, 15 minutes thinking about. I'm like, alright. I'm just gonna post it. You know, grab a coffee. I come back, and the ends blow it up. And I'll
00:45:45 Greg: And then all of a sudden, you see 1.4000000 views.
00:45:49 Sully: Oh, man. I do you have time? I have to I have to I have to tell you the story of how this Yeah. I'll be sure. Do I have time for that?
00:45:55 Greg: Yeah. Yeah. Let's hear
00:45:56 Sully: it. Okay. So because it it's so relevant to the banger tweet. So my company, we we started, like, a year and a half ago. And right this is around the time that agents like, people were talking about them, but didn't have any clue. This was, let's say, March 2023. And at this time, I I was no one actually knew of my account. I literally had I had been posting tweets, and no one replied. You know, the classic zero views. You know, that's just what happens. Yeah. Yeah. And then and I remember I saw someone else post something about auto g p t. And I saw it, and I was like, oh, it looks pretty cool, but I ignored it. And then it came up again.
00:46:34 Sully: And I was like, no. I can't I cannot not ignore this. Like, this seems something very interesting. And I'd been building actually, like, AI projects, side projects before this. And I was like, you know what? Let me, like, try this thing out. And, obviously, I tried it. And back then, I was like, dude, this is insane. Agents, AI, it's gonna be crazy. So when I was like I just posted about it. And, like, I didn't post anything crazy. And I was like, oh, yeah. This thing is kinda cool. It's pretty crazy. And it, like, got, like I think that was the first post that got over a 1,000 likes. I was like, wait a minute.
00:47:03 Greg: Wow.
00:47:03 Sully: And then I was like, hold up. Hold a second. Then I saw this trend that people wanted to do something about, like, AI agents. And it's interestingly enough, I, like, thought back to an episode of m I f like, my first million. It's getting so funny. And and I remember them talking about, like, there's sometimes you see, like, this opportunity. And I was like, dude, I gotta sit here and I gotta do 2 things. 1st, I gotta craft something. I gotta make a product that people wanna use, and I gotta figure out the right Twitter thread and narrative and story to craft to get people on it. So that weekend, I spent the whole weekend building v zero of Cognosys, which was like our previous product.
00:47:39 Sully: In the meantime, posting Twitter bangers and threads about how AI agents were going to change everyone's life. And every single post was getting, like, a 1000000 views. I'm not even exaggerating. Wow. And I was like, dude and I was like, okay. And all I would be posting, I was like it was kinda clickbaity. I was like, this is gonna change your life. And then getting, like, 1,000,000 view, 1,000,000 views, and I post the product. Like, I was like, hey. Like, here. I built this thing for you people to go and try because I know from what you've been telling me, you don't wanna go through GitHub. And I and I posted out, and it was literally built it in, like, 3 days.
00:48:14 Sully: And within, like, 2 days, we got 50,000 users. So
00:48:18 Greg: Oh my goodness. That is so crazy. The
00:48:21 Sully: the craziest 2 weeks and the most stressful 2 weeks of my life, and it started all from how can I craft a banger tweet? So I I will say it with that. That was why it's so relevant and so funny. It just shows how powerful writing well and writing with the right timing and structure given what's happening can potentially, you know, help you start a company. So
00:48:43 Greg: And with that, that is an absolutely beautiful story to end on. Sully, thank you very much for joining us today.
00:48:49 Sully: Oh, dude. It it was a pleasure. I I enjoyed it. And hopefully, my workflow is applicable to other people. People can look at it and see that, like, hey, using AI is just not that hard. You just gotta talk to the computer, and it'll do stuff for you.
# File: /content/learn/dev/ai_show_tell/tibo.mdx
---
title: Tibo, $8M Exit & Maker Of The Year
description: AI-powered workflows, mastering prompt engineering, navigating the costs of AI video generation, and the future of AI in gaming.
date: 2024-11-21
---
[Tibo](@tibo_maker), a successful entrepreneur who sold Tweet Hunter and Taplio for $8 million, shares his journey and insights into leveraging AI for product development. This interview covers his early adoption of GPT-3, his current AI tool stack, cost management strategies, prompting techniques, and thoughts on the future of AI in gaming. He emphasizes a scrappy, iterative approach, focusing on building tools for personal use and rapidly shipping products. He is currently building [revid.ai](https://revid.ai), [feather.so](https://feather.so), [superx.so](https://superx.so), and [outrank.so](https://outrank.so).
### Insights
* **Early Adoption Advantage (and Misconception):** Tibo started using GPT-3 in 2021, focusing on analysis rather than generation. He highlights the misconception that it's "too late" to enter the AI space, emphasizing the vast untapped potential.
* **Build for Yourself:** Tibo builds tools for himself to solve his own problems and streamline workflows, which allows for deeper understanding and faster iteration.
* **Leverage High-Level APIs:** He prioritizes using readily available APIs like Replicate, AssemblyAI, and others to save time and avoid the complexities of managing servers and GPUs.
* **Character Consistency in Video:** He discusses techniques like masking and inpainting to maintain character consistency in generated videos, addressing a significant challenge in the field.
* **AI Costs and Margin Management:** Tibo openly discusses the high costs of AI video generation and the challenges of balancing expenses with user expectations and rapid growth.
* **One-Day Sprints for Rapid Iteration:** He shares his strategy of dedicating one day a week to building and shipping a completely new project, highlighting the importance of rapid iteration and experimentation.
* **Mega Context Prompts:** Tibo uses TypingMind to create and manage project-specific mega prompts, ensuring consistent context and eliminating the need to repeat instructions.
* **Generated by AI, Polished by Humans:** He champions using AI for generating the initial draft and then refining it with human input, optimizing both speed and quality.
* **Structured Outputs with JSON:** He emphasizes the reliability of using JSON for structured outputs from language models, simplifying parsing and integration.
### Frameworks
#### Build for Yourself
Tibo's core philosophy revolves around building tools that he would personally use. This allows him to deeply understand the problem he's trying to solve and iterate quickly based on his own needs. It also bypasses the complexities of traditional product development processes.
#### One-Day Sprints
Tibo dedicates one day a week to developing and shipping a new project. This rapid iteration cycle allows him to experiment with new ideas and technologies, increasing the likelihood of stumbling upon a successful product. It also fosters a mindset of continuous learning and experimentation.
#### Generated by AI, Polished by Humans
Tibo advocates using AI to generate the first draft, which he then refines manually. This framework allows him to leverage the speed and capabilities of AI while maintaining quality control and adding a human touch. It optimizes for both efficiency and quality in content and code creation.
# Transcript
00:00:00 Tibo: All the people that think this today, I think they are so wrong too. Like, there's so many things to be left. Model that just got out a few weeks ago, it's much better at creative thinking and it's 10 times cheaper. Replicate is an incredible example. So from day 0, I want to save time by use super high level APIs. It was way better compared to what a human can do to analyze, summarize, cluster, and categorize stuff. It honestly saved me 1 hour by creating the bones of the component.
00:00:47 Greg: Meet Tibo. His first product had $0 in sales. His second product hardly made any cash at all. But on his 15th product, Tibo sold Tweet Hunter and Taplio for $8,000,000. In 2022, he was awarded maker of the year by Product Hunt. In this interview, Tibo walks us through which AI tools he has bookmarked in his browser, which techniques he's exploring for character consistency in video generation, and his thoughts about when to use o one and ClaudeSonic together, or maybe not together, when generating blog posts. Let's get into AI show and tell with Tibo. Tibo, I'll tell you what. So I was looking at your Twitter during my research here, and in 2021, you had a very interesting tweet.
00:01:27 Greg: And the tweet said, do you want me to have GPT 3 analyze your tweets for you? And what blew my mind was this is 2021. So when did you start with AI? How would you walk me through that?
00:01:40 Tibo: I think that with that I think it was, like, one of my first who got some decent engagements this night. And I think it was the very, very beginning of my successful startup. The thing is, I think at the time, I already had a few project failures with GPT 3 because, like, I think, like everyone else, I was trying to generate stuff. Like, I was trying to use GPT 3 to generate tweets, generate copyright copy. And at that time with these tweets, I realized that GPT 3 was super, super good at analyzing stuff. Like, it was it was way better compared to what a human can do to analyze, summarize, cluster, and categorize stuff.
00:02:31 Tibo: And so I I did this tweet where I would just use the a p the Twitter API behind the scene, getting some information, dumping everything into a g p three prompts, and just just tell it's like, analyze stuff. Tell me what's what's interesting about this profile. What's what's the main topic that's generate engagement for him and it works. And I was I remember that I was manually, like, launching my function, my GPT 3 function, like copy pasting the handle of the guy putting it putting the handle on my code, launching the function, and copy pasting the results back to Twitter as a reply.
00:03:19 Greg: Yeah.
00:03:20 Tibo: I did, like, 20 or 30. And every time I had the like, I had a mistake in my code, like, I was seeing something bad, I was I was actually fixing the prompts on the go and just trying to improve it. It was so fun. Like, I was I was seeing the likes going up and my prompts getting better and better. It It was it was so fun.
00:03:42 Greg: That's wild. So your likes go up, the prompt and performance goes up. But then also probably your motivation and your excitement goes up at the same time.
00:03:48 Tibo: Yeah. Exactly.
00:03:49 Greg: Yeah. Yeah. Yeah. And that was back when it was still magic. Because, like, 2021, that was pre chat GPT. So this stuff was still, like, relatively new all over the place. I feel like it wasn't a big crowd that even knew what GPT 3 was at that point.
00:04:02 Tibo: You know what? Like, when we joined, I think I think we needed, like, 6 months to get a GPT 3 key, like an an open AI key because it was so restricted at the time. And when we finally got it, it was like everything was done. Like, all the cool use cases was built and it was too late. Like, we were we missed the train. Yeah. And it was so wrong. And all the people that think this today, I think they are so wrong too. Like, there's so many things to be left.
00:04:33 Greg: Well, so let let's keep on going down that path. And so I love that metaphor that that was your mindset back then in 2021, and you still kinda feel the same way today. So, like, what's your advice for people who say we're too late? Like, we've built everything we need to so far.
00:04:46 Tibo: I would like, I think my the thing that worked for me and that I want to repeat at every new thing that I that I build is I want to build something for myself. Like, I I want I want to be maybe not the main user, but I want to be the user of my own product because I don't have the discipline or I don't have this case to, I think, to manage a big team and to build the right processes so that the correct product manager can learn from the clients and categorize the feedback in a good way so that another team can build the thing. Build the thing. I don't have these skills. What I know is that when I when I use a tool, I know how to fix my problems.
00:05:26 Tibo: I know how to take my daily workflow and to make it simpler, thanks to some code that I will just dump somewhere in an app.
00:05:42 Greg: Yeah. Yeah. Yeah. Absolutely. Well, I'd love to hear more about the tool stack that you're building with. And so as you're building these tools for yourself, you had a tweet that said or it it kind of showed your p and l, so it showed with your expenses. And you were talking about Deepgram, OpenAI, Fall, LUMA, Sieve, 11 Labs, all of this. Where is your tool stack settled from an AI perspective right now?
00:06:05 Tibo: I think I have a are you a are you a ARC user? No, I
00:06:10 Greg: no. I do promise.
00:06:12 Tibo: ARC is my browser and in ARC I have like folders. Like like, it's a little bit like a bookmark, but it's it's better organized and have a folder that's Could you show us? Yeah. Yeah.
00:06:25 Greg: Sure. That'd be really cool to see.
00:06:28 Tibo: So this is like my all my cool AI APIs that I'm gonna use. Like, it's if if you combine the power of file, the power of, like, replicates and and save too where where is it? This one. You have, like, you have tons of AI models that are, are ready to use and super powerful. I think, like, most people that are trying to like, I see tons of people that are trying to get a server up and up and running, like, set up a few GPUs and scroll hacking phase, find coolant models to use, and spend so much time setting them up, fixing crash, fixing configuration issues. And I really don't see the point of that. Like, those providers, they're going to be more expensive, but they're going to save you so much time.
00:07:30 Tibo: Like Yeah. Replicate is an incredible example because they have so many models. There are so many that you can't, you cannot even like explore them. You have to search for something because you have just so many things. So from pretty much day 0, because I want to have a super lean team, I've always wanted to, like, use, like, ready to use things. Like, I I want I want to save time by use super high level APIs Mhmm. Like them together with code and and make a product out of that.
00:08:08 Greg: Yeah.
00:08:08 Tibo: And I think when you do when when you are able to chain a few APIs, like chain Chargept to then generates an AI image to to then put that in the video, it's, like, so satisfying. You know?
00:08:22 Greg: Yeah. Yeah. Yeah. Yeah. Well, I took so I tell you what. I would love to hear how do these tools chain together? So what do you use replicate for? What do you use c for? What do you use Luma for? And, like, how how does your stack look like?
00:08:35 Tibo: I would try to, like I think I think file is very good as so you have, for example, like I'm using Flux a lot in in file. I think it's, like, one of the best provider for for Flux. I'm using I'm just generating images there. Then I'm going to, like, let's say, minimax, this one, or Runway. They have a few image to video models. And so you are like that, you are able to, like, have a prompt with gpt3 or Sonet or anything else, like, that would generate the image. Then you use Flex to generate the image with the prompts. Then you take this image and you put it into a runway. And just like that, you have a 3 step workflow where you're able to generate a video a short video throughout, like, basically anything.
00:09:37 Tibo: Yeah. One thing that is critical with file, and I haven't explored that, but they they make it quite easy to do comfy UI workflows, which then lets you build super specific and complex AI workflows by just chaining models between them. Like
00:10:03 Greg: Sure.
00:10:04 Tibo: Something like one thing that I haven't nailed yet, and I'm working a lot on that, is I want to be able to, like, use this kind of stuff, like train a Flux LoRa, meaning that I would be, like, training training an actual model from a few images of, like, someone or something. Mhmm. And then use that model to then generates generate a video of the same
00:10:37 Greg: person,
00:10:38 Tibo: with a strong consistency in the video itself. So, like, a a video is, like, 1 or 2 minutes long.
00:10:45 Greg: Uh-huh.
00:10:46 Tibo: Right now, it's incredibly hard to have character consistency in a video because you have to generate, many, many images and and short videos.
00:10:56 Greg: Sure. And what what what's the secret to consistency for those things?
00:11:00 Tibo: I don't so there's there's one secret that I'm, like, I'm just scratching the surface of that. That's you I I I think that is, like, a a strong like, a huge value in using mask and in painting. I mean, like, basically, you will generate an image. You will with a character, and you will just try to create a mask from about this character. You will extract this character and regenerate the correct image with by using this custom model with a tech called in painting where you just Sure. You just tell the the the model to just generates the missing piece of of the image. Does that make sense to you?
00:11:45 Greg: For sure. It's like, let the model run wild and everything in the background and the body and the sky and everything, blah, blah, blah. Because that's not what you want the consistency for. But then mask and then maybe the face because then that's what people really care about. And that's what you really need to get right is the face piece of it.
00:11:59 Tibo: And and this step is super important if you want multiple characters. If you want the the issue with fine tuned models when you're trying to generate images is that if you are using if you combine multiple fine tune models where each one has been trained on a specific character Mhmm. The image generation model will just merge all the characters into 1. And so all the characters on the image will look the same and will be a merge version of all the fine tune models.
00:12:36 Greg: Sure.
00:12:36 Tibo: Which is interesting. This is Harvard.
00:12:38 Greg: Yeah. Yeah. Yeah. For sure. It's interesting. It's like you want a model for every subject that you want consistent. And so it's almost like you have the parent image, and then you have a tree of all the subjects in And so it's almost like you have the parent image and then you have a tree of all the subjects in there that you need to have remain consistent. Yep. And so that's that's some I I'm just I'm just I'm
00:12:49 Tibo: blown away by by the fact that this has not been properly handled by like, this is a super important problem in my opinion, and there's a big value in fixing that.
00:13:05 Greg: Well, I tell you what. I mean, look at where text was even just 6 months ago. You couldn't do it. And now it's like a non issue. So I think that these things will just come around.
00:13:12 Tibo: Yeah. Sure. So
00:13:14 Greg: You had another tweet that was super interesting, and you said that AI is eating your margins. So, basically, I I read that as there's it's super expensive, paying a lot of money for it. How do you manage costs, or how do you track costs across all these tools?
00:13:28 Tibo: I have a big issue right now. It's that's like with my projects, like Revit dotai. It's it's the first time that I'm I'm working with videos. And AI videos is the most expensive stuff that I have ever seen. If you, like, if you are if you're trying to use if you're trying to, like, the the top level models, like Runway, Luma, or Cling, they cost you between 20 to 50¢ per 5 second video.
00:14:04 Greg: Wow. Yeah.
00:14:05 Tibo: And so when you have a user that is trying to create, like, a a 1 minute video, it's it's easily gonna cost you, like, more than a dollar, $2, maybe 3 if you add some some, like, voice duration on top of that. And so there is a huge disconnect between the price of those API and the expectation from the user. Like, the the users, they don't understand why it would cost that much because it's just 1 minute video. And so they are constantly expecting us to lower the price or they're just screaming at us because of the price of the products. But at the same time, our AI costs are so high. Like, they are I haven't like, more than more than 60% of the projects.
00:14:54 Greg: That's wild. That's wild. You know, one of my favorite things I don't love everything about Elon, but one of my frameworks I took from him was his idiot index, which is the price of something the price of the raw materials to actually do it. And then the delta between the two says how much of, like, you know, buffer there is from that. Do you know how much it costs hardware providers to generate those videos versus what they're actually charging you? Like, what are their
00:15:17 Tibo: margins? I have no idea. Like, this is not public. It's it might be that they are not even profitable. Like like, touch gbt is not like, OpenAI is not profitable maybe because
00:15:28 Greg: Sure.
00:15:28 Tibo: I think I think they want to, like, they want to, like, get the markets. They want to get the market share. And so they are, like, lowering their price to their very minimum they can. It's it's likely that they are not even making profit out of that.
00:15:46 Greg: Sure. Well, it's like the rumor about Grok having unprofitable tokens as well. The the, not the xai one, but the, the hardware provider.
00:15:54 Tibo: Yeah. Yeah. I see. Like, it's it's it's possible. The Yeah. Their generation is so fast. Like, it's it's likely that's it's the case. Yeah.
00:16:03 Greg: Yeah. Yeah. Yeah. For sure.
00:16:04 Tibo: For this project that I talked about, like, Revi dotai, the the issue is, like, it's there there is something that you, learned at beta school. And, for, like, for 10 years, I thought that it was not relevant to me, but it seems like it is. Like, the thing is our AI costs are more than half of what we are getting every month. And we are growing so fast, like, 50, 60, 60% month over month.
00:16:35 Greg: Wow.
00:16:36 Tibo: And and our revenue we we get the payout of our revenue 1 month later compared to what when we are getting it. Yeah. All that summed up makes us very hard to sustain the growth because we are paying right now the expenses of this month, but we are getting right now the revenue of the month before. And so since the growth is so hard, we are just barely getting enough money from the sales to cover the expenses of this month because of this delay.
00:17:11 Greg: Yeah. That's wild. It's funny how there's no better motivator to learn business topics than having too many expenses and too much money to try to deal with it.
00:17:19 Tibo: That's that's a good problem to have, I guess, but it's still surprising.
00:17:23 Greg: Yeah. Yeah. Yeah. Absolutely. You know, one thing that you were describing as you were coming through, you were talking about fall with Comfy UI, then you can do complex workflows for that. Was the mask example that you gave an example of one of those complex workflows? Or what's complex to you that you would be, you know, coding up in there?
00:17:40 Tibo: Yes. It is. Like, this this process where Meta just released some an awesome mother called called the, like, SAM 2.
00:17:51 Greg: The object detection 1.
00:17:52 Tibo: Yeah. Exactly. It it's it's able to detect the very specific edge of an object on an image and even a video, and it's able to label it. And so using that plus some in painting tech and some other advanced labeling technology, like, all that together, yes, using Convy UI is a nice thing to use. I'm not, unfortunately, because I'm not comfortable with that yet. Mhmm. I'm too I guess I'm too scrappy to to use it, so I need to, like, see my code and be able to, like, fully control it. But I'm pretty sure I would be able to do great things with it.
00:18:37 Greg: Yeah. Well, I tell you what. You you strike me as somebody who, when he wants to learn something, he has a pretty good way, a determined like, a very determined way of going into it. So say you're gonna approach Learning Comp UI. What's your personal learning plan to go do so?
00:18:52 Tibo: I would try to be like the bare minimum thing. Like, the just just I I would try to, like, get the first ID that I have and start working on it right away and try to ship something to the public as soon as possible. And I would I would treat about that even if it's, like, even if it's shitty and even if it's not, not that useful, I would just tweet about it and expect a few of my friends to test it and give me feedback. So 2 years ago, I was living in Bali for like 6 months. Nice. And at the time, I was working on Tweet Hunter, one of my most successful business.
00:19:33 Greg: Mhmm.
00:19:34 Tibo: And what was super cool is that I thanks thanks to a small group of hackers that I met there, I was able to, like, all the week work on Twitter Hunter, my main projects. But one day per week on Thursday, I would meet the group of hacker. And during one day, I would try to build something totally different. And it it can be related to Twitter. Like, many times I build, like, some me tool that was different code base, different setup, different server compared to the main projects, but could serve the growth of the main products by just being a firm thing aiming at going viral. And and I found this I found this to be quite healthy, like, to have, like, 4 days per week dedicated to my main thing and one day per week where I would try to read and ship something in just one day.
00:20:33 Tibo: Yeah. And I I did, like, many times. And I think my most viral products my my most viral tool Uh-huh. I really did that way. Doing one That's
00:20:42 Greg: so cool.
00:20:42 Tibo: One day sprints. That's it.
00:20:45 Greg: That's so cool. It just what stands out to me about that is when Sam says that the entrepreneur's way is to basically have a lot of shots on goal because you just need to be right for one of them. And so having one day a week where you can I I think the important part about that is is 2 things? 1, you have the execution to actually do it and actually commit to it, which is really nice and the consistency. But 2 is you package up the entire thing that you can ship in one day. So if you can't do it in one day, then it's it's really tough to do.
00:21:12 Tibo: I think I think you're so right. That's when we started, like, I think in, in 2021, like a few weeks after the initial tweet that you mentioned, we, we, we were in this process of building 1 project per week for and we did that for like 4 months. During 4 months, we tried to, like, ship 1 product per week because 9 of 9 product out of 10 fails. And it was, like, the 10th one who eventually got successful. Yeah. And that's I think that's how you build something truly valuable, like, by creating 10 things without knowing the one that will be valuable. If you're up for that, I want to talk about this all one thing because Sure.
00:21:59 Tibo: Because I I kind of changed my mind.
00:22:03 Greg: Oh, yeah. That's great. So let's talk about o one. What what are your thoughts on it?
00:22:06 Tibo: So the thing is I'm I'm working for, like, a few weeks on this new SEO project, where, basically, we have an interesting approach where we're trying to find the super relevant keywords to to rank for for your business. It's not launched yet, but I'm onboarding people right now in the beta phase. And it's it's working very well. And at the very beginning of the project, what's unlocked the project was the reason of o one. Because on this project, we were we're willing to spend a lot of money on prompts and LLMs and to to generate high quality blog posts. And o one was quite good at that. Like, it it's we thought that it would be the the the the piece of tech that would unlock the high quality contents.
00:22:57 Tibo: And the thing is it's very good at math. It's very good at logic. It's surprisingly good at complex problem solving. But the thing is the new release of, like, the new Sonet model that just got out a few weeks ago
00:23:18 Greg: Mhmm.
00:23:19 Tibo: It's much better at creative thinking.
00:23:22 Greg: Mhmm.
00:23:22 Tibo: And it's, I think, it's 10 times cheaper. And so we reverted to this one. And and and, yeah, we just by by prompting, by getting more in-depth about what we want, by providing some example. Like, I think I think this is super important. People don't do this enough. It's by by providing good context and example about what we wanted. I think we are able to work with SONETs and to get better result than o one. That's specific to our workflow, by the way. But I think creative thinking is a very, very interesting way where all one is not very good.
00:24:06 Greg: For sure. So I would imagine when you're doing, AI generated blog posts, there's many steps in the pipeline from candidate ideas, pick the idea, pick the title, write the actual thing, come up with the outline, write the come up with the outline, write the actual paper, etcetera, etcetera. Like, there's a lot that goes on to it. Will you use Sonnet for every single one of those steps? Will you one shot it? How how will you break it down? Like, what's what's your thought process with that?
00:24:30 Tibo: Not not entirely. Like, there is one thing where when you're trying to generate very long stuff, like, very actuates and in-depth contents, SONET can sometimes try to limit the length of the generation. The one model that I found that is not trying to do that is Gemini. Gemini 1.5 Pro is very good at generating long content.
00:24:58 Greg: Cool. Beautiful. So will you have Gemini actually do the writing for you or will it write the outline for you? Or how's that work?
00:25:06 Tibo: I cannot go into that.
00:25:07 Greg: Okay. That's fine. Alright. No. I I understand. Yeah. I totally understand. Okay. So that's the o one side. Beautiful.
00:25:15 Tibo: The maybe the one thing that's I think it's interesting and to be cool to show up is I would I would highly encourage I I will share my screen.
00:25:28 Greg: Yes. Fabulous. I love this.
00:25:31 Tibo: The I I think I think, like, you you said that you are working on many projects, and I am too. And I think that's the norm. Like, I think I think right now, most people are working on many, many projects because they have unfinished stuff. They have the new stuff. And the one thing that I do, which I think is quite useful, is I'm using typing mind. It's from it's basically like another chat gpt UI, but it's very convenient for me to to make it to work on multiple project because I have those folders. And those folders, they are that like, my all my project that I'm running right now. And if we take this one, for example, I would be able to, like, set a specific set of instruction for these specific projects.
00:26:26 Tibo: And so then every time that I will run a new prompt, on on on this project, it will have this, like this context about this project. I will not have to put everything back here.
00:26:43 Greg: Mhmm.
00:26:44 Tibo: I think this is a very simple step. Like, it's it's not crazy workflow that you will spend weeks to build, but just spend spend the time to work on this mega context prompts for your projects, do it for each of your projects. And then you have something super specific, super useful where you can ask anything anytime.
00:27:09 Greg: Yeah. That's very cool. You didn't build that project, did you?
00:27:13 Tibo: No. I didn't. I would love to if I did, but I'm not. It's a guy called Tony Dean.
00:27:20 Greg: Okay.
00:27:21 Tibo: He's he's pretty big on Twitter.
00:27:23 Greg: That's very cool. That'd be cool if after each chat session that you have with it, it's like, hey, here's the new stuff that I extracted that's not in your system prompt already. Do you want me to add it to it? And then it just comes to the top.
00:27:33 Tibo: Amazing. Yeah. That's be amazing. I I think TagePT is trying to do that, but
00:27:38 Greg: Yeah.
00:27:38 Tibo: It's it's so it's way too b to c to do it in a in a good way for us.
00:27:45 Greg: Sure. It's almost it'd b to c. Yeah. You must need you want the power user use case is Yeah. Exactly. Yeah. Yeah. Yeah. Yeah. That's the side that you're on for it. That's cool. Nice. Well, do you think that the regex demo is interesting? Like, would that be cool to walk through? Is that kinda small?
00:28:01 Tibo: No. I don't think so. But Okay.
00:28:03 Greg: That's fine.
00:28:03 Tibo: Basically basically, basically, I'm using cursor. Do do you know about cursor?
00:28:09 Greg: Oh, yeah. Big time. I'm a I'm a huge cursor user.
00:28:12 Tibo: And basically, every time I have to every time I have to use a cron Mhmm. I I triple check my my cron with with cursor. Like, I just I have I have these tools where I just press command t by selecting the the text, the the regex or the the Chrome and and just tell me if it's good or not. Something that I do a lot too after testing, I select my function or my code and I I do the command t again and I will ask it if is there a mistake here? Like, is it is there an edge case that I didn't think about? And that's it. And and just doing that allow me to go to be to be way lighter on the on the testing, like, on the manual testing and push the production very fast.
00:29:07 Greg: Interesting. And I was just speaking with Sully Omar who says that when he uses cursor and he writes with it, he actually has it write the test first, And then it's a lot easier to actually write the function to execute against that test because you have a validated test that comes with it.
00:29:21 Tibo: Yeah. That's amazing. This this is this is super interesting work for the thing is I've never I've never written a test in in my life, and I don't really I don't really plan in starting. But when you when you used to do that, I think that's very powerful.
00:29:38 Greg: Yeah. Yeah. Yeah. That's cool. That's beautiful. I wanna jump back over to the prompt side real quick. And so you said you're using a lot of Sonnet. And I know that Sonnet is works better with XML. So what sort of special formatting do you put into your prompts with regards to headers or XML or anything like that?
00:29:55 Tibo: I'm doing JSON every time.
00:29:57 Greg: JSON?
00:29:57 Tibo: Basically, I think I think that the sentence that I type the most is output in JSON. Do not output anything else. Uh-huh. Uh-huh. That is
00:30:09 Greg: So it's basically structured outputs only that come out from it.
00:30:12 Tibo: Yeah. And it's super reliable. Like, I I don't really understand why people say that it's not, but it is. Like, it's very reliable in in, like, being following this format that I'm going to I'm I'm showing it to you.
00:30:27 Greg: Yeah. Do you have a validator on the other end? Or do you just, like, accept the output no matter what and see what hopefully, it comes out alright?
00:30:35 Tibo: I have a custom made function where I validate the just not boots. And I retry if it failed to do so. But it was super useful 2 years ago when I was using a previous version of GPT 3 and GPT 4. But right now, it's not that much useful. Like, honestly, it's it's very reliable.
00:30:59 Greg: Cool. That's beautiful. Back in 2022, you said you're never gonna write you're never gonna write another commit message again. My question, are you writing commit messages still or no?
00:31:09 Tibo: Yeah. Because I'm working back with people. At the time, when I wrote that, I think I was I was using that so I was I think I was I was being a a douche, but let me tell you why. When I would when I when I wrote that, I was working with a team of people that I hired. And so I I made them agree to me using AI in my commit message, meaning that every single one of my commit message, it was it was basically me pressing common enter. And when I was pressing common enter, it would auto commit and auto push with an AI generated message.
00:31:58 Greg: Uh-huh.
00:31:59 Tibo: And right now, I'm working with people who are not not not employees. They are partner. Mhmm. And they really didn't like it. And so I had to change and and try to, like, try to be nice with them and Yeah. Write my own commit message.
00:32:20 Greg: Yeah. Yeah. Yeah. For sure. Well, that leans into one of the philosophies I saw that I really resonate that you had mentioned, which was generated by AI, polished by humans. And really, I think that that simple sentence alludes to a whole lot more about where AI fits or should fit into folks workflow, where it's you have it do the first 95% and then humans are on the other end doing the validation to make sure it's good. How does that saying fit into your workflow?
00:32:47 Tibo: That's that's how I call it right now. That's that's how I I program every day. And it's I think it's it's the beauty of a cursor is when I want to create a new feature, I would just tell it what to do, what I want to do, how I want it to work, and it will create the first draft of the components. Mhmm. And if if you set it up correctly, if you have these master prompts, if you if the project is somewhat well structured, it will create decent things. And and, like, I just I just did, like, a few hours ago. And it honestly saved me 1 hour by creating, like, the, like, the bones of the component. And I really didn't have to change, like, few lines.
00:33:39 Tibo: Sure. And I then ship it in production. It worked very well.
00:33:44 Greg: And then just ship it there. Are you using v zero or anything like that for front end components?
00:33:49 Tibo: I'm not, but I think I should. Yeah. Yeah. I I should. It's it's super powerful. And from what I'm seeing on Twitter, it's getting better and better every day. So yeah. I I should read that.
00:34:00 Greg: Well, what's amazing is having seen v zero from the beginning, they weren't investing it that much. I think it blew up. It started going nuts. And now they're throwing like a real team and real money
00:34:12 Tibo: behind it, which is really cool to see. Yeah. Yeah. Then that that's alright. Like, it's it's super valuable. Yeah. I I should I should give it a more, like, a a fair try, I guess.
00:34:19 Greg: Yeah. Yeah. Yeah. Yeah. For sure. We just need to get cursor to import v zero and have it just
00:34:26 Tibo: right before. There's actually, like yeah. I I have this. So I'm not using it, but I have in my project. Mhmm. So here I'm in cursor. And if I want to, like, create a new comp using 0, I would reference my master prompts with 0.
00:34:50 Greg: Oh, cool. Is that prompt public? Have you shared that?
00:34:56 Tibo: Yes. I think I I I got it from Twitter, so it should be
00:34:59 Greg: Oh, cool. Nice. Yeah.
00:35:03 Tibo: I'm gonna select for countries. Poof. And and by doing that, I think it should be able to, like yeah. It it creates a link. And this link, it's it's a v zero link with the prompt embedded in the query. Uh-huh. Oh, maybe it's not working anymore.
00:35:27 Greg: Wow. Oh, either way, I was gonna say, I did not know that, and I like that a
00:35:31 Tibo: lot. Yeah. But maybe the like, it seems like it's not working anymore. So maybe they they removed the support for the the query. Like, they re Interesting.
00:35:42 Greg: Well, call to builders out there. This is needed. We want this. Somebody please go do it. Just a few more questions here as we start to wrap up. What that type right or the the the the chat gbt alternative that you showed me, that was really interesting. What other mini tools like that are you grabbing for?
00:36:01 Tibo: That's a good question. Not not that much actually. I'm I'm using so I'm using TypingMine, the tool that I showed you earlier. I'm using Glip to, like, to handle customer supports, 200 tickets. It it's not that much deep into AI, but it has just the minimum stuff that, is useful to me. And I want to start experimenting with sidegpt. Sidegpt, yes, it's it's the it's a completely new SaaS that is supposed to totally, like, remove the need for a support team. So that's by by, like, by training it with the the correct documents, help center, it should be able to fully fully answer to your users with these unique capabilities that it it can take action.
00:37:10 Tibo: Like, if if a user want to cancel, it will not only show the user how to cancel, but it will actually be able to cancel the subscription for the user. And the way it it it's doing that is pretty simple. Like, it's it's quite easy to set up. And so I really want to dive into that and see if it can just help me save time.
00:37:31 Greg: That's cool. And this is sitegpt.ai. Right? Yes. Cool. Well
00:37:37 Tibo: it's it's quite it's quite old, but the maker completely revamps the software. And it's like it's the new version is completely new and mind blowing.
00:37:48 Greg: That's very cool. Tell me about what else is on your to learn
00:37:56 Tibo: list. Let me check. I have one.
00:37:58 Greg: Oh, I love this. This is so cool.
00:37:59 Tibo: It's very long. It's like it's a mix of things to explore.
00:38:06 Greg: Can you show us too?
00:38:08 Tibo: No. Because it's it's it's, like
00:38:12 Greg: Personal.
00:38:12 Tibo: So I I've I've seen that, like, the the AI provider that we mentioned earlier, it has some new cool models, and I want to explore them. Cool. I'll try to it's it's a very cool provider, and it has some things that I'm already using. But things like eye contact correction, for the spray. It's it's quite powerful. And this is just a way to, like when, like, you take this video where the you see the eyes sing around. And it's able to correct the video and to make the eye very static, always looking at the camera. Yeah. This is the kind of man that's it it might not change the game for me.
00:39:05 Greg: Mhmm.
00:39:05 Tibo: But it's it's still quite useful to improve the quality of some generated video for Revit.
00:39:11 Greg: Yeah. They
00:39:13 Tibo: have a lot that I haven't explored yet, and I really want to dig down.
00:39:17 Greg: Yeah. That's beautiful. You had another tweet that caught my eye. In July of 24, you said, I still haven't found a suitable use case where AI agents could help me for real. Have you found use cases where agents help you for real?
00:39:30 Tibo: No. And I so may maybe maybe it kinda work now with so I haven't I haven't tried with a new sonnet, but there was a few times where I built an entire agent infrastructure, meaning a set of actions, like, this this kind of infinite loop where an agent would try to achieve a goal by splitting this goal into multiple, like, smaller tasks. And the thing is because because those models that we have right now, they are, like, 95% accurate. If you if you like, if you take the those 5 percent errors that you have and you multiply them by the number of steps that the agent will take to achieve the task
00:40:24 Greg: Uh-huh.
00:40:24 Tibo: It's it it makes them unsustainable. Like, it it doesn't work because, like, most of the time, one error in the chain will mess up with the entire thing. And I understand that it's super interesting, but I just I just found that right now, I was not able to plug the correct context and set of actions to generate value. But for sure, it's happening. And probably in 2025, we'll have very interesting stuff related to agent. Like, Sam Atman Sam Atman himself said that's gonna be, like, the the the time for agent is coming. So Yeah. Yeah. It's something to watch, definitely.
00:41:09 Greg: Yeah. That's beautiful. Well, I tell you what, on that one, one question I'd like to ask at the end is you're on Twitter a lot. You see a lot of AI talk. You see a you see a lot of smart people talking about AI. What are the topics that the smart people in AI are talking about right now?
00:41:25 Tibo: I I've seen a lot of things about gaming. It's it was surprising because the I I kind of think that there are tons of people working on text for, like, for, like, 4 years right now. Pretty much the same for images. You have, like, very incredible models with 2 images. Videos is kinda works. Like, we have we have things. Like, we have very good providers. It's expensive and probably is gonna get disrupted. But we already know some of the main players, and they are out there. And gaming, it it it was, like, it was pretty silence for like for since the AI revision started, we haven't seen any good, good games and good game engines being disrupted.
00:42:17 Tibo: And maybe, maybe it's gonna come in 2025. Like, I've seen those demos about, like, Counter Strike and Minecraft being run by a neural net and not a game engine. And I found it super impressive. Like, it's how does it burn my mind? Because thinking that from just a prompt, you could create an entire game without all the game engine that would support it. It's it's such a huge change in and the thing is people playing video games, they already have GPUs. So
00:42:58 Greg: Uh-huh.
00:42:58 Tibo: It could be very easy to, like, to to them. If if it works, you can just get the games on quite powerful PCs from gamers and they could just run it without many things that makes game complicated. Totally. So again, we could see an entire industry being disrupted. Tons of people becoming, like, quite useless. And huge companies like a Red Engine or Unity 3 d be being completely useless.
00:43:29 Greg: Yeah. You know, I love in life when you have kind of, like, a frame breaker moment where you think one way, but then you hear something that's, like, completely different and completely Yeah. Shift your frame. And when I thought of games, I thought of, sure, you're gonna walk into a virtual and the the bartender is gonna have, like, a chatbot behind it and you're gonna have, like, an engaging conversation. Just some like, a chatbot basically in a game. But then when I saw those Minecraft demos, I thought, holy cow. I haven't even thought about this before hand.
00:43:55 Tibo: Exactly. Exactly.
00:43:57 Greg: Just the neural net generating the game. So it's like whole it's like this is different. Wait a minute. There's something interesting going on here.
00:44:04 Tibo: Yeah. We are so we're so like, we were thinking about how text based add ins would disrupt video games and it's really not happening that way.
00:44:14 Greg: Yeah. Yeah. Yeah. Yeah. It's always fun in life when that happens. Well, I tell you what, I wanna end on that one. Tibo, thank you very much for the conversation today.
00:44:21 Tibo: Thank you, Greg. It was awesome.