We are announcing the second edition of our Latent Space demo day event in SF on 2/23: Final Frontiers, a startup and research competition in “The Autonomous Workforce”, ”Beyond Transformers & GPUs”, and “Embodied AI”.
RSVP here! The first one was aimed for 15-20 people and ended up blowing up to >200 and covered in the Information - let’s see what a year of growth (and competition) does to the local events space in 2024.
You can find all Latent Space events here, and of course get in touch with us to host your own AI Engineer meetups like AI Engineering Singapore.
In our December 2023 recap we covered the Four Wars of the AI stack. But how do we know when it’s time to crown a winner? As we kick off 2024, we wanted to do a recap of the State of AI in 2023 to set a baseline of adoption for different products. Retool had a great report at the end of last year which covered a lot of it.
David Hsu, CEO and co-founder of Retool, joined us to go over it together. We also talked about the history of Retool, why they were too embarrassed to present at YC demo day, and how they got to $1M ARR with 3 employees. If you’re a founder, there are a lot of nuggets of advice in here!
Retool AI
In our modeling of the “Software 3.0 Stack”, we have generally left a pretty wide open gap as to the “user interface” equivalent of the AI stack:
Retool AI launched 4 months ago with some nifty features for SQL generation, and its own hosted vector storage service (using pgvector). However, as he explains on the pod, the more interesting potential of Retool is in helping developers build AI infused applications quickly, in combination with its Workflows feature.
This moves Retool down the stack from just the UI for internal tooling to the business logic “piping” as well. There are a bunch of dedicated tools in this space like Respell, BuildShip, Flowise, and Ironclad Rivet.
"We think that practically every internal app is going to be AI infused over the next three years." - David on the pod
RIP StackOverflow?
In July 2023 we talked about the impact of ChatGPT and Copilot:
This was then disputed by StackOverflow, who pointed out (very fairly so) that there were privacy-related changes in their analytics instrumentation in 2022. StackOverflow no longer reports traffic, but based on StackOverflow’s continuing transparency we can see that organic declines have continued throughout 2023.
Retool’s report comes over a year after those changes and has some self reported samples from users:
* 57.6% of people said they have used StackOverflow less; almost all of them replaced it with ChatGPT and Copilot.
* 10.2% said they no longer use StackOverflow.
We also saw a lot more tools being released in the dev tools space such as (one of our oldest pod friends) Codeium (which just raised a $65M Series B), SourceGraph (and their newly released Cody), Codium AI (just released AlphaCodium which was picked up by Karpathy), Phind (which beat GPT-4 with OSS models), and Cursor, one of the most beloved products in the dev community at the moment. Intelligence is getting closer and closer to the IDE, and the trend doesn’t seem to be reverting.
We already said that “You are not too old (to pivot into AI)“, and the advice still stands. When asked to rate “Preference for hiring engineers effective at using ChatGPT/Copilot for coding” on a scale of 1 to 10, where 10 is “Much more likely”, ~40% of companies voted 8-10. Having an AI Engineer skillset is extremely important. 45% of companies between 1,000-4,999 employees said that they increased the difficulty of technical interviews to compensate for these new tools, so the gap between users and non-users will keep widening.
Crossing the AI in Production Chasm
Geoffrey Moore’s “Crossing the Chasm” is one of the most quoted business frameworks. Every market has an initial group of Innovators and Early Adopters, who are willing to suffer through the rough edges of products initially, and eventually crosses into the Early Majority, which expects a full product.
In the AI world, ChatGPT and Midjourney / DALL-E have crossed the chasm in the consumer space. Copilot is probably the only tool that did it in the enterprise, having crossed 1M paid users. ~$50B were invested in AI in 2023, and we still only have <5 breakout products; expect this number to rise in 2024. According to the survey, only 25% of companies had real production usage, but 77.1% said their company is making efforts to adopt more. Closing that gap could triple AI adoption in one year.
The report also broke down adoption by use case. 66% of companies use it internally, while only 43% do so in customer-facing use cases. Internal usage of AI is much more varied than customer-facing one as well:
One point that David made in the podcast is that this number isn’t a knock on AI as a tool, but rather about the demographics of businesses outside of our Silicon Valley bubble:
We all work in Silicon Valley, right? We all work at businesses, basically, that sell software as a business. And that's why all the software engineers that we hire basically work on external facing software, which makes sense with most software companies. But if you look at most companies in the world, most companies in the world are actually not software companies. […] Most of the [work of] software engineers in the world actually goes towards these internal facing applications.
Beyond code models, it’s clear that the big winners of the first wave of AI adoption are vector stores and RAG. Knowledge base Q&A, customer chatbots, recommendation systems, etc are all based on them. Retool even rolled out their own with Retool Vectors. Expect the battlefield to get even hotter in these areas, with Mongo and Chroma leading the charge on a NPS/popularity basis.
It’s also clear that OpenAI won the first campaign in the AI models war, by far. Hopefully Mistral and LLaMA3 will shake up this chart when we look back at it in 2025:
TLDR: We’re really early. If you want to build in AI, there’s a ton of work to be done, and a lot of problems to be solved. You can find the full report here to dive through all the numbers.
Video podcast
Watch along on our snazzy YouTube!
Show Notes
Companies and Projects:
* Retool
* Raising less money at lower valuations
* Paul Graham's "playing house" essay
Timestamps
* [00:00:00] Introduction
* [00:02:43] Retool's founding story and decision not to present at YC demo day initially
* [00:09:08] Philosophy on fundraising - raising less money at lower valuations
* [00:12:53] Overview of what Retool is
* [00:15:41] Origin story of Retool AI product
* [00:19:59] Decision to use open source vector database PG Vector
* [00:21:29] Most underrated AI use cases
* [00:25:56] Retool's AI UX and workflows
* [00:30:38] Zapier vs Retool
* [00:32:54] Updates from Retool's 2023 State of AI survey
* [00:35:21] Who is adopting AI first?
* [00:37:40] Evolving engineering hiring practices in the age of Copilot/ChatGPT
* [00:40:02] Retool's views on internal vs external AI adoption
* [00:41:50] OSS models vs OpenAI in production
* [00:44:46] Additional survey questions to ask in 2024
* [00:47:04] Balancing enterprise sales vs bottom-up adoption
* [00:51:54] Philosophical thoughts on AGI and intentionality
Transcript
Alessio [00:00:00]: Hey everyone, welcome to the Latent Space podcast. This is Alessio, partner and CTO in Residence at Decibel Partners, and I'm joined by my co-host Swyx, founder of Smol.ai.
Swyx [00:00:16]: And today we are in the studio with David Hsu from Retool. Welcome.
David [00:00:20]: Thanks. Excited to be here.
Swyx [00:00:23]: We like to give a little bit of intro from what little we can get about you and then have you talk about something personal. You got your degree in philosophy and CS from Oxford. I wasn't aware that they did double degrees. Is that what you got?
David [00:00:35]: It's actually a single degree, which is really cool. So basically you study content, you study philosophy, and you study intersection. Intersection is basically AI, actually, and sort of computers think, or computers be smart. What does it mean for a computer to be smart? As well as logic. It's also another intersection, which is really fun too.
Swyx [00:00:51]: In Stanford, it might be symbolic systems or whatever. It's always hard to classify these things when we don't really have a word for it. Now I guess everything's just called AI. Five years ago, you launched Retool. You were in YC at winter 17 and just been a straight line up from there, right?
David [00:01:09]: I wish.
Swyx [00:01:10]: What's something on your LinkedIn that people should know about you? Maybe on their personal hobby or, you know, let's just say something you're very passionate about.
David [00:01:17]: Yeah, sure. I read quite a bit. I probably read like two books a week around about. So it's a lot of fun. I love biking. It's also quite a bit of fun. So yeah.
Swyx [00:01:25]: Do you use Retool to read?
David [00:01:27]: No, I don't use Retool to read. No, that'd be funny.
Swyx [00:01:30]: What do you read? How do you choose what you read? Any recommendations?
David [00:01:35]: I'm mostly reading fiction nowadays. So fiction is a lot of fun. I think it helps me be more empathetic, if you will. I think it's a lot of fun. I actually just want to see what it's like to be in someone else's shoes. That's what I really good about philosophy as well. I find philosophy just so interesting, especially logic. We can talk more about that for probably hours if you want.
Swyx [00:01:50]: So yeah, I have a casual interest in epistemology. And I think that any time you, you know, you're trying to figure out a way to solve a problem, you're going to have to figure out a way to solve it.
David [00:02:05]: Yeah, totally. What does it mean to know?
Alessio [00:02:13]: That's its own podcast. We should do a special edition about it. That's fun. Let's maybe jump through a couple of things on Retool that I found out while researching your background. So you did YC, but you didn't present a demo day initially because you were too embarrassed of what you had built. Can you maybe give any learnings to like founders or people who are built? I've seen a lot of people kind of give up early on because they were like, oh, this isn't really what I thought it was going to be to be a founder. They told me I would go to YC and then present and then raise a bunch of money and then everything was going to be easy. So how did that influence also how you build Retool today in terms of picking ideas and deciding when to give up on it?
David [00:03:30]: Yeah. Let's see. So this is around about 2017 or so. So we were supposed to present at the March demo day, but then we basically felt like we had nothing really going on. We had no traction, we had no customers. And so we're like, okay, well, why don't we take six months to go find all that before presenting? Part of that, to be honest, was I think there's a lot of noise around demo day, around startups in general, especially because there's so many startups nowadays. And I guess for me, I'd always wanted to sort of under-promise and over-deliver, if you will. And then demo day, I mean, maybe you two have seen a lot of videos. It's a lot of, honestly, over-promising and under-delivering because every startup says, oh, I'm going to be the next Google or something. And then you peer under it and you're like, wow, nothing's going on here, basically. So I really didn't want that. And so we chose actually not to present on demo day, mostly because we felt like we didn't have anything substantial underneath. Although actually a few other founders in our batch probably would have chosen to present in that situation, but we were just kind of embarrassed about it. And so we basically took six months to just say, okay, well, how do we get customers? And we're not presenting until we have a product that we're proud of and customers that we're proud of. And fortunately, it worked out. Six months later, we did have that. So I don't know if there's much to learn from the situation besides I think social validation was something that I personally had never really been that interested in. And so it was definitely hard because it's hard to sort of, it's almost like you go to college and all your friends are graduating, but you failed or something, you failed the final and you have to like redo it here. It's like, well, it kind of sucks that all your friends are up there and on the podium presenting and they are raising a ton of money and you're kind of being left behind. But in our case, we felt like it was a choice. We could have presented if we really wanted to, but we would not have been proud of the outcome or proud of what we were presenting. And for us, it was more important to be true to ourselves, if you will, and show something that we're actually proud of rather than just raise some money and then shut the company down in two years.
Alessio [00:04:45]: Any Sam Altman stories from the YC days? Could you tell in 2017 that Sam was going to become, like, run the biggest AI company in the world?
David [00:04:49]: Wow. No one's asked me that before. Let me think. Sam was, I think he was, I want to, I forgot, I think maybe president of YC in our batch. We actually weren't in his group actually at the very beginning. And then we got moved to a different group. I think Sam was clearly very ambitious when we first met him. I think he was very helpful and sort of wanted to help founders. But besides that, I mean, I think we were so overwhelmed by the fact that we had to go build a startup that we were not, you know, honestly paying too much attention to everyone else's partner taking notes on them.
Alessio [00:05:20]: That makes sense. Well, and then just to wrap some of the Retool history nuggets, you raised a series A when you were at 1 million in revenue with only three or four people. How did you make that happen? Any learnings on keeping teams small? I think there's a lot of overhiring we've seen over the last few years. I think a lot of AI startups now are raising very large rounds and maybe don't know what to do with the capital.
David [00:05:42]: So this is kind of similar, actually, from sort of why we choose not to demo day. And the reason was, it feels like a lot of people are really playing startup. I think PG has an essay about this, which is like, you're almost like playing house or something like that. Like, it's like, Oh, well, I hear that in a startup, you're supposed to raise money and then hire people. And so therefore you go and do that. And you're supposed to, you know, do a lot of PR, because that's what, you know, startup founders do. And so you could do a lot of PR and stuff like that. And for us, we always thought that the point of starting a startup is basically you have to create value for customers. If you're not creating value for customers, like everything else is going to, nothing's going to work. Basically, you can't, you know, continuously raise money or hire people if you don't have customers. And so for us, we were always very focused on that. And so that's initially where we started. I think it's, again, maybe goes to like the sort of presenting something truthful about yourself or staying true to yourself is something to that effect, which is we didn't want to pretend like we had a thriving business. And so the only way to not pretend was actually to build a thriving business. And so we basically just, you know, put our heads down and, you know, grinded away for probably a year, year and a half or so, just writing code, talking to customers. And I think that at that point we had raised something like maybe a million dollars, maybe a million and a half, something out of YC. So I mean, to us, to people, you know, that was a huge amount of money. I was like, wow, like, how are we ever going to spend a million and a half? The runway was like, you know, five, six years at that point, right? Because we're paying ourselves 30, 40K a year. And so then the question was not like, oh, we're going to run on the runways. The question was like, we better find traction because if we don't find traction, we're going to, you know, just give up psychologically. Because if you run an idea for four years and nothing happens, you're probably psychologically going to give up. And I think that's actually true in most startups, actually. It's like most startups die in the early stages, not because I run out of money, but really because you run out of motivation. And for us, had we hired people, I think it would have actually been harder for us because we want to run out of motivation faster. Because when you're pre-product market fit, actually, trying to lead the team of like, you know, 10 people, for example, to Marshall's product market fit, I think it's actually pretty hard. Like it's, you know, every day people are asking you, so why are we doing this? And you're like, I don't know, man, like, hey, trust this. That's actually a very tiring environment to be in. Whereas this is just like, you know, the founders figuring out product market fit, I think it's actually a much sort of safer path, if you will. You're also schooling less with employees, like when you hire employees, you have an idea you're trying to work with your customers. That's actually, I think, a lot more stable of a place for employees to join as well.
Swyx [00:08:00]: Yeah. I find that typically the sort of founder employee relationship is, employee expects the founder to just tell them what to do, and you don't really get critical pushback from the employee, even if they're a body and even if they like you as an early engineer. It's very much like the role play of like, once you have that founder hat on, you think differently, you act differently, and you're more scrappy, I guess. In trying to figure out what that product is. Yeah, I really resonate with this, because I'm going through this right now.
David [00:08:26]: Awesome. One thing we did actually early on that I think has paid a lot of dividends, especially your rituals a lot larger now is we hired a lot of former founders. So I want to say like, when we were 20, 30, 40 people, we were probably like half former founders at each one of those stages. And that was actually pretty cool, because I think you infuse sort of a, you know, get things done kind of culture, a outcome oriented culture of like a very little politics, because you know, no one came from larger companies, everyone was just like, this is my own startup, let me go figure out how to achieve the best outcome for the customer. And so I think from a cultural perspective, even today, a lot of rituals culture is sort of very self startery. I think it's actually because of sort of these like, you know, early founders that we hired, which was really, really, you know, we're really lucky to have had them. Yeah.
Swyx [00:09:08]: And then closing off on just a little bit of the fundraising stuff, something notable that you did was when in 2021, when it was sort of peak Zerp, and everyone was raising hundreds and hundreds of millions of dollars, you intentionally raised less money at lower valuations as your title. And I think it's a testament to your just overall general philosophy and building retool that you're just very efficient and you do things from first principles. Any updates on like, would you still endorse that? You know, would you recommend that to everyone else? What are your feelings sort of two years on from that?
David [00:09:38]: Yeah. I think exactly you said is correct, where we raise less money at a lower valuation. And I think the funny thing about this is that when we first announced that, even, you know, internally and both externally, I think people were really surprised, actually, because I think Silicon Valley has been conditioned to think, well, raising a giant sum of money at a giant valuation is a really good thing. So like, you know, you should maximize both the numbers, basically. But actually maximizing both the numbers is actually really bad, actually, for the people that matter the most, you know, i.e. your employees or your team. And the reason for that is raising more money means more dilution. So if you look at, you know, a company like, let's say Uber, for example, if you join Uber at like, I don't know, like a $10 billion valuation, or let's say join for a huge route, which I think happened at a few billion dollars in valuation, you actually got diluted a ton when Uber fund raises. So if Uber raises, if Uber dilutes themselves by 10%, for example, let's say it raised $5.25 billion, for example, I think employees' stake goes down by 10% in terms of ownership. Same with, you know, previous investors, same with the founders, etc. And so, if you look at actually a lot of founders in sort of, you know, the operations statistics space, or, you know, those that fundraise, like, you know, 2013, 2017, a lot of the founders by IPO only have a few percentage points, actually, for a company. And if founders only have a few percentage points, you can imagine how, you know, how little employees have. And so, that I think is actually still really, you know, bad thing for employees overall. Secondly, sort of higher valuation, given the same company quality is always worse. So basically, what that means is, if you are fundraising as a company, you could commit a certain valuation in the market, you know, let's say it's, you know, x. For example, maybe if you're lucky, and you can raise two times x, for example. But if you choose two times x, your company itself is not fundamentally changed. It's just that, you know, for some reason, investors want to pay more for it. You know, maybe today you're an AI company, for example. And so, investors are really excited about AI and want to pay more for it. However, that might not be true in a year or two years time, actually. And if that's not true in two years time, then you're in big trouble, actually. And so, now, I think you see a lot of companies that are raising really high valuations about 2021. And now, they're like, man, we're at like 100x, or, you know, we raised 300x multiple, for example. And if we're at 300x, then, you know, maybe now we're at like 200x, man, we just can't raise money ever again. Like, you know, we have to grow like 50x to go raise money, reasonable valuation, I would say. And so, I think that is really challenging and really demotivating for the team. And so, I think a lower valuation actually is much better. And so, for us, in retrospect, you know, to answer your question, two years later, we did not predict, you know, the crash, if you will. But given that, I think we've done extremely well, mostly because our valuation is not sky high. If our valuation were sky high, I think we'd have a lot more problems. We'd probably have recruiting problems, for example, and probably have a lot of internal morale problems, etc. A lot of people would be like, you know, why is the valuation this way? We might have cash flow problems because we might have to go raise money again, you know, etc. But we can't because the valuation is too high. So, I would urge, I think, founders today to, quote unquote, like, leave money on the table. Like, there are some things that are not really worth optimizing. I think you should optimize for the quality of the company that you build, not like the valuation, you raise that or the amount you raise, etc. So,
Swyx [00:12:34]: Highlight 2020, but it looks like, you know, you made the right call there anyway. Maybe we should also, for people who are not clued into Retool, do a quick, like, what is Retool? You know, I see you as the kings or the inventors of the low-code internal tooling category. Would you agree with that statement? You know, how do you usually explain Retool?
David [00:12:53]: I generally say it's like Legos for code. We actually hate the low-code moniker. In fact, we have docs saying we will never use it internally, or even to customers. And the reason for that is, I think, low-code sounds very not developer-y. And developers, they hear the phrase low-code, they're like, oh, that's not for me. I love writing code. Like, why would I ever want to write less code? And so, for us, Retool is actually built for developers, like, 95% of our customers actually are developers, actually. And so, that is a little bit surprising to people. I'll generally explain it as, and this is kind of a funny joke too, I think part of the reason why Retool has been successful is that developers hate building internal tools. And you can probably see why. I mean, if you're a developer, you've probably built internal tools yourself, like, it's not a super exciting thing to do, you know, it's like piecing together a CRUD UI, you've probably, you know, pieced together many CRUD UIs in your life before, and there's a lot of grunt work involved. You know, it's like, hey, state management, it's like, you know, data validation, it's like displaying error messages, it's like the bouncing buttons, like, all these things are not really exciting. But you have to do it, because it's so important for your business to have high quality internal software. And so what Retool does is basically allows you to sort of, really fast, whether it's a front end, whether it's a back end or whatever else. So yeah, that's what Retool is.
Swyx [00:14:02]: Yeah, actually, if you started hiring, and so I do a lot of developer relations and community building work, and you hired Krithika, who is now, who's also an AI, to start out your sort of DevRel function. And I was like, what is Retool doing courting developers? And then she told me about this, you know, developer traction. And I think that is the first thing that people should know is, which is that the burden and the weight of internal tooling often falls to developers, or it's an Excel sheet somewhere or whatever. But yeah, you guys have basically created this market, you know, in my mind, I don't know if there was someone clearly before you in this, but you know, you've clearly taken over and dominated. Every month, there's a new YC startup launching with that it's like, you know, we're the open source Retool, we're like the lower code Retool, whatever. And it's pretty, I guess it's endearing, you know, we'll talk about Airplane later on. But yeah, I think I've actually used Retool, you know, in my previous startups for this exact purpose. Like, we needed a UI for AWS RDS that they can, you know, like the rest of our non less technical people, like our sales operations, people could could interact with and yeah, Retool is perfect for that.
David [00:15:04]: Yeah, it's a good example of like, that's an application that an engineer probably does not want to build, like building an app on top of Salesforce or something that is not exciting. And so it sucks. It's very limited. It's like not a fun experience at all. But piecing together a Retool is quite a bit easier. So yeah, let me know if you have any feedback, but also, yeah, no, of course, like more recently,
Swyx [00:15:23]: I think about three, four months ago, you launched Retool AI, obviously, AI has been sort of in the air. I'd love for you to tell the journey of AI products ideation within Retool. Given that you have a degree in this thing, I'm sure you're not new to this, but like, when would you consider sort of this the start of the AI product thinking in Retool?
David [00:15:41]: So we actually had a joke internally at Retool. We are part of roadmap for every year, it was like 2019 or something. We had this joke, which was like, what are we going to build this year? We're going to build AI programming, is what we always said as a joke. And so, but it was funny, because we were like, that's never gonna happen. But like, let's add it because it's like a buzzword thing that enterprises love. So let's look at it. And so it was almost like a funny thing, basically. But it turns out, you know, we're actually building that now. So this is pretty cool. So I would say maybe AI thinking on Retool probably first started maybe like, I don't know, a year and a half ago, something like that. And when we first started thinking about it, sort of in a philosophical way, if you will, it's like, well, what is the purpose of AI? And how can it help, you know, what Retool does? And there were two main prongs, if you will, value. One was helping people build apps faster. And so you've probably seen Copilot, you've seen sort of so many other coding assistants, P0 to them, you know, stuff like that. So that's interesting, because, you know, engineers, as we talked about, do some grunt work. And grunt work, you know, maybe could be automated by AI was sort of the idea. And it's interesting. So we actually, I would say, kind of proved or disproved the hypothesis a little bit. If you talk to most engineers today, like a lot of engineers do use Copilot. But if you ask them, like, how much time does Copilot save you? It's not like coding is 10x faster than before, you know, coding is maybe like 10% faster, maybe 20% faster, or something like that, basically. And so it's not like a huge step change, actually. And the reason for that, as we think, is because the sort of fundamental frameworks and languages have not changed. And so if you're building, let's say, you know, like the sales ops tool we're talking about before, for example, let's say you've got AI to generate you a first version of that, for example, the problem is that it probably generated it for you in like JavaScript, because you're writing for the web browser, for example, right. And then for you to actually go proofread that JavaScript, for you to go read the JavaScript to make sure it's working, you know, to fix the subtle bugs that AI might have caused, hallucinations, stuff like that, actually takes a long time and a lot of work. And so for us, the problem is actually not like the process of coding itself, it is more sort of the language or the framework, I think it's like way too low level. It's kind of like anything like punched cards, like, let's say, back in the day, who designed punched cards, and AI could help you generate punched cards, okay, you know, I guess that helps me punch the cards a little bit faster, because I have a machine punching them for me. But like, when there's a bug, I still have to go read all the punched cards and figure out what's wrong, right? It's like, it's a lot of work, actually. And so, for us, that was the sort of initial idea was, can we help engineers code faster? You know, I think it's somewhat helpful, to be clear, like, again, I think it's 10 or 20%. So we have things like, you know, you can generate school careers by AI, you can generate UIs by AI, and stuff like that. So that's cool, to be clear. But it's not, I think, the step change, that I think is, you know, the, we're investing somewhat in that. But the bulk of investment, actually, is a number two, which is helping developers build AI enabled applications faster. And the reason why we think this is so exciting is we think that practically every app, every internal app, especially, is going to be AI infused over the next, like, three years. And so every tool you might imagine, so like the tool you were mentioning, like a sales operations tool, for example, probably, you know, if you were to build today, one of the corporate subform of AI. And so, you know, we see today, like, for us, like, a lot of people build, you know, I'll say sales manager tools, or retool. An example is there's a fortune, like a company is building like, sales forecasting tools. So they basically have salespeople enter their forecast, you know, for the quarter, the beginning of the quarter, like, hey, I have these deals. And these deals are going to close, these deals are not going to close, you know, I think I'm upsiding these, downsiding these, stuff like that, basically. So you can imagine it's pulling in deals from your Salesforce database. And so it pulls in the deals that actually use AI to compute like, okay, well, you know, given previous deal dynamics, like, these are the deals that are more likely to close this month versus next month was this quarter, next quarter, etc. And so it could actually, you know, pre write you a draft of, you know, your report, basically. And so that's an example where I think all apps, whether it's, you know, a sales app, you know, until it looks like fraud app, a, you know, fintech app, you know, whatever it is, basically, especially internal apps, I think, like you said, Alessio, in order to make you more productive, it's going to incorporate some form of AI. So the other question is, can we help them incorporate AI faster? So that's why we launched like a vector database, for example, built directly into retool. That's why we know launches all these AI actions, you don't have to go figure out what the best model is and do testing and stuff like that, which gives you out of the box. So for us, I think that is really the really exciting futures. Can we make every app and also retools use AI a little bit and make people more productive?
Alessio [00:19:59]: So for Wang, who's the co founder and chief architect of amplitude, he mentioned that you just use Postgres vector. When you were building retool vectors, how do you think about, yeah, leveraging a startup to do it, putting vectors into one of the existing data stores that you already had? I think like, you're really a quite large customer scale. So like, you're maybe not trying to get too cute with it. Any learnings and tips from that?
David [00:20:23]: Yeah, I think a general philosophical thing I think we believe is, um, we think the open source movement in AI, especially when it comes to all the supporting infrastructure is going to win. And the reason for that is we look at like developer tools in general, especially for such a fast moving space. In the end, like, there are really smart people in the world that have really good ideas, and are going to go build companies and they're going to go build projects basically around these ideas. And so for us, we have always wanted to partner with maybe more open source providers or projects, you could say, like PG factory, for example. And the reason for that is it's easy for us to see what's going on under the hood. A lot of this stuff is moving very fast. Oftentimes, there are bugs, actually. And so we can go look and fix bugs ourselves and contribute back to them, for example. But we really think open source is going to win in this space. It's hard to say about models. I don't know about models necessarily, because it's going to be pretty complicated there. But when it comes to tooling, for sure, I think there's just like so much, there's an explosion of creativity, if you will. And I think betting on any one commercial company is pretty risky. But betting on the open source sort of community and the open source contributors, I think is a pretty good bet. So that's why we decided to get at consumer games. Awesome.
Alessio [00:21:29]: So we're going to jump into the survey next, but we're going to put a bunch of links in the show notes about Result AI and whatnot. Is there any most underrated feature, like something that customers maybe love that you didn't expect them to really care about? I know you have a like text to SQL, you have UI generation, there's like so many things in there. Yeah. What surprised you?
David [00:21:49]: Yeah. So what's really cool, and this is my sense of the AI space overall, you know, if you're on YouTube as well, is that, especially in Silicon Valley, where a lot of the innovation is happening, I think there's actually not that many AI use cases, to be honest. And AI to me, even as of January 19th of 2024, still feels like in search of truly good use cases. And what's really interesting, though, about Retool, and I think we're in a really fortunate position, is that we have this large base of sort of customers, and a lot of these customers are actually much more legacy, if you will, customers. And a lot of them actually have a lot of use cases for AI. And so to us, I think we're almost in like a really perfect or unique spot, we're able to adopt some of these technologies and provide them to some of these like older players. So one example that actually really shocked and surprised me about AI was, so we have this one clothing manufacturer, I think it's either the first or second largest clothing manufacturer in the world, who's using Retool. They're a ginormous company, very multinational, stores on pretty every mall in the world. And so they have one problem, which is, they need to design styles every year, for the next year, basically, for every season. So like, hey, just like summer 2024, for example, and we're going to design. And so what they used to do before is they were hiring designers, and designers would go to study data, they'd be like, okay, well, it looks like, you know, big floral patterns are really hot. And like, you know, California, for example, in 2023, and like, do I think it's going to be hot in 2024? Well, let me think about it. I don't know. Maybe. And if so, if I believe it's going to be hot, let me go design some floral patterns, actually. And what they ended up doing in Retool, actually, is they actually automated a lot of this process away in Retool. So they actually now built a Retool app that allows actually a non-designer, so like an analyst, if you will, to analyze like, you know, who are the hottest selling patterns, you know, particular geos, like this was really hot in Brazil, this is really hot in China, it's really hot, you know, somewhere else, basically. And then they actually feed it into an AI. And the AI, you know, actually generates with Dolly and other image generation APIs, actually generates patterns for them. And they print the patterns, which is really cool. And so that's an example of like, honestly, a use case I would have never thought about, like thinking about like, you know, how clothing manufacturers create their next line of clothing, you know, for the next season, like, I don't know, I never thought about it, to be honest, nor did I ever think, you know, how it would actually happen. And the fact that they're able to leverage AI and actually, you know, leverage multiple things in Retool to make that happen, is really, really, really cool. So that's an example where I think if you go deeper into sort of, if you go outside the Silicon Valley, there are actually a lot of use cases for AI. But a lot is not obvious, like you have to get into the businesses themselves. And so I think we're, we personally are in a really fortunate place. But if you know, you're working in the space and want to find some use cases, please come talk to us like, you know, we're really excited about marrying sort of technology with use cases, which I think is actually really hard to do right now.
Swyx [00:24:38]: Yeah, you know, I have a bunch of like, sort of standing presentations around like, how this industry is developing. And like, I think the foundation model layer is understood. The chain vector DB rag layer is understood, I always have a big question mark and actually have you and Vercel V0 in that box, which is like sort of the UI layer for AI. And like, you know, you are perfectly placed to expose those functionalities to end users, you personally don't really know what they're going to use it for. And sometimes they'll surprise you with their creativity. One segment of this, I do see some startups springing up to do this is related to the things that to something that you've you also build, but it's not strictly AI related, which is retool workflows, which is the sort of canvassy boxes and arrows point and click do this then do that type of thing like which which every what are we calling low code? Every internal tooling company eventually builds, you know, I worked at a sort of workflow orchestration company before, and we were also discussing internally how to make that happen. But you are you're obviously very well positioned to it to that. Yeah, basically, like, do you think that there is an overlap between retool workflows and AI? I think that, you know, there's there's a lot of interest in sort of chaining AI steps
David [00:25:55]: together.
Swyx [00:25:56]: I couldn't tell if like that is already enabled within retool workflows, I don't think so. But you could you could sort of hook them together as kind of jankily, like, what's the interest there? You know, is it all of a kind, ultimately, in your mind?
David [00:26:07]: It is 100% on time. And yes, you can actually already saw a lot of people actually are building AI workflows down retool, which is what we're gonna talk about in a second. But a hot take here is actually, I think a lot of the utility in AI today, I would probably argue 60 70% of the utility, like, you know, businesses have found an AI is mostly via chat GPT, and across the board. And the reason for that is, I mean, the chat GPT is sort of a UI, you could say, or interface and user experience is just really quite good, you know, you can sort of converse, you know, with an AI, basically. But that said, there are downsides to it. If you talk to like a giant company, like a J.P. Morgan Chase, you know, for example, they may be reticent to have people copy paste data into chat GPT, for example, even on chat GPT Enterprise, for example. Some problems are that I think chat is good for one off tasks. So if you're like, hey, I want a first version of representation or something like that, you know, and help me write this first version of a doc or something like that, chat is great for that. It's a great, you know, very portable, you know, if you will form factor, so you can do that. However, if you think about it, you think about some economic productivity, more generally, like chat, again, will help you like 10 or 20%. But it's unlikely that you're going to replace an employee with chat, you know, you're not gonna be like, oh, I'm a relationship manager at J.P. Morgan Chase, and I've replaced them with an AI chatbot. It's kind of hard to imagine, right, because, like, the employees are doing a lot of things besides, you know, just, you know, generating, you know, maybe another way of putting it is like, chat is like a reactive interface, like, it's like, when you have an issue, you will go reach out to chat and chatbot solve it. But like, chatbot is not going to solve 100% of your problems, it'll solve like, you know, 25% of your problems, like, pretty quickly, right. And so what we think the next like, big breakthrough in AI is, is actually like automation. It's not just like, oh, I have a problem, let me go to a chatbot and solve it. Because like, again, like, people don't spend 40 hours a week in a chatbot, they spend like two hours a week in a chatbot, for example. And so what we think can be really big, actually, is you're able to automate entire processes via AI. Because then you're really realizing the potential of AI, it's like, not, it's not just like, you know, a human copy pasting data into an AI chatbot, you know, pasting it back out or copying back out. Instead, it's like the whole process now was actually done in an automated fashion without the human. And that, I think, is what's going to really unlock sort of big canonical productivity, or that's what I'm really excited about. And I think part of the problem right now is, I'm sure you all thought a lot about agents is that the agents are actually quite hard. Because like, you know, the AI is wrong, like, you know, 2% of the time, but then you like, you know, a score, if you let's say, you know, raise to the power seven, for example, that's actually wrong, you know, quite often, for example. And so what we've actually done with workflows is we prefer, we've learned, actually, is that we don't want to generate the whole workflow for you by AI. Instead, what we want you to do, actually, is we want you to actually sort of drag and drop the workflow yourself. Maybe you can get a vSphere or something by AI, but it's coded, basically, you should actually be able to modify the steps yourself. But every step can use AI. And so what that means is like, it's not the whole workflow is created by AI, every step is AI automated. And so if you go back to, for example, like the users are talking about, you know, with a clothing manufacturer, that's actually a workflow, actually. So basically, what they say is, hey, every day, we see all the data, you know, from our sales systems into our database. And then we do some data analysis, and, you know, it's just raw SQL, basically, it's nothing too surprising. And then they use AI to generate new ideas. And then the analysts will look at the new ideas and approve or reject them, basically. And that is like a, you know, that's true automation. You know, it's not just like, you know, a designer, copy pasting things as a chat, you can be like, hey, you know, give me a design. It's actually designs are being generated and generated 10,000 designs every day. And then you have to go and approve or reject these designs, which I think is a lot, you know, that's a lot more economically productive than just copy pasting something. So we think sort of the AI workflow space is a really exciting space. And I think that is the next step in sort of delivering a lot of business value by AI. I personally don't think it's, you know, AI chat or AI agents quite yet, so.
Swyx [00:29:50]: That's a pretty reasonable take. It's disconcerting, because like, I know a lot of people trying to build what you already have in workflows. So you have that sort of, you're the incumbent sort of in their minds, I'm sure it doesn't feel that way to you. But like, I'm sure, you know, you're the incumbent in their minds, and they're like, okay, like how do I, you know, compete with retool or, you know, differentiate from retool. As you mentioned, you know, all these connections, it does remind me that you're running up against Zapier, you're running up against maybe Notion in the distant future. And yeah, I think that there'll be a lot of different takes at this space and like whoever is best positioned to serve their customer in the way that they need to shape is going to win. Do you have a philosophy against around like what you won't build, like what do you prefer to partner and not build in-house? Because I feel like you build a lot in-house.
David [00:30:38]: Yes, there's probably two philosophical things. So one is that we're developer first. And I think that's actually one big differentiator between us and Zapier and Notion, and we're so very rare we'll see them actually, and the reason is we're developer first. Because developers, like, if you're like building a sales ops tool, you're probably not considering Notion if you're a developer, you're probably like, I want to build this via React, basically, or use retool. And so are you we build for developers, it's pretty interesting, actually, I think one huge advantage of some of the developers is that developers don't want to be given an end solution. They want to be given the building blocks liquid to themselves to build the end solution. And so for us, like, interesting point that equilibrium we don't get to, it's basically to say, hey, retool is a consulting company, and we basically build apps for everybody, for example. And what's interesting is, we've actually never gotten to that equilibrium. And the reason for that is for some of the developers, developers don't want, you know, like a consultant coming in and building all the apps for them. Developers like, hey, I want to do it myself, just give me the building blocks, give me the best table library, give me, you know, good state management, give me an easy way to query the rest of the API. So I'll do it myself, basically. So that is pretty, so we generally end up basically always building building blocks that are reusable by multiple customers. We have, I think, basically never built anything specific for one customer. So that's one thing that's interesting. The second thing is when it comes to sort of, you know, let's say like, in the AI space, we're going to build and we're not going to build, we basically think about whether it's all core competency or whether there are unique advantages to us building it or not. And so we think about the workflows product, we think workflows actually is a pretty core competency for us. And I think the idea that we can build a developer first workflows automation engine, I mean, I think after we released, you know, workflows, virtual workflows, there have been a sort of few copycats that are, I think, quite, quite far behind, actually, they sort of are missing a lot of more critical features. But like, if you look at the space, it's like, Zapier on one side, and then maybe like, Airflow on the other. And so virtual workflows actually is fairly differentiated. And so we're like, okay, we should go build that. This is the one I was going to build, so I'm just going to build it. Whereas if you look at like vectors, for example, you look at vectors like, wow, there's a pretty thriving space already, if you know vector databases. Does it make sense for us to go build our own? Like, what's the benefit? Like, not much, we should go partner with or go find technology off the shelf. Narcissus is pretty effective. And so for us, I think it's like, how much value does that for customers? Do we have a different take on the space? Do we not? And every product that we've launched, we've had a different take on the space and the products that we don't have a different take, we just adopt what's off the shelf.
Alessio [00:32:54]: Let's jump into the state of AI survey that you ran, and maybe get some live updates. So you surveyed about 1600 people last August, and I were this busy like five years ago. And there were kind of like a lot of interesting nuggets and we'll just run through everything. The first one is more than half the people, 52% said that AI is overrated. Are you seeing sentiment shift in your customers or like the people that you talk to, like as the months go by? Or do you still see a lot of people? Yeah, that are not in Silicon Valley, maybe say, hey, this is maybe not as world changing as you all made it sound to be.
David [00:33:30]: Yes, we're actually on the survey again, actually, in the next few months. So I can let you know when it changes. It seems to me that it has settled down a bit in terms of sort of the maybe like, I don't know, signal to noise, you could say like, it seems like there's a little bit less noise than before. I think people are still trying to look for use cases. I'm saying, but honestly, last year, like United States, again, and I think there are slightly more use cases, but still not substantially more. And I think as far as we can tell, a lot of the surveys, especially some of the comments that we saw, do feel like the companies are investing quite a bit in AI, and they're not sure where it's going to go yet. But they're like, right, it could be big. So I think we should keep on investing. I do think that based on what we are hearing from customers, if we're not seeing recurrence of like a year or something, there will be more skepticism. So I think there is like a, it is time bound, if you will.
Alessio [00:34:15]: So you finally gave us some numbers on Stack Overflow usage. I think that's been a Twitter meme for a while, whether or not Chad GVT killed Stack Overflow. In the survey, 58 people said they used it less. And 94% of them said they used it less because of Copilot and Chad GVT, which I think it kind of makes sense. I know Stack Overflow tried to pull a whole thing. It's like, no, the traffic is going down because we changed the way we instrument our website. But I don't think anybody. And then you add right after that expectation of job impact by function and operations, people, 8 out of 10, basically, they think it's going to, it's going to really impact their job. Designers were the lowest one, 6.8 out of 10. But then all the examples you gave were designers of a job being impacted by AI. Do you think there's a bit of a dissonance maybe between like the human perception is like, oh, my job is like, can possibly be automated? It's funny that the operations people are like, yeah, it makes sense. I wish I could automate myself, you know, versus the designers or maybe they love their craft more. Yeah, I don't know if you have any thoughts on who will accept the first, you know, that they should just embrace the technology and change the way they work.
David [00:35:21]: Yeah, that's interesting. I think it's probably going to be engineering driven. I mean, I think you two are very well, maybe you two even started some of this wave and sort of the AI engineer wave. I think the companies that adopt AI the best, it is going to be engineering driven, I think, rather than like operations driven or anything else. And the reason for that is, I think the rise of this like profile with AI engineering, like AI is very philosophical, like AI is a tool in my head. Like it is not a, in my head, I think we're actually pretty far from AGI. But AI is not like a, you know, thing that it's not like a black box where like it does everything you want it to do. The models that we have today require like very specific prompting, for example, in order to get like, you know, really good results. And the reason for that is, it's a tool that, you know, you can use it a specific way. So if you use it the wrong way, it's not going to produce good results for you, actually. It's not like by itself taking a job away, right? And so I think actually, to adopt AI, it's probably going to be going to have to be engineering first, basically, where engineers are playing around with it, figuring out limitations of the models, figuring out like, oh, maybe like using vectorized databases is a lot better, for example, maybe like prompting in this particular way, it's going to be a lot better, etc. And that's not the kind of stuff that I think like an operations team is going to really be like experimenting with necessarily. I think it really has to be engineering led. And then I think the question is, well, what are the engineers going to focus on first? Like, are they going to focus on design first or like operations first? And that I think is more of a business decision. I think it's probably going to be more like, you know, the CEO, for example, says, hey, we're having trouble scaling this one function. So like, why don't we try using AI for that? And let's see what happens, for example. And so in our case, for example, we are really we have a lot of support issues. So what I mean by that is we have a really, really high performance support team. But we get a lot of tickets. And the reason for that is, you know, we're a very dynamic product, you can use it in so many different ways. And we'll have a lot of questions for us, basically. And so we were looking at, well, you know, can we, for example, draft some replies and support tickets, you know, by AI, for example, can we allow our support agents to be, you know, hopefully, you know, double as doubly productive as before, for example. So I guess I would say it's like business needs driven, but then engineering driven after that. So like, you know, we the business decides, okay, well, this is where AI can be most applied. And then we assign the project to an engineer, and the engineer goes and figures it out. I honestly am not sure if like the operation, we're gonna have much of a, like, if they accept or reject it, I don't know what's gonna change the outcome, if you will.
Alessio [00:37:40]: So another interesting part was the importance of AI in hiring. 45% of companies said they made their interviews more difficult in the in the engineering side, made interviews more difficult to compensate for people using copilot and chat GPT. As they change every tool, like, have you? Yeah, have you thought about it? I don't know how much you're still involved with engineering hiring, I get the company, but I'm curious how we're scaling the difficulty of interviews, even though the job is the
David [00:38:11]: same, right?
Alessio [00:38:11]: So just because you're gonna use AI doesn't mean the interview should be harder. But I guess it makes sense.
... [Content truncated due to size limits]