Episode 27
· 59:13
Nitay (00:00)
Jamie, it's great to have you on today with us. Thanks for joining Kosis and myself. Why don't we start things off by giving us just kind of some of your backgrounds, mirror your roots, and then what led you to where you are today.
Jamie (00:04)
Okay.
Yeah, sounds good. mean, I guess the succinct version is I'm a shameless startup junkie. I ⁓ started playing around with the internet in my dorm room a long time ago when I was a student and became so obsessed with this powerful ability to just make stuff that people saw right away and affect people far away that I kind of spent my college years building little startups instead of going to class. And it ends up you need to go to class if you want to.
So dropped out of UCLA and dropped out to join a first startup and then kind of took it from there. So did a lot of different startups in my 20s. First is kind of an IC as everyone does and then eventually running teams. And then had a lot of fun doing that. All that took me to eventually in my early 30s, I had done like, I don't know, four startups in a row and had a lot of fun, learned a lot, but was ready for something a little different.
And I joined Dropbox, which I had kind of, the way I thought about it is I'd built teams from like three to 30 several times and Dropbox was about to do like 300 to 3000. And I was like, I've never been that part, been through that part of the journey. And I thought there was a lot I could probably learn from it. joined Dropbox and had a ton of fun and was a part of a lot of really cool infrastructure projects at Dropbox, like moving Dropbox off of S3 onto our own infrastructure. so.
kind of building distributed storage systems and then designing custom databases. um, and yeah, during there, I met my co-founder, James Callie, my convex co-founder. So it was kind of this partnership that emerged out of working together on kind of all things distributed systems at Dropbox as a leadership team. And we took that momentum and kind of carried it out to convex. But for me, the through line and all that stuff was this kind of like always trying to take on things that were new and novel and kind of ambitious.
seeing that there was different way to solve a similar problem. And just my obsession with that, I guess, is probably what's just taken me to this place. So yeah, that's my background in a nutshell is just building a lot of different things and constantly trying to refine and re-examine, re-imagine how we solve technical problems.
Nitay (02:17)
Very cool. What were some of the things that you and James worked on at Dropbox? were some of the kind of trends or the different use cases, resources, et cetera, that were shifting that you saw?
Jamie (02:27)
Yeah, so I have this weird thing where Convex is the first job. I haven't joined as an IC, although I guess since Convex was the startup, I sort of joined as an IC too. Because when it's just three people around the table, everyone's just coding. But ⁓ so when I left my company before Dropbox, I was running the engineering team of that startup and then it got acquired by Google. And I went to Dropbox and said, I just want to write code.
I just want to write code for a couple years and go deep again and learn how Dropbox ticks. The first project I promoted was something called Magic Pocket, which is Dropbox's exabyte-scale storage system, in-house storage system. That was a lot of fun. That was involved designing and building out and then moving on to millions of hard drives, lots and lots of data all over the world.
spent a few years on that. then James and I transitioned over into working on rewriting the Dropbox sync engine. So I thought it was kind fun because I worked on really extreme opposites of the stack. So all the way in the back where the bytes are on disk, one of my projects was writing a custom file system to store data as efficiently as possible. And then I went to the extreme other end, which is the library that's embedded on everybody's desktop. And so
That was a code base that was a venerable code base, I think is a kind way to say that, know, that Drew Houston, that was kind of his original code base that had evolved dramatically over the years, but it was a massively complex Python code base actually, that it was starting at the point where it so difficult to change the code base that it inhibited us from innovating, to be honest, right? Like because the consequences were so high of getting it wrong if you changed it and if you lose data on a user's machine.
you can't get it back. There's no backup, right? So like the stakes were just very high. And so the kind of next project after this big storage system was redesigning the sync engine and re-implementing the sync engine and kind of making the second generation of the sync engine. So that was kind of the second big project that we worked on there. And then after that, I kind of moved back into management and then was running teams working on all kinds of stuff. So storage and databases and James helped work on
It's kind of custom database system called PandaDB at Dropbox, which is kind of their in-house metadata layer. And I was at that point just kind of managing teams and trying to put fires out of different places across the company and stuff. So yeah, a wide variety of things, but I would say the commonality was infrastructure, distributed systems, systems that were sort of the source of truth, A lot of persistence and all the problems.
Those were all the kinds of things that we focused on.
Nitay (05:01)
Very cool. And what were some of the things with either Project or the Magic Pocket or the rewriting the sync engine, what were the things happening as Dropbox was scaling that led to it to say, we need to rewrite the sync engine, or actually just sticking stuff in Cloud S3 in a basic file format or whatever is not good enough anymore. We can do better.
Jamie (05:21)
Yeah, I mean, actually very different motivations, which was super fun to learn about. Right. So when Dropbox was started, a lot of people like maybe thought Dropbox was always on the cloud or like just, especially because that's such a common decision these days. The Dropbox happened to be born right in the precipice of the large scale public clouds. Right. So Dropbox was born in 2007 and S3 existed, but a lot of the other
services that are in AWS now weren't there yet, really. I EC2 was there and like maybe early RDS, maybe not even that. And so the way that Dropbox architected itself is it ran everything in its own server racks in collocated data centers, except for file storage. And I think that was an extremely wise decision that Jiren Arash made back in the day, because I think when they were taking on a problem like Dropbox, the one problem that would have been very, very difficult to solve directly would have been...
managing many, many petabytes and eventually x of bytes of data. so Dropbox's architecture is a combination of racks of our own servers for almost everything else and then S3 file data. so moving the file data over in some respects was the last major workload. And the reason that was done was for cashflow, right? So as you can imagine with Dropbox where you're 500 million people's stuff, storing or paying for that
that stuff on hard drives, shuffling it on and off and then storing it statically on hard drives, it's a huge portion of their costs, right? And so their margins, they were in this position where their margins were controlled by one company in many respects, right? So there's a strategic dependency on contract negotiations with AWS and stuff like that. And because Dropbox had aspirations to go public, it really wanted to have better control of its cashflow and have kind of strategic independence.
on its margins. And so that's why we did the project. it took a lot of work to do, but it ended up working out. even in Dropbox S1, they mentioned the Magic Pocket project is the project responsible for making positive cash flow happen, So because we eventually were able to dramatically store things for cheaper than Amazon, even the private pricing AWS was offering us.
So yeah, that project was really all about that kind of like strategic control of your financials, right? On the kind cog side, the cost of goods sold. And then the sync engine was very different, right? Cause that one was about product velocity and the ability to innovate. it was given the stakes, you know, we just didn't have the freedom to change things about what the client was capable of, what the Dropbox client was capable of because the code base was so complex that
it was difficult to reason about its behavior. so that one was not seeking like a marginal kind of like cost control operation. That one was seeking the ability to move faster in the long run and to be able to service like one of the things, one of the movements it wanted, the Dropbox wanted to able to do was to support much larger accounts with a lot more files, right? And that, you know, there was a lot that needed to be changed about how the sync engine worked in order to do that.
And it was incredibly difficult to do that without causing regressions. would cause data loss and stuff like that. so yeah, that's, those were two very different projects. ⁓ but hopefully that answers the question kind of on the motivation speech.
Kostas (08:41)
Jamie, one question regarding all this migration and how that could happen. So, tend to to speak more about what the technical challenge is of doing something like this. But when you are going either from on-prem, let's say, to the cloud and vice versa, there's also cultural changes that need to happen in the company and the engineering teams in order to do that.
right? Tell us a little bit about that because that's not something that you hear easily about and I think it's like super important because at the end of the day like let's say the technical problems if you find the right smart people they can put things on design right but the success at the end depends heavily in my opinion on like the ability of the org and the
company to also change in order to operate like in a different way.
Jamie (09:37)
Yeah, I agree with that. So I would go as far as to say the Dropbox project is cited pretty frequently as a justification for other companies these days making a similar move. And I actually think almost every time it's a mistake when the other companies do it. Right. And one of the reasons even talking through history with you, when I mentioned everything else was already on the cloud. I mean, part of
what made it possible for us to pull this product off is Dropbox already had the muscle for data center operations and for supply chain and hardware qualification, right? And so the project would have been so much harder if we hadn't had all that stuff already. But because we had everything except S3 already in our own wraps, the cultural change was not as great. Now I will say that the machine density was so high and that...
It significantly increased our footprint. don't know by what 4X or 6X or 7X or whatever. Once you talked about file storage, it was a lot more data centers, a lot more racks, that's for sure. But we were at least expanding. It was kind of a difference of degree, not a difference of kinds, right? The team already knew how to execute on this. How do you own data centers, have data center operations, have employees spread across different geographies that are working at these? Because that's even just a cultural thing, is everyone isn't either at home or at one office.
Some people need to live close to the data center. So anyway, all of that we did have. I do think if you don't have all that, the size of the effort you need to actually put together to even get close, especially because we're now 15 years on or 12 years on. So the clouds are even better at this stuff. The prices are even lower. To take this on yourself, it's incredibly difficult. So yeah, back to your original question.
a lot of cultural changes and all that stuff around hardware just magically appearing and the hard drives magically getting replaced. And also, there's so many things you talk about, right? Power distribution units and failure domains. And there's a lot of choices you have to make to magically do what Amazon does for you. It's not just about a software team. It's really like seven or eight teams, network engineering and network topology. And so anyway, huge topic. And part of why when we started,
convex, James and I, despite the fact that we really know how to do this thing really well, if someone was like, well, then are you going to build S3 or even RDS in order to run convex? The two of us were like, hell no. We are happy to pay Amazon to do that. Right. We know how much work that is and it's a massive amount of work. So yeah, I don't know if that kind of gets to the spirit of what you're asking, but there's a lot of details of what kind of cultural stuff, but yeah, it's, it's enormous. The cultural shift to execute on that, especially when you have goals like.
The margins need to expand, which is not easy. need to do it like the service. The availability needs to be as good and the spend needs to be lower. That takes an extremely diverse organization and cultural shift.
Kostas (12:32)
Yeah, yeah, I think, I mean, I hear you that it's like a big, big topic. And probably it's worth not even just one full episode, maybe multiple ones, but I think it's worth telling the stories and sharing the experience because especially now, like because you hear many like, even like startups
because of AI and GPUs and people are like, okay, we are going to get to like, put our own GPUs and our own racks and like, monads, all these things. They don't necessarily, you know, like have in their mind what it means to scale and operate that in a production environment. And at the end, are not that many people out there that they can share this experience. So you should do it.
Jamie (13:11)
Well, I do have, I
have a couple, I have two stories. I'll try to them pretty fast, but I think they're a good spirit of like software developers realizing how big they would be getting into if they thought of this. So one is that, you you order and qualify hardware in batches at that scale, right? Cause you're buying 10,000 hard drives at a time, right? Or 20,000 hard drives at a time, racks full of servers, right? Or whatever. And,
we got unlucky, which happens sometimes, right? Because hard drives are just things that have flaws, right? Like software works around the flaws of the physical media to expose something to you that appears perfect, but is actually highly full of errors all the time, like the magnetic media. And so we got unlucky and we got a batch of hard drives. I don't even remember the manufacturer, so I can't point fingers. but, all the like, like, mean time to failure, you're expected like...
four and a half years out of these disks or something, and we were finding they were all dying within three months, right? And so as you can imagine from a capacity perspective, that's a big problem because you were expecting, if you're a growing company, you need to load data somewhere. And you suddenly realize you have a whole batch of hard drives that have a problem. And sure, the manufacturer is going to give you new ones for free and they're going to make it right. But that takes time. And it doesn't matter what their intentions are if your business grinds to a halt because you can't store more stuff.
And that gets even more problematic when you realize that you didn't factor in hardware batch as a parameter into your diversity scheme, right? So the way that big storage systems store data is they ship lots of little parity data all over the place, right? And so they'll include things like don't put all the data, certainly can't be all on the same machine, of course, but then it can all be on same rack in case that power distribution unit goes down or the network switch has a problem.
it maybe shouldn't even be in the same row because probably the power is all the same on that row in the data center, right? So there's so many of these considerations you have to make. And then you realize, well, when we load all these racks in, even if we think about diversifying them other dimensions, they're all in the same hardware batch. And so they're all new and new data all got placed on them. So we don't have that data spread onto other hard drives that don't have this problem.
And so now we've got to move a lot of data around the data center to make sure we've got parity data kept on some hard drives that don't have this manufacturing problem on them. that, you know, and the second smaller kind of sub story there is we have an even worse version this one time when two trucks, two separate trucks carrying racks and servers got in accidents in the same month. And so we had two massive shipments of capacity that just did not get to the data center on time and we had to figure out what to do. Right. So.
The magical cloud that Amazon gives you that kind of just like the number of functions and expertises and risk mitigation, again, especially if you don't want to just over provision everything and you want your costs to be reasonable. It's a huge effort and it's really exciting and interesting, but you also need to know as like a small startup team, it's, you know, to really be competitive or even close to competitive with the level service you're getting from those big providers, it's almost impossible undertaking. ⁓
That's just kind of one example of the stuff you run into and you learn, oh wow, that's yet another thing you have to engineer around, right? Because in the real physical world that can happen. So yeah, anyway, like you said, I could talk forever about this, so, it's.
Kostas (16:32)
Yeah, no, no, we should
definitely find more time like to talk and share this story. They're amazing. And I think like super useful to. All right, so tell us a little bit about how you got from, let's say one of like, as close to let's say to the metal you can get and like storage of like raw files, deciding to go and build something much closer to the application, right? Like, which
which is convex. What inspired that? Tell us a little bit about the journey there.
Jamie (17:00)
Yeah, so I think like, you know, the convex founders probably all had a slightly different story and they just happened to line up really well, right? But for me, I had a lot of convex was more about all the startups I had built before that and teams I'd run before that, right? So, you know, I tried a lot of different ways to connect front ends to back ends. And I worked a lot on different kinds of synchronization layers and then obviously Dropbox just kind of another variant of that, especially re-architecting the sync engine, right? So.
And especially between, I would say like 2009, 2013, I had played a lot with, I wrote a lot of a backend stack in Haskell, which exposed me to a lot of ideas around data flow, like pure data flow, and how powerful that could be to kind of automatically manage data in a way that comparative languages make much more challenging. And those ideas were very exciting. And then I also at the same time was aware of things like
and Firebase, right? And I think that in my head, I looked at these two things together and I was like, every other layer, right? We used to build our own machines and then we bought machines and we used to buy our own rooms, then we rented rooms. Now we don't even know what rooms are. We used to upgrade systems and now we just have Docker containers. We never think about what the kernel is. The level of abstraction is always rising. And when you see things like Mongo and Firebase and stuff like this show up, it's because the abstraction floor is going to rise again, right?
software developers are kind of exerting their power, right? So it's not DBAs buying this stuff. It's who are the first people in the door at a new startup? It's software engineers, right? And increasingly it's product engineers, because the point of the startup is to make the product, right? So I could feel that happening. And then through things like Firebase and Parse, you're like, you can see the demand rolling, right? In Mongo and things like this. like, you know, it might take us a while to get it right, but like eventually we will raise the abstraction floor again.
And we won't have every single company recreating the same back-end engineering stuff to configure a queue and a database and even AWS IAM roles and all this. If the customer doesn't see it, it will get eliminated if it can be. And so to me, the only question is, what abstraction makes it possible to actually replace a lot of what we do on the back end? And a lot of that kind Haskell experience has informed that. And so, you know, and then... ⁓
things like Elm happened on the backend, functional reactive kind of patterns starting to make their way into programming the way that we would think about more mainstream programming. And then of course the big thing is React. So then React shows up, you're like, now the client has these purely functional kind of patterns in many respects about how data flow works, who mutates, who reacts. So it's declarative and all this kind of stuff. And I was at...
at Dropbox when that happens. But to me, it's like, we're starting to converge onto a kind of more opinionated version of data flow where now you could make the backends that kind of couples with that. And then you potentially would have the ability to really abstract away all this complicated stuff that we did on the backend. So while I was heading down to Dropbox, was just obsessed with Dropbox things. But when I picked my head up in 2020, we left Dropbox.
It was like, still felt like nobody had really solved this, right? We hadn't quite come up, dialed in the, what's the design of a new set of kind of APIs that re-imagine how persistent server state would work that marries in a way that's ergonomic with these new ways that application development is working post-React. Even if it's not React, it's every other framework after React is highly influenced by this kind of declarative reactive programming paradigm.
So yeah, I think that was kind of the background of it. And then we were specifically, though, then able to take a lot of the things we've learned at Dropbox and say, you know, it was almost obvious in a way, right? Like what to make. And because of all those experiences. So I don't know if I answered the question, but yeah, that was kind of the, it was the synthesis of all of those things. And then just kind of being passionate about that whole space.
Most people, like, this may be like a weird way to put it, but there were three different versions of like why Jamie went to Dropbox. It were all different. And like, I didn't learn about them until I got there. Right. So there was the one where if you looked at like the, worked on storage stuff, it'd be like, well, of course, maybe I went there to write infrastructure. Dropbox hired me their version of it because the company I had was running.
was one of the most innovative mobile photos companies that Google acquired to go Google Photos. And they were just starting Carousel, which was their photos product. And they were like, we're so glad you're here. We would love for you to lead a bunch of stuff on Carousel. And I was like, what? I just did mobile photos. I don't want to do mobile photos. I give them to Dropbox, because they guys do file stuff. But the third one is the reason I wanted to work at Dropbox is that
And I think I told Drew this at one point, but they had done a conference like six months earlier where they released a product called Data Stores and Dropbox Data Stores and died. ultimately didn't succeed, but it was extremely parse like kind of like future of application interface backend thing. It's like CRDT based and all these kind of like new ideas. And I thought that was really cool. And so that was the biggest reason why I looked at Dropbox is I was like, if they're looking at things like Data Stores, that sounds very exciting to me.
and I've been following all this Parse Firebase stuff. I think that maybe in Dropbox, I know how good the Dropbox team is. I've heard over and over again, I a lot of people there. And so that was really the biggest reason I wanted to work at Dropbox, is I thought the data source stuff seemed very cool. Now it ends up I worked on storage stuff instead, because when I got to Dropbox, it seemed like data source wasn't working and they weren't that serious about it. But I thought the idea was awesome, right? So yeah, it was, there's always really kind of been for me, at least this thing about.
how clients and servers relate to each other that for some reason I found very interesting to work
Nitay (22:45)
What was the Dropbox idea for data storage for this kind of how our development should be? then how did you, when you guys then left Dropbox to start convex, what was your vision for how having seen Parse and all the others, as we said, what was your vision for kind of how it should be?
Jamie (23:00)
I think there was a CRDT idealism when they first became known. And I think there's a second wave of CRDT idealism right now, by the way, with Local First. But I was part of the first wave. So we used CRDTs at my previous company. Conflict-free data types. I think I've got that right. CRDTs come. Yeah, I'll come up with the whole lookup CRDT and say what they are. But basically, automatic conversion data types that have properties where
they can be merged in arbitrary orders and converge on the same value. So if it's a set, then you always know if you union two sets or whatever, then you have the result of the operation. And it didn't matter what order you applied the operations in. And those kind of data structures allow you to kind of make distributed transactions in a way that's trivial, and you still end up with the same outcome by the time you merge all of them. And so...
I had used CRDTs at previous company which was using a database that was based on the Amazon paper. What was that system called? Amazon? was the Dynamo. Not modern DynamoDB, but the Dynamo paper, which was all about vector clocks and CRDTs and stuff like that. And there was a popular database, well, kind of
popular, it wasn't that popular, but called React that was an Erlang-based implementation of the Dynamo paper. And we had stored CRDTs in it. And I think in that era, like 2012 to 2014, I still thought CRTs were pretty cool. And I think that, I seem to remember a lot of the ideas around data stores were related to CRDTs. And it ends up, I think now that
I'm pretty, I'm not very enthusiastic about CRDTs these days, to be honest. But back then I was, and I thought, I just kind of wanted to get in the door at Dropbox to understand what they meant by this and where they thought it was going. but yeah, I think that how broad that idea was going to be, the truth is I kind of never learned, because I think that it felt like, and I don't want to misrepresent Dropbox here, this isn't me speaking for someone official at Dropbox, but like,
I think that the data storage thing was kind of like a few people's idea they really believed in, but then sort of support for it had seems like it had waned by the time I joined the company. I don't think I ever really learned what the full vision was or where it was going to go, honestly. yeah, it's an unsatisfying answer maybe.
Nitay (25:20)
And what was your experience with CRDTs that then led you to do a 180 on them, sounds like?
Jamie (25:26)
So, I mean, and this is extremely embedded in convexes design now, it was really hard lessons learned from CRDTs. I believe that one of the things I believe in the most, and James, my CTO believes in the most, my co-founder, is never bait and switch developers. I hate bait and switches, right? And there are accidental ones too, by the way. It's not always intentional, but...
There's a lot of things you can do that make day one amazing that will not work at month six and certainly not at year three. And so, and then if by the time you get to that, if you're hopelessly committed to a paradigm, it can be really difficult to retool or figure out, especially when it comes to like transactional systems, how to have one transactional paradigm exists next to the other one. It's, it's a nightmare. It's absolutely pan of the butt. And so to your original question, I believe now at this point,
that eventually most systems need something like regular old transactions. They just do. And if you want to have a more permissive paradigm that has some good developer ergonomics like CRDTs, you can always add those on top of straight transactions as a layer down the road. And so I believe the right solution for most developers, especially when you're making a product for people
experts at the trade-offs of distributed systems is to give them something that effectively feels single-threaded because that is the way they're used to thinking. And it ends up that strictly serializable asset transactions are the most intuitive way for people to write transactional code and the most flexible, right? Where all the things their app is going to need to do, even if the first thing they make is a collaborative editor and they're like, oh my gosh.
Why isn't everything operational transformer? Why isn't everything CRDTs? The longer their lablet, and it turns into a regular old business product with a lot of things it does, I find that CRDTs people are putting hats on hats on hats on hats to try to make them keep making sense once you get beyond that first allow use case. And then they end up deeply unhappy with the architecture down the road. And so that's why I don't love CRDTs is the most fundamental building block for
backends. think it, you know, once you get something basic working and then you're like, oh, I want to do something like instantly offline, automatically mergeable in this one part of my app. Great. Now you kind of know what you're doing. You got a strong foundation to operate off of. And you get kind of layered in something more interesting with more limited transactional semantics. But I don't think that's right place to start. And I don't think you should make that the cornerstone of the way.
Not if you want to be used for anything other than toys. You should not make that the foundation of your platform if you're trying to serve all developers and make them all successful.
Kostas (28:16)
I have a question. It's a little bit more related to what you said about like the paradigms and what people are used to and how easy it is like to change that. So we live in the world where there are like some dominant, let's say, technologies that has been used for like building applications, like almost like
Jamie (28:36)
Postgres.
Kostas (28:37)
for Postgres, right? Yeah. And we've seen also kind of renaissance around Postgres. I guess also this whole architecture with the plugins and making it more available to people to write extensions and build on top of Postgres has turned it into more of, not just like a monolithic database, but some kind of platform people can build on top of it and create.
Jamie (28:39)
Sure. Yeah.
Kostas (29:01)
pretty much like completely different data systems. So in a world that is kind of dominated with Postgres and Postgres-like systems, how does a paradigm like convex or even Firebase, if you want to go a little bit more in the fundamentals of where things started from, like fit, and how do you...
manage at the end of the day, like to go to an application developer who has been working on like using Postgres, it's what they know, that they trust, and convince them to be like, hey, give a try to this new way of like doing things.
Jamie (29:40)
So how do I convince it? Yeah. How would I think about the exercise convincing them to try? It has to be testimonial power, right? So it has to be somebody they respect says, trust me, it's where they're like, it can't come directly from us. If somebody is like, we all disagree Postgres is good. Give me the bullet points. I don't think there's a bullet point conveyance of what's different here that will resonate. I think it's just, you have to try it. If they try it for 15 minutes.
They're usually sold at that point. the, I think that the, the prove it kind of, statement like a priori before I tried the product, unfortunately, it's the explanation is rooted in things they don't understand yet. So there's not really a great explanation to give to be frank. I hate to be defeatist about that, but like, but that's part of why so much I think of the motion is focused on. Pure advocacy, right? Cause it matters a lot less of what we say than.
people they respect saying, I didn't expect it to be so good, but wow. Right. So, James and I have talked before about like, maybe like three, I don't want to say like the mid-wit curve or whatever, but like there's a little element of that there. Right. So we tend to have like three different kinds of conversations. If someone is extremely junior, they go, doesn't everything work this way? And you're like, believe it or not. No. Right. So, and if someone is extremely senior, the truth is they usually look at
Convicts, go, oh yeah, at Facebook we had a thing like this or blah, blah, Or like, I've built a version of this two and a half times before on top of it. Cool. But who made yours? Okay, you guys kind of know what you're doing. What's the design? Oh, this seems reasonable. Let me give it a shot. I would say it's kind of the middle where they're just like Postgres has vector storage, you know, and you're like, yes, it does. You know, so I think that's the place where I don't know if I like some kind of marketing message really, other than that thing that you've been hearing people buzz about, which is only.
recently, like, this is it, try it, right. And just see, like, wow, maybe I was wrong. had no idea kind of thing. So there's not, ⁓ yeah, I don't, I don't know if we've ever come up with how to win that argument, but we've also gotten advice from other people to make technologies like ours that like, not to try like, in other words, try to win over the people that are excited about it already, maybe like the most junior people, the most senior people. And then eventually they're
opinion can you converge those folks to try it, even though they're like, everyone knows Postgres is great. You're like, okay, sorry. So that's the best we've come up with. it is working now, but it's taking time.
Nitay (32:04)
I want to tie this question actually to something interesting you said before about investing in technology and architectures that doesn't bait and swish, that that's going to stick with you six months, three years in, et cetera. How do you appease to not only kind of the developer that wants to get their app up and running quickly and like, great, you give me single threaded view. That's amazing. I don't need to think about concurrency. Awesome. Then the person who's the power user expert who actually like really gets concurrency and wants to do the kind of cutting edge next level stuff.
How do they end up using convex and how does kind of that six month in thing scale.
Jamie (32:32)
Yeah.
So I would propose that James and I, lot of the convex team, is pretty good at concurrency. We've had to be. We would still prefer to use convex most of the time. But I think a version of your question I very much agree with is, in relapse, there are times where you maybe need to, for example, relax consistency for performance once in a while or something like that. There are use cases where like this
strictly serializable, single-threaded world. It can create too much conflict or something. One of the things we have to do is, I think on that whole not bait and switch thing is you should never promise your technology is perfect because it's not. You could do a couple things. You could say, my God, the paradigm is completely wrong because 3 % of the time they can't get done what they want to do. I think if they can't get done what they want to do,
29 % of the time, it's technology is not ready yet or you chose the wrong design or whatever. But like, if it's like less than 5 % of the time, you can't ignore it, you shouldn't deny it. And then you should like say to yourself, like, what is the escape hatch policy, right? for it, and the thing this reminds me of going back to Haskell for a second is hit like, I learned so much from Haskell, I would never develop Haskell professionally ever again. I just would not.
And there's a number of reasons why, but one of the ones that's especially, I guess, germane to this consideration is the kind system. And so some Haskell people would be very upset at me saying that, I guess. But, you know, the kind system is the ability to parameterize the class system, right? And that's incredibly powerful. But it also allows you to express computation using almost any paradigm you can imagine. And because of that, the thing I found is...
when you try to compose libraries in the Haskell ecosystem, they're all using a different paradigm for computing. And so you spend a lot of your time just like adapting one paradigm to another instead of just using libraries, right? And I think that talking about a more modern thing that's working better is Rust, right? So part of why I like Rust is Rust has a macro system and so far it's resisted having a kind system. And what's wonderful about a macro system
is the 3 % of the time you need some dynamism that can't be encompassed in the simple class system. You can just write a macro and it's kind of ugly, but it lets the other 97 % of the language stay simpler and more composable and more elegant. And so I think that instead of like trying to fix the 5 % where 5 % of the time you're to need to break out of the box a little bit and it's a little uglier, think acknowledging that and then putting as much supporting material as you can.
that to say, in these cases, you might need to do something different. Here's how you do it. As opposed to like, how can we create an even more complex paradigm that tries to eliminate that 5 %? But it makes the time spent, the 95 % way worse, right? Because you've introduced some new kind of flexibility that actually makes projects worse the rest of the time. So, like embracing a little bit of ugly part, just making sure the time spent in that ugly part is relatively small amount of time.
and then acknowledging exists and then providing documentation and support to like how to deal with these edge cases. Like that's the approach we've taken. I think that on the go-to-market side, you just have to make sure they don't hit that part in the first three weeks, right? Because like once they have conviction that this is great for most of what I do, they're more patient with, this one thing isn't great, but like there's a way around it. And then, yeah, most products are like that, ours too.
But when it comes to developers that are good at concurrency being like, I want most of my code to be more complex than this. think usually they don't. They kind of like the fact that they too, even though they know how to write just mutexes or futexes or whatever, they like the fact they can write code the simple way in convex. If they're in that mind space where like, I'm just making a product. I just want to focus on the human experience and the edge of the surface. I don't want to spend time thinking about mutexes right now.
Nitay (36:03)
What you guys?
Jamie (36:29)
Convex lets them get that job done without doing all that work, even if they know how to do it.
Nitay (36:33)
So to that point, kind of building off of the Postgres, the data storage, obviously that's one key aspect of things. There's also kind of broader like state management within the application itself. There's RPC mechanisms, there's frameworks and so on. And so I'd love to hear kind of like, what is your view of like, here's the best way to make an app or is there different kind of subsets of, well, if you're making this kind of app, you should do this versus that. And here's kind of where convex fits in.
Jamie (36:59)
I think that a lot of state managers exist to compensate for the server not being good enough in retrospect. Yes, you could say, well, there's transformation stacks in the application that are independent of the server.
In my what like we we've learned at this point, like back when we first launched convex, like three and a half years ago, people were like, well, have my Redux thing. And then, and then we would say like, well, maybe try to remove it. Right. And then they would, and then it would kind of just work. And I think over time, we've learned most of the time, those things are actually about the server, not doing what you want. Like maybe one example is under the covers, like, let's say you have five components on the page and they're all subscribed to the same query in convex. Right.
So the convex client library automatically makes those only one query. It manages deduplication. It also manages it like if those five components all depend on like they joined against some data and so does a different component on the table, convex ships the entire transactional update over as one unit and refreshes everything in one pass, right? So you don't need to own taking an aggregate return value and carving it up and dispatching it to different.
components, like at some state manager level, like convex this whole pipeline is designed to do that for you. The reason why that is good is because I really believe, especially if get back to like convex, like Facebook has had 5,000 people all trying to work on facebook.com homepage, right? Like the power in theory of an architecture like React is local reasoning, right? So I can work on this component over here. It's a product, it's props, and then it does something. And I don't need to really worry about
the seven places that data is mutated, that's just handled for me. I can just look at this file and know if I want to change the way this is rendered, all the information is right here. And that's what allows you to let thousands of people work on the same thing without creating this kind of like spider web of data dependencies to understand who updates who and who I change the way this works. Like I forgot one of the call sites or whatever. Convex makes the server do that too. You can just look at a component and be like,
use query and attach to some piece of the backend state, but you don't need to worry about things like, well, that's an extra fetch or, I really should make that one aggregate thing because that would be cheaper or whatever, or that would be consistent. There would be data consistency. Convex guarantees that data consistency at the protocol level. So a lot of the things you end up relying on state managers for, you don't need anymore. And now you can go back to local reasoning, even if server state is involved, right? Components just express a kind of lens on the backend state.
and making that efficient and making that deduplicate and everything. That's kind of our convex's job is to solve that for you. Just like it's React's job to solve that for you in the application. So I think, guess to your question, just do things the simple way, right? And I would say if there's one place where I think it was sometimes we're tempted now to do less of, it's to get rid of local reasoning for some efficiency thing, like a loader that fetches everything ahead of time.
I'd say try not to do loaders because loaders mess with composability. You need to centralize getting everything ahead of time in one place. And the bigger your application gets, that gets to be pretty fragile. There's a lot of stuff a page needs once your application is nontrivial. I also think server-side rendering is a mistake for most applications. It doesn't matter that much. And it overcomplicates your architecture, and it hurts composability of your code base over time as you scale up your code and your team.
⁓ Especially if you have agents developing it, which agents have such limited context windows. So go back to old school React, because now the servers are good enough if you're using convex, that the React paradigms from 2016 are good enough. They'll work very well. You can keep things simple. Same thing for other frameworks too, by the way, solid or spelt or whatever. So it's simple local reasoning kind of paradigms, modular.
Kostas (40:45)
Jamie, what I hear is, okay, there is like the back end, there is the front end. There's probably something in between that I think like, historically, if someone is like old enough has seen like, things like kind of moving between the front end and the back end, and who is like responsible for doing certain things, there is some business logic there that needs like to be managed. Because why
I hear from you and again, correct me. I'm probably the last person on earth who should have an opinion about anything that involves application development from then. But what I hear is that there is computation that needs to happen where we still try to figure out where these computations should be and who should be responsible. And that's both from the perspective of
the user, right? Like you mentioned like concurrency many times and like how hard it is like for an application developer, like for example, like to reason about concurrency and at the end, should they have to worry about concurrency or even care about that? Maybe not, right? But you also mentioned like with React, things that are locally computed, but they shouldn't be like locally computed, they could be like on the backend. What's from your experience?
would be the ideal state, right? Like the platonic application development framework and infrastructure. Like what is on the backend, what's on the front end. And if we need the middleware, I don't know what this should be.
Jamie (42:11)
Yeah, mean, one thing is that convex has a server function layer, right? So convex believes. Convex is not like Firebase or SuperBase where we use like some kind of row-based configuration language, and then there's direct record access. Convex team thinks that is the wrong paradigm. There should be server-side compute. It's just the server-side compute is extremely easy to write and has type safety and basically feels like an extension of your frontend.
So another way to put it is your React state graph just kind of extends into the server. And so that's what convex lets you do. There is a server-side edge, but it's owned by the product developers. And for most teams, there's no backend team anymore for the most part. So I also agree they shouldn't really ever have to think about concurrency. And so I'm happy to expand on this to get to the question behind your question, but...
The platonic ideal backend is convex, which is why I built it. And unfortunately, you're asking the most biased person in the world. That might not be true, but it's true for the person you're asking right now. And then on the front end, I think any of these frameworks, think solid is better than React in some ways. like this Svelte 5 is quite good. But to be honest, the ecosystem React is very powerful. And most of the like most important ideas to get right, like React did a good job at in.
Once you're out of... Even mobile these days is mostly React Native. Most new projects are React Native, so Web Tech is taking over everywhere. That's why we do recommend for most teams, they probably should just use React because between talent and the agents being good at writing it and the ecosystem being massive, yes, it has more warts than solid or spelled five, but it is good enough.
has a tremendous amount of momentum and increasingly code is being written by agents. And it's probably a desire to get a little bit less precious about code and focus a little bit more on higher level, just like architecture and then UX. So yeah, I mean, I guess that's my answer to the question. like I said, inherently I'm admitting you're asking that because I present a thing every day that represents what we think the platonic ideal is, I'm mostly just going to point at the
the thing we made. I'm happy to defend that though in any way though, it's specifics about it, why I believe that. But yeah, that's my answer.
Kostas (44:23)
Yeah, yeah, yeah, of course.
Yeah, 100%. Okay, you mentioned agents. And I want to ask something, which I kind of find one of the most fascinating things that has happened with with AI as a person who is also like, working primarily with infrastructure. And I think like the biggest change that happened in a way is that, you know, we've been building infrastructure, like, for the past, I don't know, like 5060 years, whatever we like to call it.
having one user in mind, which is a human being. And suddenly, this new thing comes, which is like an agent. And especially for infrastructure, I would say, especially like where you come from, where like it's more application focused, it exploded in a way. I remember, I think I wrote that about like NeonDB, where they were before like
Jamie (45:02)
and
Hmm.
Kostas (45:19)
around like the time they got like acquired by Databricks, they were saying that at some point, they were actually more neon DB instances being spinned up by agents that it was like from humans, right. And so now we have like a new persona there that interacts with infrastructure, which completely new, it's not something you can exactly like interview and like try to understand what they know and what they don't know.
⁓ But tell us like how how's your experience with that? Like what you saw happening because you are like like like what I find like really interesting in case of convex is that you started the company around the time but AI started happening also and you had like to ride the wave as it happened so you have a lot of like signal of like how the
Jamie (45:44)
and
Kostas (46:07)
AI itself change things. So tell us a little bit about that.
Jamie (46:10)
Yeah, I mean, one thing that's very true is that we got very lucky with AI, right? So the reason why is that, and I feel crazy admitting this now because now I'm frightened at this level of hubris, but like when we started Convex, we knew when like Mongo came out and Firebase came out, like the NoSQL movement was happening, even though it failed, right? So like, or mostly failed, right? So there was a...
a behavior change that was being considered by customers to like try a new thing. And there was optimism around new database abstractions. And we knew when we wrote convex, it like, everyone was kind of all just saying to each other at this point, no Postgres is great. Like we flew too close to the sun with NoSQL, that was a terrible idea. And so we knew we were going to be dependent on some kind of movement bigger than us that we could not create that caused developers to reevaluate.
tools, to be honest. We were like, we don't know what it is, but we know this is going to take a few years to build because it's really big. So let's start building a thing. But what the thing meant is we need to abstract all the complexity back in engineering behind more opinionated APIs that are going to fit the way applications are being built now and moving forward. And a lot of that had to do with types and narrower interfaces and more opinionated interfaces. Now, it just ends up that
the thing that caused everyone to re-evaluate the tools was LLM assist development, So there was, there suddenly was an energy that made people re-examine the way they were building. And even the platforms they were using were one of the things that started to reconsider. And so that, in that respect, we did get, that's one direction in which we got lucky, right? A second dimension in which we got lucky is because we created convex and because early data platforms.
are pretty much all selling to startups. All the startups that were being started when we launched our product were building AI stuff, right? And so we got to spend a lot of time with those founders, just looking at how they worked and looking at what they were making and learning a lot about the relationship between AI and a new backend abstraction. And so the movement happening at all was very lucky for us because otherwise maybe no one would have reevaluated the backend assumptions at all.
And then having that batch of founders was very lucky too. The third thing, the third way in which we got lucky and then we've, we've, we've sort of started putting spin on the ball to turn that book into being good at it, right. that LLMs are really good at writing code, right. And LLMs need feedback loops, right. Cause the issue with LLMs is they generate something, some probabilistic chance of being right. And they just need to be able to nudge. They need to be in some kind of constraint feedback environment that nudges them toward correctness.
And what we've made for convex, what we made with convex is we had turned everything into code. There's no configuration. Everything is just TypeScript, right? Everything is composable. If the AI finds the interface too parameterized or too full of parameters, so it gets it wrong too often, we can just wrap that and make a narrower interface that the OLEM gets right every time, right? So we ended up replacing all the backend with something that was 100 % code and it was typed code. And so...
LLMs are extremely good at putting together convex backends because there's no dashboards, there's no configuration files, there was no race conditions between this update and that update. All they had to do is look at the TypeScript API and then use it. And they're very, very good at that. And they also have the feedback cycle because it's all typed, right? So including the tables and fields, everything, nothing convex isn't typed. And so the nature of the thing we built was just a very good partner for that. And all we've done in the last year and a half,
two years is measure that and optimize it more and more. so early, you know, late 2024, early 2025, we started investing a lot of efforts like convex L and leaderboard where we developed our own extensive set of evals and constantly like tweaked our APIs and our contacts and MCP and everything else to just get success rates higher and higher and higher. And also just forging relationships with the labs. And we have fans at opening AI.
stuff like that. And so now a lot of the hints we used to do, like the models don't even need anymore because like the Opus is so good at convex that they don't really, we're just deleting all of our hints files and agents.md and everything else. So yeah, that's been exciting. And I'm very grateful it showed up. And I'm also, I feel very fortunate that we just happened to have, we were sitting on this platform. There was
It's like perfectly positioned to turn into it and to optimize for it. And so that's what we've done. But yeah, that's been exciting. It's been a lot of learning for me too, because most of how I've developed for the last 30 years is very different than this. But ⁓ yeah.
Kostas (50:54)
Yeah.
Yeah, and I have to ask you another question about culture again, because you mentioned like all these new things that you had to learn, you only like as a person, Jamie, but like the company itself, like, like, I can't keep thinking that we are talking about a database company. Like you started by wanting to build and build like a database system, right? And
regardless of like the use case. It is like a very specific type of like systems engineering that need like to do. And I heard you like talking about evals and leaderboards and MCPs and all these things that maybe someone from data science or like product maybe like they have like a little bit like here before but definitely it wasn't like the way that you were building systems like everything.
systems has to be as deterministic as possible, right? So how did you manage that as a team, right? And going from hardcore systems like into embracing these new ways of like building and sharing that, not just like building internally, right? Like, but sharing that with the community and the industry out there.
Jamie (51:48)
huh.
I mean, I think that real answer is pretty messy, right? Because like every human has a different visceral reaction to what this means, right? And the opportunities it presents and the risks it also brings along with it. And that's very true in the convex team as well, right? So, you know, people working on the core of the database usually start by saying, I could never let an LLM do any of this, right? Like, but I think that the
Like lot of like convex is backgrounds. Like this was the team that like, like the, you know, the early kind of like DST work distributed systems. Like we were doing a lot of that stuff at 15 years ago. And like, so like, there's a lot of this, how do you put things in a box to allow flawed things to make progress and the box like just checks them to give yourself freedom. Right. I think a lot of like,
even just lessons learned from like the Exabyte scale storage system, which was like, the goal was never lose a bite and we never did. Right. And so it was like, how do you do that? Well, the way that you do that is like humans are flawed, and tools are flawed. And so you create layers of controls and the flaw never permeates all layers at the same time. And that those layers have non-overlapping seams so that the same, you know, same issue seldom makes it through all of them. I think some of the same reasoning about like,
sure, if you just like fix my CSS, there's very few controls you need on that. If it looks right, it looks right. Let the LLM cook, right? I think that the LLM still can be incredibly important deeper in the stack, but you just need to invest more into the sort of cage you put it in, right? To sort of, so what are the invariants and what's the simulation and how do we ⁓ check all the rules still hold?
Certainly the more you're working in type systems, it makes it easier, but the more the stakes are high, you need a lot of behavioral invariants, descriptions that go much beyond what the type system will encompass. But it still can be done. And so I recently built a new, I wanted to move convex on to TyKV. It's a distributed key value store. It's based on FoundationDB and it's quite well done.
And so instead of just having convex run on SQLRDBMSes, we always figured one day we'd probably move it on to something like foundation DB like, because convex really only needs like a replicated B-tree. We can treat as a journal. Anything that does that, it can be a good persistence layer. And a lot of that code I wrote with Claude wrote, you know, but what I did is we had an incredible test suite for what behavior means that data is persistent, right? And so, and then...
And then I read the code and then provide different kinds of like, almost like if those old videos are like the robot that self writes, you kind of knock it and make sure it stands back up. So a lot of edge cases I'm used to these kinds of systems, like did you check that when X happens, Y happens and stuff like that, you need to have all those conversations with the LM to kind of, and then turn those into repeatable, like great write a test for that, after you fix it or whatever. But you can be successful in that layer. think.
the maybe the biggest reticence right is the human one, right? That like, I'm accountable for good work here and I don't trust this thing yet. Right. And I think that's a very fair response. Right. And so because convex compresses all the layers of the stack, we have everyone at convex for people writing CSS to people optimizing distributed systems, like in Rust, getting profilers out and writing new data structure to make a certain indexing approach faster. Right. So
We kind of in a small team have every one of these different, you know, personalities kind of all in that, in that group of people. And because of that, they do have a very, I would say, wide reaction in terms of their personal, their personal attitude. Getting more to James and I, which is a little bit less about how our product is perceived, right? As leaders, our attitude is you are accountable for your work, however you want to do it, because you're an adult. And we hired you because we think you're good. Right. And so.
If some people are not comfortable yet, we don't force them to use it, right? And if some people love it, you know, allow them to use it. We encourage them to use it. We'll pay for the license. And so it's almost to me, just like you start using an IDE instead of them or whatever. It's just another tool that has some productivity benefits and some trade-offs. And we're not hiring LLMs, we're hiring people. And so as they build their own confidence on, I've now learned how to use this tool within the layer I work at and I enjoy it.
And I feel confident in sort of like the risk mitigation strategies I'm using to employ it. We're like, that's great. Teach us all how you do it at your level. Right. So, but there's no force anyone to do anything. It's always doubling down again on this is a new tool for you all to use and like humans work for us that are independently have agency to make your own decisions about how to adopt these things over time and build their own confidence, conviction and excitement around. love using AI for part of how I built whatever it is I built.
So, yeah.
Nitay (56:51)
This is really fascinating stuff we definitely will have to have you on a future episode because I think there's this easily another hour or two we can spend just on some of the things you said here. Unfortunately, we're running out of time, so perhaps one last question to close us out, which is tell us a bit about the future for Convex, for AI and app development. What do you see as the road from here?
Jamie (57:11)
Yeah, we are, mean, one of the biggest things we're doing this year is even experienced engineers are really starting to switch, right? So the metaphor would be they're spending a lot more time in cursor agent mode than they are in cursor editor mode, right? And so, or another way to put it is they're in cloud code or open code now in many respects, more hours of the day than they're in cursor, right? Or, you know, and that is really changing things in a way that like,
we need to steer into both in terms of documentation and design and stuff like that. So that's one thing. And then the second kind of big trend for us right now is convex is now we've had some customers grow from, you know, hackathon teams to building their own multimillion dollar ARR companies with like large accounts and stuff. We also have a whole bunch of enterprise teams that built proof of concepts over the last 18 months that are now all on board on the platform. So.
It ends up that enterprise readiness is a real thing in terms of SLAs and professional services and contracts and features people need. So a lot of our effort this year is building out all the things that people need, both from a technology perspective and from an operational perspective to service this new batch of much bigger companies that are our customers. So those are our main themes this year is figuring those things out and then...
delivering on more and more features that developers ask us for all the time.
Nitay (58:33)
That makes a lot of sense. as you grow the enterprise, great stuff and kind of the robustness and the kind of more white glove ability of an entire suite of services makes a lot of sense.
Jamie (58:45)
Yeah, white
glove is a good way to put it. Yeah, the service expectations raise quite a lot. And so we're happy to do it, but we also have to figure out how to turn that into a muscle and not have it be ad hoc, right? It's like, how do we make everything a process? yeah.
Nitay (58:51)
Exactly.
Well, we'll definitely have to bring you back on to hear more about that and more about many of the topics we discussed on that we only probably lightly touched on. So thank you with that, Jamie. It's been really great chatting with you and looking forward to continuing.
Jamie (59:11)
Thank you both.
Kostas (59:12)
Thank you so much, Jamie.
Listen to Tech on the Rocks using one of many popular podcasting apps or directories.