Falling Into Databases: The DuckDB Story with Hannes Mühleisen

Nitay (00:02.252)
Hi, it's great to have you with us today. Thanks for joining us here with Costas. Why don't we start with give us people kind of a some a bit of history and background about yourself and then we'll we'll dig into some of the more current topics.

Hannes (00:15.302)
Hi guys, I need to hand off Sask. Thanks for having me. yeah, my name is Hannes. I I'm from Europe, where history is from, to quote Eddie Iizard. we live we are I'm from Amsterdam and we are talking to a way about things related to databases, my favorite topic. but from from a background is so it's actually a bit of a car crash of history, if you want. so I wasn't meant to do this. I was I was a you know

Studying computer science and doing things. And then I I met a girl and I moved to Amsterdam because of her. And then I was like, no, I guess I need a job because things are quite expensive here. And then I looked around and then there was this job in this database research lab. And I and in an insane sort of over-exaggeration of my own sort of capabilities, I had just applied. It's like, hey, I I know what my SQL should be fine.

And in in an b even bigger catastrophe of of hiring, they hired me. And yeah. Well so so that's how I learned about that. I actually didn't know anything about databases and as a result started working on on database systems and I think it really helped me to be this outsider because I didn't have the sort of the orthodox educat education of most database people and ended up

Coming up with this new concept of in-process analytical databases. And I think the most well-known example of that is DuckDB, it's a system that we've started building here. and now these days I'm the co-founder and CEO of Duck Labs. We just renamed ourselves. We have Duck Labs now. a company that employs most of the contributors of the system. So that's that's that's where I am. Yeah. I live in Amsterdam, I live in a boat. I used to have a Duck. what else? Yeah.

I I work on databases, what can I say? Yeah.

Nitay (02:15.0)
Fantastic. and I I love that because I I don't know many people who have just fallen into like happen to to get into databases. It's not it's not something you typically like fall into. It's like I'm gonna go, I'm gonna accidentally get into like one of the most complicated areas of computer science and engineering.

Hannes (02:29.574)
This is exactly what happened. I I I assure you, this is exactly what happened. I mean, I this was a little bit cheeky. I did work on distributed systems at some point in my life. I also worked on distributed but then but I I mean I had absolutely zero papers and databases when I joined the research one of the leading, I might add, research labs and databases on this planet here in Amsterdam across the road of where I'm sitting now at the CWI. I had no idea. And I I it's still a mystery to me that that that they hired me, but I guess I must have s you know convinced them somehow.

It's yeah. The f okay, sorry. I sorry, one one more thing about this. At some point I became the head of that research group temporarily, and then I realized that I would have never hired myself. So it's it's you know, it's like it's a bit like, okay, why? But I'm still very grateful to t to this day. So thanks Martin and Peter and Stefan for making that happen.

Nitay (03:01.55)
Clearly.

Nitay (03:22.734)
Yeah, and it's it's definitely a fantastic group you guys have over there. I and I think it's a I think it's the mark of a good interviewer when you know that you couldn't pass your own interview. I I remember putting together some interviews back with my company where I look at it and I'm like, I'm not even sure I would hire myself based on this.

Hannes (03:26.246)
Yeah.

Hannes (03:39.442)
I think it was a it I th it was weird, yeah.

Nitay (03:43.51)
Alice a bit you said something interesting there where you said kind of you came with an outsider perspective and some of that led to the, you know, kind of in memory analytical databases and later DuckD B and so on. So what what was the outsider perspective? What were the things you were seeing that others weren't like about to understand more?

Hannes (03:48.219)
Right.

Hannes (03:59.314)
Yeah, but I think I think this is the same exact thing. I didn't know anything about databases. So I was like, okay, I work here now. Maybe I should use a database. and then that turned out to be extremely difficult. And I at some point, you know, you don't have this unique perspective on a new company that you're joining in the first two or three weeks. Because you kind of see all the things that are wrong with it, because you have the outside perspective. After a month or two, that goes away and you become just one of the drones.

But but in the first couple of weeks you have this fresh outside perspective. So that's that was me. I was like, okay, I'm new. let me do something with these databases. And it turned out to be extremely difficult. And I never got that, right? Like, why is it difficult? I also never got, by the way, why it is so difficult to use Linux. Like, I remember like screwing up my hard disk as a teenager in school several times with a Linux installer, and I was always like,

Is this not the future? Don't these people want us to use their stuff? Why do we make it so difficult? Anyways. so this this outside perspective, I think the outside being out being having not been educated in that sp in that world, I think really made me maybe realize the the really ugly warts that that happened. And I have a theory about this. I think the reason for the warts is that database systems get sold on the golf course. Okay? They are not sold to developers, they're sold to the

you know, purchasing department or whatever. They're sold by sales professional salespeople to professional purchasing people. They go to a golf course, they have a great time. They talk about, you know, check marks or features. Does it do ISO, God knows what? Yes. Does it do SQL 2003? Yes. Is it nice to use? No. But that doesn't matter because it's on the golf course. So nobody's ever actually trying this. And so I think I think that's the reason why traditional databases are

so terrible to to use because nobody's ever nobody ever cares because it's not about that. It's not it's about checking some check marks. It's not about actually convincing somebody bottom up. But that's exactly what we did with with DuckTB, right? So that's we tried to say, hey, what if we make a database system that people actually enjoy using for once. So but again, outside perspective, right? Sorry. Sorry, exactly sorry, excuse me, I didn't want to talk over here. It's the outside perspective. Yeah. Sorry.

Nitay (06:14.798)
But so that's an interesting insight.

Nitay (06:20.332)
No.

Nitay (06:23.798)
No, no, and then I'm I think I think you're you're spot on. I mean for decades, right, we had back to the like IBM, Oracle, et cetera days, these were the the most enterprise of enterprise sales that you could get and so to your point. It was like it wasn't a golf course, it was like some Madison Square Garden box seats, though those those where where the where the sale was happening and certainly, you know, UX, DX, kind of all these kinds of things were not top of mind.

Hannes (06:40.592)
Ha ha ha.

Nitay (06:50.164)
jumping a little bit to to the modern times and then we can come back. Do you think that's still very much true today? Meaning like even pre obviously DuckDB has a very different approach, but even I would argue potentially things like even the days of like MongoDB, right? Kind of cater to more like developers and what they care about with like having a very nice in easy to use data model and so on. So do you think that's still very true today? How do you see it? I'm curious.

Hannes (07:14.738)
No, I think MongoDB with all its insane amount of flaws did do did show us the way in some ways, right? Like it it it was I remember when I first looked at MongoDB, it was like you clicked a button on the website, you downloaded a binary, you made it executable and you executed the binary and it would just work.

It had a web interface coming up automatically. It had like a REST interface coming up automatically. It was in unsafe at any speed, right? Like it was unsafe. It didn't Fsync didn't have authentication. Whatever. But it might you know, who cares? I mean, these days no one cares anymore. I think this is really fun. This is like the most liberating time of data, actually. We can talk about that later. But like nothing matters anymore. It's beautiful. but

Nitay (07:43.022)
They would lose your data once in a while, like minor details.

Hannes (07:59.664)
But no, but MongoDB actually showed us how you do installation. How do you how do you do like, you know, zero to database? I think that that is still a really great example of of of how to do developer experience. And you know, we learned a lot from it. And actually, yeah, I think I think they did great on on that that front. I think what has changed a little bit is that that I think not so many people buy a closed source database system anymore.

I think that is kind of like I mean obviously Oracle will cons you know carry on till the end of time because databases never die. It's one of my strong beliefs, right? They will they will never they will never go away. And no matter how terrible they are. I mean ingress is still a thing. This is this is this is ins insane, but it's it's still everywhere. well not everywhere, in some places. but it's it's database never die. But I think these days

People kind of want something open source, or at least maybe that was true ten years ago. Not sure what's the case this year, but I think still something that's open source has a has a much higher chance. Like I I do I do recomm I do commend the people that start a closed source database startup in in 2026. I I'm I don't think there's many, but but some people try. I don't think that works anymore. And I think it doesn't work because nobody trusts.

closed source system and I think for good reasons. I mean anything I've seen from from from you know commercial software in a very much I have to say it air quotes in air quotes the commercial software is has been has been pretty dramatically bad right so so I get that why would you trust them yeah but I think I think it's an interesting discussion to see where this is going next but it it's it's still

Yeah, I think you cannot really get away with a closed source system anymore. People kinda wanna see what they're buying, at least to some degree. I mean, again, you could argue Databricks isn't open source anymore, even though they claim they are, which is also very funny. Like they manage this interesting sort of thing where they both can claim they're open source and in practice they are absolutely not open source. But that's a very different discussion. That's not gonna go there. Anyways.

Nitay (10:14.902)
Yeah. No, that's an interesting point. So there definitely a lot there we can discuss. perhaps tying this back to where where we started where said, so you wanted to build something where, like you said, kind of it has a better DX, better UX. Why can't you just go and slap that on top of an existing database, right? Obviously DuckDB has a lot of

differences in the actual like nature of of how you process in the data and so on, but also has a lot of differences in the usability and the DX of it. So why couldn't you just take the existing folks, slap on and say, hey, I'm gonna give you a nice new lipstick, a new UI, a new new API, a new whatever, here you go. Great DX. Doesn't seem like that works though.

Hannes (10:35.963)
Right.

Hannes (10:48.496)
Yeah, that's a that is a great question. and we tried that actually. so we took a code base of an existing database system called Monetib, that is still around, it's still being developed. and we try to make it sort of provide the experience that we that we wanted to to provide, like this whole concept of let's have a database that's in process, that's doesn't require server management, that doesn't require

You know, like setting up users, things like that. and we quickly found out that the like that there's a very sort of fluid border between technical and non-technical requirements. I don't know if you've ever gone to a software engineering course in university. I have. They told that at some point they told us about there's a strict difference between technical and non-technical requirements. And it turns out in practice those are kind of flowing into each other. So for example, like

If you have a database system and you say we are the best at data, but it's actually slower than our data frames on most things, which is by the way true for Postgres, right? Like our data frames will run, will probably be faster than Postgres on on analytical tasks on most cases, let's just say, right? And they're single-threaded and they're terrible, but they're still faster.

like you still need to have good performance, right? You don't have to win races maybe, but you still have to have good performance. So then you have a kind of non-technical requirement of like you want to be user-friendly flowing into the technical requirement of like you have to actually have a competitive engine.

There's also like really stupid things like hey, if you have an error, you shouldn't be exiting your process because that really interrupts the user if you just exit their Python process whenever you think something's wrong. so so we've we we thought about doing the lipstick thing, we tried it, and we we realized that it doesn't really work because the design param parameters of the system that we're envisioning were sort of

Hannes (12:47.524)
overflowing into the sort of engine requirements that we that we that we saw. And and so so that that was why we didn't end up we we didn't get away with window dressing. We we we had to we had to start from scratch, which which is really hard because we had this prototype it was really nice. it was really nice. It was somewhat nice. And then we decided to throw it away and start over and we knew we were only gonna be back at this exact point in like four years. And that is exactly what happened, right? Like

Maybe it wasn't four years, maybe it was three and a half years, maybe it's three years, but it was still very painful to set yourself back three years. It's something that I think in startups would have been impossible. But we got away with this because we were working at this academic institution here in Amsterdam at the at the CRI which is like a government research lab, which which kind of allowed us very graciously to do whatever the whatever we wanted to do. So that we decided to start a database from scratch. and they were like, okay.

You know, it's like they did tell us that it was gonna take ten years and take you a hundred people and ignored them of course, but they ended up being right. So that's that's a different story as well. But but I think to to to your question, I think the I think databases it's it's hard to fundamentally change something with with like you know, painting things over. It's it's y you cannot really hide it. May many people have tried to put

Nitay (13:58.349)
Yeah.

Hannes (14:16.016)
And your user interface on top of

And it's always been sad. It's it's never it never works. The the law of leaky abstraction will always apply to that. It's you will never be able to hide it fully unless you redesign the thing fully to to do what you want. Yeah.

Kostas (14:38.104)
okay, I want to ask you something about like the open source side of things because I remember like when I first saw DuckTB and I was already kind of experienced like with OLAP systems, like there was I think the main thing that was happening with OLAP systems that we were getting into the cloud, right? So we had the OLAP systems in the past that were

Probably you have like to pay, I don't know, like a lot of money to get the mainframes together with your lab systems and then fight with the lawyers of like Oracle and like all that stuff. Then we got liberated by AWS with Redshift, right? then Snowflake came in and then it like more into like a SAS kind of like experience and whatever. But when I first tried

DuckTB. I was like, that's like the first time that I can just download the Nolab system and like run it, right? I mean someone can argue about I don't know, like maybe Spark, but like Spark is not a Spark is a weird thing. Like it's it's a different type of system. Like you can obviously like query your data, but it's a little bit of like a different thing.

Hannes (15:49.65)
Your words.

Kostas (16:00.034)
we can talk more about that. so in the OLAP world, like we had to wait until DACTP came out to show to the world that like you can also have an open source and also like a system that you can download and just like run it in your CLI and do something in just like two minutes, right? Because before that we had I mean in the transactional databases, like Postgres has been like open source for like forever, SQLite that like I don't know, no one

wants to talk about SQLite, but like SQLite is probably the most successful database out there, like in terms of like how many installations there are of this like thing. and then you mentioned MongoDB, sure. but why it took so long for the OLAP systems to catch up with the OLTP systems in terms of like distribution in this case.

Hannes (16:55.802)
Right. I I think this is an excellent question. And I and I have have some thoughts about that. So I mean, first of all, SQLite says they have a trillion running databases, and there's no reason to doubt that number. So that's really that's really brilliant. Yeah, great job, Richard and team. I think the reason is that the like even if you're a small sort of organization, you still probably want a transactional database because you want like have a a a Postgres for your

ticket system or for your, I don't know, what has a small organization, has maybe an internal yeah, a ticket system. Let's just say a ticket system. You're a small team, you need a ticket system, you need something to communicate, to coordinate, you need a transactional database. You know the analytics part, you're already dropping off several orders of magnitude of sort of user base. Because when do you need analytics databases? Well, it means you need one if you have a lot of stuff.

And that usually means you are a big organization, right? Like I think there is very few like ten people companies out there that are sitting on like petabytes of data. It just generally does I mean, yes, there are, you know, data brokers out there. I know. But in general doesn't happen. You have to have I I don't know, you have to have a a a gigantic like running a a Facebook or you have to run a a a SAS or you know, observability company or something like that in order to have real need for analytics.

sort of infrastructure. And I think that Postgres isn't really great at analytics, neither is SQLite, but for the data sizes that you know your 10 people company can possibly accumulate, it's probably fine. It will not be great, but it's probably fine. So so the the the the demand is is is like I don't know, is it like an order of magnitude or two orders of magnitude lower for a purely OLAP system? Simply because the

Kind of organizations that need it are have to have to be much bigger. And then you have this thing that these organizations that need that, they have enough money to buy like a Teradata, right? Not a problem. They will just, hey, we need analytics. Okay, we're a thousand people company, we can afford a Teradata license, hopefully. and and so so then you're you're not in this like there's not there's not such a need. Like you don't have very few NGOs that need like an analytical database, right? Like this, maybe the UN.

Hannes (19:19.196)
But then we're already that's maybe the only one. I don't know. So so there is there's just I think the like the op from an operational needs perspective, you need a transactional database. There's really no way around it. Analytics, mmm, that's like that's like af comes after, right? Like people always say you you you you need this. But to run your company, I I would argue you can probably get away with Postgres for a very long time until this becomes a problem. I think this is why there is

been less. But something that really that really pisses me off in that world is that the options that were available in the past were Excel, you know, and Spark. Where there was absolutely nothing in between. You could either do Excel, I mean this is of course an exaggeration. I would say anything, you know smallish that fits in Excel is like one one sort of class of problem. And the other one is like once that stops working, you go to

You go to to Spark. And I always always thought you could also argue that the smaller class is anything that ha fits in pandas and the bigger class is Spark. But there was really nothing in between. And that was kind of a shame because it's such a waste, right? Like you whatever some pandas can be realistically do like a gigabyte or something like that, maybe maybe ten. And Spark to run Spark on anything less than like

You know, a terabyte is kind of a crime, right? It's kind of a performance crime. because you're los you're losing so much on on the whole distribution aspect that it's only really worth it until you're really running huge clusters on huge problems. and so there was such a such a huge sort of space in between that that was just left open in analytics. And I mean we are very grateful nobody did anything about it.

so but when we started sort of targeting that area, it's like it was it was like it was like it was very exciting because lots of people suddenly were very excited about it and you know we got lots of traction from it and and we started pushing on both edges. I mean that's also interesting as a development, right? Like we're like to be kind of s slowly moving towards transactions maybe and just to capture more of this bigger, bigger sort of

Hannes (21:37.5)
Hi.

Nitay (21:38.822)
And you said two really interesting pieces of insight there. Like one was that it's this interesting like mixer on the one hand, actually all the doll not all, but like a vast majority of the dollars go towards OLP, not OLT. But like historically speaking, just in terms of like revenue rate, look at Snowflake and then Databricks and so on versus like, you know, MySQL or whatever whatever like transactional system you want to choose. But

the other hand, because all those dollars were at the big enterprises, that there was this gap in the middle that you said where like the pain wasn't really felt. And on the other hand, there's like you reminded me of like you know the age old like cost paper or whatever it was, right? Where like one machine, even single threaded, can actually beat the vast majority of these distributed systems and so on. And I'm sure you saw kind of that same thing like you said with like DuckDB versus a s Spark or whatever, that actually the vast majority of use cases you can do with DuckDB.

Curious to touch a little bit more on the point that you said at the end there, which is like, as you guys were building DuckDB, how did you then enter that into the mindset of people to realize A, at the enterprise level, actually most of your use cases are not quote unquote enterprise? Like DuckDB is actually going to be perfect and even better than than the big solutions that you're paying millions for. And B, how do you cater to like how do you suddenly create that middle gap that that existed and actually make them realize like,

Actually analytics is not so hard. You can stick in DuckD B super easily and suddenly there's a whole new market here of these like intermediate players that suddenly do want to do analytics because it's just that easy and I don't need to go to Teradata and Spark and whatever.

Hannes (23:08.402)
Yeah.

Hannes (23:14.054)
Yeah. So I think I think the maybe the second one first, because that's it's in my short term memory and it's easier to think about it. so for ex I mean, how do you make people use software? I think I think you have to solve somebody's problem, right? And and and and you have to make it easy. I think that's that's that's just a fundamental sort of requirement. So we spent like one of our guys, Pedro, he spent years on building the world's best CSV parser.

Because we not because we care deeply about CSV files, but we realize that's the first thing anybody's gonna do with this thing. So we better have a good CSV parser. And I think we I think Petro has has managed beautifully. So so you can also help creating a category by just solving a a new class of problems that that was extremely annoying so far, right? I c I think parsing a CSV file in in in Spark or in Postgres is is both equally horrifying. And and

I actually wrote a paper about this at some point about parsing CZ files. It was the first ever scientific paper about parsing C Z files. Very proud of it. And the other thing, how do you how do you you know how do you create this gap, maybe, or how do you make people think about it? It's it's like it's not it's not that we went out and did PR or anything. We don't have money for that. Like we don't have we don't have PR money. it's just it's just

You know, you start solving one problem, and then somebody else says, Hey, maybe this somebody talks about how you solve this one problem. And then somebody else goes and says, Hey, maybe the solution that went for this worked for this one problem that's this different from mine, but similar to mine enough that I can maybe apply the solution to it. So I think we started with something very specific. We started with large-scale survey analysis. Okay.

which is this problem from statistics. There is a survey in the US every couple of years. Every American gets surveyed, I think. You are you you live there, you tell me. I don't know. But there is just it's just a data set that used to be very painful to work with. and so we started we worked with survey statisticians to solve this one problem. And we we got into this whole statisticians world, right?

Hannes (25:28.122)
And from there it's kind of slowly built out, right? Like it's it's it's a very it's it was a very sort of gradual sort of process, sort of one convincing sort of one person at a time, and then then that's that person maybe told somebody else. It's not like we went out and purchased a a billboard on the 101 like some of our competitors maybe.

it really isn't that, right? Like it was really hey, it's it's we have to go one at a time. And we we really focused on these on these early sort of interactions where if somebody opened an issue with us, we say we'd really take it seriously, we'd follow up, you know, we will they do everything to see what is the problem, even if we couldn't do anything about it at the time, but we'd really try to understand what the problem was. And we've done this now.

for so many for many years. Like I think this this has been going on now for for for eight years or something like that. which is which is in my sort of twitchy world, this is a very long time, to to be iterating on GitHub issue pops up, engineering thinking.

implementation testing in you know integration and then you just do that and you have hundreds of issues, thousands of issues that tens of thousands of issues that you do this with. And yeah, which is why I'm glad that we have a team now that also helps with that. But it's it's really it's really a sort of one step at a time approach. it's not something that you suddenly so show up and you scream from the rooftops and and everybody's gonna believe you now. I think people have been burned thoroughly.

in this space as well. so so it's it's been it's been a process.

Kostas (27:05.483)
So okay, you are in academia and you are working on databases, and then you decide at some point to open source something, and not just like open source it. let's say like a lot of things like get open source like from academia right now, primarily because like there is like a demand for people like to prove that the things that they are claiming can be reproduced. Now someone can

Hannes (27:09.692)
Parsley. Yeah.

Kostas (27:33.738)
argue how easy it is to do it. But anyway, that's different conversation. But but why you didn't stick like in the academic world? Like you build, you can do your publications, which is what academia cares about. But instead of that and by the way, it's not like the first time that okay, let's say especially databases, like data based groups, they commercialize something. But usually

Hannes (27:38.577)
Ha ha.

Kostas (28:02.668)
They don't go like through open source, like at least like from the teams that I know that I've seen, okay, we have this new crazy thing that does things that like nobody else can do and we are the, you know, the wizards here that they know about that stuff because databases are hard. let's go and sell it to Oracle or sell it to I don't know, whoever. but you decided to go like on open source, right? what exactly was like the reasoning behind that? Like what

Hannes (28:25.906)
All right.

Kostas (28:31.616)
You wanted like Tarty. What did you expect from that?

Hannes (28:35.026)
Well, I think I think there's three reasons here. so I think that it's true. I could have I could have very easily stayed in academia and and sit on my chair and yell at the students and write yet another paper that exactly nobody's gonna read.

I think I think for me, I realized at some point that my definition of definition of impact differed a little bit from everybody else's definition of impact. So for me, impact is making a change in the world. Right? Not citations. Citations is a terrible proxy for that. citations just means that, you know, like you have enough buddies that you went to to drink beer at conferences with.

which is I'm not opposed to that by the way. I love drinking beer, don't get me wrong. but but it's it's not it's not what I want my my my work to to result in. So so for me and and Mark, it was always clear that

in order to make an impact, we we have to we have to get this in the hands of people because we've always already like you said, Costas, we have seen so many innovative projects disappear in non-caring corporate sort of hell holes because the team sold the idea, sold the team, I don't know, to someone that didn't know what they were what they were buying, they didn't have a place for it, and just slowly died in in in in their sort of you know silo or whatever it was.

So we didn't want that. That's not impact. That's just that's just sad. Okay. So it's impact. It's the second thing is it's so Nita is gonna love this because it's the opposite of of venture capital. It's like venture communism or something, is where we say, hey, we built this with tax dollars. So as a result, the people who have paid these tax tax dollars should be able to benefit from it.

Hannes (30:29.698)
And and I think and I had actually had very interesting conversations about this with somebody from our lovely government here in the Netherlands. They Yeah, but it doesn't matter because whatever you do, you're gonna pay taxes anyway. And I said, Yes, this is true. however, it's fair point. However, I I just didn't see that I just I didn't didn't want that that the only result on on on the world that is the economy grows by you know point point zero zero zero zero.

But no, I think I think this is really a for me, I was brought up with this like, hey, anything that that that you do based on government money is being paid by people that, you know, that had to work really hard for that money. And that and they had to give part of that to the government. And that's the money they're spending now. You better make it count. And I know this very, very unique position and viewpoint in the in the academic sector, but I have to hand it to my dad who who who

Who insisted on that point, who is by the way, not not a public servant. He is he's a you know in the in the private sector, or was he retired now. and to just make sure like whatever you do from this money needs to be accessible to the people that that that pay it. And I mean, yes, in my case, it's the Dutch population. By the way, I have a very ironic story here. So we build this thing from Dutch taxpayer money, yes, in the Dutch research lab.

And who is using it is the Americans and the Chinese. It's like we have zero like zero users from the Netherlands in the grand scheme of things. It's it's it's like it's like the the Europeans have this weird thing that something has to first go to the US and then come back to them before they believe it's good. It's but of course the Americans and the Chinese don't have any such problems. They do believe the European stuff sometimes. It's it's funny.

No, but so that was that was the second thing. And the third thing, why did we make it open source? I think it's also just like it was the it was the right strategic move. Simply, right? Like as I said earlier, like as is anybody gonna buy a closed source system from some academics from Holland? No. Is somebody gonna buy a system that has a million downloads a day that's you know running millions of queries every second? Maybe, right? Like I think I think that's that's that's a huge difference in credibility.

Kostas (32:49.056)
Yeah.

Hannes (32:49.498)
If your thing has been battle tested by everybody and their little brother versus you and you know, that there's like you're the second customer for this thing that has ever run this in in anger, that is a very different proposition. And I think we have we have shown that we can run this, as I said, like you know, millions of queries a second with very, very reasonable rate of issues being reported. So so that's that's that is a a very different sort of perspective than your

you know, your like single customer or commercial piece of software. So we thought about it. But in the in in reach even in retrospect, it's actually funny because I had dinner with Mark tonight. and we were like, is there anything we should we would have done differently in the last couple of years? And we're like, no, no. Even with hindsight, this is exactly what we should have it's kind of funny, right? Like this is this very, you know, at least very self serving. But

But but it was it was it was it was we we were thinking about it. It like, was there anything that we should have done differently? It's like mm, no. This is exactly what this was the right call and we still think it's the right call.

Kostas (33:57.048)
All right, that's great. So okay, you you open source the project. And okay, I'm sure like okay, just making like a public GitHub repo, like usually nothing happens. Yeah, exactly. And to your point, like you said, like we're talking about a system here, like okay, like OLAP systems it's like a fraction of the size of you know, like the demand that exists out there, like for OLTP systems, right?

Hannes (34:12.08)
Yeah, that's true.

Kostas (34:26.402)
Plus, we are talking about like something completely new as an idea, like an open source all up system. Like, okay, who was ever like looking for that? Like the market was not like educated for that. so okay. Do you remember how you felt like when it went out? And I assume you don't learn in Dutch universities about distribution of software. if you do, let me know. I would like to attend to that. I could

Hannes (34:49.948)
Ha ha ha.

Kostas (34:55.478)
get the help I guess for what I'm doing in life but tell us like how how was like these first I don't know months or whatever that like you do that.

Nitay (34:56.854)
Yeah.

Hannes (34:57.216)
man.

Hannes (35:05.776)
Yeah. Yeah. Yeah, this is a great question. And and I I mentioned I mentioned before we had this prototype before. And so the prototype wasn't in the end we abandoned it, but it it got some people excited. And I I I think that meant that we we had some idea that what we were that this was going in the right ext in kind of the right direction. So when we when we finally open source DuckDB

I remember this because it was at Sigmod Amsterdam that actually I co-organized this conference. So I was like, aha, I have access to the sponsorship you know bags. I'm just gonna throw a sticker into every single one of them and didn't pay for it. and we had a demo at the conference.

And I mean that I think in the end Sigmod isn't, I mean, don't tell the Sigmod people, but I don't think there's like any sort of real-world software can be st launched at Sigmod and make it into the world anymore. That's just over because the conference has been abandoned by the customers, as as Stonebreaker has famously complained about. but I think what what happened is we we launched it and

We had built some visibility in this in this stats world from our previous exploits with them. And so that was I had 400 Twitter followers at the time. And that but there were some relevant ones. And I wanna, you know, I wanna highlight people like Hadley Wickham from from the, you know, from Posit and and Wes from Pandas. And there was like some people from this data science world that that were that were already sort of that, you know, I was in contact with and talking to.

And then then maybe they boosted it. And so this went this bubbled up a little bit. And then a couple of weeks later, I think there was the and I it's really super sad, but there was the first somebody posted it on Hacker News with this horrible title. It's like it was like SQLite but with Postgres syntax, DuckDB or something like that. It was like it had didn't say anything about what it was. It just said, yeah, it's like SQLite, but it has Postgres syntax, because indeed DuckDb has Postgres like syntax.

Hannes (37:17.062)
Didn't matter. That got us our first thousand stars. And that that has been sort of, you know, and that and then from there on it was kind of it was kind of done in my in my experience experience. Obviously this wouldn't have worked if there wasn't like a underlying sort of demand for it, right? Like we couldn't have just blitzed the world with some PR and then say and then somehow that taken out a live on its own. I think I think there was an under under sort of under current of

demand for that that people just didn't know that they had once this bubbled up on Hacker News, it started sort of this life of its own where, you know, where where and now it's yeah, there's still still some people sometimes that say, hey, I've never heard about this. But it's it's it's getting a bit rarer, let's say that that so I I think it's it's been it's the the the thing is there is no course for it at Dutch universities. I I hate to break it to you, there's no course.

But but I don't think I could repeat it either. You know what I mean? Like let's say you give me a million or like okay, a million. Let's say you let's say n Nita gives me a hundred million dollars and says, Hannes, I want you to build the duck db for I don't know, what is cool? AI. And then and then I I should do basically if I would try to do the same thing again, I'm not sure I could do that. So so it's not is I I I I I I'm not sure I have this.

I have this sort of deep, deep sort of wisdom here to for people to do it again. Except for what I said earlier. It's like, hey, you know, maybe start solving people's problems. Like find so and so one of the things I tell people is like go on Reddit and look for like star sucks or something like that, right? and just find out what people are struggling with. Like, what is it that actually the problems are? And and and then then start from there, like.

I remember I was I I saw Parquet sucks some at some point because it at the time if you wanted to read a parquet file, it had you had to install Hadoop. Was an is an interesting proposition, right? I mean it's I would rather do many things than install Hadoop on this planet. and I got I and I've done I my god. but that was the that was that was the story, right? We said, hey you want to use parquet files, go install Hadoop. It's like,

Hannes (39:41.509)
no, right? And so we actually ended up I actually ended up writing one of the first sort of standalone C parquet readers, which added preductively. It was something we integrated in R and Python and people loved it. Right? It's still used actually. It's people still use it. It's called it's it's called nano parquet now. It's it's a it's a it's a it's a it's a huge project. It was a fork from my stupid

thing I built back in the day. And then I think and that was triggered by people complaining about hey XYZ ZAX, right? So so it's so if I think I think that that is a great place of start to start is to is to look at what what is difficult, right? What is the I mean, where do people struggle with and what do people complain about?

I mean we did a whole project in Duck TV where we we we we attacked the SQL socks screamers out there, right? And say, Okay, not attacked. That sounds very aggressive. Addressed. addressed and said, Hey, okay, they said, you cannot do freaking trailing commas in SQL. It's use this language like, okay, fine.

You can do trailing commas now in in a sequel if that's what you want. And hey, people love it. So so it's it's it's interesting to see that. And I I think I think I would every every PhD student in in in data, I as you know, I would say go go on Reddit, you know, find out what people struggle with. It's it's it's it's pretty straightforward concept.

Kostas (41:08.752)
Yeah, it makes sense. I I love the comment about parquet and Hadoop because I guess it makes sense like if you want to access a file like to need the file system also installed first, right? So they took it like l extremely literally like the Hadoop ecosystem. Install HDFS before you can do it like Parquet. So

Hannes (41:20.57)
And a zookeeper service.

Hannes (41:29.174)
Insane. And you need MySQL, by the way, because you need a you need a in a catalog server for something. Like a hive catalog thing or something. You needed MySQL. You dude you do you needed my SQL for something. I'd forget what it was, but it was it was it was. Hive! There you go. Okay. Because of course.

Nitay (41:36.482)
Yeah.

Kostas (41:36.982)
yeah.

Nitay (41:40.758)
My MySQL is the backing store of Hive. Yes, yes, yes. yeah, yeah. It was it we might it might as well have been like here's the motherboard, like good luck. Like you're on your own.

Kostas (41:54.232)
Yeah.

Hannes (41:55.187)
It's also a bit embarrassing to say here's my database-like thing, by the way it needs this other database. I mean you have no pride or something? I don't know. It's like it's like

Nitay (42:03.406)
Yeah.

Kostas (42:05.6)
Yeah, it's interesting, Nitai, because actually when I when we met with Hannes like a few weeks ago and we were talking, I don't remember like Hannes, I was telling you that there is this thing of data systems being built by distributed folks and not database folks, and that's why we have the systems that we have today.

Nitay (42:23.235)
Yeah.

Nitay (42:28.374)
Well, there's that there's an even worse version of that, excuse me if I if you don't mind me saying, which is data systems being built by not data folks at all.

Hannes (42:41.824)
yeah, I mean that's this is better than distributed systems people. Like maybe there's a hierarchy here. Okay.

Nitay (42:44.566)
Exactly. Exactly. Exactly. So the human systems is always a step up. But but I wanna okay. Go ahead.

Hannes (42:51.792)
I thought I cla a cla a clever person who I cannot name by name recently told me, listen, Hannes. The database people have always sucked at scaling. The distributed systems people have always sucked at data, but they could scale. And I think I and and so no so far nobody has no database person has managed to scale, so that's sort of the open thing, right? Like like what like that b and that's the reason why everything is like built by distributed systems people.

because that's the thing, you know, it might suck terribly at at any sort of you know, like at join maybe, I don't know. but it didn't you could throw a thousand ten thousand computers at it, it would somehow crunch on. And somebody somebody like Jimmy Lin, he's he did a bunch of Hadoop work early on on on in information retrieval.

He told me at some point like about Pig Latin. I don't know if you remember Pig Latin. I remember Pig Latin. You remember Pig Latin. Okay. he said, like, yeah, you know, all your all your you smart database people with your SQL. And so, you know, I don't need you because I can throw whatever I want at Pig Latin and it will take forever, but it will finish. And I think I think I think that is something. That is that is like an idea that that it it will finish. Is that that it's actually was the inter inst inspiration of one of our

of of what of our like long running research projects in DuckTB, which is the the the GalaxyQuest concept of never give up, never surrender, right? Like you will you will you will you will you will we will just we will rather rather you know we will always finish that query no matter what. And I'd say one of our guys, Lawrence has been working on this for for years now. and just just going through all the operators and be like, okay, what happens if this intermediate result becomes too big?

Okay, how can we use the the disk for this? What does this mean if it's an SSD? How does this change from the common system of s you know spooling, spilling to disk, you know, yada yada external joints, whatever? And I I think that has been extremely interesting because I think that's we are now the I think the only system out there that has anything on this. Because to my absolute amazement, all other people, like the research and practice world, has been perfectly happy of just having systems crash.

Hannes (45:11.674)
When the intermediate would run out of memory. Like SAP HANA. Is like has always been like you just need to buy more memory. Just buy more memory. And and and we I've for some reason we've been the first people that went, hang on, you know? There needs to be some middle ground between you running an external sort single threaded with Postgres and you just crashing your your high powered

JIT compiled analytical database system because you couldn't fish your fit your hash to a memory anymore. But like again, like like why has why do we have sometimes it is it what I'm sometimes a bit grumpy about maybe is that why do I, the outsider, have to be the person that yells is at the database people, right? Like I I still don't understand that. Like how has have they been doing things for all these decades without realizing I actually can't crash, you know?

This is not acceptable. It's not even it's not even it's not you know it's just not okay. I mean sorry, minor rent. It's

Nitay (46:16.834)
No, no, I think I think it's exactly that outsider perspective. I I think it's very much the operating mode of most data. Many systems I I've worked with and know personally, to your point. It's it's just the facto like if a query fails, okay, you'll try it again later. You'll add more memory, like you said, you'll try tomorrow. You lost today's data, you get the result tomorrow, it's OLAP, okay, who cares? Like it's fine. So so perhaps tying that to to what you were saying about like star sucks.

Hannes (46:38.224)
Yeah, yeah, yeah. Yeah.

Nitay (46:43.746)
Kind of what what sucks today and in particular maybe tying this to you guys have a lot of exciting things happening around Duck Lake, around Quack, around like there's a lot of interesting developments. So so tell us a bit about kind of what sucks today and what's what's how how you're looking at it.

Hannes (46:51.333)
Right.

Hannes (46:55.066)
Yeah. I mean I I I also I I wanna I wanna say we have we we we tr we tried really hard to have a friendly outside perspective and and so when I go around XYZ sucks, I don't I don't wanna I I don't I'm not gonna name I'm not gonna name people because I I don't wanna you know, positivity. The database world has been very grumpy for a long time. We try to change a little bit. That being said. So I think I think with

We have worked on client server protocols actually before DuckTV because it was one of the reasons that people didn't like database systems for was that it was extremely difficult and slow to move people back and forth. So we said, aha, it's not gonna do that. We're just gonna have an in-process database, copying SQLite, learn from the best, do it in process. You can do anything there. And we we've actually pushed this pretty far. Like you sold all solved all these data science use cases, it works really well. we then

Started looking at data lakes, and we ended up with this concept called Ducklake, which is, I think was released like a year ago only, where we said, okay, data lakes, this idea of having a zoo of bucket files, maybe the metadata should not be in Avro files, which is the worst format on the face of the earth, but maybe it should be in a database. I always I I like the expression like we have a database and we're not afraid of using it, or something like that. and

And so that's what we started to work on. And then we realized that if we want to use DuckDB to be this metadata database, it doesn't work if you have more than one computer because it's an in-process system and it works really well on one computer. But if you have others want to talk to it, it's it's it's you can end up with these sort of arrows that lead nowhere. And so we ended up building a client server protocol called Quark that basically allows us to

use DuckDB as a remote system. and one of the cool things I think it's pretty unique is that it's DuckDB on both sides, right? Normally you have a database, it's clever, and you have a Duck client that's very dumb. I think that's pretty standard everywhere. But with Quark is the first time that both the client and the server they are DuckDB. And so you can do kind of choose where you want to do query processing, you can do it remotely, you can be locally, you can do a mixture. It's it's it's quite cool, I think.

Hannes (49:21.082)
I think nobody's done it. and at the same time we had of course we wrote this paper about how terrible client protocols were and so we had to kind of spend some time on making sure that our sure that ours wasn't as terrible as everybody else's and so yeah, we we we optimized it a bit and and it was it was really fun. It's it's I think it's right now it's it's a very competitive sort of client server protocol. You it can it can sort of

without breaking a sweat, ship millions of of rows around even over slow network connections, it's it's not a problem. And that's I think that's that's again, that's like maybe again to to to come back to the to distributed systems people that now write everything to to like a distributed file system or an object store, right? In order to to coordinate. Now we've we've kind of shown yet another perspective that hey actually we can do a an RPC with millions of rows without too much trouble. So why are you

writing everything to files all the time in the first place, right? Like it's an interesting question. But but that's that's it that was it was hard for me to to to admit that I was wrong about client server. Like we always said we're never gonna do we're never gonna do it, we're never gonna do it. And then we had to we had to kind of be like, okay, here's our client server protocol. And people were like, what weren't you the guys? Somebody on Twitter said guys that's famous for hating client server protocols builds client server protocol as a response to our announcement post, you know?

and I thought, fair point. You know, it's like I was not I wasn't even mad. I was like, yep, that's exactly what it is. Thanks for pointing it out. But yeah. It's there's lots of I mean, this is also the cool thing about having having like this this ecosystem, right? Like DuckDb has become this ecosystem where there's like hundreds of extensions, there's hundreds of or thousands of projects that embed DuckDB in some re f in some way.

There's all these distributions, all these integrations. There's way more happening in in Duck D B land than I can kind of keep track of, which is which is which is kind of crazy. Like you know, I mean I every day I go on I go on Blue Sky, the the world's most insuccessful compu you know, social network, and I find something new about DuckD B and I'm like, Great. You know, it's like it's it's it's it's it's kind of wild to see that. But it's it's also gotten alive on its own. If of course, you know we can can only

Hannes (51:45.702)
Wait, smile and wave.

Yeah.

But yeah, it's still going on. Sorry.

Nitay (51:51.702)
Tell us bit about you. You mentioned some of that that you were you guys were taking kind of a slightly different take on on client server and that you had kind of learned from from some of the past. And tell us a bit about like what what does that mean?

Hannes (52:03.714)
from Client Server. So so the client so this is this is coming back to our seminal paper. no, but basically client server protocols. I I looked this up, they came about in they started being a thing with Cybase apparently in 1984, roughly, which is why I was born, full disclosure. and that's kind of the time before TCP. so these protocols were kind of designed to work on a

You know, whatever they always say a w a wet shoestring, like a network connection that wasn't really very good. with lots of packet loss, with like no really guarantees about order and like basically a you know, a a UDP connection over over Wi-Fi or something like that, right? and as a result, they they they had to basically reinvent a lot of things that TCP gives you for free. and especially things like the Oracle protocol, you can really see that. That it's like it's basically retrofitting a lot of these things that just didn't exist when the

database came first came around. So so we had a big advantage of of designing something in two thousand twenty six with without any of the now forty years of historic baggage of protocols that all you know like if you need to be backwards compatible on the wire, like that's a terrible, terrible job. But there's like a really real reason to not break wire protocol compatibility because you have this world rollout problem, right? Like you have like

Cannot roll out your client and server at the same time. Your client's probably gonna run for 10 years without being touched. You don't wanna break it, you don't wanna make the database stuck in time because the clients can't be updated. Like you have all these wonderful problems, and I can already see thousands of Oracle programmers screaming in the background. but yeah, if you don't have any of these restrictions, you can kind of just go and decide, you know, design from from what what is it in today that we want in a protocol and

I mean it was pretty clear very quickly that we needed to di have this to be on an HTTP because it's like there's a fun fact here that routers will actually be like all the hardware is actually better at handling HTTP traffic than anything else, just because it's what everything is optimized for. So if your protocol some f somehow fits an HTTP, it's gonna automatically be faster. It's great. and yeah, so so that's we wanna make it work in the browser, we wanna make it work you know, encrypted, we wanna make all this sort of stuff.

Hannes (54:25.402)
And it's all just there. It's just a it's just a protocol on top of HTTP with some custom serialization optimized for as few as possible round trips because those get really penalized when you have lots of latency and just a good way of of shoving bulk data over the wire because that's just that's something, for example, that the Postgres protocol really, really suffers from. like I don't know if you have a look at the Postgres protocol, but I have.

And in Postgres every row in a table can have a different schema, and you think in on the protocol level, right? And you think, why? And the reason is it started live as an object oriented database where every value could have a different type. And because of that, the Raya protocol allows to encode a different type for every row. every every field, not just every row, every field. Which of today today nobody uses this. Like if you have a

you retrieve a table where you have a column has a type and everything in that column will have the same type. But they never changed the protocol for this because of the described reasons. And as a result they have like a factor like for simple tables they have like like a factor factor two overhead over the actual payload data or something. It's insane. Like they just shove so many stuff, so much real stuff over the wire. And of course you cannot win from this. So that's that's one of the things that we were able to do better.

and yeah, it works. I mean people people are gotta got excited about it. It's actually it's like a couple of like three weeks post release or two weeks post release or something as we're recording this. And now I've been really I've been really excited about the the the the the uptake. Like people have written so one thing is really funny. We wrote this thing so that we could have cli the client on the s like I mentioned, the client and server are both duck T B and I was like, Yeah, Duck T B runs everywhere, so why would you ever you would just

You wouldn't need a separate client, you just install DuckTB and use it to talk to a Quark server. But people were like, no, no, no. And there's already like tons of GitHub projects out there that basically wrote a without DuckTB client for the Quark protocol. And you just go like, what? This was not meant to exist. We have never actually made any we've never tried to make this easy to do. It's it's using our own internal encoding, which is like, you know, internal.

Hannes (56:50.47)
But there's that didn't stop people from building clients for it, for example. So there's already a bunch of clients, native clients for Quoc that don't have duct v client side, which is kind of crazy to see that. So maybe I was wrong again. But that's fine. we just try to solve people's problems, obviously.

Hannes (57:12.166)
But yeah, it's a fun it's a fun world.

Nitay (57:13.582)
Yeah, it always amazes me when I mean this is kind of the the the the beauty and gift of open source is you get to see all the amazing but also all the terrible ways that people use things and you're always always always surprised in both directions. okay, so yes, absolutely.

Hannes (57:23.439)
Ha ha ha.

that's true. Yeah. Yeah, we have a maybe an anecdote on that. Sorry, one second. Like we have Brazilian laptops, is one of our favorite the targets, right? Because Brazilians live in a warm place. Their laptops they tend to be hot. and as a result, they got a lot of bit flips in the memory. And there's lots of Dug TV bugs that are extremely inexplicable. But then we ask them, Are you a Brazilian? and they say yes. And then I mean.

We say we don't say are you Brazilian? We say can you measure your laptop temperature, please? And then it's pretty clear that it's pretty clear that that is the problem. So that's the kind of issues you get with open source indeed.

Nitay (58:08.054)
That's amazing. They can be like a test bed for all the computing in space where you have like radiation bit flips and everything, you know, if they could be like the the test center. fantastic. Okay, so we're we're coming up at the end here. you know, we didn't we didn't touch at all on kind of the the current age of AI and so on, which I'm sure you have many topics on, so we'll have to do maybe a whole nother episode just on that. But just to to close us out, anything else in terms of kind of where you'd like to see the ecosystem going, future of of DuckTB in the community? Any any last thoughts?

Hannes (58:16.304)
Absolutely.

Hannes (58:38.75)
I I think I think it's pretty clear where we want to be going. I think we I think for for DuckTee the the the the hope and dream aspiration, whatever, is that it just becomes a default that people just, you know, grab it without thinking about it and we are working very hard toward it and we you know we would love to see that. in terms of the the the community, I think it's it's it's a very, very excited like you said, it's a very exciting time to to see like the the people that are that

The entity that interacts with the database changing from humans to machines. And I don't think anybody's really figured out yet what that means, what that means for like for query optimization, for query languages, for error reporting, for you know, like all these things that that that we thought about when trying to make a great developer experience. Now we have to make a great machine experience, which is which is a very different problem. And that's exactly what we're kind of working on right now, as you can imagine.

Nitay (59:39.276)
Okay, well we'll definitely have to cut cover that in future conversations. But with that, thank you very much, Hannes, for joining us. This has been an absolute pleasure.

Hannes (59:48.572)
Thanks, Nita. Thanks, Kustas. Thanks for having me.