Feeding the Agents: Fast Structured Data Retrieval with Arrow — Ian Cook, Co-founder & CEO of Columnar

Kostas (00:00)
Ian, hello. We are so happy to have you here today. We'd to hear a little bit about yourself before we get into the very interesting topics of ADBC.

Ian Cook (00:12)
Thanks, Kostas Nite. great to be on the podcast. Yeah, I'm Ian Cook. I am the co-founder and CEO of Columner, which is a data infrastructure company that is trying to make database connectivity much faster and easier than it's been in the past. So a little bit about me. I grew up in New York. I went to school for statistics and then continued grad school for statistics.

Really fell in love with the R programming language. So that was for many years that was my that was my programming language, kind of got involved in that community as a a package maintainer and got got very good at R. But I moved after grad school to the Bay Area to work at AMD as a software engineer statistician hybrid type of role. and at that point I re I've really fell in love with data infrastructure side of things, realized

I enjoyed building the tools and maintaining the databases and things like this kind of more than I enjoyed the analytics. that started a long period of interest in tool building, which brought me to a couple of software vendors after that. I worked for a couple of other Bay Area data infrastructure software vendors, Tibco and then Cloudera for several years during the heyday of Hadoop.

and at Claudera I met Wes McKinney, who was on your podcast a few months ago. Wes needs no introduction for people in this field, but he's the the creator of p pandas for Python. And this was a few months, years before Wes created Apache Arrow. so I got to know him a little bit and saw kind of the genesis of the Apache Arrow project. So one thing lets you that I ended up working with Wes McKinney at a couple of companies.

I got deeply involved in the Apache Arrow open source project. And then a couple years ago, we had an opportunity to start a new company around a specific sub project of Arrow called ADBC. And that's what caused me and a couple of other folks from that Arrow project to found Columner.

Kostas (02:11)
That's amazing. for the people who are not aware of, what is ADBC?

Ian Cook (02:16)
Yeah, so ADBC stands for Arrow Database Connectivity. it's it's a it's a little bit of a mouthful. people want to say ABD ABCD, but you know it's ADBC. So if you are familiar with the way that a lot of applications connect to databases, you've probably heard of ODBC and JDBC. These are standards for database connectivity that were created in the nineties.

And they have stuck around since then and become very widely adopted as a sort of standard way to connect apps to relational databases, data warehouses, et cetera. those were created in a different era. and so with ADBC, what we are doing is dragging them into the modern era by ripping out the slow, inefficient way that they move data and replacing it with Apache Arrow.

Kostas (03:08)
Okay. A few questions here. So first of all, let's talk a little bit about the relationship between ADBC and Arrow because Arrow, started as, I mean, it's more of an ecosystem probably like today. But at the end of the day, Arrow is primarily a way to store data, columnar data, like in memory, right?

So it's pretty, I would say like low level kind of thing of like how to organize information and put it together. So then you can build on top of it, database systems like probably. But when we are reaching like the level of like connectivity, how a client can connect to a database, for example, right? There's a whole stack that needs to be built on top of Arrow to reach the point of

having this need, right? And I can see why JDBC and ODBC became a thing. JDBC from the Java ecosystem makes sense. Like you built this whole thing. You definitely want your Java developers to have like a way to connect to databases. ODBC, I like the same thing, but like from the Microsoft ecosystem. But here we start like from...

Like the needs kind of like emerged like from a different layer, right? Or it is associated at least like with a different layer, like in terms of like the projects. Tell us a little bit about the story behind that. How from Arrow and how we efficiently stuck strings together in memory, we end up like wanting to build something like a DBC.

Ian Cook (04:46)
Sure thing. I will go all the way back to 1992 and give a little bit of history because I think that helps make all of this pretty clear. So in 1992, that was when ODBC was created. And ODBC was the first in this standard set of relational database APIs that attempted to sort of standardize the way that applications connected to databases. so

In nineteen ninety two, yeah, a group of of companies, including Microsoft, Simba, and a few others, got together and created created ODBC. But at the time, all databases were what we now would call transactional databases. They were row oriented, and they were not

designed for like large scale analytic queries. so if you think today about databases like MySQL and Postgres QL, these are internally row-oriented databases that are very efficient for inserts and updates, less efficient for large scale queries, analytic queries. and internally they organize data, yeah, in rows. and in in the 90s all databases were like that. So of course

When ODBC came around, it also used a row-oriented format to move data from those databases into client applications. fast forward to the mid-2000s through the 2010s, there was a total transformation in the standard architecture of anal of database systems, where this column orientation became ubiquitous in systems of that type. So

you know, starting with Vertica, which I think was one of the the first mainstream column-oriented databases today, like Snowflake, Databricks, ClickHouse, every large-scale database you can think of now uses columnar formats in order to accelerate analytic queries. And just for the intuition behind this, like why are columnar formats and columnar architecture more efficient for analytic queries?

typically tables in databases have many columns, but typically queries have touch relatively few columns. And so if you have to jump around between many different columns or scan through a bunch of data that you're not querying, that's very inefficient. So there was this this finding that if you array the data in memory on disk in columner orientation.

That it's dramatically more efficient for a lot of analytic purposes. so that was kind of the the reason behind columner systems. But all these column all these systems, these database systems started adopting this column architecture, but the connectivity APIs between them and the client-server protocols that actually sent data over the network remained the same. So they kept this row-oriented.

pattern from the nineties. And the reason why is just because there was so much inertia behind that. There was so much adoption of these row-oriented standards like ODBC. the network effects were very strong that even though it emerged as a bottleneck, people kept using it. So ODBC continued to be used very widely and is still used widely today. somewhere in the late 2010s, a a few

the Apache Arrow project emerged, and like I said, ADBC is part of the Apache Arrow project. But the Arrow project emerged for a couple of different reasons. One, as you said, Kostis, you know, it's a in-memory format for storing data in a columnar columner orientation. before Arrow, the parquet file format started to become popular.

And this was an on-disk format for storing tabular data in columnar format. but parquet was not really able to be widely adopted or as successful as it could have been until it had an in memory counterpart. So like think about it, if you're loading a parquet file, it really helps if there's a standard in memory format that you can deserialize it into. So then once it's in memory, you can benefit from the same standardization.

As when it was on disk. so that was one of the reasons why Arrow arose back then. Another reason was systems like Apache Spark were you know, some of the earliest systems to create like one single engine and then create a bunch of different language APIs on top of that. So Apache Spark natively, it's written in Scala and Java, JVM languages, but over time, like

Py Spark became a very popular way to use Spark. And moving data sets, tabular data between the Python layer and the JVM layer was very inefficient. So we needed a standard format in order to kind of move tabular data between different language runtimes. That was the second reason why Arrow became very popular. So Arrow was founded about 10 years ago

and started off really low level, kind of solving those those problems which were mostly unique to like on a single computer, just moving data between languages, between runtimes more efficiently. But then it grew and grew, and it turned out that it was also a very efficient way of moving data between processes and over networks. especially as all of these column oriented databases sort of went into to broader use, having a columnar format in between them.

became a really a great way to speed up interop between applications and and databases.

Kostas (10:10)
Okay, question. So, I understand the difference between like a columnar representation and why we need that, compared to like going through like rows. But here's like where I have a little bit of like a difficulty to understand the connectivity part why it needs to be more native to the columnar paradigm.

One of the reasons that we need like a columnar representation is because usually when we run like analytical queries, right, we go take a whole column and do an operation like to the whole column, right? So instead of going through the rows and extracting the column that you need, it's obviously like much more efficient to go and get like the, column. Now when a client connects to a database, right.

connects there with like something like JDBC connects there like emits a SQL query. The database does its magic, whatever. Doesn't really matter if it is, let's say like a transactional database or like a columnar database. At the end of the day, a relation is generated, like a table is generated that you need like to emit back to the client and the client is going like to present that. Right. So at the end of the day, doesn't

in my mind, least, unless the client needs to do some additional processing that can benefit from one paradigm or the other, or something happens on the wire while the data is transferred. It doesn't really matter what the database is doing when it goes to the client. So why ODBC and JDBC being row-oriented?

are not enough, are not good enough for connecting to Clickhouse or to Snowflake as a client.

Ian Cook (11:58)
Yes. One of the key things to understand is that it's not just source systems like databases that adopted this columnar architecture for storing data. It's also destination systems. So if you're pulling data from ClickHouse, but you're pulling it into Power BI, or you're pulling it into Tableau, or maybe you're pulling it into a local in-process engine running on your on your laptop for further processing, like Apache Data Fusion or DuckDB.

All of those systems also represent data in columnar layouts internally. So by putting data into this row orientation, when you move it from a system like ClickHouse to a system like Tableau, you're basically converting the data from columns to rows only have only to convert it back to a format that is almost identical to the origin format. and what that does is it makes the whole process of extracting data from a database.

Not actually network bound, but CPU bound. So if you have a gigabit connection, you might wonder why when you pull large data from a database, a remote database, to a like a local tool like like Tableau, why is it not saturating your full network speed? And that's the reason why is because it actually is pegging your CPU in order to do all of this conversion, jumping around to different locations in memory and you know, converting the data and then converting it back.

and so like that is the main reason is that when we have a source and a destination that all want the data in one system, obviously it's the most sensible thing to keep the data in that format in the middle, also. there's a little bit more to it as well. I don't want to get too into the weeds here, but let's just say that like ODBC and JDBC and of these other formats invented in the 90s, APIs invented in the 90s, they layer on some additional 90s jank.

even beyond the fact that they put data in a row orientation. They also tend to make a lot of redundant extra copies of the data. You know, back then, a lot of systems that were designed at the time didn't have the benefit of as much knowledge as we do now about how to like zero copy things and really efficiently use memory and use multi-core systems. So there's a lot of just tech debt in the design and implementation of some of these older APIs as well. So we've had the benefit not just of a world

where all of the source and destination systems are increasingly columnar, but also just this you know all all the innovation that's happened since then about how to use like SIM D extensions in you know CPU architecture and things like this. And ADBC avails itself of all of that in order to achieve speedups, not just the simple fact of it being column-oriented.

Kostas (14:36)
Okay, that's great. give us, because you started like giving a few examples, but you know, like the most typical thing like someone thinks when they are thinking about databases and applications is, you know, like I have like an e-commerce website. So I have like a transactional database there, or I am a bank and I need like to monitor like all the transactions that people are doing like on their bank accounts, et cetera, et cetera.

But you mentioned something very interesting and I think like it would be nice like to understand like to learn a little bit more of like how like also the market like has been enriched by new like use cases out there. So now you say like there are cases where we need to have these columnar representation in both sides. So you don't just like extract from a columnar.

data warehouse, like for example, and you just present, let's say to an executive, you mentioned like Power BI, like some other like tools, but give us some examples, some like interesting like applications out there that they are more of like these database to database kind of like connectivity at the end of the day. It's not just a client that just consumes like a table.

Ian Cook (15:50)
Absolutely. This is one of the funny things I think that's poorly understood about tabular data and analytic architecture today. So when databases first came on the scene, as you said, they're they targeted these use cases like banking, where you know, when you make a withdrawal you need there to be an update or an insert, series of updates or inserts made in a transaction on some remote database to

record that with proper fidelity. So the original use cases around databases were kind of inspired by by those needs. Over time data volumes grew and it became much more common that the type of query you might run on a database is not making an insert, not making an update, but just trying to extract some kind of information or insight from all the data that's previously been stored. So today, a huge proportion of

All of the times that something touches a database, that is what it's doing. It's not making an insert, it's not making an update, it's just trying to fetch fetch data from that database. And also traditionally, analytics, what that looked like was there's someone sitting in a back office, a wood-paneled back office in the 1950s, and they're they're trying to extract some business intelligence out of a database system.

And I think a lot of the original tooling around analytics and business intelligence was centered on that use case. But if you think about it today, think about all of the consumer experiences that we have, the experiences you have on your phone and apps that involve large amounts of data being pulled from a database and displayed to you in a graph or in a table. So fitness trackers, sleep trackers, any kind of personalized information,

That is basically an analytic presentation of data that you as a consumer are are having, you know, given to you in in an application. So that is increasingly common. I think this this analytic architecture has become part of so many experiences beyond traditional analytics. so yeah, I, for example, use a personal budget app and called Wineab. A lot of people might use it. If you're a fan of that,

That budget app. one, that's great. Two, I'm sorry, because they, you know, it clearly uses inefficient row-based formats like JSON to store data internally and pass it around. and I'm increasingly very frustrated every time I use this app because it is basically just trying to display large tabular data to me and it takes many seconds to do trivially small queries and to display them to me. so that's an example again of a an experience that

is is really a very operational consumer experience, but that inherits some of the problems that traditionally we would have thought of as back office analytics.

Nitay (18:41)
help us understand maybe like like the you know this this budget app you're talking about they've got some destination as you said that's doing some sort of you know even basic level of analytics let's assume it it was a more not not as bad as as maybe you're you're describing and they had some sort of columnar format but wasn't necessarily arrow it's probably there's there's many destination systems out there that just have like some homegrown inbuilt thing. So it sounds like in order to get the full benefit of arrow and A to B C, they have to

Kostas (18:41)
Okay.

Nitay (19:10)
overhaul the internal system to use arrow internally and then overhaul the connectivity that they're using to use A D B C as well. And then they get the full end to end kind of value. Is that right? Is that the way you think about it? Or is there ways other ways of like kind of migration along the way? Like how should people be thinking about kind of this life cycle?

Ian Cook (19:27)
Right. So before I answer that specifically, I'll talk a little bit more generally about this. So in the in in the early days when we were working a lot on Arrow, there were some proposals to take the database client server protocols and implement them in an Arrow native way. And this is before ADBC existed. So there was some work that Wes McKinney and others were involved in.

to create a couple of Arrow projects. One was called ArrowFlight. That was a remote procedure call system that was Aero native internally. Another one was called ArrowFlight SQL. This is like a database client server protocol built completely on Arrow. These required that the vendors that control the databases and data warehouses add support in their service on the server side for emitting arrow data.

that way we could really a hundred percent of the time we could keep the data in a columnar format end-to-end, right? It would come out of the database like Click House in a in Apache Arrow format would go over the wire, in Apache Arrow format it go into your application in arrow format, and it was very fast. That that was great, but there were many in in the early years, there was one or two vendors who actually implemented that full vision, like Dremio did it, right? Nobody else did it.

and so I think what we saw was a shining example of how to do it right that no one implemented. ADBC adopted more of a flexible model. So if you have a row oriented database, you have a client server protocol like the Postgres protocol, which is just a fact of life. a vendor has implemented it, we can't change it, the server side's gonna do what the server side's gonna do, we can't change it. ADBC still works there as well.

Because what we can do is we can create an ADBC driver that speaks that protocol, whether it's Arrow native or not. So we have a Postgres driver that speaks the Postgres protocol under the hood. and then it converts the data from Postgres into Apache Arrow format as it comes out out of the API surface into your application. it's not going to be as fast as if the entire system end to end, including the wire protocols, spoke arrow.

but it is a transitional step that we can use to to create a consistent API that clients can can bind to that produces arrow data. so because of that architecture, what we see is a lot of applications like Power BI, like DBT, ha going ahead and saying, Okay, we are going to standardize database connectivity on ADBC because we're not blocked by vendors.

waiting to support it on the server side. We don't need them to support it on the server side. We can just create client-side drivers that implement support for any database. so I'd say that, yeah, it's accurate that in some cases, if you're dealing with a system that introduces bottlenecks elsewhere, ADBC might not actually give you more speed. It might give you a little bit more speed because, like I said, we're able to do things a little more efficiently, avoid making redundant copies of data, just implement

you know, implement the driver in languages like Rust and use SIMD extensions and just optimize the heck out of it as much as possible. But it's not going to be as fast as this system that's end-to-end arrow native. so I think ADBC is like a a good example of, you know, don't let perfect be the enemy of good. but it is part of a broader system. And getting to your question, Nite specifically, arrow ADBC is part of this broader sort of whole philosophy about how to build systems.

which is getting away from this idea that you should just use each individual language or frameworks native ways of representing data, and introducing a whole different set of primitives for representing data internally. we saw Parquet do this to file storage, right? What did it it used to be that when you were emitting data from a specific language, you would like emit it in some, you know, a pickle file.

For for Python or for R, there was like an R data file format. Each language had its own kind of serialization format that we used to see a lot of use of for dumping like tabular data and moving it around. Now you don't see that so much. You see people using parquet as a standard way of representing data because it's optimized for that and it's portable. and even in memory, we're seeing the same thing, right? You can choose to architect your application using arrow arrays instead of primitive integer scalars.

and if you do that, you get a lot more efficiency when you're trying to do operations on tabular data in memory. So Arrow is a pretty complicated project, don't overwhelm, you know, everyone. but yeah, it is it's a whole way of working, a whole way of building applications. and you can sort of take the whole thing, or you can you can just take little bits of it. And you know, both both models will will give you some advantages.

Nitay (24:20)
And you had said something there before about, you know, some projects and some end systems, the bottleneck will be in such a place that bringing in ADBC might or might not necessarily matter. Whereas other systems it sounds like they've tuned around the edges enough that actually like bringing in ADBC is like the perfect time to bring it in. So help us understand a bit more of like what is that separation, what kind of things should people be thinking of, this is where I should bring it in. And in those cases

What sort of performance numbers, what sort of things are you guys seeing in terms of like what sort of impact does it actually have?

Ian Cook (24:51)
Yeah, absolutely. So as I mentioned, since the the like twenty tens, we've ended up in a world where most analytic database systems are column oriented. So if you're using Trino, Click House, Oracle X data, you know, a lot of these sort of analytic focused Postgres engines, Redshift

Athena using Snowflake, at any of the like large scale cloud data warehouses, they're all column column oriented internally. so on the on the engine side, think any any case where you're using those those engines, large scale data warehouse systems. the slam dunk use case for AD B C is when you're using system like that and you're extracting large amounts of data, so like more than

you know, 10,000 rows, say. And then the format that you need it in on your destination is also column oriented. So as I mentioned, any time where you're pulling it into another analytic database system or engine locally, like DuckDB, DataFusion, CHDB system like that to do further processing on it locally, or you're pulling it into a data frame library like pandas or polars, or a BI tool like like Tableau

Or Power BI. All of those destination systems are column-oriented systems. When you are pulling large data, columnar source, columnar destination, you get massive speed ups with ADBC. You get 10 to 100 X speedups compared to these legacy 1990s vintage database APIs like ODBC and JDBC. In other cases, if you're hitting a transactional database,

And your destination format is row oriented, you'll get either no speed up or you might get like up to a 3x speed up. just because we, as I said, are able to take advantage of various other fast paths and implement everything in more modern ways. so the baseline is it's not gonna be worse. And the optimum case is it's a hundredx faster. I think like there's a great example where the the DuckDB developers

Did a benchmark of ODBC versus ADBC and they saw about a 38x increase. And that's kind of what we usually see. So these are not subtle multiples, they're they're pretty big numbers. I'll say also we tend not to be too braggy about benchmarks. We our philosophy with ADBC has been mostly to let people try it themselves. Your mileage may vary.

I think the best thing for us to do is just to make it as easy as possible for everyone individually to try this out for themselves and see. So don't take my word for it. and I'm sure we'll have a chance a little bit later to talk about how specifically like you as the listener can can try this out for yourself.

Nitay (27:38)
Very cool. actually one of the only number first of all though those are very impressive numbers. so it's a great work. I'm actually surprised by the number you said there of like if you've got a essentially transactional role based, et cetera, system going through then arrow and then back to the destination where it's not columnar at all, it's just returning a couple rows or whatever. That even there you might get three X. Like I get that like okay, it might be negligible, but how do you even get three X there? Well, you're going from row to to row ultimately and and in the middle through this like columnar based thing.

That that seems actually quite impressive.

Ian Cook (28:10)
Yeah, it there's this is where it's a little bit harder to follow the intuition. but one thing about columnar data is that it compresses better than row oriented data. So by taking like values and smushing them together into adjacent locations in memory and creating a big buffer out of similar values of the same type of data, you end up with a buffer that is far more compressible so that when you do throw like Z standard compression at it.

it shrinks a lot more than heterogeneous data that you've packed together. yeah and you know that that's what that's essentially what columnar formats do is they put homogeneous values

together in adjacent memory locations. So yeah, because of the better compressibility, that's yet another reason why even in cases where your source and destination systems are row oriented, you can sometimes benefit or at least not do any worse by having the format in the middle be column oriented.

Nitay (29:10)
Very cool. I wanna perhaps shift gears a little bit and talk to us about you know, the other thing that's obviously happening everywhere today is everybody's building agents, doing stuff with AI, all that sort of stuff. How and where does A D B C and Aero and this whole ecosystem blend with that and in what sort of ways should people be thinking about it?

Ian Cook (29:30)
Great question. On everyone's minds, yes, there's especially in the sort of Silicon Valley ecosystem right now. I think every you know founder and every company needs to have its AI story tight. so you know, right now data retrieval for AI use cases is obviously critically important. I think there's there's been this realization in the last few years that you can get a lot of alpha out of your AI investments if you just make

all of your business's data accessible to AI. And this has driven interests in things like MCP and you know various other ways of of pr providing s data and semantic intelligence also to to AI models from all of the data warehouses and other sort of stores of of structured data and unstructured data, semi-structured data that you've got in your business. Well

Arrow is specifically focused on structured data. and if if you look, there's a lot of of applications for structured data retrieval in AI that are bottlenecked on retrieval speed. So if you have a AI agent that is doing reasoning and then it is spamming SQL queries on your data warehouse between reasoning steps, AI is very fast at reasoning.

Right. And so it in many cases will actually bottleneck the entire end-to-end agenc analytics workflow on data retrieval steps and on various other like tool calls associated with retrieving and processing structured data. this is a really a big departure from the past because when it was humans

That we're doing reasoning, humans are slow at reasoning, right? I like to think I'm fast at it. I have two degrees in statistics, but I see how fast Claude and Codex are at reasoning, and I have to admit that I as a human are s am slow. So when I used to be frustrated by how slow legacy data database connectivity APIs were, well, it would, you know, I could always just go get a cup of coffee. because it's really at the end of the day, it was my ability to

comprehend that data that was going to be the bigger bottleneck in the process. Agents are really fast. I think with agents it's become even more important that we eliminate the retrieval speed bottleneck. And that's exactly what ADBC does. So we've invested a lot in making sure that this whole ecosystem we've built around ADBC works very well with AI agents. We have agent skills,

And you know, just apart even apart from the agent skills, we've been working pretty hard to get intelligence about Apache Arrow and ADBC into the foundation model training sets so that, you know, you can get good advice from Claud or Codex or your favorite model about how to to build applications end-to-end efficiently using using this modern set of standards.

Kostas (32:20)
That's very interesting. going back to the conversation we had with Wes, one of the things that I remember chatting with him was that the way that LLMs and agentic systems right now are built, they don't feel really, how to say that, like native when it comes to working with tabular data.

Right. You mentioned like you touched one side of that, which is let's say the latency and like the throughput that when it to reasoning, like a human takes much longer to digest what's in front of them compared to an agent. But there is also another side to this, which is, and I of have experienced that using like, let's say like an MCP server for

something like Snowflake, right? There's a lot of data that can come back, right? That just saturates the context window that you have and literally make the models out there like done, right? So it feels like...

All these systems, they have been built so far with a very fundamental assumption that at the end of the day, the user is a human. Right. But now that we have a second person out there, which is the agent, some of these assumptions do not work anymore. Right. So from your experience, okay, one thing is how we can make retrieval.

faster and not bottleneck the agents. When it comes to retrieving data and talking about tabular data, that's what we're talking about here. What else is missing out there to make the agents actually perform with that in the same way that they were performed with textual information? Even if you get the Linux kernel,

repo, example, that is, I don't know, like 10 million lines of code or whatever, right?

That's nothing compared to a very mediocre sized database about transactions out there. Right. And it is a retrieval problem. So I'd love to hear like your thoughts on that.

Ian Cook (34:30)
Yes. Wes wrote a great blog post about this, which maybe this is what he spoke about on your podcast. It's called Can LLMs Give Us AGI if they're bad at arithmetic. and he made some really good some really good points in this blog post. and I d I do think that especially I mean, go back a few months and I think that Wes's points were especially true. model architectures keep defying, I think, our skepticism.

So you know, in the couple days that I played with with Fable Five before it went away, hopefully it'll come back, you know, you started to see some glimpses of higher level intelligence that maybe solve some of the problems Wes described in that blog post about, you know, these these model architectures not demonstrating high intelligence reasoning over data. So I think that there are a lot of things that models

Like if if you use a a model like Opus Cloud Opus 4.8 to and you ask it to do an agen agentic analytics task, you'll see that one of the things it does is make a lot of tool calls. and this is because I would say that there's a weakness of these models to innately understand data. And the way that they work around that.

Weakness is by using lots of tools to help them analyze data. So there's a certain class of questions that I think today's model architectures can nail because they just know exactly how to construct a tool call and write the code and write the query to answer that question brilliantly. There are some more fuzzy questions about the shape of data or the vibe of data that you don't get as good answers from.

So if you tend to ask like under specified questions, you'll get wide range of quality answers from different models. and you'll start to get these you know, you'll start to get into these use cases where you really wish there was more kind of human intelligence going into it. So I am very interested in some of getting to your question about maybe what we need in order to

Have a maybe 10x improvement in the quality of you know AI reasoning over tabular data. I'm very interested in some of the work that's happening around tabular foundation models. So there's a couple companies in the space that are are doing this work. And you know, if you go back to the classical ML architectures, tabular data was everything, right? You get the best results if you structure your data as tables. and then

the transformer era happened and suddenly like we all forgot about tabular data for a while and you know started to think about text and about unstructured data. I think that there is some work happening that refocuses us back on tabular data as a modality. and you know there's been some some good investment going into that work. But I'm really interested to see that meld together into

you know, the foundation models that are widely commercially available as like a first class modality. Because if you use Chat GPT, you use Claude, it has modalities for text, of course, it has modalities for images. You can drop an image and Claude can understand your image. Why can't you drop tabular data in? it has to make tool calls to understand that. So I'm eager to see some of this

this research around tabular foundation model sort of emerged from the lab into experiences that we can interact with as consumers and to see how that sort of gets blended together. I think that's the number one thing that I'm that I'm excited to see happen in this in this space.

Kostas (37:56)
So you don't think that, I mean, the other approach to that, right, is what kind of like models are doing right now, which is let's make them true savants in like writing code, right? And use that, right? And I'm saying that because why we cannot do that with just like SQL. Let's say like we create.

I don't know, like somehow we train this thing and becomes super intelligent, like using and manipulating like SQL and use that to interact with, with the data and reason about the data. And the same way that today, if you ask them like to go and do something about data, most probably will write like Python, like for example, maybe a year or two ago, would.

for some reason go and try to write it in JavaScript and React because they were just over-optimized over the use cases that were more frontend related. so what I'm trying to see how you think about these, do you think it's more of how we can fine tune the existing LLMs on how they use the tools or let's say they appropriate languages? I don't know, maybe...

SQL or some other like version of that, or it requires like a fundamental different model out there. Something that as takes tokens right now, it will take tabular tokens or whatever we call them and spit out like again, like tabular token again.

Ian Cook (39:21)
I think that there is there's been a lot of work into how do we do the tool call workflow successfully. And I you know, if you look at companies that have products that do agentic analytics

like Hex, for example, they've done some incredible work, right, to to optimize the sorts of tool calls and how they're made and when they're made and what sort of intelligence comes out. I think that there's more work that needs to be done to optimize that, but I think that comparatively the work to make LLMs or you know transformer models internally reason

about data without tool calls is relatively in its infancy.

you know, and hasn't made it into experiences that we actually can use day to day for most people. So I'm I'm I mean I'm interested in both, but I'm particularly interested in model's ability to sort of natively reason about data just because it's something that we haven't had a chance, most of us, to play with and use day-to-day. if you think about it, there were some things like vector embeddings and rag, which were early glimpses of this. We got this idea

That if you take tabular data and you create vector embeddings on top of that, there's some intelligence in that vector embedding that we haven't constructed manually. We don't know what that intelligence necessarily means. We don't know what its semantics are, but it means something about the structure of the data, right? but that sort of step of performing vector embeddings was always was like this thing that existed outside of the model architecture that you had to call a separate system to do. I'm sort of

More interested about whether things like vector embeddings, sort of a generalization of that, can be something that the models develop the ability to do just internally on tabular data as part of a standard conversational prompt instead of needing to set up a bunch of separate architecture. So I think this is more of a like an integration problem. I think some of the technology art exists today, it just hasn't been integrated into a unified system that's easy, easy for us to use.

Kostas (41:18)
Yeah.

Makes sense. And where did you see ADBC fitting in this whole story? And the reason I'm asking is because while you were talking about that, like there's something like that came in my mind and that's the data sets library, like from Hugging Face. Which by the way, I really like it. I find it very, very interesting for a couple of different reasons. I think

Obviously they, let's say they experienced like this problem from like a little bit of a different angle. because if you think about like, start from, like data is like a very important part of like training models, right? And like all these things. And it's really hard to use git semantics on top of them with the git.

infrastructure that we have. And when I saw like data said, which by the way, it's like based on Aro, like there's a lot of, let's say, I think it's like a great example to see the value that something like Aro and the whole ecosystem can bring. I was like really, really impressed. And I think that data, okay.

Obviously the training part is like the first one that has been tackled by something like that. But I can see, or I have a gut feeling at least that in the consumption, in the inference part also, you might need like a similar infrastructure in order like to work with data. So yeah, tell us your thoughts on that.

Ian Cook (42:54)
Yeah, there is a literature from academic research about what is the optimal format to represent data in to enable LLMs to reason on that data. So th this is I'd say not a very mature field of research, but there are a bunch of papers on this and it's pretty wild, right? So it turns out that people have compared like if you have data in CSV versus JSON versus other formats like

And you just dump that data into your chat prompt and ask a question about it. What will, you know, what will the model's success rate be? and the results were a little bit counterintuitive and disappointing. I think one paper that I read said that actually HTML tables were the best format to put your data in. And this kind of makes sense because if you look at like the training set for these foundation models, there's a lot of HTML, right? So maybe they developed some ability to reason better over HTML tables.

but that's a little disappointing because I think everyone knows that HTML tables are far from the most efficient way to represent data. then there's some more recent sort of ideas about how to optimize this. So a few months ago, some people were playing around with using JSON and they realized that if they made a few small optimizations on top of JSON that they could sort of have it be

more sparing in its usage of tokens. and some folks spun up this new data format called Tune, T O N, that was based on this. And you know, it's sort of a clever idea, but it it's a it's a bit of a head scratcher like where this work is going in the long term. Because I I think everyone sort of has

Like a they should have an idea that like the the the ultimate best solution here isn't gonna be some hacky variation on JSON and it's certainly not gonna be HTML tables. but I think there just hasn't really been as much investment into really identifying like like what binary format might represent that data in the truly most efficient way. you know I think Arrow

started off ten years ago, before any of this AI hype cycle was around. and we have had a lot of success pursuing use cases that are not inextricably tied up in AI. and I think the AI community has yet to really take notice of what we've been doing in Arrowland. There are some examples of this, like the datasets work that you mentioned

In Hugging Face, which uses Arrow extensively. But I'd say the broader community of frontier AI labs, the awareness of Arrow is low to moderate, and that we've got a lot of mileage to get in the future out of increasing that awareness. It's one of my long-term goals that I always wish I had more time.

to to dedicate to. But I think, yeah, like I think we will find that Arrow optimizes many more of these use cases, far beyond those which it's already optimized, once that awareness grows. And I think this is generally true of like all things Arrow. it's awareness is the hardest thing. and partly it's because Arrow is not a human readable format. If you have an Arrow

a set of arrow data, well one, it's not even a file, it's an in-memory format. So that's a little bit hard to intuit about. And then also, you know, you can dump it to disk, but then it's a binary file format and you need tools to read it. You can't open it in notepad. There's not like a Mac OS native extension that'll open it up. So I think some of the the the technical aspects of Arrow that are exactly the things that make it so fast and optimal for this also make it a little hard for us as humans to reason about.

and this has made awareness a challenge. So yeah, I hope I I certainly hope that we'll have more awareness in the future and that some of that awareness comes from Frontier AI labs because I I do think that there's many, many cases where arrow might be optimal.

Kostas (46:53)
Yeah, that's it.

That's very interesting because like, as you were talking, I remembered, I remember something interesting. So when like deep stick came out, the lab there, started like releasing some of the infrastructure that they've built. One of the things that they've released was also the file system that they are using, like for the training infrastructure. And it's built on top of that DB. Like it's like heavily using, that DB.

That's why I find it interesting when you say that the labs out there are not that much aware about Aero because it feels like, especially at the scale that these labs are working on with data, which at the end of the day, yeah, sure, they are working with unstructured data, but it's not that much different than classical ML at the end of the day. There's a lot of pipelining there that...

takes these unstructured data and creates some tables that they are going to be consumed for training at some point. And I assume there's a lot of efficiencies that are potentially important there. So it would be interesting to see, I don't know if you hear of any other lab out there that has been interested into that or if they've shared information because I know that they also like...

Not everyone's like deep sick to go and share their secret source of the infrastructure that they are using.

Ian Cook (48:14)
Yeah, the the Deep Seek story was fascinating. Small Pond was their internal infrastructure they use and it has DuckD B and it has Arrow in it actually. one important fact to understand is that DuckDB internally internally uses column oriented data for efficiency. DuckD B is not based directly on Arrow, but if you look at any system, DuckDB

the internal format used inside tableau, for example, they all are like within spitting distance of Apache Arrow. when you look at how they organize data in memory. That's because if you just start from first principles and you endeavor to create an optimal in memory columnar format, you end up with something that looks like arrow, whether you knew about arrow or not. You know, it's really it was built from first principles around around optimality.

So there's a lot of convergence of ideas around Arrow. And I should also add that Arrow is like the only game in town when it comes to standardized in memory columnar formats. There's no like Parquet is there, but well, there's also ORC and you know, Iceberg is there, but there's also Delta well Arrow, we're we've been fortunate, right? We converged the whole ecosystem around one thing, which has been fantastic. But

But yeah, I'm not an ML expert, so I'll I'll say or I'm not an AI expert, so I'll I'll say that up front. but I do think that there are many places where I see inefficiency that Arrow could solve. and I do scratch my head sometimes and wonder, sort of if if the teams behind some of these products know it exists. just steering away a little bit from AI, another community that I think, gosh, could really, really use some efficiency boosts from.

From Arrow is the Postgres community. I think there's this large database community that has coalesced around the Postgres project for many good reasons. It's an incredible database piece of database infrastructure that can be reused and permuted in a million different ways to great effect. But at the at the edge of the Postgres infrastructure is the Postgres protocol, which is a fairly terrible database access protocol.

so terrible that the duck the DuckDB founders, before they founded DuckDB, when they were just researchers at CWI in Amsterdam, they wrote a paper that heavily features a story about how bad the Postgres protocol is. makes some very efficient, inefficient choices about how to serialize data, how to move it from you know the server to the client. So yeah, I think in AI, in in in database, in application development, there's many, many places.

where where we'd we'd we'd gain efficiency and interoperability through adoption of arrows. So sometimes I do feel like my job is to be a technologist, but other times I feel like my job is to just be a missionary because we've already solved the technical problems. And it's really just the sort of propagation of this existing standard through the ecosystem where a lot of the the value has yet to be derived.

Nitay (51:14)
Actually, touching on that point and perhaps stitching it with a couple other things you said, I want to talk for a couple of minutes about the the business of columnar and building a business on top of ADBC. You know, you you've said a few times here kind of you know the missionary awareness. I I I like to joke that the attention was all you need. Paper couldn't have been better named. that that has to be the most epic name ever. I guess attention is a AI like technique, obviously, but also just in terms of the broader what's happening in the ecosystem.

and I wanna actually bring it back to an interesting point you said before, which very much I think gets to the heart of being this kind of, you know, amazing technology, passionate group of of great engineering, where you said we don't care too much about not we don't care, but like we we're not pushing benchmarks on people. We're not we're kind of like, you know, that you your management vary, try it out for yourself. My experience with database vendors is actually only the exact opposite.

Typically VS and vendors will push every benchmark down your throat. Most of the time it'll be their benchmarks, or it'll be something where they took and they wrote an entire piece of their code that literally says, if TPCDS, go do all this so that I operate better on the benchmark. Like there's an entire world of benchmark games that happens for database or data system-related vendors to win in sales. So since you're kind of, it sounds like taking a slightly different approach, talk to us about.

what it looks like to to build awareness and to build a broader business on top of this.

Ian Cook (52:43)
That is a great great point. So just to address the the the initial question of like why aren't we more public about benchmarks? Why aren't we dropping incredible benchmarks where we're, you know, a thousand times faster than the comp competition. Well, one reason why is that if you've been around the database world for a few years, you might remember that a bunch of vendors a number of years back

Ceremoniously dropped the famous DeWitt clause from their terms of service. This was the clause that prohibited competitive benchmarking. So for many years, vendors prohibited anyone who's using their product from using it for competitive benchmarking purposes, which really put ice on the entire world of competitive database benchmarks. But I think you know there was some idea that actually

Benchmarks are good. They allow the best product to rise to the top. And so a number of years ago, they dropped that clause from their contracts and they said, Have at it, benchmark. You you're welcome to benchmark Databricks, welcome to benchmark Snowflake. Well, what you might not realize is that the third party companies that Databricks and Snowflake contracted to get their ODBC and JDBC drivers from kept that anti benchmarking clause in their terms of service. So we went out you know, recently and said, Hey, we're gonna publish some

competitive benchmarks, ODBC, JDBC versus ADBC, and then we said, shoot, we can't do it because their terms of service don't allow it. So that's one thing. that so if you're a database vendor, take a look at your connector TOS. and therein might lie the reason why we haven't published a a comparison of that to ADBC. But I would say also that you know like

I am a member of the project management committee of Apache Arrow and my two co-founders, Matt and David, are as well. And we're very serious about just being good stewards of this ecosystem and of the open source project. and you know, that is that is very important to us. So our reputation is tied up with the reputation of this widely trusted project that really its value as a project, more so even than the code that it's created.

is the consensus that it's created among all the different players. So it's very important to us that we not alienate any stakeholders by sloppy benchmarking. and so as a result of this, we tend to steel man all our benchmarks, you know, the competition in all of our benchmarks. And again, only put out numbers that we're highly confident in. And then, you know, invite people to to object. So I don't know. That's just partly who we are and what we're trying to achieve.

and I think, yeah, to your sort of other question there, Nitay, like one of the things that we've we one of the opportunities we see is to create broader consensus around Arrow and ADBC, you know, among all of the different stakeholders, players, vendors, open source projects in this space. with Arrow, we've been relatively successful at doing that already. there is

Scarcely a data stack, vendor product, data warehouse that doesn't have Arrow in multiple places in its architecture already today. Arrow has become a massive success and the low levels of the stack, it's deeply embedded in almost every workflow and vendor product that you can imagine today. somewhere, right? It depends where. And you might n some of the the vendors might not even know it's there because it's a dependency of a dependency.

but the main Arrow libraries are downloaded, you know, multiple billions of times per year. so Arrow broadly has become a a real success story. And it's become that way, not by creating enemies, but by creating friends. So there have been a few times when real challenges emerged to Arrow's ability to kind of unify the whole world of tabular data. one a few years ago.

the DuckDB community and a group at Meta who was working on this product there called Velox approached us with a gripe that the way that Arrow was representing string data was suboptimal. and they were sort of like ready to drop arrow and create their own thing. so what we did was we mobilized a group of people, we got some funding from Meta and a couple other groups and we actually convinced the whole ecosystem

That we should adopt this better way of representing strings, which there was academic research backing, as a first class type in Arrow, and that became string view. So we now have an alternate string representation in Arrow that is widely supported across many arrow implementations as part of the Arrow spec that came out of evolving the standard in response to academic and industry feedback. So that's been our our game all along is collaborate and grow and

when there's threats that might sort of fracture the ecosystem, we absorb them and we solve the problem and keep keep the ecosystem together. So very much not in our interest to to fragment this ecosystem at all, to create enemies. and yeah, as a result of that, maybe we're a bit bit more polite. We have a maybe a British level of politeness about about all things. But yeah, like I like I said, we'd much rather have folks try this out for their own, on their own and and and you know

experiment with benchmarking things independently rather than just take our word for it.

Nitay (58:00)
a lot of sense. we are unfortunately coming up on time here. but perhaps to close this out first of all I want to say thank you for for coming on and thank you for all your contributions, the Aero Project and ADBC, fantastic pieces of engineering work for those that haven't looked at them or or tried them out yet, highly recommend. And so perhaps to close this out, just anything you want to say in terms of the future of ADBC and the broader Aero kind of ecosystem and everything.

Ian Cook (58:25)
Yeah, absolutely. As I've as I mentioned, speed is the obvious reason why you might prefer ADBC to alternatives. But another one is convenience. One of the things we've done over the past year, 18 months, is to make it just extremely easy to install ADBC drivers for 20 plus different database systems. And so we have a tool called DBC.

That is a driver installer for ADBC drivers. It's heavily inspired by UV for all you Python folks out there. It's a very UV style CLI that you can use to install drivers. So check it out, columner.tech slash DBC. We've got a bunch of other tools built around that that just enable you to download and install and use ADBC drivers with incredible ease.

so check that out. I think in terms of what's next, our biggest thing right now is just expanding this ecosystem. you know, we want it so that if you are attempting to connect to any database from any language or mainstream application, that you should have a path to do that using ADBC. so if you are a you work for a database vendor and there's not an ADBC driver for your database, love it if you could talk with us.

we are committed to creating an open source ecosystem where all these drivers are available for free. Again, just one CLI install away. we'd love to call collaborate with open source projects and database vendors to enable that.

Nitay (59:55)
Very cool. All right. Well with that, thank you very much for joining us and we look forward to k following the ADB C and the broader aero project and seeing how things develop.

Ian Cook (1:00:05)
Thanks so Nitay Kostas, for having me on.