Rebuilding the Robot Stack: Why Robotics Needs a New Real-Time OS with Guillaume Binet (Copper Robotics)

Nitay (00:02.578)
Yeah, I'm great to have you today on the podcast here with Costas and I thank you for joining us. Why don't we start with give us kind of a rundown from background of your work history and so on. then we can dive into interesting topics.

Guillaume Bient (00:15.757)
Hi, nice to be there. So I'm Guillaume, I'm the founder of Copper Robotics. My background is really I'm a software engineer at art. I really started very young, like with computers, like those magical boxes you can actually kind of compute stuff with, like with basic and so on. But there was always something that was

really attracting to me, like, and I think that's something that I really kind of pursue through my career, is when a computer can actually act in the real world. And yeah, that's what I did. So I went in engineering school, but not in software development, but in telecommunications, because back then, like in the 90s, that was the way you could actually interact with the world with a computer.

And that really kind of followed me along. I was always tinkering like a hardware and so on. I went through the crazy dot-com booms. I was in the US back then, came back to Europe. And after some travel, went to South Africa for some time, came back to Belgium for a few years.

I came back to the US, but this time around in the cloud space, where that was kind of the intersection between software and networking. So I a stint at Google at some point, Trilio, so I was one early employee. And one day I got a phone call from a robotics company. It was in the autonomous driving space. And I was like,

my God, this is my dream, right? I came back to that moment where, wait, that's the way you actually interact. You make your code interact with the real world. so that's how I did like, I don't know, little bit more than 10 years, like a decade in robotics, mainly in self-driving. And at the end, was like a CTO at the end. mean, at the end, before Copper, I was CTO at Skyways, like in the drone space.

Guillaume Bient (02:42.045)
and in the middle I was at Motionhall and Argo AI where we were putting self-driving cars on the public roads.

Nitay (02:50.802)
Very cool. Tell us, I actually want to take two seconds and take us all the way back to you said something fun there, which is like that you were always tinkering with stuff. Take us to like, what was one of your original like favorite projects that you did as you were, you know, just getting into the business and so on, even, even before the sky was and everything that someone of your.

Guillaume Bient (03:08.285)
yeah, so I can talk about like really how I got better at electronics and that's interesting. So I got some nostalgia at some point and I was like, yeah, I miss those like computers from the 80s. They were very simple and so on. And I started to restore some old computers. And I really remember I was in Pittsburgh and my wife, Koniko, told me like, hey, you know what, you don't spend any money.

out of this activity. Okay, she's managing the purse. And I was like, okay, then I'll basically buy computers, fix them, and actually resell them working. And I had almost like a side business. I was doing that every week, really love that. And that's how I really got good at electronics because yeah, by troubleshooting those old computers that are actually simple and

I was going through the time, right? I started with really simple ones from the 80s, then the 90s, then more recent ones. And that's how I kind of caught up with electronics on my side.

Nitay (04:17.522)
That's awesome. Go ahead.

Guillaume Bient (04:17.965)
Oh, and now I have an entire lab completely fully packed of instruments and so on so on. Yeah, because I made a lot of money with this side business finally.

Nitay (04:29.296)
You

Kostas (04:29.838)
Okay, I have a question, Guillaume. So if you have a computer from the 80s, how do you deal with stuff like the floppy drives? I'm sure there are probably young people that don't even know what a floppy drive is. Or the hard disks from back then, right? Because these are... I remember my... And I'm asking you because I remember my first computer was like...

probably like a late 80s, beginning of 90s. Now we're disclosing a lot about our ages here, so have to be a little bit careful. But that computer I had was having like 40 megabytes of hard drive and it was like an insane thing, but it was like so thick, like I remember like the size of this thing. Yeah, yeah, exactly.

Guillaume Bient (05:03.789)
There we go.

Guillaume Bient (05:20.653)
They are basically something like 12 centimeters or so.

Kostas (05:25.23)
Yeah, yeah, yeah, yeah. And I'm sure like, don't, unfortunately, I don't have it anymore. But I don't think like this thing would survive like to date.

Guillaume Bient (05:35.767)
Yeah, so you're right. The storage format, the magnetic formats in general, could be tapes, could be floppy disks or hard drives that are super finicky to restore. There is almost an art about actually getting one of those old floppy disks and actually trying to get a digital image of them. But we are kind of lucky because there is an entire community that is doing that and trying to find a working copy of whatever you want to find.

Kostas (05:42.115)
Yeah.

Kostas (05:54.754)
Mm-hmm.

Guillaume Bient (06:04.749)
If it is not your work, if it is a unique piece of work, you cannot do this. But for stuff like games or programs and so on, usually you can find that in the community and they have disk images you can emulate. You have little emulators of floppy drives or hard drives. You can plug into your old computer and the computer just, okay, there was another write there or a floppy case. But you're right, even back then, I remember in the 2000s, like 90s and

2000s, like those from the Middle East, they were already not available back then. So you can imagine like 50 years later, or 30 years later, they are mostly dead.

Kostas (06:39.298)
yeah.

Kostas (06:45.582)
Yeah, yeah, a hundred percent. I'm, I don't know. I really sparked my curiosity, like to see like what's the survivability of these things. Because I remember also there were a couple of different options. So you have like the 525, like the very soft ones, like the big floppy drives, right? Like the very thin ones, which I think they were also like more sensitive in general.

Guillaume Bient (07:04.183)
Yeah. Five, five and a.

Nitay (07:07.762)
Yeah, it's 5.25 and 3.5, I remember.

Kostas (07:11.36)
Yeah, yeah.

Guillaume Bient (07:11.789)
Exactly. Oh, you are also 10 inches. Yeah, the usual, a real floppy. Those ones are insane. And there was also switches, like in Europe, had like Amstrad and so on that are like those little ones. They are different formats, but like, yeah, the most like common ones were the PC ones, like the 3.5 and 5.25.

Nitay (07:15.538)
the huge one, yeah, that's right. The huge one.

Kostas (07:16.856)
Kostas (07:26.327)
Yeah.

Kostas (07:32.088)
Yeah. Yeah. Then there was like the introduction of like the zip drives. I were like so slow. I remember like, you're like, that's like so much like capacity in here. And then you had like to connect to like, I remember it was like, I think it like on a parallel port or like a serial parallel port. Yeah. But still it was like insane and slow. was... So anyway, I'm very curious to see like what's... I don't like...

Guillaume Bient (07:46.583)
Parallel parties. Yeah, parallel parties.

Yeah, 40k per second or something.

Kostas (08:01.696)
I'll look into these communities, will be very interesting, this kind of restoration of... Because I'm sure people have stored very interesting things from back then. So anyway, that's super cool.

Nitay (08:19.982)
So, okay, so moving on from that. now that's a great piece of nostalgia for all of us. Okay, so going on from that, you were talking about Argo and Skyways. Tell us a bit about kind of some of the roles you did there and then what pieces and how it touched kind of your love of, you know, physical robotics and engineering.

Guillaume Bient (08:40.183)
Yeah, so recently in my career, I moved into more like managerial jobs. don't know, I was like manager maybe 15 years ago or so. But in the meantime, got director, then VP, and as Skywise, I was CTO. And CTO, like what you say, Nita, is the only time for once I had the purview of the thing end to end. So basically,

At Skyways, we are building big, like, drones, like carbon fiber drones. So, yeah, it's from the design of that carbon fiber construction or build up to EE, all those boards you have in terms of flight controllers and all the peripheral management on this thing, mechanical engineering, all the moving parts, and of course, the software on top of it with like...

autonomy. yeah, no, definitely a role I really enjoyed having that end-to-end where I could really put the pieces together from what I gleaned from my career.

Nitay (09:54.224)
curious, know, I having been a CTO, can say CTO is my favorite kind of job and people to run into because it's such a role with strong with a big variety in it in terms of the types of CTOs you meet. And some some CTOs are deeply technical in the architecture, others are kind of doing more managerial, some are more external conference speakers, some are like there's you run into a whole wide spectrum. I'm curious to hear like tell us a bit more about

the organization that you had and what specific goals and where you like to spend your time versus where other people were doing things and what were you trying to kind of push forward in terms of the drones and the business overall, how you thought of things.

Guillaume Bient (10:34.817)
You had a good question and one thing that is very specific for me about the CTO role is that you need to align the technical aspect of your business with the business, like itself, where we are going and you are very involved with customers, you go talking to them or actually we brought customers also like at Skyways.

We are kind of dividing the workload between the CEO and CTO for that. So, yeah, you have to really switch your mind into, okay, I'm deep down into bits and bytes and technicalities. And suddenly, oh, yeah, you need to talk about what's going to be the plan, the contract for the next iteration of the project and so on with your customers. So, definitely, that is a huge stretch in terms of

kind of spectrum of what you do during your day. In terms of like engineering, we were back then a very small team. So we were, we are actually a good team for like electrical engineering or like mechanical engineering. So I'm glad because that was not my number 140 in terms of skills.

But we lacked a little bit in the software side, so I was helping actually hand-to-hand, like really hands-on testing and being sure that the autonomy side of the house was actually moving along. So it's often like that when you are managing a team, you kind of jump in where you have a of a weakness in your organization to help beat, like if you have directors.

or directly with your team when it's a smaller company like SkyWiz.

Nitay (12:35.61)
So tell us about maybe like one of the most like...

or one of the things you're most proud of that you guys achieved, one of the technical aspects of the drone.

Guillaume Bient (12:48.141)
For me, was really transforming this thing that was basically kind of a prototype. We are out of manufacturing. At some point, we had discussions internally because you had so many variations of that prototype that we were going through with our customers.

to something where, okay, we started to make the same one and we started to converge towards something where we were thinking about how do we design for manufacturing, for example, how do we scale up that business? So, yes, SkyWise was really an interesting experience on that side too, in terms of like, for me, not at all my specialty, but yeah, you need to learn along the way and adapt.

Nitay (13:41.126)
realize I should have asked this maybe earlier, but so give us a lot of folks here might not have heard of Skyway. So give us maybe kind of the higher level of like, what did the company do? What was particular about your drone as opposed to other drone companies? Where did you guys focus and how did that maybe like shift over time as well? It was kind of the overview.

Guillaume Bient (13:57.239)
Yeah, I I'm at Copper Robotics now, but at Skyways, we were doing, when I was there, transportation, like drones for transportation, like more logistic drones. Imagine like you want to, I don't know, move from like a package from point A to point B through like islands or like on a work site on top of, let's say,

like what they do now, like those wind turbines, right? Like you want to drop something there. So really it's about moving parts or like loads from one part to the other, usually through the ocean. And with ranges that goes from the small one that is roughly 300 nautical miles and up to 1,000 nautical miles, so that can go very, very long distances. So you can imagine our customers where

mostly in defense. Because again, when you are watching wars, most of your activity is actually logistics. And those kind of drones were actually very good to do this. Like literally, you have something to urgently deliver to anywhere on the war theater over water. You got it right. So no, SkyWise is really cool. That actually staged me well for Copper Robotics, the startup I started like two years ago.

Nitay (15:26.146)
about that then so what is it you saw at Skyways that led to creating Copper? What was it?

Guillaume Bient (15:32.427)
Yeah, very good question. Really myself, to come back to kind of live with my story, as a software engineer coming into robotics, I had kind of like, you know, this beginner kind of visibility into what you're doing, because I had no idea about robotics like 13 years ago, like nothing, like algorithms and so on. I learned everything while I was there.

For software engineer, found the tools like be it at Skyways or Motionhall or Argo or UBITG, the software sucked. It was so bad. literally, was like how people can even manage to actually build a robot with this software. And the funny thing is like what usually companies do, especially those like autonomous driving companies, for example, they get like a lot of money. So suddenly they have the incentive to say, okay, let's build our own.

like kind of runtime or operating system to actually run our robots. But that's very hard, right? Like those activities when you are building basically the tool to build your product at the same time as the product you need to deliver like urgently, of course, everything is for yesterday, you do a bad job. Literally you do a bad job. cannot actually have like kind of stop the clock, think about what you're doing, where you want to be. No, it's going to be, oh, whatever you can grab.

actually make it work to unblock the team. So that has been my experience and all my entire robotics space, like Skyways included. I've seen the gap between what, for example, if you want to build like Google today, let's say, like a big cloud distributed thing, you will have a lot of amazing tool communities and so on. A lot of things are actually available.

Robotics, no. You will have something to experiment a little bit, but nothing to actually build like a product scale, like a real-eval product that you can actually put in the hands of customers.

Nitay (17:42.96)
So what is, give us maybe a bit more like some of technical details there. Like what is needed, right? Like a lot of our audience might be thinking exactly as you said, like there's Linux, there's Kubernetes, there's all this stuff. Like, why can't I just stick that all in the robot and containers and like, great, it works. Right. Like, but clearly the robot is different. I see you laughing because clearly, yes, that's never going to work. So tell us a bit about like, what do you need there and why are there, what's the requirements that are so different?

Guillaume Bient (17:56.609)
Yo.

Guillaume Bient (18:05.581)
Buh-uhl.

Yeah, a lot of people are trying to put containers and so on in robots. I'm like, what the hell are you doing? So a robot is very different from the cloud. Imagine a robot is like a finite thing. You have the computer, that's it. You cannot scale out. I mean, you could, right, but it's not safe. For example, you cannot really offload anything real time outside of the robot because you never know if the, I don't know, communications would be lost and so on. So you have that finite entity.

That finite entity needs to react in real time from the real world. Most of the time, for a robot to be useful in the real world, need to be near humans. So, you have that tension where suddenly you need to cram a lot of algorithms into that computer.

It needs to have high bandwidth, like lot of images or don't know, LIDAR scans for a self-driving car and so on. But it needs to keep up with real time and it needs to be safe. It needs to also kind of like self-reflect, saying like, maybe I don't know, I got late in that decision or I'm in a state where I should maybe stop if I'm a self-driving car or a drone should kind of land in emergency somewhere and so on. So that's...

That tension is what we see today in the market. have lot of companies, like thousands of startups, that are basically clubbing stuff together to make a demo. You could see humanoids dancing and doing stuff like incredible, but there is a dissonance. Why are there so many videos of robots? But where is my robot? Where is my robot in my home or in my sky or on my...

Guillaume Bient (19:58.967)
public roads. for me, that's the gap in terms of tooling and certainty you have when you start your activity to actually push a product out of the door, is that lack of software and lack of maturity in the ecosystem when you think about it to actually build a reliable product.

Nitay (20:23.602)
It sounds like everybody, each one of these kind of robotic shops that we see in these demos that you're talking about, they're all building basically the entire stack up and down themselves. And there's very little kind of communal standards that have evolved over time. You said something key there though. You said real time. And some folks here maybe have heard of the word real time OS versus a regular OS. Is that fundamentally different? Like, why can you not just take Linux and say, like turn the flag on and say, okay, it's a real time OS now? Like clearly that doesn't work.

So what is needed there and how does that tie to kind of the rest of the robotics stack that you need to build? And how do you think of it in terms of like what parts are standard, what parts should be standard across the community there?

Guillaume Bient (21:06.231)
Yeah, so good question. The real times means that every single step in the chain from getting data from sensors to actuation on the other side has a known execution time. And that can be repeatable and never exceeded. That's what real time is. That's really hard to do because you can imagine like, I mean,

When you have a simple system, that's pretty straightforward. say you have a PID, a way to find equilibrium for a balancing bot or whatever. That's very easy because those operations are very small. You know in advance that you have 10 operations to actually come from

the position of your robot to what needs to happen to make it balanced. That's a very simple example. But robots today, they have a huge mix of algorithms. Sure, you have those low-level ones that people master pretty well. But then suddenly you say, yeah, no, you need to have perception. You have perception, so your robot will start to look around.

Okay, you realize that you recognize, let's say you are with a self-driving car. You are on a road, you see five actors on the road, fine. But what happens when your self-driving car arrives at the exit of a concert? Thousands of people are suddenly rushing in the street and your self-driving car needs to understand and track all those people at the same time. Suddenly, your budget in time starts to fight the...

you start to have too much to do on the fixed increment you have to get a decision. So there is an entire way of thinking about those systems where you need to adapt to a viable load, but a fixed time. And that's the entire secret of making a self-driving car, for example, successful.

Nitay (23:23.586)
The fixed time key pieces is really interesting piece of it. So I worked a little bit on robots in my past as well. And I always found that to be like one of the very interesting constraints of it because at the end of the day, you have X milliseconds or whatever time span with which to react to something. And if you don't by then, then it doesn't matter because by the time if you've passed that latency, then by that point, there's a new input and a new signal. And so reacting to the old one doesn't matter anymore.

And so if you take too long, you're like forever behind. And so you're constantly like taking what is, you know, known algorithms, algorithmic algorithms and thinking about big O and all these different things and making estimations and making like, like shrinking them down in order to fit within these latencies. Is that a lot of what the kind of stuff that you were seeing as well?

Guillaume Bient (24:10.647)
Yeah, exactly. so, Nite, you're on something where I see on my building copper, basically an operating system and runtime for those real-time robots. I see even for my customers a lack of maturity about this. I'll you a simple example. So often robots are distributed, so that adds another layer of insanity into your system. So you can imagine, let's say you have a

a high level reasoning that could be on the compute like with a GPU and so on, starts to analyze the world or even sometimes end-to-end algorithms. But often you have other boxes around like with microcontrollers or other computers that take a part of that stack. Often customers come to me and say, yeah, we need a reliable connection between those two computers. We need TCP IP. No, you don't.

For exactly the reason you said, because TCP IP guarantees that you will send your data over the wire. And if you cannot, because the other computer is loaded or whatever, it will retry. It's too late. Drop it, get to the next piece of data because only the real time is relevant for your robot. And that is a discussion I have over and over and over with my customers.

which really, I think that really symbolized the gap between the understanding of like, yeah, you have a real time system that needs to live in the present and yeah, and compared to something that is more like a cloud service where yeah, of course you have time, you can use your TCP and retry and have like available connections. So very interesting discussions we often have with our customers.

Kostas (25:56.942)
It's very interesting because the more I listen to you and what you are saying, the more I feel like the whole stack that we've been using to make compute available to humans, it's kind of like we have to rethink it. Because the assumptions that we've made in terms of what the reliability is or...

Like the problems that we are trying to solve, right? Like, okay, we're talking about like TCP, like connection, TCP. When you have like network over like great distances, you are optimizing for a different problem than networking within a system that is, as you said, like very constrained in space, right? Like the completely different problem. So what is used today? And by the way, like, it's like robotics is a new thing. Like people were doing robotics for a long time.

maybe not as mainstream as it is like today, but people were building robots, deploying robots. These robots need to have some systems like to run them. So what exists out there and what is missing also?

Guillaume Bient (27:11.757)
No, very very good question. one thing in terms of like if you look at like to answer your question in terms of like probably what we have like usually in front of us is not what you need for a robot is very true because look at I don't know I'm looking at my laptop right now. A laptop is not actually built for low latency at all like absolutely nobody cares.

It's completely built for bandwidth. Because you want to do a lot of things at the same time on your laptop. Now it arrives within five microseconds or 10. Nobody cares. You will never notice. And that, because of type of silicones and so on so on, we hit a limit in terms of, for example, for one core, CPU core, to achieve a number of gigahertz, let's say.

Now, yeah, we have a multi-core machine. A multi-core machine is just about bandwidth. It's not because you add a core that your latency is actually going down. No, it's going to be the same. So everything like that has been optimized for latency and for bandwidth. And even in the cloud, you have that. When you think about microservices, they are not optimized for latency. You might have a bunch of microservices back to back that adds up in terms of latency.

your salvation when you are Google and so on. You can have thousands of them in parallel and serving a lot of customers. And that architecture is actually super good to actually serve reliably a lot of customers, a lot of bandwidth. Guess what? So the framework that is actually in use today for robotics, their model are those microservices. Think how it is broken.

It's deeply broken. of course, with network communications between all those modules, adding latency, and discovering a very dynamic discovery. Something also I never understood. So I'm talking about ROS, the robot operating system that everybody is using. So those components, they discover themselves. Why?

Guillaume Bient (29:40.627)
When do you have a robot that suddenly invents itself like a new task that needs to actually appear in the robot? Like your drone flying in the sky, so you will deploy like a microservices in the cloud, you would deploy that in the cloud, but on your drone, no way, right? There is no way you actually kind of benefiting from that flexibility while you have so many like very, very, very constraint like resources and timing you need to match. That's like

go a little bit deeper into that mismatch between the tooling and what is available. This is really the gap at Copper Robotics we are feeling. don't do that at all. When you develop the same kind of system as ROS, you basically describe your system statically at compile time and Copper will build an entire operating system.

I'm not kidding, that's an operating system with a scheduler, with a file system that is completely tailored to the shape of the robot you gave. So that's the complete opposite way with the priority number one is to be deterministic. So we'll talk about that. But the second one is performance. when I'm talking about performance, performance is about latency, like the less latency from

from sensors to actuation, the closest past. And we are at a few nanoseconds of override between tasks instead of milliseconds, maybe over a roth.

Kostas (31:20.854)
Okay, so help me understand a little bit, like when we're talking like of an operating system here, how is like a, and it would be awesome if you can, I don't know, like describe, let's say a simple robotic system as it is like today in terms of what components we have there. Like, because, okay, we know like on a laptop, but like we have like in front of us, we have like our CPU, the CPU has CAS.

There is memory, hard drives, IO paths there like for expansion. We have like almost, I think like everyone who's using like a computer, more or less have like a mental model of what an operating system is doing just because of the parts of this thing has, right? And of course, like the main thing that like an operating system does like on a laptop is to make sure that the software we are running has access to the resources.

in a fair way, right? Like on your machine. When we're talking about robots, are we talking first of all about like the same silicon? Is it like, again, like if I use an AMD CPU or like, I don't know, like an Intel CPU on my laptop, am I expecting to see the same like silicon like in a robot? So it's like, I don't know, I have an operating system that might be like Linux or something, and then on top of it like

the robotic operating system or we are talking about an operating system here that it goes like the firmware like manages like all the stuff that like Linux kernel would do, right? Like I'd love to hear more about that. You can see like I'm not that aware of like what's going on with robotics and it's super like fascinating like to learn the basics.

Guillaume Bient (33:05.802)
Yeah, so, yeah.

Guillaume Bient (33:16.341)
Yes, so usually you have a mix. Why? Because you usually have a mixed type of workload on your robot. For example, if you do some AI, you need some AI accelerator somewhere. And those ones, you do have some microcontrollers, like very simple computers, that could have that. But usually you want something like an NVIDIA GPU or like

and they do actually deliver platforms like the Jetson. So they have a bunch of various sizes of Jetsons that are system on chips. Basically, they take a computer and they compress everything from the IOs and CPUs and so on, all on one chip. So usually that serves well those kinds of high-level workload that you probably would...

run on that real-time Linux or like UNX or those kind of hard real-time OSes. But often you do have more than this. For example, you would be on a drone, you would also have a part of the drone running on a pure microcontroller, for example, a flight controller. And that gets functions that need...

very high rates of refresh. We are talking about 1, 2, 3 kilohertz of taking the sensors and getting an output with the quadcopter with the propellers, with the motors. Usually you try to adapt the silicon to the type of workload you are

you are doing. When I'm talking about copper robotics, sure we can run on Linux as a process, but you can compile almost the same code on an MCU because we don't depend much on the operating system. We are an operating system ourselves. That's one of things where I feel the current ecosystem kind of missed the mark of that Ethereum node geninous.

Guillaume Bient (35:34.113)
compute, right? Like where you have various computes on your architecture, where you suddenly use C with a very low-level type of things to make one algorithm on your, let's say, controller. And suddenly you are in C++ or whatever on your raw system. For us, no, that's the same. You could have another five controllers and three pieces of compute.

same Rust language we use for them all, you take the description of all those subsystems and you can actually replay them all on your recompiler and replay them all even if they were from MCUs to your laptop, no problem at all. So yeah, again, that's good question because that's also I feel like there is a gap, like that was a gap between the software side

to support those complex architectures on modern hardware.

Kostas (36:34.894)
Okay, so if I understand correctly, like what you say is there is like a host OS that you are using, let's say like Linux, right? If I understand also correctly, you have to configure it probably like differently than like a vanilla Linux installation that you have like on your laptop. And I'd love to hear like what that means, like what it means like to make real time the operating system, right?

Guillaume Bient (37:03.031)
Yeah, so, yeah, good.

Kostas (37:05.356)
Just a second. And then you have your operating, the robot's operating system, right? Like the amount of your building of copper robotics that runs like as a user process inside these operating systems. Is that correct?

Guillaume Bient (37:22.765)
Yeah, or either on the host on the Linux or straight bare metal on MCU, both. We support both. Okay, so let's go into the Linux, how you would configure that. So usually what you do is you use a real time kernel. So now it's actually mainline. So those like those patches of the Linux kernel were for a long time outside of the kernel. Now, any even the laptop on your computer actually usually you can switch it to like the real time mode.

Kostas (37:28.898)
Okay, okay, you can do both.

Guillaume Bient (37:52.6)
What does it mean? It means that you add predictability into the scheduling of your kernel, right? And like what you were saying before, Costas, you were saying like, you know what, people understand well that you have resources on your standard computer, Linux and so on. And the OS try to be fair, tries to be able to actually push as much bandwidth again through those.

processes to and scheduling them so okay they can squeeze as much cycles from the platform as possible. The real-time mode is not like that at all. The real-time mode is like okay tell me which processes are critical like literally if your OS is really real-time and you put like a maximum priority and so on it basically dedicates everything if you're

process doesn't give up and a hard loop or whatever, it gobbles the entire resources of your system. can almost lock completely your system out of yourself if your process is actually saying, okay, I'm going for it. And that makes sense because again, what you want to be able to do is to lower the latency for those processes. so basically the DOS decides, you're up, it's yours. I'll see later to do my own.

I don't know, my own thing as a kernel, really prioritized like hardly on those processes. And those ones, you can actually share them. can say, okay, 50, 50 or 25%, 75%. So that's basically the extreme of like scheduling when you ask for something to be prioritized, it will be. On the MCU side, Copper is the actual operating system. So that's easier because Copper knows

what needs to be executed because you basically described it before you even constructed the OS. So the OS is like a kind of a real-time OS, but instead of having rules that are pretty configured on the platform, it's going to be, no, I know that I need that piece of information from that sensor to be able to actually

Guillaume Bient (40:09.165)
and get this one too and have that intermediate task before I can execute this one. So you can imagine like a real-time operating system that is prescient. Basically can guess what's going to be next because of course you told it. So that's a little bit like a kind of hybrid like when you think about it like in terms of operating system. A very specialized operating system. It's not generalistic at all.

Kostas (40:34.446)
Yeah, yeah, no, that makes sense. And you said something like at the beginning, that's very interesting. you say, okay, like in, in a robot, we live in a predefined world, right? Like, it's not like a laptop where, you know, I wake up this morning, I see the news, I see that I don't know, like a new version of my app is out there or like, me, Ty told me about this new sexy app, but I should try and I go download it and run it. And, know, I do it like on

while I'm drinking my coffee, like robots are not like that. So you're not going to have like, as you said, a drone flying and then suddenly you'd be like, oh, let me out like, I don't know, like something that makes, I don't know, like a choreography for a party or something, right? So.

Kostas (41:26.202)
What's the experience of actually building on top of this operating system the behavior that you want your system to have? Because at the end of the day, when we are talking about robots, we are talking about behaviors. When we program a robot, it's because we want it to have a specific behavior towards specific goals. So I assume, again, I might be wrong, correct me, that when someone's working on a platform for a robot,

The whole idea is that I'm going to build my algorithms and my software and all these things to make these robots achieve these goals. So what's the process of building that? And how is this different than building microservices for the cloud? Because I assume also it's a very different development process.

Guillaume Bient (42:16.653)
Yeah, no, no, very good question. Usually what you do when you are building a robot, like let's say from scratch, you try to reuse things because you have so much going on and you have things that usually are common, like say a driver, you don't want to re-implement a driver and most of the algorithms too, right? Like you have a lot of algorithms like CPE or don't know, SLAM algorithms, like lot of algorithms usually that you can actually reuse.

But the thing is that you will never see your robot doing anything until you have a baseline. Usually your first goal, even if it is completely crappy robot, you try to start very simple, very basic things, so you can have something running. Because also in terms of software, you have to imagine you are not alone. The dependency between hardware and software is

super high in robotics. what you want to do is first exercise this thing end to end, really in a crappy way, measure, always measure what you have and focus to where you need to improve it. It could be an algorithm. So you say, okay, maybe those two tasks, we use like classic algorithms. Let's try an ML algorithm, measure, you see that the behavior is better. You start to iterate on those and in a very modular fashion.

And I mean, it goes to extremes. Like today, a lot of robotics companies do an end-to-end algorithm. Like basically, literally putting an ML algorithm with inputs, outputs. That's a trend with the robot as they were building it. And it can actually behave itself just as a black box end-to-end. We can go into the safety and so on of that. But to give you a little bit of the story of robotics.

You need to start early, your software running, crappy, then you iterate and measure and actually are going towards very measured goals in terms of performance overall.

Nitay (44:26.61)
And remember one thing even from my early days in robotics is like, there's a mental switch because I think most non-robotic software engineers are used to, know, the vast majority of software they're writing has either the form of it starts, it does some stuff and it ends, or it starts, it kind of lives forever, requests come in, it handles the request, does something and that request ends, which is still kind of start and end, but just per request as opposed to like on the whole. Whereas with the robot, it's more like a

forever while loop and that forever while loop is constantly taking in an interrupt or a signal or some input, doing some action and doing it again and again and again and again. And it's just like different mode of thinking. I want to maybe slightly shift gears because we talked a lot about kind of, so there's a core piece of the real time OS, which is this determinism, this kind of different types of scheduling. You mentioned one piece that was interesting there, which is once you start using more and more the MCUs and you have these kinds of

you know, custom chips and custom things. A big thing with robotics is all the different integrations and connectors and different actuators and different sensors and different things. And so how does like, does copper robotics fit into that world? What are people doing today? And how do you kind of go from the situation you just said where you start small uterine, but now you want to add an arm, you want to add to this, you want to add a camera sensor, you want to add like, how does that work?

Guillaume Bient (45:48.279)
Yeah, so here the name of the game is to be able to have like a composable system. You really want to be able to have things where, okay, let's say you want to add like a depth sensor camera. We don't have the video I could show you one, but you want to add that, right? Like what you don't want to do is actually have to reinvent the wheel and actually build an entire driver and so on so on. Because often...

Not only you don't know what you are doing on the software side, but you don't know neither on the hardware side. say, we'll try your stereo camera and we'll see if it works. And if it doesn't work, okay, you try something else. So you need to have this kind of balance between like the modality of the system that cannot be too modular because otherwise you go into like the ROS programs we had before.

And the extreme other, you hand code everything manually and you go into a very, very static loop you have to build yourself. So yeah, there is kind of an interesting balance there. And also, that balance, for example, at the beginning of your project, you will favor a lot flexibility. Because again, you are exploring, you're trying to find the best algorithm for the best hardware.

And when you are going through your project, you start to specialize and specialize and specialize, and then you start to optimize and have your part of your system more like a feature freeze in terms of subsystem. You have that, it works well. You need to build something a little bit more reliable end to end. so in terms of spectrum, like when you are talking about ROS, they are very on the experimental side, very, very flexible, which is interesting because like, yeah, at beginning it's very easy. That's why you can see all those videos.

pretty quickly because to make a demo, ideal. If you go to the extreme order, like if you do everything encoded and so on and you need to actually build your loop manually, you will go into a local minimum where it's going to be very hard to actually evolve from it. And Copper is little bit between the two. So it has really the performance of this low-level encoded thing with a little bit of flexibility, but not as much flexibility as what ROS would give.

Nitay (48:08.146)
So what is, us a bit more about like, what is that flexibility? How do you choose where to draw that line? And what does that mean technically? Because you said a few really interesting things along this conversation. Like, you you're going away from the heavy microservices model. You're building on kind of the analogy of like, you know, UDP not TCP, right? Like it's not a network thing, but like that kind of analogy of like say synchronous, it's not, and so on. And yeah, do you want.

level of reliability or maybe a better word, determinism, and some level of flexibility but not, you know, a thousand microservices. And so what does that look like? Give us a sense of like...

Guillaume Bient (48:46.081)
Yeah, so that's a lot about compile time versus runtime. Where do you want your flexibility to be? Okay. And like we were talking about before, the runtime side, I don't see it because again, like, okay, sometimes you want to have, I don't know, like something in the lab where you have something appearing on your network or so on. Cool. That's runtime flexibility, but all that has like a bunch of costs. So.

where we draw the line is basically you have a lot of flexibility before compiling. That's really the key point. Once you have the binary, that's it. You deploy it and that's going to be your release. But before that, if you want to change your driver, add a driver, add a task, add some monitoring, change configuration, you can change that on the robot. you see what I mean? It's very flexible on like a...

kind of ahead of time. added like, that's interesting actually, this conversation, because like we also realized that sometimes a robot doesn't do exactly the same, like all the time. Sometimes you have like, let's say you have self-driving cars, you could have like a part of the fleet that is actually self-driving, getting customers and so on. But a part of the fleet that is actually mapping, for example, for another activity. And we...

added a little bit of flexibility on the runtime side. When you start your robot, you give it a mission, like saying, okay, what do you want to do? What's going to be the shape of your robot? So that could be either like variations in hardware or variations in missions, like actual missions like your robot is doing. But still, when the mission is running, you need to stop the stack and restart the stack to actually change that mode. So that's where we give enough flexibility.

But we are not completely dynamic at runtime.

Nitay (50:45.776)
So there's something really interesting you said there, which is, you know, I'm thinking to the analogy of like, to your point, the Linux kernel on the one hand, you can ship like a Linux kernel that has every dynamic module. You plug in a device later on, you say, I need this camera. You hot load that module that works right on the other end. You have like, I statically compile my kernel. know exactly. need these five drivers and that's the copper robotics way essentially for robots. And so what I'm wondering is how does that affect for your users, developers of robots using copper?

It must change their SDLC in some ways because the actual configuring and compiling like the copper robotics kernel itself and the OS on top of all of their robot code must become part of their SDLC now, right? So it kind of changes maybe how they're like used to working. Like how does that look?

Guillaume Bient (51:34.113)
Yeah, sure, there is way more compilation, which is kind of good. With Rust, actually, it gives you a lot of certainty about what you're doing when you are regressing on something, for example. So when you change something, even if you change the shape of your robot, you add the task or whatever, you need to recompile. So that's bad. You lose time because you need to redeploy and so on. But we compensate by making that as fast as possible.

like let's say kind of an incremental compilation, that works well when you add just a task or so on, that's how we compile the entire thing. So it's not as fast as just like copying like a Python file somewhere on your robot, but still like, I don't know, in a few seconds timeline. But then also we really simplified the deployment side. So we do build our

robot in one executable. That's it. That's the guarantee we give. I will never compromise on this because you know what? That simplifies everything. That it is statically compiled. For example, if you deploy that on Linux, you don't depend on anything on the system, which is the complete opposite of ROS where you have like a

millions of system dependencies. So you need actually a Linux that is a specific one with the correct modules and so on so on. Copper, no, we grab as much things as possible in the self-contained executable. And then you put that in whatever devices. Our only dependency is basically the Linux kernel. So that helps a lot. And that makes the cycle to actually deploy and retest on the machines or on the simulators, like a lot of people are working in.

in virtual worlds to test your other robot. That makes that very, very easy. So that's how we kind of try to balance the ease of changing something where we pay a penalty. We try to compensate that with a way easier deployment scheme.

Nitay (53:50.066)
Very interesting. And perhaps shifting gears slightly again a little more. Tell us a bit about kind of what the community reception has been like, right? Especially you're coming from a community that's to your point heavily used to thinking in ROS and then just kind of hacking it and things break and so on and there's issues and then they build a lot of stuff in house themselves and you're kind of trying to move the community to like, wait, there's a better way. I have a new robotic operating system that has a new architecture, new whole approach to this.

That the perception then and what's been kind of the community's feedback towards us.

Guillaume Bient (54:22.123)
Yeah, so that's so funny because it feels like a kind of a, I don't know, a two-speed thing. Like you have like this like cohort of like early adopters. So a lot actually of students actually really enjoy copper, which has been surprising to me because that's not my target at all. When you think about it, like ROS would be their target. Like they want to experiment, they are at the university. They don't really care about like their robots. They are just like experiments and so on.

And I would have thought, not really interesting for the university world, but no, even research is starting now based on copper. I'm like, what? And to see the gap between where those people are in terms of thinking about systems and where the tooling is. And so there is a cohort of those people. There is another one where you have companies starting or deciding to go into

robotics like proper. So they become like customers or users of the platform. And they are strong advocates because they usually they are people that they already know, right? They already know that if they pick ROS, they will eat like a wall at some point. And they know exactly what the wall look like. And they look at copper, they say, okay, I understand why you build this thing. That's because yeah, because he wants to avoid that wall at some point.

So lot of people, like more experienced people that goes into like creating startups and kind of really foresee the misery and say, okay, we need a product. We want to use a copper. And that was like the third category of people coming in in my Discord server are the people that are eating the wall. Basically they use Ross, they're like, crap.

We will never make that safe. So we can talk about a little bit of safety. And they are like, what do we do? And they start to look around and say, there is this thing called copper that looks like compatible with us. So we can interface with their own existing system. So they don't have to drop everything and rewrite everything. So we can go incrementally. And that are the third category of people joining in. So now we are, I don't know, like.

Guillaume Bient (56:45.293)
several Android people active on our Discord server. So the community starts, we have a lot of contributions coming in. And that's interesting to see, yeah, those people coming from like VIVE stories.

Nitay (56:59.602)
Very cool. Yeah, that's very interesting. Different use cases approaching and the academic one is a surprising one. That is a very interesting one. You mentioned something interesting there that you said about safety. What is it about safety in ROS versus copper?

Guillaume Bient (57:12.493)
Yeah, one thing about when you are building a safe system, let's say you want to put your software in a drone or an autonomous vehicle, is that at some point you will have an auditor, whatever domain you are in, on the road with ISO 2662, it's like a norm for safety on the public roads, or aerospace as the same, medical as the same, and so on.

At some point, will have someone like an auditor coming into your company saying, okay, now prove me that your thing is actually safe. The basic principle usually when you are building like an autonomous robot is that you cannot really prove like in a classic way the safety of your system. The classic way meaning, you know what, I have specifications, I build an algorithm

According to those specifications, I made a set of tests that are formalized for those specifications. I test them and then I verify that my tests and so on, I actually passed and on the real world. And you have like a paper trail for everything and whatever the auditor will actually point at in your system, say, yep, yep, I have the test, it's proven there and we tested that, we verified that. The very V classic thing.

Now with an autonomous robot, let's say, even like perception, let's say you have just a camera looking at the world and say, there is a person in front of me, don't eat it. That you cannot, literally you cannot prove it at the level of rigor, like a safety certification would require. basically.

You cannot say, let's say you want to be very, very safe. You cannot say, I explored all the inputs, like literally all the combinations of the pixels of your camera. And you have a defined output and you say, hey, I match them and like 100 % of that space is actually covered. Impossible. You don't have enough atoms in the universe to actually go through that space. So what do you do? You sample the space. You say, you know what, what the robot can observe?

Guillaume Bient (59:34.029)
is continuous, meaning that suddenly there is no things that are slightly different that are to react completely differently. So you said your environment is basically multidimensional but continuous and you sample it. Okay, you get a lot of logs, either like in simulators or in the real world, you sample it and you prove your safety case to your your auditor saying, hey, you know what, we tested all those cases, we assume we don't have like crazy stuff between those points.

in that operating domain, we are safe. Cool. But the one thing the auditor will ask you to prove, cool, you have those tests. But how can you prove me that the thing you run on that cloud system where you replayed those logs and so on, and you prove me that it is safe, can you prove me that what you are testing is actually what exactly the robot will?

actually achieve in the real world. And that's where there is like a huge gap between ROS and COPPER. So COPPER is 100 % deterministic. We prove every time we build COPPER that when we get logs on your robot, if we replay those logs, we will replay the same thing over and over. And that's the missing link for those like autonomous systems to be safety certified. And that's where we...

So we are working with our customers to actually have that missing link formally proven so we can actually have that building block for them to build their safety case towards their own applications.

Nitay (01:01:16.474)
What's that? That's really fascinating. So there's some, in order to do that translation from the cloud simulated environment that is verified to the actual robot, there's some sort of like replayability you're saying that Ross just architecturally, technically like can't do or doesn't do or what, what, what is it?

Guillaume Bient (01:01:33.175)
Yeah, so you actually mentioned that a little bit earlier in the conversation. You mentioned that a robot is very asynchronous. And if you think about it as a software engineer, it's like you have a huge data race in your robot, like a distributed system where things arrive asynchronously.

Nitay (01:01:39.846)
you

Guillaume Bient (01:01:59.711)
and compete for the resources and when they're going to be executed and so on. If you let that go, you will never know exactly what actually happened. The DOS actually took that slice for that activity, took that event, it unlocked that thing. so you see it's like a huge data race that you have no real idea about how they actually executed.

Because Copper is an actual operating system, we decide what we run. So ahead of time, we already know what's going to be your while loop you were mentioning. We know it, we built it. So we can actually replay that while loop as it is exactly and log those outcomes like tick by tick. It's a little bit more complex than that. We have some parallelism and so on, but still, that's the concept. We basically log all those ticks and

at any point of time, basically, because it's deterministic, the arrow of time, have no meaning. We could jump earlier or later in that log, and we can reconstruct the state, the entire state. I'm talking about all the fields of all the tasks with a perfect accuracy. That, for an auditor, that's gold. say, you do have your one-to-one...

As a closed system, you could model and actually replay.

Nitay (01:03:27.716)
That's awesome. For the data geeks out there, it's like you guys took robotics and you made it serializable. And everybody else is out there with asynchronous distributed health, basically. Well, you guys can actually, that's awesome. Cool. unfortunately, we're coming up on time here. So maybe just one last question, which is what's the future for copper, for robotics? Where do you see things going?

Guillaume Bient (01:03:34.41)
Exactly.

Guillaume Bient (01:03:41.237)
Exactly.

Guillaume Bient (01:03:54.061)
Yeah, so for Copper, it's all about community, like you kind of hint towards. Like, my goal right now is really to spread the world, the world like I'm doing today. kind of having that robotics community kind of see the light about like, yeah, I have a kind of the 10,000 feet view, but like, what are we doing? What are our goals and how can we be successful and help them? Like, ever through our open source project, which is

like ready level and the tooling we actually started to ship. So we do have like, for example, like what we call a time traveler, a debugger that is kind of like a miracle for a lot of like a roboticist because like, yeah, this hour of time, you can see it, you can drag an index and actually restore your stack in any state. And yeah, build on top of that, like really offering not only

the core of the copper, but like a kind of an ecosystem that is building on top of it, be it tooling, be it like components and so on. And yeah, that's how I see success for copper really. Having those thousands of like startups that are today kind of exploring the space, realizing like, yeah, we need to get serious and come to the come joining us.

Nitay (01:05:13.106)
Excellent. All right. So with that, thank you very much, Biyang. It's been an absolute pleasure. And we'll definitely have to do some follow-up episodes about robotics and AI and maybe some formal methods. Sounds like there's some interesting verification and serialization stuff you guys are doing there. So yes, thank you again. This was this meeting great.

Guillaume Bient (01:05:29.143)
Thank you so much.