OCSF, Schemas, and the Convergence of Security and Data Engineering with Zach Schmerber

Kostas (00:02.325)
Zach, welcome. I'm very happy to have you here today. We have some very exciting and interesting things that I'm looking forward like to chat about at the intersection of data and cybersecurity. So how about like starting with you giving us a little bit of your background story and we take it from there.

Zach (00:24.396)
Yeah, sure. Super excited to be here. really enjoyed your podcast that you've been on in the past. So excited to tell you about my story and how I got to be on this podcast. so I came from the the old school cybersecurity world. Back when Splunk wasn't really a popular tool in cybersecurity yet. And some of the labs and technologies we use were invented in in you know

US laboratories and the the evolution of cybersecurity has been really fun to watch for me. it is been kind of like hunting for rabbits in the wild where you set traps and you find things and it was very rudimentary and the evolution has gone to a very fun place over time and I'd like to tell you a little about that story. so for me I

Started in a hospital industry about fifteen years ago. Or I guess it would be the not the hospital industry but the healthcare industry. And it was a huge attack surface at the hospital and I had to learn a lot really quickly. we we used

data to identify anomalous behavior. And in hospitals there are a lot of peripherals, a lot of attack surface. And so I was really kind of thrown into the fire there. And the challenge with that is I had to learn on the fly how to identify attack and try to stop it. And so what I've done since then is is

Evolved the stack using Splunk at the time. And I don't know how are you guys familiar with Splunk at all? Have you used it before?

Nitay (02:19.522)
We actually had the founder, Eric Swan, the pod at some point too.

Zach (02:22.83)
nice. Okay, so so that was like the original security data stack, right? And and that was schema on read. And it was really useful for identifying simple attack, non-complex word matching, string matching, things like that. And it worked great for the f the the

problems we were facing 15 years ago. You you pull in all your data, you do schema on read, you identify what you're trying to find, and you find it. And and that was really effec effective. And we and I think that was part of the success Blunk has had over the last 10 years is that anyone could do it, right? It had a a domain-specific language called SPL.

People enjoyed learning that language because it was so simple. It was even easier than than SQL for a lot of people. So your average help desk technician could become a security engineer in a short amount of time and leverage the stack to identify attack. And so that was me. That was kind of my story. I I was the guy who learned how to use the stack, learned how to use SPL really well, was able to identify attack using different logic strings to

Build detections and then you know stop a few attacks here and there. And the the challenge with security has always been like you can stop a couple attacks, but in reality, the attacker is usually going to win. So it's really not about if you're compromised, it's about when and how often. So, you know, I I was put into the frying pan there. I learned a lot, and then I started realizing like

This could be really profitable if I went into a different industry, to be honest with you. So I looked at the pay scale in in in in that industry and then I moved into banking. And banking was very serious about their security. It was a it was a different world, right? It wasn't like, there's an open IP on a fish tank sitting in the doctor's office that can be attacked. It was like you have a very small attack surface and it's very well guarded.

Zach (04:29.08)
There's lot of auditing that goes on, a lot of federal requirements, and and that was a really interesting shift for me, right? It going from sort of freeform security development to making sure that you're FedRAM compliant and that you have checked all the boxes on anything the government wants from you. And so that was my sort of my second step was moving out of

a chaotic industry with a huge attack surface to a very regulated one with a small attack surface. And so in that process I got to see, you know, they use the same methodologies and technologies, but security controls were much more critical. and quite frankly, it made the work a lot easier in a sense that if your attack surface was smaller, the burnout was less, to be honest with you. Like people weren't getting as exhausted from

trying to work these late nights because they they caught an attacker and there's there's five other attack services that are simultaneously being hit. the the other fun interesting part was I got to meet some really great individuals in that industry who were exceptional at what they do. And I think one thing we don't talk about in security enough is sort of the the ecosystem that that each company has.

have a a great group of six security engineers who are just absolutely focused on solving the security problem it it can it can actually work like you can you can stop attack and you can make your your environment more secure and so then the there I learned how to really do security prop like properly and at that point

We had solved a lot of the security problems that we had. We were we were finding advanced persistent threats, trying to log into our systems and stopping them before they got there. We were building complex detections based on MITRE attack kill chains and user behavioral analytics. We were solving, you know, the problems of creating a security security barrier around our our corporation. And that was sort of the second step forward.

Zach (06:53.358)
So third step was moving to big tech, right? there was a headhunter that came along like every every time, right? And they said, Hey, could can you do our security logging for for this tech company? And I said, Well, no, not unless you pay me X dollars. And they said, sure. And I was a little surprised. And I said, Okay, well, I'll hop over there. And so that was the next step. And what I found was really interesting. And

What was interesting is that bankers want to protect what they have. big tech companies want to move fast and break stuff and build the modern the the most, you know, build things quickly because the value of a tech stack, especially SaaS companies, is to evolve quickly and produce new technology. And that is not conducive to having a super secure stack, right? Like it it's a different vertical. You don't have the same type of audits and it was a little more chaotic.

They also have a ton more data, it turns out. So like that industry has just giant swaths of data. the banks have maybe like here's your login to your bank account, here's your one or two apps that we provide for you, and here's your thick client for your phone. That's about it. And if you can secure those data, th those touch points, you can secure the bank. Move to big tech and you've got we you know a com a a

multi-billion dollar corporation that has bought five different companies that all have their different tech stacks that are all in different clouds that are all public facing and have many downloads, many inputs, many outputs. they they move quickly and they change even faster. So how do you how do you solve for giant amounts of data that constantly come into the environment? How do you solve for geographically distributed teams?

Doing different things at different times. These were all big questions that we didn't know how to answer. And quite frankly, when I started there, my job was to optimize their Splunk stack because it couldn't handle the volume. And that's when things got interesting. the the things that we we noticed is that the the Splunk stack was great at a certain scale. But are you familiar with how?

Zach (09:17.588)
Splunk functions in its distributed compute

Kostas (09:23.04)
we are, but I think it it's gonna be useful for our audience like to hear about it. So please do.

Zach (09:24.536)
Yeah.

Zach (09:28.3)
Yeah, okay.

So like the basically there are indexers in Splunk and there are search heads. And how they work is the search head will query the data on the indexers. And the indexers, the more of them you have, the more distributed your compute is and the faster it can return the answer. The problem is when the indexers are both your storage and your your compute for query, you have to scale those.

with your data. And so when you scale compute with storage simultaneously, that can actually distribu it's not always optimized for speed. It could be just for long-term storage. It could be optimized for speed, but not long-term storage. The two are difficult to manage together. So if you can imagine, you know, it it works great. I mean I would I would say like why would you switch if you were had a fully built platform from from

200 megabytes per month to 500 to it, you know, 800 terabytes. You could probably handle it pretty well, especially if you cluster it correctly. But you start getting into the petabyte scale, and you're talking about thousands of these ser these systems running simultaneously. And at any given moment, two of them may go down. I mean it 2,000 systems and 365 days a year.

What are the odds that one is not working that day, or two, and if those two both carry the same replicated data set, now you're starting to get data loss. Or you don't have enough servers, so you can't query the data fast enough and you want to look back a month. Well, you can't look back a month because you don't have enough compute to query that data in a reasonable amount of time. So we started to run into these roadblocks, these problems, and and the problem became not

Zach (11:27.03)
That the indexers of the technology are bad, but the general methodology that they used with schema on read wasn't fast enough to keep up. So then we started turning into we started turning to different technologies and we started looking at the modern data stack, right? We started looking at at table structures and and schematization of our data before it lands in our data store. And and this is where

li like my life shifted quite a bit from being a classical security engineer to move into data engineering. And that evolution has been really exciting, but also kind of chaotic because I don't know if you guys have noticed it, but in the last five years the modern data staff has shifted quite a bit. things are moving faster than ever. the technology stack is, you know, should I use iceberg? Should I use Hoodie? Should I use Delta? Like

What is what is the king of s of table formats, you know, and and when we first came into it, we had to pick a schema to to use to develop our table structure and build on our top, build our data architecture on top of. And this is where I I stumbled into the OCSF, the Open Cyber Security Schema Framework. And

This is also where I learned what community, why community is important to cybersecurity. And something we don't talk a lot about is how important community is to building complex data stacks. it's hard to know what to use. It's hard to know what technology I should be using in and and how to implement it. And so I ran into

Paul Agbabian about three and a half, four years ago, back when the OCSF was in release candidate two and we were getting ready to they were getting ready to deploy it into its first production version. and the OCSF, for people who don't know, is a cyber security schema that was

Zach (13:30.869)
made public. It used to be the ICDX schema created by Symantec ten years ago. And they you know Symantec was bought by Bluecoat. The technology stacks that they have all were built on top of the ICDX schema a long time ago. And I think Broadcom eventually owned all that technology and they just said, you know, you you know you can open source it. We're kind of doing they're they're they let us have the schema and at that time

AWS was looking to use the schema to commit compete with the unified data model that Google has, right? And the the big the big evolution was like, okay, they're building one. what should we use? we thought there's common information model that Splunk created a long time ago. It's not very verbose, it doesn't lend itself well to data typing. there's Elastic Common Schema, same problem, very string-based. also.

somewhat locked down by Elastic. So, what open schemas were out there that we could use that could have some gravity to them that are just going to disappear and not be supported? And that was the only one at the time. So we shifted to the OCSF. I became part of the community, I joined the OCSF Slack channel and became one of the first implementers of that schema at scale. And in that process, I got to meet engineers from

AWS, from Cisco, from there's 200 other different companies that are part of the OCSF consortium. you know, and I got to learn what it is to do real data engineering. And it's it was very different, right? Like understanding how to transform your data, schema validation, moving huge amounts of data in through Kafka. all of these were huge learning experiences and valuable.

In a sense that I could understand all that all the reason one why Splunk is so successful is you didn't have to know how to do all that. Schema on read kept you away from that type of challenge. And two, how fast you can read data when it's in columnar storage and properly formatted. So those were huge learnings that I had by joining the OCSF schema. The people

Zach (15:55.019)
Is what I really want to talk about though, in a sense that it was one of the first times in security's a very like closed-knit community, right? Like people don't like talking about what they do, the people don't like talking about how they solve the problems. everything's hush-hush. I don't know if I should talk about this. And it was the first time where I I was with a bunch of senior security practitioners and they openly talked about.

How to work with security. You know, open source communities are like that. You can talk about whatever you want because you're promoting an open source tool. And I got to learn about how data typing and different table structures could be optimized so that we could query the data faster, how Colmer Storage and Delta and Iceberg tables have revolutionized the variant type so that that is available to you when you have a JSON structured schema.

and need to embed things within your types and I got to learn about, you know, optimizing table structure for customer use cases. And so that was a really huge turning point in my career so far. And and I still feel like a novice. So a lot of the words that I'm probably using, if anyone out there is listening and going, what is this guy talking about? He doesn't sound like a data engineer at all. That's probably because I'm not by trade.

I I came from security and so the verbiages that I use are probably more aligned with a a SOC analyst or a security engineer.

Kostas (17:24.769)
No, actually I think you're doing a great job like talking as a data engineer. okay, many things here and many interesting things that you said and obviously like a lot of questions that come in my mind to ask, but before we get into like the details of like the technologies and like all these things that you mentioned, I I I want to ask you something about the people. because I noticed

You were talking about the technology, but you were always coming back like to the people. which makes sense. But I want to ask about the people, but I want also like to ask about the bad actors here. something. So you mentioned you you talked about like the differences in cybersecurity from like one industry to the other. You talked about like healthcare, then like into like the banking and then like into like the technology sector.

and how cybersecurity from the point of view of the practitioner, like the the good guy, is different. But how the bad guys are different. Like w what is different in terms, I don't know, like their techniques, their motivations, their like sophistication. if they are, I don't know, maybe they are not, but

Zach (18:50.034)
you know, the to follow that the the funny thing about their goals is they're very, very aligned with making money, right? So if you want to find the bad actors, you follow the money. And if you're a bank, that's why they're so locked down, because banks store money. So you know if you look at a hospital, yeah, they make money off the patients, but it's not like they're storing it in one central place. The danger at banks is someone hacks a peripheral and they have control of your pacemaker.

That's not a it's a different thing. That's life and death and it's really important. But what do they get? They get they don't get money out of that. it could be extortion, something else. It it's not the same vertical. And and so what it really comes down to is in tech, you know, abuse i is a huge problem where okay, you you are let's say you're your GitHub and you have a pl a bunch of repos that are being accessed and being used to store bad files.

how do you maintain how do you monitor for that? How do you ide identify how it's being abused and so they can abuse it to use it to for their financial gain? so the fun part about attackers is they they usually want the same thing unless they're state actors, right? And state actors have different prerogatives and they they want to damage infrastructure, they want to have coordinated attacks on things so that

it can have political advantages for them in some way. they the those are uncommon and you know though they're though they're uncommon you also have to look out for them. The challenge is they're also usually way better at what they do than your average script kitty or or someone that's just trying to you know fish or vish somebody and and the those are the ones that are actually the most fun to try to catch also. So you know it's it's it's it's fun to see

Every industry has different methodologies, but I think it's based on how they're being attacked, if that makes sense. So if you if you have money, if you have valuable information, or you have something that can be extorted easily and and and used, those are the places that people need to look first. And and quite often threat modeling is something I think is way underutilized.

Kostas (21:08.151)
Mm-hmm.

Zach (21:12.172)
where you model the type of threats you can expect for your business use cases and then focus on those instead of just getting all the data in and seeing what you can find. oftentimes practitioners forget the basic rule of you know, you want to stop you want to make sure you don't have any low hanging fruit. You don't need to stop every attack. You need to make sure that you're harder to attack than your neighbor. And that's the way that you survive in the security industry.

Kostas (21:42.52)
So because you you you you mentioned on some point that a big part of like what you're trying to do is like to identify behaviors, like even like customer like behavioral analytics like kind of thing where you try to find people behaving like in a weird way, I guess, and probably flag that like as something like to investigate. Does that mean that

Because okay, at the end everyone wants like every bad actor wants to make money. but because the way that they are making money is different, changes also, let's say, the behavioral patterns as a result, let's say the the rules or like the detections that you need like to employ there. Because like

the behaviors at the end are going like to be different. And I the reason I'm asking about that is because okay, I'm not obviously like an expert in cybersecurity, but from the little that I've seen around, like I I I I've heard like the like the the the the term like detection a lot where from what I understand, from what I see is, you know, like kind of like rules to detect specific patterns, like behavioral patterns.

And I wonder like how much this can be generalized, but instead well actually cannot maybe, and it's like very vertical specific at the end of the day of like what like the detections should you go after.

Zach (23:18.726)
Yeah, I mean that's a great question. And it it's there's different levels of maturity, right? And so the the challenge in user behavioral analytics starts at a fundamental level. And there's a lot of ontologies that we have, security that we use to identify the behaviors. And it starts with understanding what you have. First, you need an inventory of what's in your environment. So you can start coalescing together things like

that Zach as an actor has this computer and he's and this environment and he usually accesses these things. When he deviates from accessing these things and all of a sudden he's accessing something he doesn't normally, that can be an alert. But to get to that point, you have to know who Zack is and what he normally touches. To know what he normally touches, you have to look at his behavior. To look at his behavior

You have to have all the systems in the environment monitored simultaneously. And that's when we get into the big data problem, right? The more systems you have, the more you have to monitor, the more VPC flow traffic you have, the more DNS traffic you have, the harder this gets. And the there are things like the miter attack kill chain out there, which can that have sort of classified normal attack pattern behavior where

If you are scanning an environment followed by several failed logins into that environment and a successful login, and then you move horizontally inside that environment to another machine and you drop a a binary on a machine. After that, that that's a kill chain. Those are behaviors specific to an attack pattern. And if you have all the data and you've labeled each of those different activities with a MITRE attack kill chain identifier.

You can look for behavioral analytics that match a attack killchain. That is what I've been seeing in the historically to be the most successful methodology to use user behavioral analytics in a sense that when you notice a behavior is abnormal for a user and that user's behavior matches a killchain, there's enough filtering there that the true positive rate is actually pretty high. So

Zach (25:31.295)
When we talk about user behavioral analytics, i it's not the it's not the the the hard part is the prep for that, right? Getting your data ready to the point where you can identify that in giant swaths of data. And that takes years. The easy part is writing the alert to identify that behavior after that cleaning has been done in valid hood.

Kostas (25:54.816)
Yeah, okay, that's great. And I want to bring Nikai also like in the conversation now for because here's the here's what I hear, okay. you mentioned like many times that okay, like securities kind of like data problem, the modern data stack, the data infrastructure, you talk about like all like the the fancy new technologies like iceberg, Delta.

etc. etc. Now, in in the general case, all right, like data infrastructure is primarily driven by BI use cases, right? Like that's the main use case. And the way that we deal with data there is fundamentally different to what you are describing, but it's very close to what MITI was doing for a very long time.

which is the CDPs, the customer data platforms. So typically like in BI, right, you don't care that much about like sequences of events. Like you don't care that much about like Zach did this and after this amount of time did something else and then did like a third action, right? Like this

sequence of events, like the the the timing necessarily like, and especially like on an individual like level, like doesn't matter that much. Like BI traditionally is all about aggregating sums, right? It comes from finance, actually. It's like I want to go and add all the expenses and get like a sum at the end. and the reason I'm saying that is because my feeling, and I'd love to hear from both of you on that, is that if we go on the system level,

If we go like on the data platform and like the technologies, even like the the way we store the data, the way that like we are able like to analyze the data, we do need like different systems if we want to do that at scale. Obviously, like if you don't have a big enough scale, like you can even take DuckDp and like go and just like transcript numbers there, right? so and that's like I'll

Kostas (28:17.373)
give the microphone to Unicai first and first of all I want to hear your opinion if well I'm saying it makes sense. and take it from there.

Nitay (28:29.226)
Yeah, I mean so so I mean, Zach, I think you highlighted something very interesting that that I wanna as part of this maybe come back to, which is kind of this converging of the two worlds between the security world that traditionally had kind of its own stack, if you will, and this modern data stack that has is as you pointed out has been having its own levels of evolution, but they the two worlds seem to be coming together. I I think the point you're you're making, Kosas, which which is right, is that you know you know in in

In BI sort of workloads, it's meant to do quote unquote arbitrary slice and dice. Group by this, aggregate that. Although and I say quote unquote because most even most of those systems are are tend to be highly optimized towards, well, I know these are the dimensions and I know these are the measures and I'm gonna treat this like this and that like that and so on. And if you think about a lot of the like cases that I think Zach was alluding to, and certainly we saw in the like the C D P world, at a very, very simple level, what you're basically doing is saying group by user, sort by time.

Then do something. Right? So you said so so like you can you can I think reduce a lot of the workloads in a way from from a conceptual perspective at least that Zach was pointing out, as well, I'm grouping by, I get all the all the events for for Costa, I line them up, and then I run some anomaly detection, some behavioral analytics, and et cetera, on top of that. And you can absolutely, I think.

shoehorn that into existing systems. It's just not going to be the most optimal. And you can't, but and and instead of that, you can build specialized systems. And the whole world of like if if you kind of I I actually oftentimes a lot of the stuff that Zaki we were talking about, I almost tie to kind of the broader world of like observability, right? And the observability space has long had its own set of technologies like time series databases, for example, right? That are tailor fit for these kind of time series type of

Analytics essentially and and workloads. So so I do think that each space has its optimized specialized technologies, but that what we've seen at the same time as there's there's been this like evolution of the modern data stack is this desire to have this like this dream of this one holistic broad layer underneath that all of these other folks just pull from and kind of tie to. So there's not these disparate systems, and I don't have to build two massive lake houses or whatever.

Nitay (30:52.192)
Right. But I'll let you what what are your thoughts, Zach?

Zach (30:55.967)
You know, that that's r that's dead on. And in fact in in the OCSF we're adding metrics components regularly to make sure that we can handle both observability and security in the same stream because everyone, once you reach a certain size, doesn't want to manage two separate data stacks. It's quite expensive. And some of the data between those two are shareable. So why would you why would you waste storage and compute when you could have both? the challenge though is building the ontologies that work for both. So

How do we have metrics-based information that is both security valid and useful? it gets very complex and every business is different. So totally agree with that. I think some of the other big challenges between common data engineering BI use cases and security is that there are unknown unknowns in your data. You don't know when you're going to need some of the data that's in there, and it may become valuable later down the road. Often

BI use cases, this is our use case, this is what the data is for, and this is its purpose. It has an end and a be a beginning and an end. Often in security, we're storing this data and not knowing if we need it in a year from now to identify that, yeah, that system was compromised a year ago, and here's the proof of it. So

I think the unknown unknowns are some of the biggest challenge in security data and that you have to store it, but you don't know if you're gonna need it. So what should I keep? What should I not keep? how how do I generate signals that have the semantic meaning in it that are adequate to both prove attack and simultaneously not so voluminous that it's gonna

cost me a a million dollars in S three storage every year just to keep that seat belt available to me in the future. So yeah, but in general I totally agree with what you're saying there.

Nitay (32:53.186)
And I want to tie that to something else, which is you made an interesting point a couple of times here as we're talking. You were alluding to the the kind of Splunk approach. And and I think there's a really interesting thing that happens and will forever happen in the world of data, which is there's this pendulum of of kind of approach. On the one end of the pendulum you have you shouldn't need to structure everything, you shouldn't need to go and do the all this work up front. Like just give it to me. I'm this magic machine and I will make sense of it all.

Right. And that's like kind of the the at least at least the marketing perspective messaging of something like a Splunk. And it certainly was the the was the perspective of something like Google, right? Early days Google. I'm not talking kind of the modern AI, Google, but like just traditional Google search was we're gonna take all this massive, you know, messy web data and we're gonna make sense of it. And on the complete other end of the pendulum, you have you should structure everything first, and then you're gonna get the absolute best quality out of it.

And the web version of that was semantic web that some people may remember. It's the idea that the entire web should actually be XML with very str specific and rigid markup and all very kind of things, right? And I think that in every aspect of data infrastructure and data systems, this pendulum is constantly kind of swinging back and forth, and people are exploring both approaches. And and so with that, I'd be I'm curious to to hear your your thoughts on like, you know, you you kind of told us.

what OCSF is. But like in this mindset and and to the exact point you just said of like in security, you kind of have to store everything because you don't even know what you're gonna necessarily need future on. So how do you think about this as like store everything, restructure everything up front and and kind tell us like the why OCSF, if you will?

Zach (34:39.009)
Yeah, that that's a great question and kind of at the crux of a lot of architecture for security. And that's because there are many factors polling at you when you make these decisions. One of them is you know, you you have to have the original log for audit. to to say this was an untampered log, so what should I do with it? And so oftentimes we have two copies. We've got the original raw log and we've got the transform log that goes into our detections engine.

And maybe the transform data is one third the size of our raw log archive. And and what happens is we we we try to pull the critical fields and useful fields into our detections platform that and instead of trying to just do the detections on top of the raw data, which gets really expensive at scale, we tend to take the critical values out, transform them into a common model so that they can be queried effectively.

at scale. So I'll keep my DNS logs, but am I gonna keep all VPC flow internal VPC flow logs? Maybe not, because they're so voluminous that the cost of storing them and keeping them is greater than the danger that is proposed by not. And when we look at you know

How this pendulum has swung back and forth between, you know, structuring all the data or just keeping the raw data and transforming what you want in place. I I kind of see the OCSF trying to solve that in essence by shift the the what I should really say about the OCSF is it was designed to be a producer schema. It was designed to be the schema that all different companies use to emit their logs.

And what that does is solve the real problem here. And the real problem comes down to devs not be having a common way to represent the activities that are happening within their software. And if we look back at all the security industries that exist today, I think they would be one-tenth the size they are now if there was a schema that told devs how to log their security information from their platforms before they developed their technology and emitted whatever they wanted to.

Zach (36:59.261)
And that's simply because all we're doing is cleaning data at this point when we talk about Splunk or the data stack, is we're cleaning data and pulling out what's valuable. If it was already clean and valuable when it came out, we wouldn't have to do all this work. And quite frankly, we'd be able to hone into the key values that really mattered, and our data stacks would be one-tenth the size. So

I guess the pendulum is is the the recourse of trying to solve a problem that is continually growing. And that problem is standardization.

Zach (37:43.053)
So yeah.

Nitay (37:43.584)
So this this is becomes an interesting one because it's a like you said, it's a it's a developer problem, it's a human problem. Well, well, these days maybe an agent problem, which we'll get into, but it but it's a it's a producer problem, as you said. and so how how do you see that kind of shifting in in today's times and and how people are kind of producing these security events and and using them and where where's the modern data stack cross security stack going?

Zach (38:11.231)
Yeah, so the way I see it evolving is, you know, there's twenty percent year over year growth on average on d in data volume per for companies. So what we're feeling at the the big data level is going to trickle down to the medium-sized corporation before too long. And as that begins to happen, we're I think we're going to see a big shift towards s schema the schema on on right.

So like instead of schema on read, we're gonna have our schema written to its to the data store and it'll it will exist schematized. I also think that the need is what drove the OCSF to exist. People understand that they need a schema to under to for their security data and that devs don't want to make bad logs. They just do because they don't have a an architecture to follow.

So, what I'm hoping is going to happen is that as the OCSF becomes codified into the ITU, which is something we're attempting to do right now, it will drive nation states, it will drive developers, and it will drive help drive LLMs to help generate your logs from the producer side. And that will reduce some of the lift necessary to standardize and structure all of your data to the point where.

you need a team of ten data engineers to to clean your data just so that you can write detections on top of it. My hope is that we we start using the technology that we have. LLMs are particularly good at this where they can read the structure of a data and transform it into another structure. That that's kind of what they were designed to do, right? They can translate languages, they can translate locks.

what what we've been finding is that it's not as hard to to build these these mappings and these transformations anymore. So if we're talking about landing raw data and dealing with it afterwards versus dealing with it in ETL version or extract transform load versus extract load transform, I feel like ETL is going to start winning because the cost of

Zach (40:30.741)
Transforming your data after it's landed is higher in a lot of cases. You want to do the transformation as far left in your pipeline as you can. You want to do it on the endpoint if possible, emit it so that you can send it on the wire in a more compressed state to its final destination. You want to you want to have a simpler environment, is what it comes down to. So if I were to look at it from a lens of like, you know,

Splunk and and schema on read versus structured data. I I think in the end the structured data and the table formats are gonna win. Where it's getting really interesting is real-time detection capability, where before you even land your data, you're you're reading it in transit, transforming it and alerting on it in milliseconds. And that's also where I think there's gonna be a lot of interesting opportunity.

Kostas (41:28.083)
I want to add something here because I think there's something a little bit like unique and different when it comes like to the standardization like in security compared like to let's say like the more general like data infrastructure. So usually like data folks when they talk about schema

And you but you use that term a lot like Zach, like schema on read and schema and write and like ETL and all that stuff. but when data engineers like think about that stuff, they primarily think about the schema, about the structure of the relations, right? Yeah, there are let's say obviously there are things that are related like to the payloads of the data themselves, like you have, you know, like in

You need like to take into account like the actual values. But the main way that these systems work, right, is by trying to keep the actual instance of the data as agnostic as possible, right? Like the best case scenario is we care about like the structure of the relations, we care about the data types.

But when we start getting into let's put some cases there because like the values should also shape how the da the the schema itself should be structured, right? Things start like getting hairy. And they get hairy for a reason, because like the the the foundations of like the like the systems that we are using, like the relational algebra, like all that stuff, they are primarily primarily care about structure. They don't

get into like the the data that carries semantics, right? Because that's that's what we are talking about here. Now from our conversations, one thing that I see that is let's say much more important in cybersecurity compared to let's say like to at least like to vanilla use cases like with data engineering is that when you're talking about schemas

Kostas (43:45.226)
It's not just the structure, right? of like the what are like the column names and like the high hierarchies there, blah blah blah, and like foreign keys, blah blah blah, whatever. It's also the values, too. Like many times you need to take into account the values themselves in order to and you standardize that at some degree at least, right? now there are again if we go into like the traditional like data systems like

design there is the concept of constraints. and in theory, at least like you can have constraints there to enforce like specific shape of also like the values that you have there. But I would say that that's rarely the case, at least when you write, you might have it more of like that's why we have a whole part of the industry with like the data quality that comes afterwards, actually, right? Like you usually write and then like you go and check like the qual the quality.

So, what I hear from you is that it is important to resolve as much as we can before we actually write the data in terms of the syntactic and the semantic structure of what we write there. But there is a tension with what the best practices of the industry all these years are like for data at least are going towards to, which is

Well, probably things should be somewhere in the middle. That's why we have like variant type. So we can make sure that we write. And some of the stuff that needs to be, let's say, figured out, we can do it like a little bit later. So tell us a little bit about that and like how do you see also, especially in the world where let's say like the two worlds like merge. So let's say in an ideal world we don't have

like cybersecurity and data, we have like one word and just different, let's say, consumers of that. How this can work and if you see that there is actually like some kind of like friction there.

Zach (45:59.808)
Yeah, that that's a great question. And this gets into a sort of semantic modeling of the data itself. At a certain point, the OCSF data that we transform is trying to carry semantic meaning. That's its whole point of existence, is telling us a story about what happened at a certain time and date by a certain person or entity. And what's happening as the schema grows is it is becoming larger and larger to the point where we're trying to

To structure and standardize the entire world of data. Because everything has some sort of security relevance eventually. So, what's starting to happen is the schema is so big and complex that we need to build a model on top of the data that is structured. And so that's where I see the two converging, is in a semantic layer or a model of some type where the canonical starting point.

of your data is structured at this point. But it to make sense of it, you actually need models. So if we're talking about standardizing the entire world of data, of course there's going to be a million fields in there. But at least the field remains the same field. It doesn't change per vendor. It doesn't change per per engineer. Every engineer puts their data in a relatively similar place if they're doing it correctly. And that's the other channel.

Making sure everyone's doing it the same way, right? And and that's where I think LLMs can really help too. In the scheme, the if we're making a canonical starting point, and that's our goal in the OCSF, it we have this opportunity now where as the schema definitions and descriptions are being written, we can have an LLM evaluate those to see if it understands what that field is for. So as we build automatic mappings that map the raw logs.

to a particular to the the schema or class in this case inside the schema it will know where to put the data and the data will be put the same place every time or at least almost every time 95% of the time and if we have a canonical starting point and we start creating a model that works in parallel with that canonical starting point I think that's where we can get the use cases for each of the different archetypes and

Zach (48:27.127)
That's think I like to think of it like a training of a skill. We can have a semantic model for someone who is particularly focused on vulnerability engineering. You know, like this person's gonna look at these fields, they're gonna want to know about these things, their focus is on on these relationships. And the semantic model will understand that when I ask a question like, tell me all the vulnerabilities that are C VE level nine or higher.

And calculate which ones are actually at risk. It could do the correlation of looking at the CVE score or the vulnerability, seeing if it's an externally facing system, and then identify helping that user identify the threat level based on that. If it if the semantic layer has the necessary context that it can now pull from those schema definitions, we're starting to talk about the convergence of

metrics-based information and security-based information depending on the context brought into the semantic layer itself. And that's that's where I finally see the convergence happening is you've got semantic information molded together with metrics and hot paths for that data that can all be contained into a dictionary of where to find it and what type of personality type needs to find it so it can help you identify how to use it.

Kostas (49:55.646)
Mm-hmm. So going back to like something that like Nitay mentioned like earlier, he said that traditional BI like has been built heavily around like the concept of like metrics and dimensions, right? And that's like what pretty much what semantic layers all also are. Like if we take out like the kind of like a sexy keyword of like semantic, at the end of the day, like the semantic layer is pretty much like a standardization around

metrics and dimensions and what can be mixed together and what not, right? And building like rules around that. Is this abstraction good enough for security? because you were mentioning, let's say, like the type of questions that you would like to ask to that. And again, it goes back to like my initial question about

How things are heavily influenced by traditional BI, and how at the end of the day, like this traditional BI like modeling is actually sufficient for the use case in this case, right, of like cybersecurity. Or we need maybe a new semantic layer model, right? Or like a new way with like new primitives that let's say they are like more

aligned with like the expressivity that is required for something like cybersecurity.

Zach (51:24.203)
Yeah, I think that that's dead on. It I would call it a metrics plus security semantic layer. So you you'd have your metrics, you'd have your lineage, you'd have your history of how that data got to be where it is, you'd have those standard BI components, but you'd also inject things like a place to put the miter attack kill chain or ontology that you expect to be using that's sp like specific to cybersecurity.

And you would inject things like, well, you want to you know use join these tables specific to the OCSF in in many cases. And so I would say an OCSF specific semantic layer in parallel with the the metrics you would normally see in in BI w is what we need. And so in in in essence

Most of the power comes from the BI itself. Because like if you look at the who's using what fields most of the time in big data, that's going to tell you where the semantic information that that type of user would need. And you can use that to help drive the evolution and building of the semantic layer itself. So if I've got a vulnerability engineer and I look at the metrics of what they're querying inside the data stack, I can determine

what a semantic what what a skill or semantic layer for that particular user type is. And this is where I see the industry going is that there are a lot of repetitive tasks that a vulnerability engineer does. How many of those can be automated and brought in and done by an agent? And now the vulnerability engineer is an agent orchestrator, orchestrating many different vulnerability engines and and just validating

like a human in the loop that they're doing the right thing. That's kind of where I see it going.

Nitay (53:26.21)
How do you think about so you mentioned kind of the the agent and the loop and all that and and so so for folks here that are might be hearing OCSF the first time, they're thinking, okay, this sounds interesting. Given I have AI and LMs, as you said, you can kind of toss them at them and just because I happen to call it like IP adder instead of IP underscore address or whatever, like I didn't conform to a particular thing, they they probably can make sense of it anyways.

And so where and how should I be thinking about putting in OCSF and how is it gonna help kind of the on to end security defense perimeter that I have and now that there's kind of this world of agents, as you said.

Zach (54:03.357)
That is such a good question. in the sense that it it carries a lot of weight and financial implications as well. In a sense that agents will continue to read until you tell them to stop. If they're reading unstructured data and trying to find a needle in the haystack, the cost of that token usage is gonna be extraordinary over time. So the way I like to think of it is the

Everyone's using agents to build technology. Well, what is the number one thing we have to give them? We have to give them context and good context that is accurate to for them to produce an answer for us. The the the problem in security is is those agents need the same type of context on your security data to be able to find valid security threats in your environment. What the schema does is provide that context.

for the agent. So when they do query something, they can do it quickly and cost effectively. the fact that we're actually making sure that all the descriptions and definitions in the OCSF itself are agentic friendly, I'll just coin that term. they though that agentic friendly definition is going to excel it's going to reduce your cost when you've got agentic workflows and improve the the quality of the answers that are returned. So

I kind of see the OCSF being the the I would call it distillation of context for the agentic systems that'll be running on top of it.

Kostas (55:38.872)
Alright, we are close to the end here. I feel like we just like scratched the surface to be honest, of all that stuff. But tell us a little bit about like the future of like OCSF and like what excites you about it. I think OCSF itself, at least like from my perspective as an outsider, has

And it's still like evolving in a very interesting way. I mean, started as kind of like an attempt to standardize the way that information is structured. So it started like on a very like the basic level of like, okay, let's not just have every vendor in invent their own log formats. Let's figure out like one way because at the end of the day how different they are. And to be honest, like I think

kind of amazing and that's because of the community in cybersecurity that this work because you can see a lot of standardization attempts out there similar to that that they just keep failing but like in cybersecurity it seems that it's like working but it's moving as it seems like it okay after the problem of like the structure is solved more into like the semantics the meaning and getting like a

different let's say like kind of role in terms of like the importance and like the the function inside like cybersecurity. So tell us tell us a little bit about that and like what excites you.

Zach (57:18.539)
Yeah. w what's exciting for me is I feel like we finally have critical mass, right? We've got enough vendors that have promised to start using the schema that it may actually happen this time. And I I I think it's been needed for twenty years, but it's like herding cats, right? We live in a democracy. You can't just force someone to to to emit logs in a certain format unless they comply. And and I think what what what this means is it's the very beginning.

of the next phase of this common war that we have going between the attackers and the defenders. And currently the defenders have been at a major disadvantage because trying they spend 80% of their time cleaning data and preparing the data just so they can do basic detection. What I'd like to see is more complex detection capability being standardized along with the data itself.

So once you have a foundational data architecture, what we found at the bank was, okay, we've we've we've structured all of our data. Now let's start doing some interesting things with it. I think what the schema can do is give you that capability. If you give the input in a common way, now the industry can focus on how do we write detections and creative traps for attackers.

On top of that common data set, and we can share that amongst each other because it's not sharing business information, it's sharing an open schema with each other. the beauty of that is that the community that has built this data foundation can now use that data foundation to uplift their security and become better at what they do.

The advent of LLMs is actually accelerating attackers as well. So just because you know the defenders might have a better shot doesn't mean this isn't also accelerating the attackers. So it's exciting to see that we have a standard that is going to help, but it's also a little scary to think of how efficient attackers are going to get at the same time. This is not a stagnant industry in security, it is a constantly moving one.

Zach (59:30.809)
And it is also a war zone. And in war zones, technology evolves depending on the defender and the attacker. We constantly try to meet each other. And it'll be interesting to see where this goes in the next five years.

Kostas (59:45.942)
Yeah, one la one last question. So obviously like there's been a lot of like noise about LLMs and cybersecurity, primarily because of like the ability that they have like to discover new vulnerabilities like at scale that wasn't like possible like before, right? and this is I I assume like the most like visible use case out there for obvious reasons. Helps with marketing a lot also like for companies that

They want to go IPO right now. So, but I'm sure like there might be other, not that visible, but probably equally important, I don't know. like examples of how LLMs like in practice are like being used right now, like in cybersecurity. And I'd love to hear from someone who leaves these things like from the inside. Where else have you seen LLMs having like a big impact right now in cybersecurity?

That might not be as visible as like the vulnerability detection.

Zach (01:00:49.409)
That that's a great question and something we we're actually implementing into the OCSF now, where the lineage of, let's say, processes that are spawned by an an agenc system and monitoring the lineage of that like if an LM star spawns another LM, spawns another LLM to do a process, to do a thing, how do you track that? How do you

Know who started this and who is ending this. And so the telemetry for agentic technology and entity resolution on that telemetry has been critical. things like, you know, for for some of the companies I've worked with, fun things that they're doing are using LLMs to monitor prompt injection from other LLMs outside of the platform. So

You're using LLMs to monitor each other. So this is where I think the agentic swarms and agentic orchestration with human in the loop is the future of security, in a sense that you're going to use LLMs to monitor LLMs. And what does that mean? That means we get to watch them and we tune them and we train them to identify attack. and that that is where I see it going. That's some interesting things that maybe people don't think about, but every prompt has the possibility to be a malicious prompt.

How when there's a million prompts coming into your in into your your gateway, how do you know which ones are good or bad? And I'm sure this is happening at the Frontier Labs right now, where they've got entire teams that are just looking at the prompts that are coming into their system, trying to identify ones that are injecting or stealing or continuously asking questions to probe to get the models to expose themselves.

There's not someone just sitting there watching these prompts and reading them. There's definitely a Blue Team agent system orchestration that's going on to monitor what's happening. So those are some interesting other angles that are happening. there's also ontologies that are trying to be built on top of agent telemetry that we're trying to slowly read and sort through to determine which ones are best to incorporate into the OCSF.

Zach (01:03:01.641)
it's it's moving so fast that it's a little frightening to try to codify these different methodologies into a schema because they may not be there in three months. so I guess what it comes down to is the semantic layer is getting even more important in a sense that it's one thing to have a standard methodology to emit logs. It's another to have a model that can shift depending on how that structure changes that is more ephemeral.

So I think it's even more critical that we focus on having models that live in parallel with the data standardization.

Kostas (01:03:36.055)
Awesome. All right. Zach, thank you so much. can't wait to have you again, like in not that far future, because as you said, things like move like extremely, extremely fast right now. and yeah, discuss more about cybersecurity, data and AI, obviously.

Zach (01:03:57.055)
Awesome. Thank you so much for having me. I really appreciate it.