Episode Transcript
[00:00:13] Speaker A: Welcome to Security Serengeti. We're your hosts, David Swineger and Matthew Keener. Stop what you're doing right now and immediately follow us on twitter at serengeti sec.
[00:00:23] Speaker B: We're here to talk more about cybersecurity. Today we're going to be discussing a recent report from Ponemon on the economics of security operations centers.
[00:00:31] Speaker A: And as usual, the views and opinions expressed in this podcast are ours and ours alone and do not reflect those views or opinions of our employers or.
[00:00:40] Speaker B: Chancellor Palpatine and the galactic senate. It's very important.
[00:00:43] Speaker A: Yeah, I don't want to get that letter again.
[00:00:46] Speaker B: I don't want to be force choked again without consent.
[00:00:50] Speaker A: Anyway, that's fair.
[00:00:51] Speaker B: Never without my consent.
[00:00:55] Speaker A: All right, so as Matt mentioned, we're going to be talking about this ponemon survey results, actually, and this was a survey that was sponsored by FireEye, and it's called the second annual study of the Economics of Security Operations center. What is the true cost for effective results?
[00:01:14] Speaker B: Did they actually answer that going through it? I don't think they actually ever provided, like, a straight answer.
[00:01:19] Speaker A: Not directly, no.
[00:01:21] Speaker B: Yeah, I was really looking forward to just a straight number that I could take to my boss and be like, look here, $1.2 million. That's what it takes. Give me it and I'll make it happen.
[00:01:30] Speaker A: They should have put that in the summary. Just one line. Executive summary. Spend this money.
[00:01:35] Speaker B: Exactly. Can you think of how many maseratis.
[00:01:37] Speaker A: That would buy me? I don't know. Two? Yeah.
[00:01:40] Speaker B: Well, that's one for each of us. That's enough. How many maseratis do you need?
[00:01:45] Speaker A: I don't know if I could insure the insurance on it. Probably just have to sit in the driveway and look pretty.
All right. And as I said, this is sponsored by FireEye, so there'll be a link in the show notes to FireEye's press release on this, as well as a link directly to the article. So the article is free. But of course, you have to give away your details in order to get the link to the download. So, as we mentioned, the executive summary is not exactly one line, and it didn't exactly come directly out and say, this is the true cost of an effective stock. The executive summary was a paragraph or a page.
The gist of that summary is in a quote from Chris Triolo, which is the vp of customer success at FireEye, in saying that the findings of the Ponemon economics of the SoC report show that organizations are facing an onslaught of rising security operations costs. But despite these increasing investments, are still unhappy with their ability to combat growing cyber threats. Many security teams are now seeking new technologies that can provide greater efficiencies and visibility, while cutting alert overloads and eliminating mundane tasks to improve analyst morale.
So that's their basic summary of the overall report, and we'll get into some of the details as we talk further down.
And as I would categorize it, that these are survey results. So it's really all about self reporting for these organizations and may or may not reflect the true reality of each organization's situation. So you have to take this stuff with a bit of a grain of salt. But I would summarize it saying organizations are spending more money without getting the real return on investment they're expecting with the increased spending.
They're attempting to correct this by doing more outsourcing and increasing their spending on technology.
And there is something that's not highlighted in their summary, which they actually kind of, it's not exactly buried in the report, but it's farther down in that some organizations within the survey state that they are, quote unquote, high functioning. And there are some characteristics of those organizations, at least as they self reported, that we'll talk about later in there, that hopefully can give you an idea about where a better return for your value is at.
So the survey is about 35 pages, including tables which have the details for the questions asked and the numbered results back from those answers. So they asked 17,200 organizations and they got a reply back from 400 and some, and had to cut some of that so that the final count for the results is from 682 different organizations, which is 4% of the total requested. Now, of those, most of it was from the questionnaires were answered by either security or IT leadership and 13% was done by or answered by security analysts themselves.
Of those who responded to the survey, 58% had a role in security and 30% were in it, and 12% did not state their affiliation within the organization about what part of the organization they were in.
[00:05:40] Speaker B: Kind of curious where they got the people they sent, because they sent this out to 17,000 people and like you said, they got 682 results. Wonder where they got their source data.
[00:05:50] Speaker A: From since FireEye sponsored it. My guess would be that FireEye gave them a list, hopefully, but it's hard to say considering it wasn't actually stated in there about how they chose their survey pool.
[00:06:04] Speaker B: It's just interesting that such a large percentage are it. I don't know how much their answers on some of the spend and things like that. How accurate those will be?
[00:06:14] Speaker A: Well, it could be that it's people within it that are doing security, but they don't have that title. Maybe, like I said, it's hard to say because we don't have those details. But continuing on with some of the statistics around the report itself, 77% of the organizations who responded had 25,000 employees or less.
And according to the survey, the average cost of a breach is $3.6 million.
[00:06:42] Speaker B: That's cheaper than the cost of the mssps. Later, maybe they should just have the breaches instead of the MSSP.
[00:06:48] Speaker A: Actually, it's not cheaper than the MSSP. It's cheaper than security engineering.
But we'll get into those numbers here in a minute.
But in another Ponamon study in 2019, the percentage of companies reporting data breaches increased from 59% to 63%. So not only is the average breach cost 3.86 million, but more companies are experiencing breaches overall.
[00:07:15] Speaker B: If they don't have a sock and they're not detecting any breaches, there's nothing to report.
[00:07:21] Speaker A: That is frighteningly logical.
So going through the survey, we're going to attempt to answer a couple of questions, and we'll state the question. Then we'll get into some of the details surrounding that question. So the first one is, what does this say about the state of the Sock based on these survey results? So 81% of the survey respondents said the complexity of the sock is very high. They don't get any details around what makes a SOC complex.
[00:07:53] Speaker B: We don't need details around that. We know that. We've talked about it. We talked about it in last week's episode. We'll talk about it over and over again, that the level of complexity in it in general, and therefore the level of complexity in the SOC because they have to monitor that, is frankly probably security's biggest enemy in general at this point.
[00:08:14] Speaker A: Well, that would be the assumption. But is that really what they're thinking about when they think about complexity?
[00:08:20] Speaker B: It's got to be right, because it's the difficulty in taking an ids alert on the network and connecting that from an ip address to a hostname, then connecting that to a user account that actually performed the action, and connecting the network connection to a process, and then that process to an executable.
[00:08:42] Speaker A: I don't dispute that. I'm just saying that since you don't explain what you mean by complexity, there's room for interpretation, is all.
[00:08:54] Speaker B: All right, I have no follow up.
[00:08:58] Speaker A: So other information in here about the state of the SOC is the average analyst. Turnover is that they will have, or a SOC will have three analysts resign or fired within or every year, and they attempt to prevent this by increasing salaries, which I think is the wrong.
[00:09:22] Speaker B: Way to do it.
[00:09:23] Speaker A: I don't disagree, and I think we'll get into that a little bit when we talk about the high functioning organizations.
[00:09:28] Speaker B: Yeah.
We should actually have a real deep discussion on keeping analysts, because I have a lot to say on that.
[00:09:35] Speaker A: Well, we'll make a note of that.
And analysts stay, on average, 2.5 years.
So you've got really high turnover rate, even if you aren't having problems with your analysts, since they're only staying 2.5 years. So it's not just the low performers or the high performers necessarily, that are moving on in that short period of time.
[00:10:06] Speaker B: I don't know, though. Isn't that about the normal turnover these days?
[00:10:10] Speaker A: Well, that's what it says. I have no idea, actually.
[00:10:13] Speaker B: I would have assumed that, I don't know, maybe the normal turnover for all positions is lengthened and that's still shorter than average.
I would have assumed that the SoC, one analyst would have been way shorter than kind of your normal security tenure just because of the grind of the position.
[00:10:30] Speaker A: Right. And I don't know. I've listened to an interview with the founders of LinkedIn a few years ago, and they were talking about how turnover isn't necessarily a bad thing if you are leaving on friendly terms and the organization. Some places I've been have this problem where if you leave the organization, the organization or the people who work there feel like you're betraying them by leaving the organization and going someplace else.
And the LinkedIn guys were saying that really, you need to consider this as a benefit for both you and them, because if they leave that organ, you leave the organization, which they're working, and they go out and they get additional or better skills from someplace else, and you have a good enough relationship with that individual or those individuals. So if they come back to your organization later, then they bring that additional value back to your company where they may not have been able to collect it or get that in your organization because of this reason or that reason, or you're not doing this thing, or at the time you don't have that capability, but you can grow into it kind of thing.
[00:11:46] Speaker B: That makes sense.
[00:11:47] Speaker A: Yeah. So I'm not overly concerned about, really, the average analyst stay time or the number of analysts that resign within a year.
The challenge, of course, is then refilling those billets because being short on a 24 x seven security operations center is really challenging when you have fewer people to run that 24 x seven than you did the week before.
So this might be an opportunity for organizations to try to work better with your HR in order to try to get the amount of time that your billets remain empty shorter. See, that's something I would have liked to have seen in this survey that I think would have been helpful, as well as from the time you post a billet to the time you fill a billet, how long does that generally take? Because if you're turning over three analysts a year, but your billet fill average is like a week or two weeks or something like that, then your cost to replenish that is smaller than it is versus taking a lot longer because you're down the capability as well as spending the actual amount of money to advertise for the position and so on and so forth, and going through the motions to do the hiring.
[00:13:14] Speaker B: Especially for tier ones, it's usually just from previous experience. It's usually pretty easy to find and hire tier ones relatively quickly, but finding really experienced analysts, because this doesn't seem to differentiate between tier one, tier two, tier three. So losing one of those guys is a severe hit. I mean, I'm just remembering somewhere where we worked before. I don't think we found anybody in the year we looked. We only ended up interviewing even two candidates that kind of match the requirements of the position. And neither one of those worked out for various reasons.
[00:13:44] Speaker A: Right? Yeah, this seems to be mostly focused around tier one.
All right, so the next question is, what does it say about the value proposition for the stock?
So where's the money being spent and is it worth it?
As I mentioned earlier, you have to take these survey results with a little bit of grain of salt, and we'll try to suss that out a little bit when we're talking about some of the numbers here.
So they say that the extrapolated cost for an MSSP is $5.3 million a year, which is up from 4.4 million in 2019. And that's a 20% increase year over year. But according to the survey results, they say that mssps are being more effective. But they're also saying that fewer organizations are leveraging mssps, which is 16% in 2020 versus 20% or 21% in 2019. Now, the extrapolated outsource is overall 34%, and most companies are outsourcing eleven to 50% of their alert monitoring, which is.
[00:14:58] Speaker B: Really interesting given how much they're paying for mssps.
And I get that that's an extrapolated kind of.
But, and I understand kind of the mixed model, know, maybe exporting your really boring stuff out to an MSSP and keeping the higher fidelity alerts internal or maybe alerts the MSSP doesn't have the skill to review AWS or something like that. But that seems very low as a percentage.
[00:15:25] Speaker A: Right? So if you're paying 5.3 million for 30% or 50%, depending on which number you're looking at, that sounds like ridiculously high, because how many sock analysts could you hire for that?
[00:15:36] Speaker B: Yeah, 5 million. I could build a really bang up sock for 5 million a year.
[00:15:42] Speaker A: Well, and that's just the MSSP cost. We're not even talking about the security engineering costs, which we'll get into in a bit here as well.
[00:15:49] Speaker B: Yeah, I could build a really large and I think fairly effective sock for that total amount of money. That blows my mind that people are outsourcing for this much. So, heck, you know what? I could build a really large and effective sock and still outsource the boring tier one stuff for less than that.
[00:16:05] Speaker A: Right.
And one of the reasons that I think these numbers are a bit skewed is when you drill down into it, 31% of the respondents said they are spending more than $5 million a year, which means that 70% or almost 70% are spending less than that. And there were 3% which were spending over 25 million a year for their MSSP, which is really. That blows my mind. That is crazy.
[00:16:33] Speaker B: That skews the average for sure, having that number of people. And what's interesting is, I think, what was it? There were 600 and some recipients, and 30% of them said that they had an MSSP. So 3% of the people that have the MSSP, which is 3% of a third of 600, is like six people or something like that. Six respondents said that they had. I wonder if they're all from the same company or if there are actually six companies that are spending more than $25 million on an MSSP. It's a waste of money. Come hire me. Give me a million bucks a year, and I'll save you 60% of that.
[00:17:12] Speaker A: Well, I guess that's one of our assumptions here, is that at least it was. My assumption is that each of those 682 respondents were from different companies.
I had not even considered that. Well, you're going to ask the stock manager and an analyst in the same company these questions.
[00:17:31] Speaker B: Yeah. Do they tell us how many companies? I don't think I saw that listed as no.
[00:17:36] Speaker A: They just say respondents and people who they were sent, the number of surveys they sent out, they didn't specify any distinction between organizations. So I certainly hope that they were individual organizations and not individual people within.
[00:17:54] Speaker B: That's a gap.
[00:17:55] Speaker A: Yeah.
[00:17:56] Speaker B: They need to draw. They need to describe how many organizations these came from.
Like 682 results from 400 organizations.
Then all this should have been done by organization. That's a shame.
[00:18:11] Speaker A: Well, I think there might be challenges there with saying it's by organization, because actually it's individuals who are responding, not the organization officially responding, which the survey results would probably be like ten companies if the organization had to provide an official response or something like that.
[00:18:33] Speaker B: I mean, it's got to be more than that. Looking at the percentages for the industry that they're in, it definitely looks like it's at least like 100 different companies, minimum, judging by the percentages there.
[00:18:46] Speaker A: Yeah, but it would certainly be helpful for them to specify or make that more explicit within the documentation for it.
[00:18:54] Speaker B: You know, actually looking at that list, this might explain why they're spending so much money on those outsourced socks. 18% of the results are from financial services companies.
[00:19:02] Speaker A: They're just rolling in money.
[00:19:04] Speaker B: Yeah, I've definitely heard that's where you got to go if you want to make that money. Money dollar, dollar bill, et cetera. Whatever the kids say these days.
[00:19:12] Speaker A: I don't know, but I don't think Scrooge McDuck was a banker. It's kind of interesting.
So, moving on to some of the other costs in here. So a large cost to the sock overall, as well as the MSSP, was in engineering and tools. So tools, just with the three largest tools, which organizations were leveraging, which is SiM, XDR and Soar, was almost a million dollars with engineering costs at 2.7 million.
And even with all this spending, 51% of the SoC surveyed say that their ROI is getting worse, which is up from 44% in 2019. So we're spending more money, have higher turnover rates, and yet the return on investment, as subjectively noted by the respondents in the survey, is getting worse.
[00:20:09] Speaker B: I don't know how they're. This has got to be kind of a feeling based ROI, though, right? Because I've never worked anywhere that's managed to put real good numbers on how much does each detection we catch save the company?
Nobody that I've seen has put a number on, like, one. Phishing email has any given. Phishing email has a 0.5% chance of turning into a breach if we didn't respond to it. And the average cost of a breach is 3.8 million. Therefore, by analyzing and stopping this email, we have saved $200,000 or whatever, the math works out. Or $200.
Yeah, I'm sure there's people working on it, but I've never seen a real good ROI for security.
[00:20:50] Speaker A: Yeah, well, one of the questions I would have is how much time and effort do you put in figuring that ROI out? And is that figuring out that number worth the expense? Which is maybe why a lot of organizations don't have that explicitly figured.
[00:21:04] Speaker B: Know, though it doesn't even really have to be accurate as long as it's standard, because then you can kind of compare. If you had some sort of standard method, maybe NIST released a method of figuring it out where you could at least compare kind of between companies and between different tools in a way that maybe it's not accurate in terms of dollar amounts, but it's accurate. It's at least standard. And you can compare directly between calculations. That might be useful.
[00:21:31] Speaker A: Actually, that might be something that organization like Mitre or somebody could come out with, like they did with the mitre framework to say that this is how you should figure out, or this is a methodology for figuring out ROI, because even if you do have one organization figure it out, that's not going to translate to other organizations. So you can compare apples to apples.
[00:21:53] Speaker B: Yeah, I think this would be a real interesting place for someone like Mitre, like you said, or NIST to work with the insurers and maybe having a trusted third party look at the data they have. I keep going back to this, I need to find something else to harp on, but they've got all the data about the breaches. They've got kind of the information about what protections are in place and stuff like that. And they might actually be able to get some real correctly configured firewalls. Reduces the percent chance of breach by 75% or something like that.
[00:22:22] Speaker A: Yeah, right.
We talked about this before, that insurance companies are really going to be where we're going to get these numbers crunched in the future to get.
[00:22:30] Speaker B: They're the ones with the financial. They're the ones with the financial incentive to do it.
[00:22:36] Speaker A: Yeah. We just have to hope that insurance companies will be sharing that kind of information amongst themselves, which is going to be a challenge as well.
[00:22:43] Speaker B: Yeah. So I have a weird feeling about the cost in tools there.
They're saying they're spending 2.7 million on engineering resources and tools and about a third of that is in SIM, XDR and Soar. And the numbers seem real low for the tools to me, versus, I mean, if we assume that a fully loaded cyber engineer is around 200,000 per person, I think later, somewhere else in there, they say the average number of engineers is like six, right? So, like 1.2 million in engineering and 1.5 million in tools.
I don't know. Maybe I don't know enough about budgeting and budgeting cybersecurity, but it just seems like the tools, the cost of the tools is a lot cheaper than I'm used to, because something like splunk, they're saying, I think, what, 200,000 for a sim, on average, spent on a sim. And I know splunk, that's a real small splunk installation.
[00:23:40] Speaker A: But you know, what this might be telling us, which isn't in the survey, is how much they are trading software costs for labor. So it could be that the costs of the tools are down and the labor is up because they're relying more on open source, which takes more work, but lower upfront costs. Oh, you know, it's pay me now, pay me later kind of model here.
[00:24:08] Speaker B: You know what, though? This actually also matches up with another thing you pulled out later that you called out about what percentage of SoCs are looking at logs. And that percentage is very low. And that's the main use for Sim, right? Is to correlate logs. I wonder how many of these socks aren't even using a sim. Or they're using, like, a real simple one that's just correlating, like, alerts and being used kind of as a workflow and ticketing system rather than as a sim.
[00:24:32] Speaker A: So you're suggesting that people might be saying that excel is a Sim?
[00:24:37] Speaker B: No, not necessarily, but I'm just saying that, for example, again, let's use splunk because splunk is licensed by volume. If all you're feeding into it is alerts from your ids and your av and your cloud alert, your default guard duty alerts or something from Amazon, then you're not really using that much license. And frankly, you don't even need splunk at that point. You can use an elk stack or logarithm or really anything. So maybe they're not using the sim to pull in large numbers of events. They're using the SiM instead as kind of like a ticketing system for it to bring in all the alerts and maybe do some light correlation on the alerts from their other system and then serve as a place for the analysts to market completed. Certainly explain why it's so much cheaper.
[00:25:23] Speaker A: Right? So they could be using it ticketing system in lieu of something else and saying that's what it is.
[00:25:32] Speaker B: I don't know. Yeah, the number just seems wildly low from my experience with Sim, that's all, because we've worked together on kind of mid size splunk deployments, and they were maybe more than that. And I've worked on maybe larger splunk deployments, and they may be a lot more than that. So again, although again, maybe it's just because I'm used to working with splunk and arcsite, so I don't know necessarily what some of the other vendors charge.
[00:25:58] Speaker A: That's true.
[00:26:00] Speaker B: The other interesting thing about that comment there is that the soar product, the average extrapolated price based on the responses, there's a lot of people that are paying more than $300,000 for a sore product, which, given that they're still kind of immature and kind of on. I haven't talked with too many people that have fully implemented a sore product. Pretty much everybody I've talked to that has one is still within their first year or two of kind of trying to make it work.
[00:26:23] Speaker A: It's kind of funny you say trying to make it work.
So it kind of sounds like Sim is where, or soar is where Sim was.
You implement it and then you spend the rest of the time trying to make it effective.
[00:26:37] Speaker B: There's certain things that are real easy to make effective and come out of the box, like the enrichment stuff, but the remediation stuff. Like I wouldn't try to use the remediation and containment stuff straight out of the box. That requires some man, some person time to get working.
[00:26:51] Speaker A: Yeah, that's something for later maturity. You have to have a level of comfort for that.
[00:26:57] Speaker B: This is where my own background is probably coming back to bite me. From my experience, soar is much less than the cost of Sim for an equivalent sized organization. And that's part of why I'm confused, because again, maybe my own specific background with the specific sims and soars that I've worked with, it could be a.
[00:27:14] Speaker A: Matter of where the value is at. So they might say, well, we're going to cut costs here, but spend more here, and they think that falls out in we should spend more on soar and less on SIM.
[00:27:26] Speaker B: I can see that actually thinking about it, the SIM does require a lot of dedicated time if you're going to use those events that are not already security events, go in and find them, correlate them, maintain them. When vendors suddenly change the format and it breaks, all that is super expensive. I think I can see simplifying your worldview, maybe by leaving off those kinds of alerts and focusing more on kind of the pre built content from tools.
[00:27:58] Speaker A: All right, so what does this say about the work in a sock? So 85% of those who responded send that working in the sock is painful or very painful, which is up from 72% in 2019.
[00:28:17] Speaker B: Yeah, and this is not really a surprise. This kind of gets back to the complexity we're talking about before, where most SoCs don't have their systems tied tightly together. I know you were a big fan of that threat and tell exchange and enrichment layer from McAfee, but all the different stuff, not talking to each other, especially when you're trying to hit an alert every ten or 15 minutes, and some of these socks that are really cranking up the number on you, like trying to pivot back and forth between figuring out who owns the box, what processes we're running, that's all kind of wasted time from an analyst perspective. That's not time they're spending doing anything interesting. They're just doing super repetitive and boring tasks that should be soared away.
[00:29:00] Speaker A: Yeah, no doubt. The idea is to take those what can be automated and automate it, and that's where they're spending their additional money on. Which is why the sore number and the XDR number are higher is attempting to do that in order to reduce that pain level. Because 80% of the people who replied said that the reason is the increased workload causes burnout, which is why it's painful to work in the sock. So if you can automate that away, then you can reduce that burnout and maybe reduce the turnover rate.
[00:29:35] Speaker B: Part of it's that, and then part of it, too, is the number of false positives. I'm sure that was on the list. Somewhere where we experience this, we've run into vps that say that we monitor everything and we can't turn off any alerts and we have to watch it all. I think a big part of this is related to that. I have talked with people that work in socs now where they see one alert an hour or less, and they have time to kind of deep dive into each and every alert and spend some time lovingly crafting an investigation around it, really figure out exactly what's going on there. And then there are people that work in kind of the MSSP, or turn and burn socks, as I call them, where you're expected. You get like five minutes to go through an alert and it's on to the next one. That is definitely 100% a burnout. And that's probably why we're seeing, like we were talking about the averages there. And the averages are probably hiding those socks that are losing six to ten analysts a year because I don't think.
[00:30:33] Speaker A: That anybody should overload them.
[00:30:34] Speaker B: Yeah, I don't think that anybody should spend more than six months doing that kind of frontline. You get five or ten minutes to crank through alerts.
And part of this is related, we talked about this before, for sure is related to not having good standard content that everybody has. Because if you just turn on the default alerts from your ids, you're going to drown. If you just turn on the default alerts from your, actually, the malware, the AV stuff and the ADR stuff is a little better, but there's still a lot of false positives in there. And you just open up the fire hose of all of your ids alerts and funnel it into your sim and say, all right, guys, go to work.
That's just a nightmare.
[00:31:15] Speaker A: Well, you have to have a concentrated effort on false positive reduction, just not monitoring alone.
And either you have a dedicated analyst or a dedicated engineer to bring those numbers down, or you have to have a philosophy about what you're going to monitor for figuring out how many events can an analyst realistically investigate a day. Because if you get over that number, are you going to really doing monitoring at all? Because they just can't grasp it. They can't be done.
[00:31:49] Speaker B: That's a leadership function that's got to be done at kind of the leadership level. They've got to say, I am willing to miss alerts by turning this down. And I've yet to encounter someone and kind of the management level who's been able to say that. They always say the same things like, well, if we did better false positive management, if we had better logic and better detection, we wouldn't have to ignore stuff, which is great, and it sounds really good, but the reality of the situation a lot of times is you're going to have to ignore something.
A lot of times I think that happens when analysts find something or they find a new source and they realize that, oh, you know what, it doesn't look like it's very valuable. And then maybe they just don't mention that source. They don't bring it up and say, hey, I found a new source. I think this is important. We should definitely monitor this well, leadership.
[00:32:39] Speaker A: Has to look at it in the context of risk. It's just another risk function, and organizations accept it and manage risk every day.
I think you're just running into security management or security leadership who's afraid to say, we are going to accept this risk of turning that off and saying, this is a risk function, that's a risk determination. And maybe if you could jigger together an algorithm or your organization and say that anything at this false positive rate, based on these metrics, anything below that, we are going to, in our documentation, say that is an acceptable risk for us and get someone to sign off on it. You really need the lower tactical leadership being able to pitch that to your CISO and CIO level leadership.
[00:33:39] Speaker B: I think actually the disconnect there is probably, in terms of verbiage, I think you're right. I think I haven't been phrasing it as a risk based decision rather than I think using the word ignore probably has a real negative connotation there.
[00:33:52] Speaker A: Right. Versus assessing level of risk and accepting a certain level.
[00:33:56] Speaker B: I just want to do security. I don't want to talk about all this business.
[00:34:03] Speaker A: All this risk stuff.
[00:34:05] Speaker B: Yeah.
That's all. Compliance.
[00:34:08] Speaker A: Yeah.
[00:34:09] Speaker B: Tell me where the bad guys are.
[00:34:11] Speaker A: I used to have friends that said, friends don't let friends play risk.
[00:34:17] Speaker B: That's a good starter game when you're in middle school.
[00:34:20] Speaker A: All right, so getting into some of.
[00:34:23] Speaker B: The monitoring stuff, this is real interesting. This is real interesting. So one of the things that I noticed on here that I called out is no single tool or service showed up in all of the responses. I could not believe that. I have a hard time believing that 100% of the socks are not monitoring antivirus. Some of the other ones I get. I get UeBA. Yeah. Not everybody has it. Not everybody's monitoring it. Cloud stuff. Maybe the isn't in the cloud. Maybe security is a little bit behind the eight ball there, but maybe web proxy, maybe they don't have a web AV and network ids or host based ids. That's the bread and butter of a sock. How do you not monitor that? Ids comes built into most firewalls now.
[00:35:15] Speaker A: Yeah, it's really od that you didn't see a uniformity there.
And some of the other stuff is that the numbers changing over time also didn't seem to make sense. So more organizations were monitoring next gen firewalls and antivirus in 2019 than they were in 2020.
[00:35:39] Speaker B: Did they just decide to stop? The firewall doesn't provide us much value. We don't need it.
[00:35:46] Speaker A: You know, what it might be also is a change of responsibilities. So if organizations may be shifting away from who's going to do what, and in shifting that away changes what's being monitored and what's not. So if you have your it organization doing more of the day to day management of some of those infrastructure security apparatuses, it's possible that that's where those changes are taking place and that reduction is happening.
Could be.
[00:36:19] Speaker B: And this may be part of where the 30% responses in it may not know exactly what they're monitoring too.
[00:36:27] Speaker A: Right. And that comes down to, we were talking about definitions and stuff like that before, is that some Places say that if it goes into the log management system, that's a check marks for monitoring, but some other places say you send it to the log management system and then you generate alerts off that which show up in your sim. Now that's monitoring.
So you're going to get some nuance here as far as how all these things are going to flush out based on people's perspectives.
[00:37:01] Speaker B: Yeah. And this may be related, as we mentioned before, about the Sim cost seems awful low, too. Whereas if you're not bringing in data from 20 different sources and you're not bringing in the endpoint data and you're not bringing in the firewall data, you may not really need a big beefy sIM with a license for multiple gigs of data.
[00:37:20] Speaker A: Right. And if you're filtering stuff at your log management solution to weed out, then you can take advantage of those lower SIM numbers overall. All right. One of the other interesting things that came out of these numbers here was that the perceived importance of IR has increased while the importance of collaboration with it has gone down.
[00:37:44] Speaker B: I can kind of see why IR has increased in everybody's mind. There's been a lot of kind of really public. I'm not talking about the last six months because I know this was for 2020, but especially towards the end of the year, solar winds and a bunch of high profile things. I can see why IR has gone up. Not sure why collaboration with it has gone down, though.
[00:38:03] Speaker A: Yeah, I don't have a hypothesis there because it doesn't make sense to me. Unless security has taken on more of the management role in it is taken less, which doesn't make sense considering what I said a minute ago.
[00:38:18] Speaker B: So this is unrelated.
[00:38:19] Speaker A: Kind of.
[00:38:20] Speaker B: But I recently heard I had a recruiter hit me up and ended up not doing anything about it. But the recruiter mentioned that IR has been real big lately. There's been a surge in need for IR consultants and employees starting in about April last year. So I don't recall what happened around April. I don't know if there's a big incident, but yeah, he said that there's been a huge surge in looking for IR people.
[00:38:48] Speaker A: Maybe it was just Covid.
I don't know.
[00:38:52] Speaker B: Yeah. The work from home thing, maybe. Maybe it led to a lot more stuff that their it help desk couldn't help with since everyone's remote.
[00:39:01] Speaker A: All right, so now drilling down into the financial numbers around the numbers and the financial numbers around the actual people within the SoC. So the estimated SoC size is twelve with an estimated engineering size of six.
And 30% of organizations said they will hire six or more analysts in the next year, but only 38% say that they can get the right talent, which is interesting because everybody's been talking about a talent shortage within security. And I think the key point here to highlight is the right talent. So I don't think there's necessarily a lack of individuals that are available within the cybersecurity space, but the individuals that fit your need are more difficult to find than most organizations would like.
[00:40:12] Speaker B: Yeah, we've seen this before. I mean, I remember when I was about to go into college in 1998, dating myself here, and at the time I was going into computer science and all the talk my freshman year was about how mcses could make $100,000 and that was the certification to get because that was what was in demand at the time. And the supply hadn't risen up to the level of the demand yet. And then for a couple of years, there was the whole lawyers thing. Lawyers were making crazy amounts of money. And as soon as people saw lawyers were making all that money, they went into school. And about a decade later, the salaries for lawyers has dropped way down since then. And that's just where we are in this stage right now, because there's been such a huge additional demand. The salaries are going up, people are struggling. I've seen this in the past where it's easy to get tier one people, but it's really hard to get experienced people because there just aren't that many and they can demand higher prices. But what's going to happen is in five or ten years, the schools are already pumping out like the tier one socks right now. But as they get that five or ten years experience, over the next five or ten years, all those high level roles are going to be filled, the salaries are going to drop.
We're in kind of the golden age of security. Salaries right now, I think. I think that if you're in security, you probably know you're making pretty good money. You're not making surgeon money, most likely, but you're making real good money compared to the average salary in the US. And it's not going to last. We've got another couple of years of it, and I don't know what's going to happen at that point. I don't know if we get to keep our salaries. If they're going to start firing us, they can hire cheaper people, but we are definitely in a salary bubble.
[00:41:51] Speaker A: Yeah. I think what this doesn't show, which you would think, would be that there would be something here that would show that organizations are spending more time taking their tier ones and moving them or progressing them up to the higher tiers. Because if it's really hard to actually find external sources for higher level tiers, you'd think you'd see more spent on building those capabilities within the organization. But with the turnover rates that they're showing here, it's like people are going from tier one over here and probably getting hired as a tier two over here versus moving up within the organization from tier one to tier two. Because if you're leaving that organization in less than three years, it's certainly possible for you to move up a few levels, but you're not going to get very high in that short amount of period of time. Yeah.
[00:42:50] Speaker B: Corporations are really resistant to rapidly promoting people even when they're worth it. Definitely seen that in the past where people that come in, maybe especially career changers, where they have a lot of transferable skills and they can progress very rapidly. They go faster than HR expects them to. They go faster than HR rules allow them to. It's definitely an internal problem. I agree, though, especially with the people saying that only was it, 38% can get the right talent, you'd think they'd be really focused on finding the good ones out of that tier one and really kind of pouring money into their education and really focused on keeping the right people.
[00:43:33] Speaker A: Yeah. Well, if you're in one of those organizations that's just throwing alerts, tons of alerts to analysts, they just might not be willing to put up with it that long to move up to the next level.
[00:43:47] Speaker B: If you're doing that to your tier one analyst, you don't actually care about your tier one analyst anyways.
[00:43:52] Speaker A: True.
[00:43:52] Speaker B: Yeah.
That's a hot take, but I'll stand by it.
[00:43:56] Speaker A: I would agree with that. All right.
And the final thing in here is that 46% of the respondents said that they expect salaries to increase in 2021 as well.
[00:44:11] Speaker B: It's not really surprised. I mean, as long as we're in the bubble, they're going to keep going up until it either becomes unsustainable or the supply catches up, right?
[00:44:18] Speaker A: It's a matter of reaching that market clearing price, if you will. For Sock analysts, what I find amazing.
[00:44:27] Speaker B: Is I don't think you hit the estimated salary for a tier one SoC analyst, and I even called out a tier one SoC analyst. I think what the median so 50% of the respondents said that a SoC analyst made 50,000 to 100,000, and then 50% said a tier one analyst made more than $100,000. So that makes the median $100,000 for tier one analysts. And that is utterly and completely ridiculous. As some examples, I would like for you to, dear listener, to compare these in your head as to whether they are more valuable than a tier one SoC analyst or less valuable than a tier one SoC analyst. A garbage man makes 35,000 a year on average. Would you rather have another SoC analyst or would you rather have garbage men?
Starting teachers make 38,000. On average, a network administrator makes 60,000. On average, a Windows administrator makes 75,000. This is actually what I was talking about with the MCSE. Windows administrators were in shortage 20 years ago. They could make more money, although admittedly that was hearsay while I was at college. Who knows if that was real or not, but that bubble popped. Accountants make 52,000, lawyers make 76,000. I'm not citing any sources for these. I just went on Google and searched for average salary lawyer, so feel free to update it with your area salary or whatever. But the salaries for the tier one Soc analysts are utterly and completely ridiculous. They are not represented in the skill and knowledge level needed for a tier one analyst. I would say that most windows administrators probably know more than tier one Soc.
[00:45:59] Speaker A: Analysts, which is why you should not necessarily turn them down when they come to apply for your tier one analyst, even though they have no security background.
[00:46:07] Speaker B: And this is something that I found when I was breaking into the business. I started off as a teacher and then I moved to a network administrator and then I moved into security. And each time that I moved I got a higher salary. But that higher salary was the starting salary of the place I moved to.
So it's just super weird to me. I happen to be married to a teacher. I know she works way harder than I do, and it blows my mind the way that society prices these things and I get it. It's profit based. It's what makes the company the most money and or a supply and demand thing. And there's lots of people that want to be teachers. Don't think there's lots of people that want to be garbage men.
[00:46:47] Speaker A: Well, most teachers are also public sector, so you expect those salaries to be somewhat depressed.
[00:46:53] Speaker B: Yeah, I would love to see a little more market dynamics at work there. Let's introduce some. Yeah, I was a teacher for a couple of years, and the chemistry teacher at my school who worked super hard, she stayed late every day setting up experiments for students. And the government teacher who left every day and goofed off all day and sent his kids to all three lunches. They both got paid the same because they both started the same year.
This is not a podcast about salaries or about my own personal feelings about school teachers.
[00:47:22] Speaker A: All right, so let's get down to some of the last couple of items here. So Soc organizations who claim to be high performing, these are some overall information. Unfortunately, there's not a lot of details around here, either what we're going to relay to you or actually what's in the report. But higher performing SoCs have lower turnover. They do more hiring. Their organizations are supposedly less complex, which means less painful.
They have a better opinion of their mssps, and they are less impacted by the transition to work from home. And those last two, the MSSP and the work from home, were hugely different from what the other organizations reported. Like 40% different, whereas the lower turnover, higher, more, less complex. They were much smaller percentages, 10%.
[00:48:28] Speaker B: That makes a lot of sense. I feel like, especially for the MSSP relationship, if you don't really interact with your MSSP and you don't try to force them to meet your needs, they're just going to give you the big box service.
Here's our black box. Here's our standard content. And standard content is probably pretty mediocre and super generic. So you've got to work with them. You've got to tell them, like, we care about this stuff. We want you. Here's some custom content we thought about. Does this work with you? Here's some new data sources we found. What do you have here? All right, well, let's work with you, and let's create some content for it. If you're not performing those actions with your MSSP, then you're getting subpar mediocre service. 100%.
[00:49:08] Speaker A: Yeah. So that relationship, if you're going to invest in an MSSP, you have to invest in time, in the MSSP as well. And not just the, not. You've.
[00:49:19] Speaker B: You've got to have people that work with the MSSP every day. You can't just completely outsource it and then look away and just ignore to, you have to have at least one person who works with them and probably more.
[00:49:32] Speaker A: I think the assumption is you hire an MSSP and then like, well, call us if you need us.
[00:49:37] Speaker B: Yeah.
[00:49:38] Speaker A: And they know are waiting for the call from the MSSP. And that's it.
[00:49:43] Speaker B: It's also interesting about the impacted by work from home. I think that probably has to do with hiring better people.
[00:49:51] Speaker A: Well, actually, my theory on that one is that because they have higher quality employees, because they are already remote and they hire people from wherever and not strictly tied to the SoC resources or the cybersecurity resources within a certain location.
[00:50:12] Speaker B: No, you're right, because I imagine, like a company headquartered in, say, dayton, Ohio. Not that Dayton, Ohio is not, I'm sure, from nominal, but their tier one SoC analyst population may be somewhat constrained.
[00:50:24] Speaker A: Well, imagine they're tier three or tier four SoC analysts.
[00:50:27] Speaker B: There's like two in the whole city, and they're already hired by your competitor.
[00:50:33] Speaker A: So I think they were less impacted from the work from home because they already had that kind of model.
[00:50:41] Speaker B: Yeah. This doesn't tease out cause versus correlation.
[00:50:44] Speaker A: Right.
[00:50:45] Speaker B: I find actually it interesting that the high performing socks describe them as less complex. I would have a hard time believing they're actually less complex. Well, at least maybe I just want to believe in my heart of hearts that they put effort into making it less complex for the analysts. And again, coming from kind of the position we talked about before, where maybe they put the effort into the soar to automatically tie identity to machines and maybe automatically so ip addresses to the processes that made those network connections, stuff like that. So the analyst has to spend less time hunting for the bogus, boring stuff that wastes their time, their highly paid time, and just have all the pertinent information at their fingertips.
[00:51:27] Speaker A: Right. I think what they're really saying here is not that they are less complex, but they're less complicated. They're using automation to reduce that complication. But the automation is itself complex because you can be complex without being complicated. I think that's really breaking my brain. Talk about the distinction in language here that they are probably actually more complex than the lower performing socks, but not as complicated.
[00:51:57] Speaker B: Yeah, I would have loved to have seen they didn't talk at all in here about how many alerts SoC analysts are reviewing, but I would have loved to have seen, like a direct correlation.
I'm sorry, I guess a direct comparison in the number of alerts being seen versus how well off the SOC employees are and how happy they are with their job and how complex they say it is.
I'd like to think that there's a direct line correlation there in terms of keeping SOC employees and making them happy versus how many alerts you make them review.
[00:52:27] Speaker A: Right. And I think if you could tie that also to the spend, you could say, oh, and this is the average cost of investigating an alert.
That would be interesting. But I think that would also probably be a swag for most organizations because they're not tracking that. They don't have the metrics around it, probably.
[00:52:48] Speaker B: I feel like we've had this conversation before, but not on the podcast. But it would actually be, because I would love to see that compared to the true positive and false positive rates of an alert like this alert fired 1000 times last year, each alert took, on average, 15 minutes. This took 250 hours of our time to investigate average analysts. Let me see, 100,000 divided by two, it's $50 an hour. So 250 hours times $50 is 2500, times five is $12,500 to investigate this alert. And then you can kind of make a comparison there that may be better at tying it to business things like actually putting dollar amounts on how much each alert took to investigate. That'd be interesting.
[00:53:33] Speaker A: Well, you know, if people are leveraging their ticketing system, well, I guess that doesn't necessarily mean that their ticketing system is tied into like a PPM, where they have the hours for resources within there. But if you're doing that kind of thing, you could maybe leverage your ticketing system or your PPM system in order to kind of tease those numbers out.
[00:53:54] Speaker B: Yeah, because then you can actually figure out, how much are we paying to find one bad guy and compare it across ticket types.
Our AV finds bad guys at the rate of $10,000 per bad guy found. And our network ids, we spent $20,000 because it generates all these false positives. Well, you can tie that too with that.
[00:54:15] Speaker A: Well, you could tie that back to that conversation we had a minute ago about accepting risk, say that keeping this alert on costs $50,000 per year, but the expected risk of failing to respond to that is 30,000 or something like that.
[00:54:36] Speaker B: Yeah, because then that's the other side of it is you don't really know necessarily, because one company doesn't have great data on how often bad guys are detected by ids versus AV versus which one's more likely to catch an actual real attacker that has a higher chance of actually ransomware in them versus the random bs commodity phishing emails or whatever.
[00:55:02] Speaker A: That's why I think it's important to have your incident details figured out within your case management system so you can tease that out and feed that back into your sim. To say, we had this many alerts investigated. This many turned out to be true positives. And the source of these were alerts for these tools. So in the past year, AV was a winner, firewall was a loser, blah, blah, blah, et cetera, et cetera, on where you're getting your return on your investment for your tools.
[00:55:33] Speaker B: Hold on. No, actually this is where you tie in that threat intel component because if you can tie it to one of the generic gangs like, oh, this is from the emotech gang, it's commodity like, it'd be bad if it hit, they'd ransomware us or whatever, but it's commodity versus this one doesn't have any threat intel hits. It might be custom. And then maybe you could at least assign sort of a high or low impact to the attacker, but you can't tell exactly what they do. You can't tell, but you might be able to at least kind of give an idea of high value and low value attackers. That would be really. I kind of want to do that now.
[00:56:06] Speaker A: Good luck.
[00:56:07] Speaker B: Yeah, hold on. I'm going to have to take some notes after we get off because I'm kind of thinking about.
Yeah, because then what you could do is you could turn around and you could say, like, it costs us this much for this alert to find high value attackers and this much for low value attackers, and then you combine all of your alerts and then that gives you a framework to compare a new alert against. When you do a new alert and you're doing your testing and you go back a month or two in your sim and you look at what would we have found?
You might be able to kind of get an idea for, oh, if we'd had this alert, it would have found two low value attackers and no high value attackers. But you can't just willy nilly throw new alerts at an analyst. You've got to have some method of determining what alerts are better and what alerts are higher value than the other. And this might be an interesting way to do that.
[00:56:53] Speaker A: Yeah. If you had a content development team, that if you actually had a broken out content development team where they were separate from the monitoring team, they could be more focused on identifying and collecting those metrics and figuring that out.
[00:57:05] Speaker B: Yeah, that'd be a tough row for a small team, for sure. All right, I'll stop distracting us, but that seems interesting.
[00:57:13] Speaker A: All right, so the last item on here is, what does it say about the future of the sock?
[00:57:20] Speaker B: Yeah, I marked down a couple of things on here about. I was really looking kind of at the services that aren't being used by a majority of socks, figuring that they're probably fairly cutting edge. And it was kind of interesting because there are only three services that were not offered by all the socks, at least not in their environment. And the first one was threat hunting. As a service was only offered in 46% of socks, incident response as a service was offered in only 45% of socks, and deception and countermeasures were only in use by 17% of socks. I don't understand the incident response only being a service in 45% of socks. I don't get that at all. It makes zero sense.
[00:57:58] Speaker A: The only thing I could figure is the outsourcing of it.
[00:58:01] Speaker B: See, that's the part that you insource. You outsource the detection and monitoring, and you insource the incident response.
[00:58:07] Speaker A: Although maybe you have Mandian on retainer.
[00:58:09] Speaker B: No, but Mandian is not called in for everything. Like incident response includes. This guy clicked on a phishing email and his account was compromised. That's incident response. It's not fun incident response.
[00:58:19] Speaker A: But I was being a bit facetious.
[00:58:21] Speaker B: No, but you're probably right. And that's probably what they were thinking. They probably weren't considering the minimal malware mediations. They were probably only thinking about the big ones.
[00:58:30] Speaker A: Right. They were thinking breach response, not necessarily everyday incident response.
[00:58:36] Speaker B: Yeah, I find that deception countermeasure is only in use by 17% of socks is interesting.
It's not really a surprise, honestly. Like, their job is already complex enough, and adding deception and countermeasures as an additional layer on top of that is really typically something only really mature socks do. That's not really a surprise.
[00:58:55] Speaker A: Yeah, because they're already buried in what they currently have. And since deception and countermeasures is new, it's like, do we really have the cycles to take on additional detection? And does your engineering team have the cycles to deploy such thing?
[00:59:10] Speaker B: Yeah, and then keep track of it too. That's something I've seen in the past, where you create fake accounts or fake servers, and then a year or two, someone leaves, somebody else comes back. People are, like, spinning up this weird vm they found or logging into this weird account they found.
Right?
[00:59:26] Speaker A: Yeah. If you're going to build that in house, you got to make sure you have a good, solid foundation for not only deploying it, but maintaining it over time.
[00:59:36] Speaker B: I'm not really surprised about the threat hunting as a service and only 46% of socks. That's definitely yet another on the maturity scale one. And that's one that everybody loves to talk about, but people hate to really devote resources to, because again, you've got to stop monitoring something else. If you've got seven analysts and they're all super busy all day, and you're like, we need to do threat hunting, you've got to shut down some alerts or find some way to open up that time.
[01:00:00] Speaker A: Right. I mean, it would make sense to have, like, a dedicated threat hunter spot and rotate people through it.
You're the threat hunter for this month or something like that. And then that also may help reduce the burnout because it adds different challenges to the day to day for analysts.
[01:00:18] Speaker B: Yeah, I am 100% a fan of rotating duties. I know I talked with somebody, I don't even remember the company, but it was one of the big four, and they were talking about how their analysts are only on for one week at a time, and then the other week they're working on projects or they're doing kind of private projects or threat hunting, which seemed to me to be of kind of a very enlightened view of security operations and ir work. So really help either prevent the burnout and then also make sure that they had time to work on tools to automate the stuff they're doing right, and.
[01:00:46] Speaker A: It gives them a different perspective and adds to their understanding when they're back looking at the console.
[01:00:55] Speaker B: Also, I added some future data sources. These, again, were data sources that were used by under 50% of socks. I don't know if I trust the original data here because, again, this is the one where not 100% of socks were doing antivirus, but cloud native security tools is only 48%. It kind of makes sense to me. The cloud is still new on a lot of people, a lot of smaller companies. Radars, web proxy, 44% of socks. That's kind of a surprise. Proxy is a pretty mature technology. I am really surprised that so many companies either aren't using the proxy or aren't watching the logs.
[01:01:28] Speaker A: If they are, well, and proxy, at least from my opinion, is crucial, really, in reducing the success of phishing campaigns. Having those links pre identified by your proxy as potentially dangerous and in the proxy not allowing out.
Hopefully your proxy would be doing website categorization and saying that, all right, newly registered domains, you can't go out to those. Or uncategorized domains, you can't go to those. I've seen both of those two things alone, preventing a huge number of infections.
[01:02:12] Speaker B: Yeah, 100%. So that seems to me like one of the basic, I mean, your basic is your network ids, your host based AV, your web proxy, your email protection. I wouldn't even start a company without those protections in place.
Next one down was CASB cloud access security broker, 43% of socks. That kind of makes sense. Again, not everybody's in the cloud yet, and those that are, a lot of them are just kind of think it's a fun fairy wonderland where you can do whatever and the security is taken care of by the cloud provider.
[01:02:44] Speaker A: I think the value in the CASB for organizations that aren't in the cloud yet is figuring out actually that they are in the cloud but don't know it yet because anyone can subscribe to a SaaS service.
[01:02:57] Speaker B: You're right when I say that not everyone's in the cloud yet. What I mean is not everyone knows they're in the cloud yet.
[01:03:04] Speaker A: Right?
[01:03:05] Speaker B: Yeah. And then finally UeBa at 39%. I'm actually surprised that's as high as it is. I don't understand how UEBA is a data source looked at by 60% of socks, and yet things like firewall are only looked at by 60% of socks. I really don't understand that at all. It makes almost no sense to me. But sure.
[01:03:30] Speaker A: Well, that's some of the stuff we don't understand about this survey. Yeah.
[01:03:35] Speaker B: And I've heard things about Ponymon in the past. I don't know if those things are true or not, but I generally are some concerns, and there's some things in here that are a little weird, and some of their questions, I feel like, could be a little bit more precise. But.
[01:03:51] Speaker A: So that's a summation and some of the high points or areas that we thought were of interest within that survey. Take a look at the show notes and pull down the survey for yourself and look at those numbers and let.
[01:04:05] Speaker B: Us know how you feel about this. Is this going over kind of a report like this in detail, something that you think is valuable to you? Come be follower number three on Twitter. And we got follower number two during the podcast today, by the way.
[01:04:21] Speaker A: Wow, that bot is finally starting to work. Awesome. All right, well, thanks for listening to security Serengeti and follow us on Twitter at Serengeti Sec and download and listen to us on your favorite podcast app. I'm.
