ISMG Editors: Breaking Down OT Cybersecurity Challenges

Transcript

This transcript has been edited and refined for clarity.

Anna Delaney: Welcome to the ISMG Editors' Panel. I'm Anna Delaney, and this week we're delving into a significant cybersecurity discovery, exploring the implications of a backdoor found in a critical Linux utility. Also a CISO's comprehensive approach to cloud security and key insights shared at the Cybersecurity for Critical Assets Summit in Houston last week. Joining me today are Tom Field, senior vice president of editorial; Suparna Goswami, associate editor at ISMG Asia; and Mathew Schwartz, executive editor of DataBreachToday and Europe. Good to see you all. Mat, let's start with your story. Because it's a big one. A security researcher has uncovered a backdoor in a critical Linux utility, as I mentioned in the intro, believes to have been deployed by nation-state attackers, aiming to gain full remote access. Now, we know that the vulnerability was identified and mitigated before widespread exploitation. But it is a huge story, as I said, in terms of what the implications might have been had it not been detected. So why don't you just talk us through what happened?

Mathew Schwartz: Yeah, we dodged a big bullet with this one. This could have easily been worse than SolarWinds, security experts are saying, because of just how far reaching Linux is these days. So we're talking about a piece of code called XZ Utils. This is a compression and decompression tool that is built into pretty much every distribution, every major distribution anyway, of Linux. Thankfully, this erroneous code in XZ Utils got discovered by a developer at Microsoft. He noticed that the utility was doing some funky stuff, it was taking too long, and just there was some weirdness. And so to his credit, he dug in, and thanks to him, looking into this and posting to an open-source mailing list, there is a backdoor in this, we averted disaster, it was probably just a few weeks away from when this would have worked its way into major Linux distributions, like Red Hat Enterprise and just been everywhere. Tear downs of this code are still underway. And they're complicated somewhat by the fact that this appears to have been a sophisticated effort, which is what has a lot of people saying nation state attack. And there's some other reasons for that as well that we can touch on in just a moment. Researchers aren't even exactly clear how this would have worked. It looks like code was altered in XZ Utils, so that at a later date via SSHD, or remote attacker could introduce code to the system and have it run. But they were looking to get the capability to do this, so embedded that it probably wouldn't even have been noticed for a while. And it would have allowed attackers to execute any code of their choosing, this would have been incredibly powerful. So where are we? Where we're at is the person who is developing this appears to be blameless, like a lot of critical parts of today's software, open-source ecosystem software, there was one person maintaining this very useful and widely used tool, and this person has other stuff on their plate. This was a hobby, maintaining this. What was interesting about this is it appears to have been a very patient nation state attacker, who, two years ago, and we're deducing here, but what appears to have happened was a couple of years ago, someone started saying to this code maintainer, oh, I need this feature, I need this functionality. How come you're not spending more time updating this code, you should look at this, and this isn't fair. So turning up the heat a little bit, psychologically speaking, after a little while, through a confluence of events, lo and behold, this miracle developer shows up, he's like, hey, I got some free time, I see all this flak that you're getting for not maintaining this code. Let me introduce you to another friend of mine who's a great developer, he can help you out. So there's this setup, where they socially engineer the person who has the right to maintain the code. And he ends up sharing the code maintenance capability with this miracle developer who has been parachuted in. And that is where it appears that the backdoor code has been inserted. So again, a lot of things that we don't know about the attack. What we do know, it's been stopped. The backdoored code did get into some beta versions, or some rolling development versions. And people have flagged what those are, and said, please downgrade to what we know, to be safe. The guy who is maintaining the code, legitimate guy, Lasse is his name. He said, look, I'm digging into everything that's happened. Use the safe version for now. I will put out a newer version that we know is safe. And I'm sure you can expect everyone to have a close look at that for obvious reasons. But that is forthcoming. There's a lot of tear downs, like I was saying still happening of the backdoored code. A lot we don't know. But the longer the short of it is, thankfully, this was discovered; raises some big questions about if any other code has been backdoored. And we just don't know it. There is a huge attack surface here. And this is the first time we've seen people attempt to mess with open-source components, won't be the last, raises big provocative questions about if anything is going to be done to fix this sort of thing. It's going to take time, attention, funding, and all those sorts of things that we don't seem to see in abundance, unfortunately, with a lot of open-source software.

Delaney: I think the most interesting part of it is how the backdoor got there in the first place that you explained very well there. How do you think this incident affects the trust in open-source software? And what are you hearing in terms of solutions? You mentioned resources there, but what can we do in terms of securing this open-source world?

Schwartz: We've seen some efforts in the past when massive vulnerabilities have been found in critical open source components. The Linux Foundation has some funding that they have been putting into components. But I don't even personally understand the scope of the problem here. You have a compression decompression utility in Linux, that people were able to subvert in such a way that they could have made Linux do whatever they wanted at any point in the future. All these critical components, I don't even think we have a full understanding of the supply chain risk here. So hopefully this is going to prompt a lot of probing questions, but this isn't the first time this has happened. And it's not like the floodgates have been opened. And all these open-source components are getting the time attention and love that they need to be getting to help prevent this.

Tom Field: Days for Log4j, Mat, and I'm thinking that even a year ago 25 to 30% of new downloads of Log4j with the infected version. So prediction, we're going to hear a lot of moaning, we're going to see some hand wringing. But I don't know that we're going to see any significant changes. I don't know what significant changes could be imposed. And we're going to wait for the next incident.

Schwartz: Yep. And it's going to happen. Yep, I agree.

Delaney: Any advice to organizations right now?

Schwartz: That's a good one. Kudos to the Microsoft developer who dug in and found this. I think we're still waiting to see what if any takeaways we have here, except that we need to show more love to the open-source ecosystem.

Field: It is open source, it's maintained by volunteers. As Mat says, there are too few hands on it too few eyes watching it and too many potential vulnerabilities. This is another incident just waiting to happen.

Suparna Goswami: Tom said too few people watching it, as too many people using it as well. Mass of people who just swear by open source.

Delaney: Now, Suparna, you've recently interviewed MatchMove CISO about his comprehensive approach to enhancing cloud security. And I know that you covered a range of issues, including compliance challenges, and bridging skill gaps through training and automation. Just tell us about it.

Goswami: Yes, to the CISO was one of the award winners at DCISO awards for cloud security. So before I begin, I must say that DCISO Awards not only recognizes the best from the industry, but also so much to learn from these case studies. For example, the interview I will be speaking about today with some Samrat Bhatt, who happens to be the CEO of MatchMove, which is a fintech company here in India, he won the award for cloud security, but likes to call his project, integrated security transformation project, because he believes that if unless you take care of all other things, you can't just secure the cloud. So you have to take care of other things as well, like endpoint security, automation, everything needs to be done. So before he started, it was a startup MatchMove was relatively a small company when he joined. So there were a lot of areas that he identified that needed immediate attention. So one was, of course, because it's part financial industry, compliance with regulatory standards, data protection were top challenges, then there were various issues in your identity and access management, vendor risk management, incident response preparation, cloud governance, and those need to continuously monitor and DevSecOps. So these were some of the priorities. And of course, one of the critical challenge that he faced, in fact, he said that, for him that was the most challenging part, was the lack of cloud security skills, both among the IT staff as well as the security stuff. So of course, now that he had to implement cloud security, he just couldn't focus on that, he had to have a concrete plan, which took into account all these things. So that's why he named his project integrated security transformation. And for him, an integrated approach was probably combining training programs, your automating tools, then implementing advanced security solutions, specifically cloud-native solutions. Then, of course, he took care of the endpoints and targeted awareness initiatives; that was majorly something that he focused on. And he divided the project into four quarters. So when I spoke with him, you'll hear in the interview, which should be published this week, he explains it very nicely, I'll just give a brief of what he's said. So he divided the project into four quarters. So the first quarter was focusing on uplifting IT skills, because he said that unless and until those skills were imparted, he couldn't go ahead with the project. So an interesting thing that I would like to mention here was that he trained his IT staff to have knowledge on cloud as well, because he believed that going forward IT admin will need to have knowledge of the cloud. So he tackled the industry-wide skill shortage by focusing on training and development. And he specifically had vendor supported training a lot. Every week, there were vendor supported training. And he emphasized those both on technical training as well as the concepts of information security. And needless to say, he had to automate a lot of tasks to manage the skill gap and enhance operational efficiency and I'd tell you later, like what percentage of operational efficiency improved. Plus another thing that he mentioned, which I found interesting was that he wanted the IT and the security teams to work closely. So he organized sessions every week where the IT team will teach security teams about specific tools in the IT and vice versa. So they could both work together. So if one was absent because it's not a huge team that he had, the others could for the time being take care of things. So that was how the first quarter was spent. Then second quarter he focused on automation. He automated DLP monitoring, he automated exception monitoring, vulnerability analysis. So all these things were automated. And in the third quarter, he implemented next-generation security in which to secure the endpoints, he went for the endpoints first. And here he what he mentioned was he held regular meetings with the C-suit the business, hence, developers, to let them know from where the threats are coming, to let them know that what exactly were the challenges in the endpoint. He said that it was very, very important to let all of them know that this is a threat, this is what it will result in so that they take all these things very seriously. And the last quarter, and finally, the fourth quarter was about the cloud-native security solutions. It was basically on AWS. So this is how he said the project was deployed. Of course, the interview has the details. But what particularly impressed me was that though I initially approached him only for having an interview on cloud security, but the way he explained me the way he approached the entire project that caught my attention, and that shows that equal attention needs to be paid to everything; if you're just securing the cloud, it will not help; how he had to secure everything to ensure that okay, the final product is successful.

Delaney: Did you discuss the benefits to the business? I'm just curious to know, what were the tangible and intangible benefits to the business and how does Samrat measure these?

Goswami: Well, of course, so like I said, operational efficiency, he said it led to 20% reduction in manual processes. And it improved ROI by at least 30%. And intangible benefits. Of course, there were a lot he said there was a culture shift towards security. He said employee confidence increased as far as security is concerned, agility, adaptability, then, of course, it enhanced the reputation also. And the cybersecurity score rating which was in 60s and 70s, when he joined late 60s, early 70s was above 97-98 out of 100 by the end of the project. So there was quite a bit of ... and that probably what impressed the juries as well, when he won the award, that in one year there was a massive shift in the overall culture of the organization. So it was a lovely conversation that I had with him. So it was pretty interesting. Great. Well, thanks for that, Suparna. We look forward to watching it soon. Tom, you've just returned from Houston, where you were busy interviewing speakers and attendees at the Cybersecurity for Critical Assets Summit. What were the key trends, takeaways that stood out for you?

Field: But first a commercial because OT is the word of the day - operational technology. And I want to make sure we have this announcement to our audience here that just last week announced at the Cybersecurity for Critical Assets event. We have launched our newest media property, OT.today. So you can look that up OT.today. It is the latest media property from ISMG, it is focused entirely on operational technology. Now, backdrop for our conversation today. Maybe two weeks ago, we as an editorial group were visited by Dawn Capelli, from Dragos, who heads up the OT-CERT. And she shared some information with us that I think would be shocking to almost anybody talking about small public utilities, where the person who is in charge of cybersecurity is also in charge of cutting the lawn. And we hear and I hear this consistently talking with CIOs and CISOs, about oil and gas, chemical, manufacturing facilities that all have that one or two critical systems that are dependent upon Windows 95. And long, outdated technology. It's as frightening as the picture that Mat painted about open-source in the software supply chain. So great focus on OT, we went to this event in Houston, which was hosted by our new acquisition QG, entirely on cybersecurity in the OT space. I spent time there, we put up a pop up video studio where I was able to interview about a dozen security leaders - people in IT, people in OT, people that are in both disciplines, and talking about a lot of the issues that are at the heart of being able to secure these critical systems. So I wanted to share some of these today, but the videos aren't ready. We'll have up on our site probably within the next several days. But among the topics we discussed, were life cycle management for these legacy systems. How you approach that how you get funding for that and make this a business risk that the organization has got to pay attention to. We talked about building and maintaining cybersecurity programs for OT, where this hasn't been a long standing discipline, as has been with IT. We talked about workforce management, getting the right skills, being able to develop the people you need to manage and maintain these systems. So there's a lot of fundamental work going on to create and maintain these security programs for systems that are at the heart of our critical infrastructure in so many ways. And of course, the challenge with all of these is, how do you address cybersecurity issues, and be able to have such resilience and such quick response, that you don't take down production cycles, or interrupt a utility? , these are systems that can't come down for any reason, and how do you maintain them, well, without having to take them down. So lots of topics, I'm excited about our new property, the opportunity to talk to more thought leaders, create compelling content and educational opportunities, because as much as I've been talking about, as Mat has, software supply chain for the last couple of years and have concerns about that, the concerns are very much parallel with operational technology. I'm glad that we have this new property that where we can talk about this.

Delaney: Lots in there. Did your interviewees propose any insights or solutions for overcoming these cybersecurity challenges in critical asset environments?

Field: I guess, one, acknowledging there's an issue with step one, I hate to make this like a 12-step program. But it almost is. We have to acknowledge there is an issue. And that comes from all aspects of the organization. Doing that we've got to build cybersecurity programs with legs and with teeth that can sustain and can be able to bridge cultural gaps between IT and OT. This is a space where people that own this technology have been on their own for many years, and they have suspicions of people coming in and trying to offer new solutions that might impact their production cycle. So there's a lot to be done here. And as we spoke to one leader in particular, a CISO that has been an IT security for decades, now, OT is part of his remit. And there's a big educational curve there for him to learn more about this. I think that's not particularly unusual. I don't think he's necessarily the exception. So there are lots of discussions to have, it's nice that they're having those and this was a well-attended event. So I'm glad that we're giving them the forum to move forward here. But there's a lot of work to be done quickly.

Delaney: Well, we can't wait to watch your interview. So they're coming up, I hope in the next couple of weeks.

Field: Look forward to it.

Delaney: And finally and just for fun, if there were a book club for cybersecurity enthusiasts, and I'm sure there is, what non-technical book should be on their reading list and why? And go creative with this. Tom, I know you're going to share something with us. Now we know why you say good at communicating Tom. I'll to read it. Great suggestion. Suparna, go for it.

Field: I am. This is the book that I have given away more copies of in my life than any other book, Strunk & White's Elements of Style. To me, it is the best book on writing that has ever been prepared. And for people that have got to communicate via email, via text, via video, however they're communicating, there is nothing better than spending the 45 to 50 minutes it takes to go through this book, and learn just the fundamentals of communication. Starting with and ending with eliminate unnecessary words. As a writer and editor, I have given away scores of copies of this book, and I would recommend it for anybody that is working in the cybersecurity and technology fields.

Goswami: I thought the first thing that struck me was when you said it can be as fiction as well was 1984 by George Orwell. So the novel's overall theme is about surveillance, is about government control, lack of privacy, a thought provoking read, so I thought it would be a good read for the cybersecurity professionals.

Delaney: Perfect.

Field: Good choice.

Delaney: Mat?

Schwartz: Wow, yeah, from authoritarian regimes and writing best practices, I'm going to go with a book by The Visual Display of Quantitative Information, which, like Tom is talking about how to how to write well, this is talking about how to consume visual information well, and also how to design it as well. So one of the famous examples in Tufte's book is, I'm not sure which way you pronounce it, but is the cholera map prepared in 1854 by Dr. John Snow, where he looked at cholera deaths in London and visualize those in a map up with dots. And then he plotted with crosses where the pumps were. And there was a very immediate, visual way of figuring out what had happened, what was causing these cholera deaths. And there's numerous examples he brings up to highlight how displaying information visually, can help communicate in a way that nothing else can with an apology to Strunk and White, sometimes the visual display is the most useful display. So I would suggest that.

Field: Brilliant book, I've had read it as well, Mat, love it.

Schwartz: Yeah. Hard to have one that you love. They're all great.

Delaney: Excellent choice. And next time, you're all in London, we'll go to the John Snow pub in Soho, named after John Snow in his honor. Well, I'm going to turn to master William Shakespeare, because I think his works are a compendium of human emotions and vulnerabilities. And in particular, I'm going to recommend his play A Fellow, which explores themes of manipulation and deception and the destructive potential of jealousy or unchecked jealousy. And I think we can all agree that these are themes that resonate, perhaps with cybersecurity professionals as they deal with social engineering and misinformation and the human factor in cyberthreats every day. Well, thank you so much, everybody. This has been excellent, informative, and fun.

Goswami: Thank you, Anna.

Schwartz: Thanks.

Delaney: Thanks so much for watching. Until next time.