Improve Security With Automated Image Scanning Through CI/CD

Alex Williams: Hi, I’m Alex Williams, Founder and Editor-in-Chief of The New Stack, here at KubeCon + CloudNativeCon in Copenhagen. I'm here with Liz Rice, who this year is co-chair of the CNCF, congratulations. You've done a magnificent job on stage and, I think, bringing some great focus. I really kind of feel a focus when you're up on stage talking about the project, so thank you for that.

Liz Rice: Thank you very much!

The Aqua Security MicroScanner

I've been writing about it a lot recently, mostly because I've been doing some work on a product called MicroScanner, which is a free tool for developers to use where they get to use our Aqua package vulnerability. It's the same kind of vulnerability intelligence that we use for our commercial product. But it's free for individual developers to use on their projects.

Our enterprise customers will typically be building scanning into their CI/CD pipeline. The product kind of hooks into your Jenkins or whatever your favorite CI/CD tooling is. MicroScanner, on the other hand, is more of a stand alone thing that you put in-- actually it's a line into your Dockerfile. So it's an easy way for individuals-- that might not really be the right approach for an enterprise to take, but for an individual developer to check their own images are not full of vulnerabilities and their packages, it could be a useful tool.

Alex Williams: Okay. So when we think about CI/CD in Kubernetes, it's much different, right? It's a lot-- it's an entire different approach and Kubernetes is a much different architecture than people are accustom to. The Micro services are constantly crossing each other all the time. How do you think about that? How do you think about that from a CI/CD perspective when you're thinking about an infrastructure that really is like a hive.

In the Kubernetes world, suddenly you're running these containers, these bits of software somewhere. You don't necessarily even know where. You've got these thousands of instances and for a traditional-- if you've come from a traditional security background, at first glance, this sounds really scary. And the reality is automation is your friend, right?

Automation on Kubernetes

Automation is in Kubernetes deploying that software, deciding where to run that software. Automation is part of your CI/CD pipeline, can help you secure it and make sure that you're only deploying images that your confident in because they've been scanned. They pass your image policy. You might have rules around what levels of vulnerability you're prepared to accept or who has built that image. Maybe even some checking on the prominence of those images. There's all sorts of things that you can automate so that when Kubernetes does start running a container-- running a pod with a container inside it, that container image is-- you've got confidence in it.

Alex Williams: Right. So it's the confidence and it's the trust that you're really trying to garner here?

Liz Rice: Yea. The automation is key because the scale of these things, you couldn't possibly manually check all these different images when you're shipping potentially hundreds or thousands of deploys in a day. That scale is beyond being able to do manually, but automation CI/CD is the key. Hooking those security pieces into that process.

Liz Rice: Totally. So a lot of the customers that we have at Aqua Security will have policies around the base images that their developers are allowed to start from. Maybe they've got a different base image for each different language that they might use. You know, there might be a Python based imaged and a Node based image and a Java based image. But the security team will have looked at what's contained in those base images and they'll have some confidence and making sure that every image you then build has been built on top of one of those approved images. It can give them some confidence.

Alex Williams: So what are some of the vulnerabilities that are emerging that come out of this? What are the vulnerabilities that you're trying to address now that are growing in scope and going to take on different forms and--?

Liz Rice: Well, there are vulnerabilities, new vulnerabilities, discovered every day. Some of them maybe got introduced into a piece of software really recently. Some of them could have been around for years and years. I'm pretty sure that Heartbleed was in code that was years old. So you never know where an exploit's going to come from. Every time there is an exploit, there's this process around the vendor, the person who wrote the software having a patch, getting that patch applied to distributions into packages.

It's actually kind of complex because all the different Linux distributions have their own package management processes and they might handle-- some of them will take a new update, some of them will back port a patch. So you really want your vulnerability scanner to have knowledge of not just the basic vulnerabilities but also which distributions have those vulnerabilities in which different levels of each package. And that's something that differentiates different image scanners. Aqua has a lot of intelligence from a lot of different sources, plus a research team, putting a lot of work in to make sure that our vulnerability reports don't-- the big thing is false positives. You tend to get false positives if you don't have all those sources.

Liz Rice: It's really the different sources bringing in information from the NVD (which is the National Vulnerability Database), but on top of that applying knowledge about the distributions, applying knowledge from upstream vendors. Sometimes vendors like the Database Software might have their own security disclosures, language specific disclosures. Plus, actual manual intervention. We have this-- The research team in Aqua have this whole tool around where there's doubt about a-- whether a vulnerability is really there in a package or not. They'll go and check it. So we have this huge amount of research going into exactly which packages in which distributions contain which vulnerabilities.

Alex Williams: Oh my gosh, you must be learning all kinds of things out of that.

Liz Rice: Yea. I mean, there's so much detail in that. It's really impressive.

Alex Williams: How are you organizing that data then?

Liz Rice: They have internal tools to manage that.

Alex Williams: Right. So what are the other components of it that Aqua looks at? You look at it from the scanner perspective. We also talked about the runtime perspective.

Liz Rice: Some of this stuff is just so fascinating to me. I could talk for hours. What it really boils down to is knowing what a particular piece of software should be doing. We learn what normal behavior looks like and then we can spot when something unusual happens, when something anomalous happens. Like, for example, accessing a file that shouldn't be accessed or running as a user ID that shouldn't be run.

Particularly if we are talking about microservices, it's really reasonable to think about what a particular service should be allowed to do. What network traffic should it be allowed to talk to? Should your microservice-- maybe you've got a product search microservice. It has no business contacting a database with user data, for example. And being able to constrain what that containerized code does so it can't talk to that user database and it can't, I don't know, starting pinging for different network components that it might be able to-- you know, it can't start exploring its own local network. The more you can constrain-- we talked about shrink wrapping the behavior of a container at run time so that we can flag-- well either alert or prevent, something outside of that normal behavior. Just really powerful.

Alex Williams: Interesting.

Liz Rice: And actually if we're talking about CI/CD you kind of need to build that into if you're deploying a new version, you might need to re-learn what that behavior is because the new version might have a change in features. That means it's going to behave slightly differently. Maybe it does need to access a different file. So being able to keep those runtime profiles up to date is also important.

Alex Williams: So how do you keep the runtime profiles up to date?

Liz Rice: Re-profiling them automatically.

Alex Williams: Just re-profiling them. How are the profiles, then, getting defined? Like what are you thinking about in that respect?

Alex Williams: Yea, because then they could get into anything, right? Yea. Yea. How do you prevent them from getting-- I guess, the question is how do you get them from getting into all the containers? You don't want SSH in there for that reason alone.

Liz Rice: Yea. Nobody's every going to say everything is secure because risks are coming-- it's all a question of risk and attacks come from new sources every day. I'm going to be talking tomorrow about some privilege escalation issues that maybe people don't realize. People don't realize their running as root half the time. So that's what I'm going to be talking about.

Alex Williams: Is it because they don't have the interpretability to some extent?

Liz Rice: It could be. Yes. It could be. I think having tooling that lets you know when you are-- I mean, actually that's another thing you can see from the Aqua Security console, is containers that shouldn't be running as root but are. That's something else we can highlight and also give advisories on. Perhaps you want to run this container not as root.

Alex Williams: Just a thought.

Liz Rice: Yea. And, I guess, you could hook that kind of audit logging into broader log management or seam management systems.

Alex Williams: Yea. I guess that leaves a lot of questions about monitoring the components across the environment.

Liz Rice: That whole observability world is-- there's so many aspects to it and we touched very briefly on all the different projects in the CNCF yesterday and several of them related to observability, tracing, logging. It's a big field.

Alex Williams: So what are the projects, then, related around security that are of interest to you?

Liz Rice: There's a couple of new projects that have just joined the CNCF. One of them is the open policy agent, which is a general purpose engine for policies. So I think we'll be looking at that-- we're obviously defining policies related to images and run time. So it's early days--

Alex Williams: You might be able to offer that as an image that you can--?

Liz Rice: Yea, it's really early days for that project. But it's obvious that there's some overlap there that will we'll be looking at. And the other one that I'm keen that we look at is Spiffy, which is this secure identities. We obviously see more service mesh and more secure connections between containers, which is a very good thing. The way that those containers identify themselves to each other would be really good to have that integrated into Aqua tooling as well, so that there's common identity.

Alex Williams: I've noticed more discussions about these communication layers that we're starting to see emerge. So if you can't have that communication. But, for example, I would think in a security context between the containers themselves, right? But then also how they're behaving on the run time for example.

Liz Rice: Yea. So, network micro-segmentation another component that we've had in the Aqua platform for a while, which is the idea that containers should only talk to certain other set of containers. Really related to some of the tooling that we see in service meshes which can control which services can talk to each other. We do it at a container level, so it's a slightly more granularity there. And I think also looking at the configuration of these things is going to be really interesting moving forward, configuring a service mesh is not something you're going to just take five minutes over.

Liz Rice: Yea, there's definitely some things I need to look into more, like Brandon was talking about this morning.

Conversations about Security

Alex Williams: Yea. I guess going back to kind of continuous integration/continuous delivery in Kubernetes is security issues that are being discussed now. You know, there's been this odd relationship between developers and security teams for a long time, right? You always hear at a conference like this, "Get out the checklist." And you won't get that check and that means forget it. What are the conversations that you're having with developers now that are striking to you about security?

Alex Williams: But what are developers, I mean at an event like this, when you're talking to developers, what kind of questions are they asking you? You know about scanning and CI/CD and the role of security?

Liz Rice: Actually, the biggest question that's come up about scanning has been to do with this whole question of false positives. And I think some people have been bitten with scanning tools that just take a single source of information. They scan, they find they have a lot of false positives, and then they're kind of like, "Well, this isn't giving me anything useful. What's the point of doing this image scanning?"

Alex Williams: "False alert. False alert." You know what they say.

Liz Rice: Exactly. Yea. So I think one of the things we want to do with MicroScanner is give people a tool that is less prone to fal-- You know, I'm not going to say there's never going to be a false positive, but it should be a lot less prevalent than perhaps we've seen with some more straightforward tools that just deal with the NVD.

Alex Williams: So what was the technical architecture that you built for that scanner that does distinguish it?

Liz Rice: So this is going back to the combining of the intelligence from all these different sources?

Alex Williams: Right. Which--

Liz Rice: So the MicroScanner is like a small executable that you can build into your container image. It calls out to our, we call it the "Cyber Center," which is the repository for all this vulnerability information that we've collected.

Alex Williams: But you're collecting from all the scanned images you're doing?

Liz Rice: Yeah. And, well, from all these databases, the security advisories, from the distributions and the vendors and what have you.

Alex Williams: Which, you guys well know from your practices over many, many years, and how to kind of build that data?

Liz Rice: Yea. Yea. And we're constantly updating those sources. Yea. And then that's exactly the same Cyber Center that our commercial tool is using, but available for free.

Alex Williams: Right. Well, I just want to, maybe in conclusion, just kind of getting a picture of what is your vision for how we're going to be thinking about building images and for Kubernetes?

Liz Rice: So, I guess one thing that is definitely afoot in the image building world is the need to build images unprivileged. So we've kind of fallen into a habit of using tools (no disrespect to Docker) but you shouldn't need to have root privileges to build an image. And I think we're seeing a lot of activity around that. That will make integration into CI/CD pipeline much easier because nobody wants the CI/CD pipe-- they don't want their Jenkins deployment having root privileges over their container deployment, their Kubernetes deployment. So removing those privileges, it will make things a lot more secure and yea. Things are on the up. They're improving.

Alex Williams: Nice. So what are some of the projects that you're looking forward to seeing developed? Like Spiffy, I would expect then. And OPA, I guess.

Liz Rice: Yea. Spiffy, Open Policy Agent, Grafeas. I know there's been some interesting things going on related to Grafeas, and I haven't caught up with them yet, so--

Alex Williams: Well, we'll have to catch up on Grafeas another time, but thanks Liz for taking the time to talk and congratulations on a great event here.

Liz Rice: Thank you. I'm glad it's going so well. Great.