The Best CI/CD Tool for Kubernetes Doesn’t Exist

Alex Williams: Hello, Welcome to The New Stack Makers, a podcast where we talk about at-scale application development, deployment, and management. I'm Alex Williams, Founder and Editor-in-Chief here at The New Stack, this episode is a part of our upcoming Ebook CI/CD With Kubernetes. I had the pleasure of being joined by CNCF Cloud Native Ambassador Chris Short and CNCF Developer Advocate Ihor Dvoretskyi at KubeCon plus CloudNativeCon earlier this month in Copenhagen, Denmark. Before we begin, I'd just like to say a huge thank you to KubeCon plus CloudNativeCon, our sponsor for the Ebook and podcast. KubeCon plus CloudNativeCon conferences gather adopters and technologists to further the education and advancement of cloud native computing. The vendor neutral event features domain experts and key maintainers behind projects such as Kubernetes, prometheus, gRPC, Envoy, OpenTracing, and more. KubeCon plus CloudNativeCon is a conference under the Cloud Native Computing Foundation, a project of The Linux Foundation. Now, on to the show.

Alex Williams: It's Alex Williams with The New Stack, here for another episode of The New Stack Makers. Joining me today are Chris Short ... Hey, Chris, how are you?

Chris Short: Doing good, how are you doing?

Alex Williams: Good, and the famous Ihor of the CNCF. How are you?

Ihor Dvoretskyi: I'm doing good, how are you?

Alex Williams: Good. I'm here to talk about CI/CD in Kubernetes. You guys are with the Cloud Native Computing Foundation, because the Cloud Native Computing Foundation is the sponsor of our ebook series on Kubernetes, and this last book we're doing is on continuous integration and continuous delivery. I was just chatting with you guys about what is this shift that we're seeing? What is this that is apparent with Kubernetes that it's forcing enterprises to think through? I hear one thing is this technical gap, this technical understanding is there, but it's just totally different in a continuous integration environment. You used to describe the artifact and then load it up, essentially, Chris, isn't that right?

Chris Short: Yeah. The trend I'm seeing is there's a lot of ... not necessarily misunderstanding but this nebulous thing that you're trying to codify now. You're going from this pipeline of the build server builds your artifact, your delivery pipeline grabs this artifact and then deploys it somewhere. Well, now you're grabbing a container, or you're grabbing a configuration, that calls on all these other things, like a Kubernetes deployment, you pull in all these disparate pieces, put it all together, and then off it goes.

It's kind of a shift. You're going from nothing to a fully running application, essentially, where before, it used to be, you've got to put in a ticket with infrastructure to get some VMs, you've got to put in a ticket with networking to get all this load balancing and VLANs and everything else configured. Now, it's just ... your boxes are ready to go. If they're running a Kubernetes distribution of some sort, they're already set up. They can talk to the world. All you have to do is plug in your deployment, check in your code, and then off you go. Excuse me.

The hard point is that mind shift of ... Okay, now we have to say we care about networking as an operations team and as a development team, now we have to do these things like load balancing and all these other bits and pieces that we never really considered or had expertise in. How do they pull in all these different pieces into one functional group and then send it off?

Alex Williams: That's the shift. Everything's ready to go, all you have to do is just basically build out the platform to make it go, don't you? That's the problem, right? That's the issue. What is this new system that you built? How are you seeing this from a community perspective, helping change that kind of ... helping companies adapt and change as social constructs to be able to think from a contextual point of view about how you think of this socially, and how you think about actually enabling and making this process work?

Ihor Dvoretskyi: From the general perspective we have to move back to the early ages of devops and pre-devops days and the pre-container days when the operators had to deploy almost everything manually. Switching to the different paradigm with the early and first CI/CD systems with the monolith applications was a big step forward from the previous experience, when again, the operators had to deploy everything manually, and the general operations partners were all completely manual.

Today, with the containerized technologies and with the cloud-native technologies, we can see how the general market of the CI/CD systems, how the general ecosystem of the CI/CD systems, devops tools, and devops-related tools is totally different. As Chris mentioned, today you simply build a container. You can build locally, you can build it remotely, you can build some configurations and simply go to a different direction, or you can simply transfer some predefined configuration that you can build in some different place.

It's a big shift, and within the ecosystem, we can notice how many new projects have been born during the last few years that satisfy today’s operators and devops engineers that are strongly designed to work with the containers, to work with the containerized technologies, to work with the cloud technologies, and they are totally different from the previously born technologies of 10 or 15 years ago.

Alex Williams: The changes, then, that these companies are making internally both try and develop that devops mindset. Where do you see that intersection, Chris, with customers, and what are the technical challenges they're trying to tackle first? What do you recommend?

Chris Short: A lot of the technical challenges are easily solved nowadays. Five years ago, it was hard to overcome some of the technical challenges, but now, because of things like Kubernetes and a wide and growing field of CI and CD tools, it's more of narrowing down what works best for you to solve your problems, as opposed to the actual process of change. The harder part, I think, is the cultural shift that's required with it in the sense of, we're not going to wait around on tickets to get done, we're not going to wait around on someone to finish this thing, it's literally everybody's moving as fast as they can towards pushing something out into production.

One of the clients I'm working with at the moment is trying very hard to push a full-on ELK stack out into production today. The idea of doing that ... They've had some pieces and parts put in place, but the idea of doing that in a matter of one sprint would have been a completely foreign concept three, four, five years ago. Now you can do it in two weeks, and easily do it in two weeks.

Ihor Dvoretskyi: I'd also second the cultural shift, and also, for example, comparing ... I'm coming back to history again. You can probably remember if you've been using some operating systems, some desktop software, 15 or 20 years ago, do you remember how often you received updates to your software? For example, if you'd been using Windows 95, how often have you received Windows updates to your operating system?

Today, we are all using mobile phones with their mobile operating systems, for example, and we are using the web applications, and today, we are not receiving updates to these software technologies once per year or once in a few months, but we can receive a few updates in a single day. That significantly changes the culture, the culture of development, the culture of operating these environments, and that is truly one of the major areas where the typical operations and development became devops.



Kubernetes and CI/CD

Alex Williams: There's this thinking that there's this initial challenge with containers, right? They've historically affected organizations and how they think about translating their current development solutions into Dockerized development solutions, right? We saw that emerge, we saw that people started to understand that better. Now we're starting to see, I would say, more of a sophisticated look, and we're starting to see that adoption spread at organizations. It seems like the social issue is a big one.

Now it also seems like there's this gap in the technologies themselves where there's been a long reliance on continuous integration using platforms and technologies like Jenkins, but now, in this new world of Kubernetes and CI/CD, we're looking at a lot more raw data that's available, and we're seeing the use of graph databases like Prometheus to help provide some better monitoring and observability. What are the gaps that you're seeing in the tool sets now that are going to be important to be addressed as continuous integration, continuous delivery spreads more to the mainstream IT organization?

Ihor Dvoretskyi: I can notice one ... I'm not speaking right now about the technology gaps, but I'm speaking on the gap from the open-source world perspective. Speaking about the cloud-native world and about the projects that landed with the Cloud Native Computing Foundation, today, we have almost every project for every single purpose that you may have in the container world. Fortunately, we have CI/CD-specific projects in this ecosystem. That moves the consumers, operators, and the companies that like to use these technologies to select some proprietary technologies, probably to use some open-source technologies, but they can be ... nobody from the open-source world can guarantee that they will, despite their desire in the real cloud-native compatibility. I would say that is the market, the open-source market gap.

Alex Williams: Okay. Chris?

Chris Short: The hardest thing is gluing all the pieces together. I think CNCF is doing a good job of finding projects that can work very well together, adding them to the CNCF landscape, bringing them into the sandbox and saying these things should work together with each other very well in their specific functions. You mentioned Prometheus. You can feed anything into Prometheus. I've seen people using Prometheus to feed in AWS billing data and correlate that with their usage, traffic, other metrics to get their business logic out of Prometheus faster. They understand what their business needs, what metrics are very important to them and their applications, and to do that out of the box, that's pretty hard.

There is a knowledge gap that exists as far as going from that traditional, you mentioned Jenkins, deployment model, I mentioned deploying Java apps, for example, to this new world of ... whether you're running a cloud inside your own datacenter or cloud with a provider of some sort, taking all that data and actually correlating it with all these different business metrics, there is a greater understanding of what your business is considering key metrics, or some people call it true north, and bringing that into the operations and development world more closely makes for a more challenging job for the typical people that for years were like, "I do the Linux," or, "I do the network."

Chris Short: The hard part for a lot of people is that they've had these huge support contracts with one company that, for a lack of a better term, was some kind of value-added reseller or a big service provider. That company did all the gluing together for them. Now it's very much they have to get down and understand how to communicate with different APIs, the different tooling that is required is ... it's no longer, "Oh, I have to have this thick client installed to interface with this huge application that does all of our orchestration for us," now it's just in cURL, Postman, or whatever API integration tool you want to use to just test things out and say, "When this event occurs, push it this way. When this change comes into the infrastructure, do this."

You need to think more holistically about your systems and getting a deep understanding of what you're working with. That is the biggest challenge. The glue is very much a ... it could be a simple, "This API needs to send this data to this API when this event happens," to, "How do we move from Jenkins to some kind of different CI/CD tool that can better manage these cloud-native primitives simpler and more effectively?" That's where I think things like Kubernetes is great at helping you make things live in production, but there needs to be that understanding on the backend to come in and say, "You can't just take your own tooling and expect it to work the same way."



Tooling on Kubernetes

Alex Williams: Ihor, that must be leading to a lot of new tools that you're seeing emerge, isn't it, to help with this process?

Ihor Dvoretskyi: Exactly. The market is growing. The landscape of the open-source tools related to CI/CD, and CI/CD systems through the last years.

Alex Williams: What are specifically some of these tools trying to solve ... that help solve this problem that Chris is talking about, where you might have previously been working with one organization, maybe it's a consulting group or a systems integrator who was doing this for you, and now you're responsible for it yourself? What are the tools that you're seeing them use? One thing we're hearing is there's a lot more need for tools that have better observability.

Ihor Dvoretskyi: I would say that all these tools today have two major requirements. First of all, they have to work natively with version control systems, especially with Git. Almost every organization is using Git in some different way, so it can be GitHub, for example, for some open-source projects, or GitLab for the private projects. I would say that more than 90 percent of source control systems today in the market are related to Git, and all these systems have to work with it natively.

Another challenge, as Chris mentioned, is about native compatibility with the containerized technologies, especially Kubernetes. Again, to see Kubernetes as highly dominating technology in the container world. It could be even more dominating in terms of the share, comparing even to Docker, because Docker, it's usually been used as simply a daemon and the container runtime, while Kubernetes is much bigger. Kubernetes is not only the technology, not only the project, it's the huge ecosystem itself. I would say that these are two major areas where these technologies have to be native enough and have to work in a desired way if they'd like to be successful today.

Alex Williams: Chris, do you see any other requirements that are critical? Making it native with Git seems critical. Almost every organization is using Git in some way. Any other thoughts on what's going to be needed?

Chris Short: I think the biggest thing is that whatever you're deciding to use for your cloud infrastructure, whether it be AWS, Google Cloud, Pivotal, the tooling has to work with that very well. You're not going to take something that's designed specifically, and not many people are making tools designed specifically for one cloud provider, and they're trying to shoehorn them in with another. They're going to look to, can you work with Git, GitHub, GitLab? Can you work with AWS? Can we push our tooling and infrastructure changes out through these disparate tools?

They're not going to say, "We have these favorite darlings any more," they're going to say, "These are our requirements specifically," based off the choices they've made. If it doesn't support that, it's not even going to get looked at, which is, I think, a big shift than when I first got started in IT. If it didn't come from HP, Microsoft or Dell, it didn't even get looked at, whatever organization your company decided to pair up with, that was it, that was the Holy Grail. Now you can do anything you want so long as it works with the choices you've made in the past.

Alex Williams: What are you seeing as popular? What are some of the use cases you're starting to see emerge, Chris?

Chris Short: I see a lot of people that are trying to move from 100 percent on-prem into more of a hybrid environment or a cloud environment that's on-prem. I see a lot of people latching onto GitLab CI. I see a lot of people latching onto more lightweight CI tools. The idea of the CI pipeline having some overarching team that admins it is gone. Everybody's contributing equally to improving that pipeline delivery platform so that it enhances services to users, and those users are often paying customers, in that case.

The tooling that's emerging is from the on-prem side. It's a lot of stuff that you can run in your infrastructure as-is. On the cloud side, it's so expansive now. You're seeing a lot of tools like Spinnaker and other thing coming out of companies getting open-sourced and then pushed into either CNCF or some other open-source forum of some sort to get an adoption that the community then forms around and improves those products respectively. It's an interesting world to live in now with everything just kind of saying, "Okay, we can develop this internally, but what if we open-source it? That'll make it better."

Ihor Dvoretskyi: Spinnaker is one of the best examples of this.

Ihor Dvoretskyi: Spinnaker is one of the best examples of this when the single company is using some technology internally, but they are open-sourcing it, and it becomes valuable to everyone in the market. Today, Spinnaker’s technology is one of the most popular technologies in the CI/CD world today.

Alex Williams: We write a lot about Spinnaker in the book. One of the criticisms I hear about Spinnaker, though, is that it's built with a different type of ... It was built in a different context, right? It doesn't fit exactly into the architecture that represents Kubernetes. As I understand, it's more of individual identity versus the identity of the node and everything else. Is that correct? Is there a difference in the architecture for Spinnaker that's forcing it to be adapted to Kubernetes?

To take it a step further, we're working integrated with Kubernetes better. The Spinnaker open-source that allows for that thing to happen, the change to happen, the Kubernetes tooling, it's going in its direction, Spinnaker's going in its direction, but they meet somewhere, gracefully, sometimes, in the middle. Sometimes it needs to be shoehorned in, for lack of a better term. The idea that it's ... Yes, it was developed out of a different need and maybe even a different era in cloud computing. That doesn't mean that it's not an effective tool. It just means that your thought process has to shift a little bit. Everybody needs to be a little more flexible nowadays and be willing to be more flexible faster, because that's the world we're living in. No-one waits around for the perfect tool to show up to release their products. They release what's available.

Alex Williams: This speaks to the direction, then, of CI/CD and Kubernetes, doesn't it? We're kind of going through the evolution right now, and we're seeing the maturity of these CI/CD environments. If you're going to look at the current state of the market, and then looking forward, what are some of the core issues that you want to address and build upon the success that Kubernetes has had?

Chris Short: I think for me, I want to build systems that are, of course, loosely coupled, but easier to use. People say Kubernetes is complex, and it's hard to use, and it's hard to manage. Well, that's kind of fading into the background. That might have been the case two, three years ago, but it's getting easier, especially if you're using a cloud service offering. Even if you're using it internally, it's, okay, you got a problem with the node, well, shouldn't your node be somewhat ephemeral? Unless it's the master, it should be easily replaceable.

The idea of making IT and general technology easier to use is something that I've kind of built my life around. Technology should not make our lives harder. Choosing a specific technology should not change the way you do something very basic so drastically that it's harder to use, as opposed to easier to use. That's the whole point of technology. Pacemakers exist to keep people alive. Kubernetes exists to speed up infrastructure changes and change systems more quickly. I think it makes it easier to do that when you look at the overarching goal of what Kubernetes is here to do.

Alex Williams: Ihor, we're starting to see interesting developments that come with this evolution of Kubernetes. You were talking about Git and how these systems have to be native with Git, and we're starting to see concepts emerge such as Gitops. Maybe you could help our listeners here understand what Gitops intends to do.

Ihor Dvoretskyi: Gitops is pretty simple. It's probably one of the simplest technologies and the simplest concepts in this complex world. It's been originally developed and initiated by Weaveworks, one of the major contributors to Kubernetes ecosystem. The idea of Gitops is pretty simple. It's operations by pull request. So for an operator, the only thing that you should do on the front end is to submit a pull request to your Git system, so with a single action, you can trigger some different actions that will trigger different actions, and so on.

It's pretty simple for almost any kind of developers and operators that are using Kubernetes today. It doesn't require some extremely specialized […] as we probably would see before the devops era, when we had the group of people that had been working on operations, and they would then be working on the operations full time. We had a group of people that had been simply developing code. Today, with the tools like Gitops and similar tools, we can have the ability for the developers and for the people who contribute to simplify their typical working time.

Alex Williams: Great. That speaks to that Gitops story emerging. I'm curious on how those practices, then, end up affecting the tool development itself, Chris. Do you see that the devops practices like something that I think is really manifested in what we see in Gitops, and do you see that affecting the tool development itself?

Chris Short: Potentially. I think Gitops may be ... To me, it's the holy grail of software and infrastructure management. I make this change, I push it, and off it goes. When I say off it goes, I mean there's very little that you have to do to intervene between it and the code or infrastructure change going into customers' hands. It's a little new to say that it's changing the tooling quite yet. I think the premise of it has always been in the background, like that's what people want. A lot of my life has been dedicated to making it possible to do that, and now it's getting down to the point where it's going to become the standard. I need to be able to Git push, and it just handles the rest all on its own.

The it is irrelevant to some people, whether it's Jenkins, Spinnaker, Kubernetes, or magical faeries with unicorns running around, they really don't care as long as that Git push gets tested, gets checked for any vulnerabilities or infrastructure changes that are unsafe, and then ends up in customers' hands within minutes, hours, not, "Oh, yeah, we've got to put it through change review, and we've got to do all these things, and it's got to go ... we have to build these Jenkins jobs around it, and all this other tooling has to come out of the Git push. Gitops, to me, I think, has been the underlying state of where we need to be for a while. The tooling that will emerge from what is now possible I think will be simpler, easier to use. Maybe you can tweak some things, but out of the box, it'll fit most use cases very easily.

Alex Williams: Looking ahead, we are seeing this shift, Gitops is a great representation of that, but we're also seeing the evolution of these resource ... I call it resource-driven event processes, almost, which is serverless, essentially. We will see, more deeply, integration of serverless into the CNCF, into Kubernetes, and there's a lot of open-source projects out there. You guys now have a serverless working group focused on these efforts. What are the things that we're starting to see with serverless affecting how we think about continuous integration, continuous delivery in Kubernetes?

Ihor Dvoretskyi: One of the biggest benefits of having, of using the serverless technologies is almost zero operational cost to use it. From the concept standpoint, it really can be compared to Gitops […], where you can simply trigger some action, something happens behind the scenes, and you simply receive the desired result, or undesired result, probably, but you can work with that. Again, serverless technologies allow your organization, your team and yourself to have almost zero extra operational charges. If you are a developer, you can simply focus on developing the code and pushing one magic button and everything will happen […] and CICD systems that will be focused on serverless and are focused on serverless will globally satisfy this request, this demand.

Alex Williams: How about the development of ... We're starting to see ... At KubeCon in Copenhagen, we saw the emergence of this concept around operators. The concept of operators were essentially kind of a ... If I'm managing a very complex Java application, or if I'm a database provider, the controller serves as almost like that Kubernetes control play. Does that have any material impact on how we think about continuous integration, continuous delivery in the context of where we are today?

Chris Short: It certainly provides a lot of that glue that we were talking about earlier. It definitely eases that transition, or simplifies the management of those various services. I see this happening very quickly, but the idea of all these operators coming out, especially around KubeCon, the state that I want to be in is these operators are just becoming the standard. We updated MySQL version X, well, here's the new operator for it. We released this new tool, well guess what, it comes out of the box with a Kubernetes operator of some sort.

People don't have to figure out, "We've got to use this latest version, but it doesn't have this Kubernetes operator, so now we've got to reconfigure all this stuff or retool some things." No, give us the operator out of the box and make that part of your development and release platform. That way, that glue becomes a Kubernetes primitive, essentially. You're talking SQL or Kubernetes, you're not talking this weird middle layer that you have to configure the two together.

Alex Williams: That seems to speak for what's to come. We're going to see more of these declarative environments, such as these operators, that will continue to force us to think about how we look at the overall Kubernetes infrastructure and thinking then about that raw data that's coming out of it and how you can better understand it through monitoring, alerts, and everything else.

Chris Short: The integration of operators into the environment as they are—It’ll allow for a lot tighter and potentially faster changes to infrastructure and the underlying services themselves. When I say services, I mean if you have an operator for Kafka or a new database or whatever, you'll be able to have those things running a lot smoother, I think, and the ability for you to accidentally put edge cases into your infrastructure goes away, or gets significantly diminished. The idea of speeding up development, I think, is a huge gain in some of these operators.



The evolution of CI/CD and Kubernetes

Alex Williams: In conclusion, the CNCF role, you guys' role in the evolution of CI/CD and Kubernetes, what is your mandate for the community?

Ihor Dvoretskyi: It's a good question.

Chris Short: I think as far as the community goes, as an ambassador, I'm not here to say, "Use this tool, use that tool," I'm here to say, "This is how you can use the CNCF tooling together and solve problems with it in your organization." As far as mandates go, I don't think there's anything specifically saying we need some kind of blessed, perfect, cloud-native, computing foundation solution to CICD. That's not why we're here.

Alex Williams: Your job is to be the guide and testing the tools. I guess you mostly spend a lot of time looking at the ... I could probably talk to you for hours on how you go through the process of defining all these tools, their roles, and how you assess them, but we can save that for another time. That seems like a big part of your work, isn't it?

Alex Williams: ... understanding the technologies out there and then providing people guidance about them?

Ihor Dvoretskyi: It is, not only the guidance but also provide the general understanding and sharing the knowledge, as Chris mentioned. CNCF, we all here are not pushing someone to do something, but we can strongly recommend, provide and advise why this technology is better for your specific use case compared to the different technology can be better for someone else, for the different use case. The second technology still exists in the market and is still available and for some reason, some other people are using it. Chris mentioned that technology A is better than technology B, because all these technologies have some value in this world.

Alex Williams: Wonderful. Thank you, Chris, thank you, Ihor, for taking the time to talk to us about CI/CD and Kubernetes from your perspective at the CNCF. We appreciate your support with the ebook. Look forward to seeing you soon in the coming months. There's lots happening in this world, and we'll be looking to you guys for some guidance and perspective. Thanks for your time.

Ihor Dvoretskyi: Thank you, Alex.

Chris Short: Thank you, Alex. Appreciate it.

Alex Williams: You bet. Have a good day.

Alex Williams: Huge thank you to KubeCon plus CloudNativeCon, our sponsor for the Ebook and podcast. KubeCon plus CloudNativeCon conferences gather adopters and technologists to further the education and advancement of cloud native computing. The vendor neutral event features domain experts and key maintainers behind projects like Kubernetes, Prometheus, gRPC, Envoy, OpenTracing, and more. KubeCon plus CloudNativeCon is a conference under the Cloud Native Computing Foundation, a project of The Linux Foundation. Listen to more episodes of The New Stack Makers at thenewstack.io/podcasts. Please rate and review us on iTunes, like us on YouTube, and follow us on SoundCloud. Thanks for listening, and see you next time.