DevOps-as-a-Service: Our Internal Solution
As organizations change their development models, the tools that support these organizations must transform. At NetApp, we’ve seen a change in the development landscape, from monoliths to microservices. To help evolving teams achieve rapid, frequent, reliable code delivery, and to enhance engineering productivity, we began offering DevOps-as-a-Service (DaaS) to NetApp internal teams.
This solution provides automated installation, configuration, and deployment of services such as Jenkins, source control management, and the plugins needed for the workflows. DaaS enables teams to significantly reduce their cycle time, from the time code is written to the delivery.
Features and Value of DaaS
Based on Kubernetes, our DaaS solution offers centralized management of the entire software development tools ecosystem — such as Jenkins, source control management (SCM), artifact storage, and monitoring. Helm and Docker Engine are used to run the container and deploy the solution. Trident manages our persistent volumes, so any storage — NetApp ONTAP, Element software, Cloud Volumes Service, or Cloud Volumes ONTAP — can be used. A set of internal services were developed using Flask, and CouchDB is used to store information on pipelines. Bitbucket, Perforce, or GitHub can be used as SCM, and the solution can be on the premises or in the cloud.
The key value of DaaS for us is the automated deployment of Jenkins. Along with the deployment of the Jenkins image, the following are taken care of automatically, without any user intervention:
- Customizing the Jenkins image.
- Installing additional plugins.
- Enabling the Kubernetes plugin.
- Connecting to SCM to detect new code commits.
- Starting new jobs as soon as pipelines are created.
DaaS Architecture
Automated Installation Using Helm
This step, run by a Kubernetes administrator, installs base services using Helm. Once the base services are up and running, the web user interface can be used to deploy the solution.
Creation of Base Helm Charts in a Namespace
User interface to deploy the solution
This step enables a solutions owner or an admin to configure and deploy a containerized Jenkins instance, and integrates it with the selected SCM – along with deployment and configuration of all the necessary plugins. What used to take an admin several hours is now completely automated and can be accomplished in just minutes with a single click.
Deploying the solution using Helm charts (Infrastructure as code) makes this procedure programmable and repeatable, saving developers a lot of time. With everything in Helm and source controlled, additional modifications to the project can be saved, changes can be reverted, and information can be stored safely before upgrades.
Web UI to Select and Install the Required Services
Once the required services are installed, Jenkins is installed and deployed along with all the configurations from the Helm charts. No manual intervention is required.
Jenkins pipeline creation automation
This feature automatically creates a multibranch pipeline, pulling code from the SCM and using dynamic Jenkins agents to execute the pipeline steps.
Each time a new project is created, new pipelines need to be created for each repository. By using dynamic Kubernetes pods as Jenkins agents, installing all the needed plug-ins, connecting to SCM, and creating pipelines are all taken care of automatically.
Automated Creation of the Pipeline
The SCM service account is used to populate the list of projects and the repositories associated with the selected project. The SCM URL is auto-populated based on the project and the repository selected.
This step significantly cuts down the time and manual effort required from a project admin. Once a pipeline has been created, the following things are taken care of automatically:
- Every time a new branch is created in a repository, that branch is automatically pulled in and a pipeline build for the branch is triggered.
- Every time there is a code commit in an existing branch in a repository, a pipeline build is automatically triggered for that branch.
- When a pipeline is deleted, the Jenkins folder and all of its configurations are automatically deleted; and the Kubernetes resources that were allocated for the pipeline are released.
Jenkins Templating Engine (JTE) for Pipeline Management
This framework enables projects to have templated workflows, which can be reused by multiple teams simultaneously — regardless of the tools they are using. Projects can be complex and their Jenkins files can get quite long, with multiple pipelines and many lines of code. JTE simplifies and breaks it down into reusable templates that are easy to understand and debug.
Key features of JTE:
- Reusable Libraries.
- Governance rules can be imposed on the workflows at multiple levels (configuration level, library level, and project level).
- Project-specific custom scripts.
- Define branching and deployment strategy as code (advanced feature).
- No configuration is stored on Jenkins Server.
With JTE, we can abstract the way the CI/CD pipeline is constructed, and a single template can be customized and tailored to every workflow.
Data Monitoring and Metrics
Linkerd and Splunk are used to monitor the solution. Linkerd is a service mesh for Kubernetes, and in combination with Splunk it’s used here for operational and development metrics.
- Operational. Monitors the infrastructure for failures:
- Base metrics such as success rate, latency for deployments and pods.
- Live calls over HTTP, TCP, and gRPC calls.
- Debug services on the application layer.
- Developmental. Monitors the pipelines, pods, and automation for failures.
Grafana is used for visualization of these metrics. It provides actionable dashboards for services out of the box, such as deployment, service, and namespace details, health monitoring, cluster monitoring, network I/O, and CPU and memory usage. It is possible to see high-level metrics and dig down into the details, even for pods.
Summary
In conclusion, our DevOps as a Service is a complete end-to-end Kubernetes-based CI/CD solution that provides centralized management of the entire software development tools ecosystem (Jenkins, SCM, and Artifactory). It enables effortless project onboarding; automating the installation, configuration, and deployment of services and saving significant time and manual effort for the development teams.
By leveraging features such as JTE, the solution promotes simplification and templatization of Jenkins. The combination of Linkerd, Splunk, and Grafana monitors the infrastructure and automation, so that our developers can focus on development.
Most importantly, our DaaS provides best practices that can be reused across development teams; along with the automation and integration of tools such as Jenkins, SCM, Artifactory, security compliance, metrics, and monitoring.