Secrets Management: Doppler or HashiCorp Vault?

A “secrets manager” is a centralized system for storing sensitive information, such as API keys, database credentials, or even files (e.g. certificates and private keys). This post focuses on comparing two secrets managers, Doppler and Hashicorp Vault.

Jun 22nd, 2021 11:38am by Ryan Blunden

Featued image for: Secrets Management: Doppler or HashiCorp Vault?

Feature image via Pixabay.

Ryan Blunden

Ryan loves the challenge of breaking down complex topics into easy-to-understand educational resources for developers of any experience level. He's a Developer Advocate at Doppler, where his mission is to help developers learn how to manage application secrets securely.

Security-minded developers realize that scattered and ad hoc credential storage methods such as .env files pose significant security risks and are difficult to manage across applications in different environments.

If the term “secrets manager” is new to you, it’s a centralized system for storing sensitive information, such as API keys, database credentials, or even files (e.g. certificates and private keys). In addition, it’s responsible for ensuring secrets are encrypted during transit and at rest, as well as a host of other security-related features such as strict access controls and audit logs.

But which secret manager will best suit you and your team?

A single blog post couldn’t hope to compare every secrets manager adequately, so today, we’ll focus on just two: Doppler and HashiCorp‘s Vault.

Let’s get started!

High-Level Comparison or TL;DR

This article is a long read, so we’ll start with a high-level comparison, then ease our way into the details.

Target Audience

Vault is primarily aimed at Security teams and is undoubtedly the most enterprise-capable and configurable secrets manager on the market.

At its heart, Vault is a Key/Value store for secrets, that with its incredible flexibility and configurability comes a domain-specific learning curve with topics such as secret engines, authentication methods, and access policies.

Doppler, by contrast, is built for developers while still containing the features security teams expect from a secrets manager. It’s designed to provide the specific features needed by developers when managing secrets for deployed applications.

Hosting

Vault is open source and can be deployed in several different ways, each with its own tradeoffs. HashiCorp also provides Vault Enterprise, a fully managed version running on HashiCorp Cloud Platform (HCP).

Self-hosted costs are also a factor, as a non-trivial amount of resources is required for Vault to be considered highly available.

Doppler, in contrast, is a fully managed service (see historical uptime here) — nothing to deploy. Check out Doppler’s security page and related docs to learn more about how Doppler is hosted and stores secrets.

Structuring Secrets

Doppler organizes secrets by grouping them by application or microservice (what it calls Projects) with a customizable list of environments for environment-specific values.

While HashiCorp offers a very detailed reference architecture, Vault itself is largely un-opinionated about how the secret key path should be structured or whether separate Vault clusters should exist for each environment.

Vault is primarily CLI and API driven with its Web UI designed for administrative tasks.

Doppler differs here by putting its Web UI dashboard at the heart of its secret management workflows. As a result, it makes regular tasks such as adding, updating, or removing a secret in every environment easy, completed in just a few clicks without learning a new CLI or API.

Fetching Secrets

Doppler has opted for a CLI application runner that injects secrets as environment variables into the application process. This ensures application code has no direct coupling to Doppler, with the added benefit of environment variables being language and framework agnostic.

The decision to use environment variables is informed in part by platforms such as Amazon Web Services‘ Lambda and Cloudflare Workers that provide a Key/Value store to supply secrets at runtime, injected as environment variables.

Environment variables are the ideal choice for these platforms as they are language agnostic, eliminating the need for platform-specific SDKs for secrets fetching.

Vault has seamless integrations for Kubernetes, Spring, and officially supported client libraries for Go and Ruby.

Developers using other languages, frameworks, or platforms with Vault must choose between fetching secrets using the CLI, REST API, or community-maintained open source libraries. While this is not overly difficult, it’s another design decision facing teams that can take time away from shipping new features.

Vault authentication methods for secret fetching must be chosen, and even once an auth method such as App Role is enabled, additional precautions such as Cubbyhole response wrapping should be considered.

Doppler makes authentication simple with CLI tokens for local development, personal tokens for API access, and Service Tokens to provide read-only access to secrets for a specific application and environment.

Integrations

Doppler recognizes that many platforms such as GitHub, Azure, AWS, GCP, and Vercel have built-in secrets storage solutions. As a result, Doppler can be configured to sync secrets to these storage providers, focusing on being the central source of truth for secrets, regardless of the platform or cloud provider.

Vault has an impressive list of integrations, primarily focusing on authentication and secret storage instead of secrets syncing to external platforms such Cloudflare Workers.

Summary

Vault is a highly configurable secrets manager, offering more than 20 ways to interact with secret data, Key/Value storage being just one of them. Hosting options range from free and open source to managed Vault instances on HashiCorp Cloud Platform (HCP).

Doppler is a fully managed service and takes a streamlined approach by orienting itself to the way developers naturally think about managing secrets for an application, organizing them into projects with a customizable list of environments.

Doppler and Vault offer tight integrations with cloud providers and the flexibility to choose the underlying secrets storage backend, with Vault offering additional options compared with Doppler, such as database engines and Consul.

In any case, and most importantly, Vault and Doppler make insecure secrets management practices such as unencrypted .env files a thing of the past.

This is only a high-level summary, so now let’s dive deeper.

Getting Started with Vault

To get started with Vault, you’ll need to decide how your instance will be hosted.

Many options exist, such as:

Vault’s Getting Started guide recommends downloading a pre-compiled binary for running in development mode. While great for testing locally, you’ll need to deploy your own instance or choose a managed offering such as HCP to evaluate with your team.

If deploying Vault, ensure it’s configured to be Highly-Available (HA) which requires a multi-server, load-balanced cluster with an HA-capable backend (e.g. three-node Consul cluster) plus request forwarding and client redirection correctly configured for failover scenarios.

An initial deployment relatively simple, but what’s trickier is adequately maintaining and supporting all of the resources on which Vault depends, in addition to Vault itself.

Solutions such as Vault Helm charts are great for standing up a Vault instance, but forking Vault’s Helm chart to version control internal changes while also rebasing and merging upstream Helm chart updates is a non-trivial amount of work.

If running Vault in dev server mode locally, you can use the Getting Started Guide, containing over an hour of educational content covering the basics of secret storage and retrieval using the Key/Value v2 secrets engine, as well as an overview of the Web UI.

It’s a solid introduction to using Vault, but you may need additional time, e.g. familiarizing yourself with Vault’s Reference Architecture Guide if deploying the open source version.

Getting started with Vault requires a significant investment of time and potentially money, depending upon the demands on your Vault instance. As a mission-critical enterprise application, you’ll want to carefully evaluate whether self-hosting Vault or using a managed option such as HCP is the way to go.

Getting Started with Doppler

Doppler subscribes to the assumption that for a new developer tool to gain traction and widespread adoption, it must demonstrate immediate value with little to no friction in a short period, meaning minutes to hours, and certainly not weeks.

And preferably with no initial financial outlay. More specifically:

No credit card required to sign-up
No trial periods that require a credit card to continue
Sufficient functionality on a free plan to evaluate a real production-facing application

Doppler is designed as a fully managed service with:

A fault-tolerant, multi-infrastructure service with active DDoS mitigation and a historical yearly uptime of 99.99%
A free tier with unlimited projects, secrets, and users

For those wanting a visual demonstration, check out the four-minute video of Doppler going from local install to production.

Getting started with Doppler is straightforward:

Sign up to create an account
Name your Workplace (could be company-wide or org unit)
Create a project
Enter the secrets for the Development environment
Install the Doppler CLI locally
Use the Doppler CLI to run your application locally (injecting secrets as environment variables)

The benefit of Doppler being designed as a managed service means the only thing developers need to do is add secrets to a project, then use the CLI to fetch those secrets.

They can also set up integrations with all the major cloud providers and platforms, as well as Kubernetes and Terraform.

Doppler aims to deliver maximum value with minimal friction and no new concepts to learn as environment variables are language and framework agnostic.

Structuring Secrets in Vault

The KV secrets engine version needs to be chosen, balancing the v2 benefits of versioned secrets with rollback functionality vs. potentially reduced performance and larger data storage requirements due to the extra metadata and version history retention.

Another key decision is how applications will access secrets specifically for the environment they’re running in, which comes down to three main choices:

A single Vault instance that includes the environment in the path (e.g. /my-app/production/db-username).
Each environment has its own Vault instance, enabling the paths to remain constant (e.g. /my-app/db-username).
Vault’s Enterprise Namespaces feature, allowing each environment to act as an isolated Vault within Vault.

Vault Enterprise customers will likely use namespaces, while open source deployments will need to select from one of the other alternatives.

Typically, most large companies have security teams responsible for mandating a security policy for how applications are deployed, which includes how secrets are accessed. A centralized platform or infrastructure team will then manage access to Vault and provide implementation advice and patterns for consuming secret data.

The more friction development teams face in adopting a new tool such as a secrets manager, the greater likelihood some teams will continue using existing insecure practices such as .env files. It’s therefore vital that policies, processes, and documentation be in place ahead of time to ensure adoption is as easy as possible.

Structuring Secrets in Doppler

Doppler is a universal secrets manager, built for a microservices world, grouping secrets into Projects, each with a configurable list of environments.

Each environment consists of a root config and optional branch configs, enabling secrets to be customized for a specific platform (e.g. AWS Lambda vs. Heroku) and allowing developers to override secret values from the root Development config when developing locally.

Authorized team members can add new custom environments such as Test, CI, or QA. This not only helps to organize secrets but aids with applying the principle of least privilege, as every user with Collaborator or Viewer permissions must be explicitly granted access to each project and environment.

Grouping secrets by application and environment is essential, regardless of which secret manager you use. Doppler just has the advantage of having this structure built-in and configurable with a few clicks, as it was designed specifically for developers managing secrets for deployed applications.

Managing Secrets in Vault

If using Vault in dev server mode, you’re likely using the root token (providing unrestricted access) and v2 of the secret engine, as this is enabled automatically. This makes getting started easy but is not a production-ready scenario. However, if you’re after something resembling a production install, you can use the listed methods in the Getting Started with Vault guide or a managed instance running on HCP.

Vault is primarily CLI-driven, so the shell will be where most of your secret management operations occur.

Before creating secrets using Vault’s CLI, take precautions to prevent commands with secret values from being recorded in your shell history. Or better yet, use files to provide secret values.

You must also decide whether a secret will contain one or multiple Key/Value pairs. I would recommend a single Key/Value pair per secret, but this can depend on the type of secret.

Let’s now take a look at creating and fetching a secret using the Vault CLI.

Using sample code from Vault’s static secrets tutorial, let’s create an API key as a single secret:

sh
tee apikey.json &lt;&lt;EOF
{
"key": "AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI"
}
EOF

tee apikey.json <<EOF

{

"key": "AAaaBBccDDeeOTXzSMT1234BB_Z8JzG7JkSVxI"

}

EOF

# 2 Create the secret:

sh
vault kv put secret/my-app/development/api-key @apikey.json

1 2	sh vault kv put secret/my-app/development/api-key @apikey.json

# 3 Delete the secret file:

sh
rm apikey.json

1 2	sh rm apikey.json

# 4 Fetch the secret:

sh
vault kv get -format=json secret/my-app/development/api-key

1 2	sh vault kv get -format=json secret/my-app/development/api-key

# 5 List all secrets for my-app in development:

sh
vault kv list secret/my-app/development

1 2	sh vault kv list secret/my-app/development

#6 Delete the secret:

sh
vault kv delete secret/my-app/development/api-key

1 2	sh vault kv delete secret/my-app/development/api-key

Creating, fetching, and listing secrets with the Vault CLI is straightforward, with namespaces providing the best option of segmenting secrets by environment for Vault Enterprise customers.

Managing Secrets in Doppler

Doppler’s goal is to create a delightful user experience for managing secrets.

Doppler groups secrets into Projects with a configurable list of environments such as Development (or dev).

Branch configs (highlighted above) are a Doppler-specific concept where secrets are inherited from the root config to which they belong, allowing for secret values to be added, deleted, or overridden while not affecting secrets in the root config.

Why? The two most common use cases for branch configs are:

Enabling an application deployed to different platforms (e.g. AWS Lambda and Cloudflare Workers) that require platform-specific secrets in addition to the secrets found in the root config.
A developer needing to override a value for a specific secret from the root development config for local testing or troubleshooting.

In short, branch configs provide an inheritance model where secrets can be customized to suit a particular scenario without duplicating secrets unnecessarily.

Doppler also includes features to keep secret changes in sync across environments.

A visual indicator will display if a secret is added or removed from one environment but not others

Comparing a secret’s value across environments (that the user has access to) is also built into the dashboard.

Let’s now discuss Doppler’s secrets access model.

Doppler (in comparison to Vault’s Policy syntax) takes a more straightforward and UI-driven Team approach.

A user with Admin or Owner permissions is required to explicitly grant access to the environments within a Project for a user with Collaboration or Viewer access.

For the most part, this means developers will be restricted to the Development environment with SREs and DevSecOps able to access additional environments to work cross-functionally.

While Doppler has fewer auth backends than Vault, support for Google Email SSO, SAML SSO, and SCIM enables organizations to grant access at the group, team, org unit, and individual level as needed, with the configurable default access level usually set to Collaborator or Viewer permissions.

While the Doppler dashboard is where you’ll perform the vast majority of your secret management tasks, the open source Doppler CLI is also capable of the majority of secrets management functions, plus there’s also the Doppler’s API should you need it.

Doppler has strived to build a developer-friendly secrets manager, providing the features and workflows Developers and DevSecOps need in an easy-to-use web UI while still providing a fully-featured CLI and API.

Application Secret Fetching in Doppler vs. Vault

While Vault and Doppler both provide a RESTful API for fetching secrets, they differ in what’s expected of the developer or application to access secrets at runtime or when deployed.

The comparisons I’ll draw here are reasonably high-level, as the difficulty a team may experience when integrating with a secrets manager depend upon several factors such as the platform, application server, or implementation support from the Security and Platform team.

If you don’t have a dedicated Security or Platform team, keeping things as simple as possible is especially important.

If we take it as a given that most organizations will deploy applications:

In different languages
On different clouds and platforms (e.g. AWS, GCP, Azure, Vercel, Netlify, CI/CD)
With different packaging (e.g. AWS Lambda zip vs. a Docker image)
And different runtimes (e.g. Python Lambda runtime vs. Python in Docker vs. Python Cloudflare Worker)

A secrets manager should aim to work as seamlessly as possible for any language, platform, and runtime, ideally without the need for secrets manager-specific glue-code.

Doppler describes itself as a Universal Secrets Manager, which in concrete terms, means it is “universally” focussed on integrating and syncing secrets to platforms with built-in app config and secrets storage such as Vercel and GitHub Actions as well as other secrets managers such as AWS Secrets Manager.

Why would a secrets manager want to sync secrets to another platform, let alone another secrets manager?

The fragmentation we see in deployment platforms is only going to increase. The most effective way organizations can manage secret sprawl across these platforms is if their secret manager controls this sprawl on their behalf through integrations.

Effectively a hub and spoke model with a secrets manager at the center, pushing secrets to providers where possible, instead of forcing applications to directly request secrets from the secrets manager.

Or, in simpler terms — Provide a secure path of least resistance for secrets fetching that is as native as possible to the platform or runtime.

Vault’s Kubernetes integration is in line with this thinking, shifting the responsibility of secrets fetching and injection to the platform layer so the application itself can (ideally) remain blissfully unaware of how its secrets were populated.

What makes things difficult for Vault is the variability in authentication methods and token delivery that make it harder for applications to prevent Vault implementation details from leaking into their application code or CI/CD jobs.

For example, life is relatively simple if running on Kubernetes and using the Kubernetes auth method, but if a more generalized auth method such as App Role is chosen, you’ll also need to consider Cubbyhole Response Wrapping for greater security.

By contrast, authorization in Doppler for production environments is performed using Service Tokens which provide read-only access to the secrets of a project for a specific environment.

Service Tokens can be instantly revoked, and customers are encouraged to specify a trusted list of IP addresses in CIDR format for each environment (Vault also supports CIDR block ranges for App Role authentication).

As an added layer of protection, GitHub officially supports scanning for leaked Doppler service tokens.

Summary

Doppler and HashiCorp Vault are both excellent secret managers that enable any development team to make insecure practices such as using .env files a thing of the past.

Vault offers unparalleled flexibility for secrets storage, authentication, secrets access, and hosting. However, this flexibility comes with a steep learning curve where design decisions and implementation support from Security and Platform teams will be highly advantageous to ensuring Vault will be adopted by development teams organization-wide.

Doppler is a fully managed Universal Secrets Manager. It’s built for a microservices world, natively organizing secrets into projects, each with a configurable list of environments using a simplified Service Token model for secret access in production environments.

It uses the language and framework agnostic method of environment variables to supply secrets and enables teams to go from local development to production in less than a day.

I highly recommend evaluating both to see whether the flexibility offered by Vault or the speed and simplicity of Doppler’s operating model is the best fit for you and your team.

I welcome your feedback and you can find me on Twitter or email me at ryan@doppler.com.

A big thanks for HashiCorp for reviewing the portions of this tutorial concerning Vault.