Securing Automated Database Access
Over my years as a software engineer I’ve seen quite a few different approaches to CI/CD credential management ranging from secure to egregiously bad.
A common one that I’ve encountered is the “vault and forget” approach. In this approach, an engineering team uses a credential manager like lastpass or 1Password to store their long-lived shared credentials that their different automation services use. These could be things like passwords, API tokens, certificates, SSH keys, etc. — all living in one central location. Then when the automation service accounts are deployed, they make a request to the vault server, which then returns the key.
This way no credentials have to be stored in git repositories, or — God forbid — on sticky notes taped to the monitor. While on the surface this may seem like a secure practice, storing these credentials in a vault introduces a litany of problems.
The Trouble with Shared Access
Perhaps the largest problem with sharing credentials is that these tokens are almost always long-lived. In some cases they are even permanent, having no automatic-expiration date.
This makes credential management much easier for engineers because they don’t have to rotate these keys and redeploy automation pipelines, which causes production downtime.
However easy, this is NOT GOOD. Due to their long-lived nature, if any of these credentials get out, attackers can use them indefinitely to grant access to secure systems and wreak havoc.
Without frequent rotation, ex-employees can still have these credentials stored, exposing you to potential leaks. And without good monitoring and auditing of your different automated services, it can be difficult to pin down exactly which cred has been exposed, increasing response time for putting out the fire.
| Type | Pro/con | Strength |
| Legacy Authentication Method |
|
It’s better than managing passwords yourself. Only recommend it if you are already using one and the cost to switch is too prohibitive. In other cases you should probably upgrade. |
| Strong Password Encryption for Authentication |
|
Good for isolated, non-shared resources that have a small attack surface. |
| Certificate-Based Authentication |
|
Ideal for diverse, distributed infrastructure at scale. Requires a certificate authority. Teleport is an open source product that provides a CA as part of its access plane. |
Rethinking Automated Access
Of course, this problem is not unique to automation; human engineers face the exact same problems and have shared credentials to manage as well. For humans these problems have largely been addressed with identity-based access controls: tying a human engineer, says “Alice Smith,” to a set of role-based access control (RBAC) roles, forcing her to authenticate with a single sign-on (SSO) like Okta for example to access sets of shared resources. At Teleport while developing open source secure access tools for humans, we thought: “What if we could bring the same identity-based access to every microservice, bot and resource in our CI/CD environment?” And thus MachineID was born.
With MachineID, we aim to change the way you think about automation access, assigning every resource and bot across your entire infrastructure a unique ID mapped to specific RBAC roles, and authenticating every request with short-lived X.509 certificates that automatically rotate in a fine-grained configurable manner. Because every resource has its own identity, this makes it possible to audit all the different bots’ activity across a variety of resources in one central location.
Previously if you really wanted to secure a database, you could configure your own certificate authority (CA) that would sign and issue SSL certificates whenever you wanted to add a service or user. This is extremely costly, however, and I’ve seen companies with whole teams dedicated to the maintenance and upkeep of such systems. In contrast, Teleport acts as its own CA, allowing automated service accounts to connect with the database using short-lived SSL certificates with the database credentials and configuration baked right in.
MachineID Demo
As an example of how to configure an on-prem MySQL database with Teleport MachineID, we have created a little demo here. This demo is powered by Instruqt, which sets you up with a sandbox environment right from your browser.
Let’s walk through what you can expect in the demo.
For this example, we’ll be assuming we have three nodes: the Teleport proxy, our MySQL database server and our machine that we will run our automated service account on.
Step 1: Create the Teleport bot user
In this step, we’ll be creating our Teleport bot user. We’ll use this bot user to assign an identity to our automated service, configure RBAC, and allow us to audit all the bot’s activity. Once we create this user, we will then configure our bot service with MachineID using a short-lived token to connect to the Teleport proxy.
Step 2: Configure the database
Once we have our bot user, we will then configure our MySQL database server to connect to Teleport using SSL certs generated from the Teleport proxy. First you must export the SSL database certificates generated by the proxy to the database host. Then to configure MySQL to use these certificates, add the paths of the exported certs to your MySQL configuration file, mysql.cnf like so:
|
1 2 3 4 5 6 7 |
``` [mysqld] require_secure_transport=ON ssl-ca=/path/to/server.cas ssl-cert=/path/to/server.crt ssl-key=/path/to/server.key ``` |
Additionally, your MySQL/MariaDB database user accounts must be configured to require a valid X509 client certificate by adding the REQUIRE SUBJECT field on the database user. This will later allow us to access our database remotely with our automated bot account.
Step 3: Start the MachineID service and connect to the database
Finally, we’ll start our MachineID service on the bot machine, open a proxy tunnel through the Teleport node to our database, and actually retrieve some of our data. Once this configuration is complete, you can set up automated services to securely access and modify data in the database however you see fit. And all of this activity is centrally logged in the Teleport node, providing you with an easy-to-follow audit trail for your bot accounts.
The way this works is through a lightweight agent called tbot. tbot is a small binary that comes bundled with a standard Teleport installation. This binary communicates with the Teleport proxy to facilitate the automatic reissue of database certificates. These certificates have both the identity of the robot user and the database credentials baked directly into them after the bot user connects with the one-time join token from the first step. Once tbot is authenticated, it will continually receive reissues of these end-user access certificates from the Teleport proxy.
Conclusion
When thinking about automated database access, it’s really important to find a solution that minimizes the blast radius when things go wrong. Whether that solution is something you create in-house from scratch or open source Teleport MachineID, it is key to have the same level of accountability for your human developers as your robotic ones. 🤖