CircleCI to Customers: Change All Secrets and API Tokens NOW!
We all know keeping secrets, such as passwords, credentials, keys, and access tokens, in our code, is a bad security idea. But, thanks to code-driven automation with secrets and continuous integration/continuous delivery (CI/CD) we do it anyway. And, sometimes, it comes back to bite us: Hard.
Such is the case with CI/CD company CircleCI. While the company doesn’t know yet exactly what went wrong with a “security instance,” CircleCI CTO Rob Zuber, has blogged: that you should:
- Immediately rotate any and all secrets stored in CircleCI. These may be stored in project environment variables or in contexts.
- Customers review internal logs for their systems for any unauthorized access starting from Dec. 21, 2022, through today, Jan. 4, 2023, or upon completion of your secrets rotation.
Zuber added, “Additionally, if your project uses Project API tokens, we have invalidated those and you will need to replace them.”
That said, Zuber said customers can still build their projects on CircleCI’s service. “We are confident that we have eliminated the risk that led to this incident,” Zuber wrote.
This is bad news with a capital B for CircleCI’s million-plus users. As one user put it on Reddit, “F**k, there goes the rest of my week… My company deploys several hundred websites through CircleCI, each with their own secrets…”
CircleCI promises to tell everyone about the incident as it completes its investigation. In the meantime, do as its say and change out your secrets. Users can continue to use
Mind, you don’t have to put secrets in your code. Or, if you do, make them temporary and auto-rotate them every hour or so. Or, better yet, use environment variables to hold secrets. Then, manage these secrets with programs such as Azure Key Vault, HashiCorp Vault, or AWS Systems Manager Parameter Store. You should also use role-based access (RBAC), so only authorized people can access even the encrypted secrets and variables.
CircleCI will support you in this. For example, the CI/CD service now supports OpenID Connect identity tokens. This enables your CircleCI jobs to authenticate with cloud providers that support OpenID Connect, such as AWS, Google Cloud Platform, and Vault, while avoiding the use of static secrets.
That said, you should also note that CircleCI appears to be in trouble. In December 2022, the company laid off 17% of its staff. CEO Jim Rose warns that this meant, “it’s not business as usual.” Potentially leaking its users’ secrets certainly counts as “not business in the usual” in the worst possible way.
