Supply chain security has been a big topic of conversation over the past several years, and while many of the conversations have revolved around insecure third-party components in codebases, there’s another part of the supply chain that could have a negative impact if not secured properly: secrets.
Max Power, product lead for Bitwarden Secrets Manager, said that from a development perspective, secrets include things like API keys, certificates, and SSH keys.
“Any chain is only as secure as the weakest link,” said Power. “The same applies to organizations. We have seen in the past multiple examples of massive data breaches as a result of accidentally leaked secrets, particularly secrets that were either hard-coded or pushed in Git repos.”
According to GitGuardian’s 2024 State of Secrets Sprawl Report, 12.7 million secrets were detected in public GitHub commits in 2023, which was a 28% increase from the previous year. Over the past four years, the problem of secrets sprawl has gotten four times worse, as in 2020 only 3 million secrets were detected.
Power says that when it comes to security, it’s important that everyone take responsibility for the codebase, from development to production to deployment, and ensure that secrets aren’t being hard-coded.
According to Brian Vallelunga, founder and CEO of the secrets management company Doppler, there are many ways developers share and store secrets, and some are better than others. The least secure method is storing them in files on their computer. Unfortunately, Bitwarden’s Power says this is one of the most common ways secrets are stored.
A step up from that are the people storing secrets in their cloud provider tools or building their own tools, Vallelunga explained. Developers may be storing secrets in the built-in AWS tooling, for example, but that becomes tricky because it means your secrets are all tied up in one tool. And then there are companies out there building their own internal tools for this purpose, but then start running into scalability issues eventually, he said.
The most secure method would be to use a dedicated secrets management provider that is designed for this specific purpose. Vallelunga explained that some of the added benefits of using these tools are that it makes it easier to share across teams and also offers things like access controls, auditing, and automated synchronization.
To put this into a real-life example, say you’re integrating with a service like Stripe, which requires you to have an API key that is needed throughout the development life cycle, explained Nic Manoogian, engineering manager at Doppler.
“So local developers, if I’m integrating with this new service, I need a test environment to try this stuff out,” he said.
He said that secrets tend to be more secure in production environments for companies with a mature security practice, but then less so in local dev environments. “Maybe your company has a really mature process for managing secrets in these upper environments and these deployments, but in the local development environments, it’s kind of like, well, I don’t know, call your manager and ask for the .env file, or we’ll just check it into code. And that comes with a whole bunch of other issues,” said Manoogian.
Vallelunga believes that in order to successfully implement good secrets management practices, teams should put up as many safeguards as possible and make it work with their workflows so that it’s as easy as possible for developers.
When developers feel that they need to start taking shortcuts in order to get things done quicker, that is when security incidents happen, he explained.
Vallelunga believes that as organizations begin to grow and mature, they tend to take a closer look at risk and thus address their problems with managing secrets.
“I think companies kind of go into two modes, the first mode is to build something that is valuable,” he said. “And then once they reach that point, then it’s to protect the thing that is valuable as it’s growing. And when they get into that protect mode, they start looking at all the areas of risks. And when you’re looking at the keys to your digital kingdom, that’s probably one of the biggest areas of risks you can have. And that’s when companies really start to think about that.”
