[kolla][barbican] Secure password management in Kolla-Ansible using Castellan
Hi all, I have deployed an OpenStack environment using Kolla Ansible. I'm looking for a way to avoid storing service passwords (e.g., database, keystone, etc.) in clear text inside the configuration files. Is there any recommended method to integrate Barbican and Castellan so that services like Nova, Keystone, or Cinder can fetch secrets securely during runtime using oslo.config? If not natively supported, are there best practices or known workarounds for achieving this securely in a Kolla-based deployment? Thanks in advance!
Why not using Ansible-Vault to secure passwords? On Thu, Jul 24, 2025 at 12:57 PM <elnazcloud@gmail.com> wrote:
Hi all,
I have deployed an OpenStack environment using Kolla Ansible. I'm looking for a way to avoid storing service passwords (e.g., database, keystone, etc.) in clear text inside the configuration files.
Is there any recommended method to integrate Barbican and Castellan so that services like Nova, Keystone, or Cinder can fetch secrets securely during runtime using oslo.config?
If not natively supported, are there best practices or known workarounds for achieving this securely in a Kolla-based deployment?
Thanks in advance!
Thank you for your response! I'm aware of Ansible Vault and currently use it to protect secrets at rest during deployment. However, my goal is to also avoid having the passwords appear in clear text within the final configuration files on the deployed nodes. I'm specifically looking for a runtime secret management solution, ideally integrating Barbican (via Castellan) with oslo.config, so that services like Keystone, Nova, and Cinder can fetch secrets securely at runtime, instead of reading them from plaintext config files. Does OpenStack currently support such integration in a production-ready way, particularly in Kolla-based deployments? If not, are there any best practices or patterns to follow for achieving this? Thanks again for your guidance.
It sounds like they might be – but the complaint is perhaps that once kolla-ansible generates the config, that config has the passwords in plain text (i.e. if you SSH into the machine, docker exec into the container, the /etc/nova/nova.conf has the keys built from the ansible-vault password.yml) I do not think there is a way around this, nor do I think there needs to be – if the passwords are only in “plain text” in the config on the container only accessibly by root, then privilege escalation to root is the vulnerability you should be protecting against anyway – even if you somehow worked out a way to have secrets separate in a way that wasn’t just a big house of cards, if there’s a privilege-escalation to root on your host nodes, you’re going to have massive problems, and some configuration passwords are going to be the least of them. @elnazcloud@gmail.com<mailto:elnazcloud@gmail.com> can you confirm you’re using the ansible-vault password encryption for your passwords.yml? Kolla-ansible has supported it for like 8 years: https://opendev.org/openstack/kolla-ansible/commit/684194ff9dbd442a93278bec2... Here is the link to the ansible documentation: https://docs.ansible.com/ansible/latest/vault_guide/vault_encrypting_content... Let us know if you need more information. Kind Regards, Joel McLean – Micron21 Pty Ltd From: Satish Patel <satish.txt@gmail.com> Sent: Tuesday, 29 July 2025 1:29 PM To: elnazcloud@gmail.com Cc: openstack-discuss@lists.openstack.org Subject: Re: [kolla][barbican] Secure password management in Kolla-Ansible using Castellan Why not using Ansible-Vault to secure passwords? On Thu, Jul 24, 2025 at 12:57 PM <elnazcloud@gmail.com<mailto:elnazcloud@gmail.com>> wrote: Hi all, I have deployed an OpenStack environment using Kolla Ansible. I'm looking for a way to avoid storing service passwords (e.g., database, keystone, etc.) in clear text inside the configuration files. Is there any recommended method to integrate Barbican and Castellan so that services like Nova, Keystone, or Cinder can fetch secrets securely during runtime using oslo.config? If not natively supported, are there best practices or known workarounds for achieving this securely in a Kolla-based deployment? Thanks in advance!
Yes, we are using Ansible Vault to encrypt our passwords.yml, and all sensitive values are stored encrypted in that file. However, our concern is that—even though the secrets are encrypted at deployment time—they still end up as plaintext in the final config files inside the containers (e.g., nova.conf). As highlighted in the OpenStack Summit talk, there's an initiative to improve secret handling by avoiding storing passwords directly in service configuration files, even on disk. Instead, passwords would be referenced or pulled dynamically from a secure external source, such as Barbican or HashiCorp Vault, via Castellan integration. So while encrypting passwords.yml with Ansible Vault is a good first step, we're interested in going further — specifically, preventing plaintext secrets from appearing in config files altogether, as part of a more secure design. Thanks in advance!
On 2025-07-29 07:20:39 -0000 (-0000), elnazcloud@gmail.com wrote: [...]
However, our concern is that—even though the secrets are encrypted at deployment time—they still end up as plaintext in the final config files inside the containers (e.g., nova.conf). [...]
I think oslo.config's remote_file driver was intended for this: https://docs.openstack.org/oslo.config/latest/configuration/options.html Of course you still need the backend securely configured with per-service client certs authorized for the specific blobs, and this only works for services relying on oslo.config. I'm not sure if there's a documented walkthrough for this sort of deployment, nor whether Kolla has any integrated support for setting it up. I've added ops and security-sig to the list of subject tags in case it bubbles this up to the attention of anyone in those circles who knows the answer. -- Jeremy Stanley
On 29/07/2025 15:54, Jeremy Stanley wrote:
On 2025-07-29 07:20:39 -0000 (-0000), elnazcloud@gmail.com wrote: [...]
However, our concern is that—even though the secrets are encrypted at deployment time—they still end up as plaintext in the final config files inside the containers (e.g., nova.conf).
i tired to help as best i can inline but im not entirely sure what you gain by doing this. in general if someone can acess the host filesystem and read the nova.conf your security evnolope has already been breached. there is a minimal gain but unless you have already move away form using password/user login ot using revocable rotatable application credentials for service creditals there are probaly lower hanging furit you can adress first in hardening your openstack deployment. the main isue to me with saying "i dont turst storing secrete on the host file systme" is the classing boot straping probelum to regrive data form barbican or vault or any other Castellan backend you will still need a local root of trust, be that a secret sotred in teh host tpm or the client cert used for the remote_file backend. if the attacker can read that then they with a littel more work can just load the data form Castellan if that datasouce is remote not local like an infra barbican then you can audit the access and rotate the key but that has the disadvantage of intoduceing a single point of failure in your deployment. the keystore. so you need to be careful not to use any part of the cloud that uses the keystore to provision the keystore or you can have a chicken and egg situation if you have a pwoer outage. anyway if you figure out a nice way of doing this then im sure others would be interested in hearing about it.
[...]
I think oslo.config's remote_file driver was intended for this: https://docs.openstack.org/oslo.config/latest/configuration/options.html
the remote file diiver is not nessiarly a perfect fit but it an exampel of the plugable interface. you could write a vault or castellan driver for oslo.config but there was one developed in the past https://review.opendev.org/c/openstack/castellan/+/599589 oslo.config provides a stevedore entry point for this interface https://github.com/openstack/oslo.config/blob/68cefad313bd03522e99b3de95f178... the remote file driver is here https://github.com/openstack/oslo.config/blob/68cefad313bd03522e99b3de95f178... as a refence. it does not support http basic auth but i belive if you provide a client key and cert you can do mtls on the remote webserver. the plugin loading s done here https://github.com/openstack/oslo.config/blob/493a08645fe7339b9883f71748f341... going back to your question about castallen specifically you could install castallen in contaienr that dont already have it via a config overried there are exmples of this in the kolla docs https://github.com/openstack/kolla/blob/master/doc/source/admin/template-ove... https://github.com/openstack/kolla/blob/master/doc/source/admin/image-buildi... writing your own oslo.config driver is not common but i blelive at least one operator has written one to use dns svc or txt records to store the service config in dns so if the https://github.com/openstack/castellan/blob/master/castellan/_config_driver.... driver does not work for you should be able to provide your own. you just need to install castellan and add the following to your config overrides [DEFAULT] config_source = castellan_config_group [castellan_config_group] driver = castellan config_file = castellan.conf mapping_file = mapping.conf to provide the mapping.conf and castellan.conf you will likely need ot use https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration... to add the files to the reslevent container that need them and have a separate role to ocpy them.
Of course you still need the backend securely configured with per-service client certs authorized for the specific blobs, and this only works for services relying on oslo.config. I'm not sure if there's a documented walkthrough for this sort of deployment, nor whether Kolla has any integrated support for setting it up.
there is a bit of a chicken and egg situraiton depending on what backend you wanted to use i.e. if you wnated to store thing in barbican you would have to dlepoy it first without useing this then the other sevices. but if you were ot use an external barbican or vault you may be able to make this work today with enough effort. you would have to manulaly overreid the password or other filed you wnat to come form castellan using kolls genering service cofnig overried method. https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration... then manually config the secrete in the relevent data store. but to do this properly in a maintainable way you would want a new flag to disbale some or all of the config generation. you may also want to automate the process for manginging the configs in the secret backend but im not enritly sure if you would want ot do that in kolla-ansible. ansibel itself can use a number of inventory datasouces beyond just files so i guess you could perhaps do that security but if your just going to use files on disk to provie the data to kolla it kind of defeats the exercise of doing it this way in the first place.
I've added ops and security-sig to the list of subject tags in case it bubbles this up to the attention of anyone in those circles who knows the answer.
On 2025-07-29 21:27:51 +0100 (+0100), Sean Mooney wrote: [...]
`in general if someone can acess the host filesystem and read the nova.conf your security evnolope has already been breached. [...]
It's not uncommon in naïve security policies that are primarily concerned with "data at rest" to require all sensitive information on disk be encrypted. Their authors are typically focused on what happens if a decommissioned hard drive or stray backup tape falls into the wrong hands. As you note, there's a substantial Catch-22 here, in that the client keys for accessing the remote service represent a similar risk unless kept isolated, e.g. in an HSM. What this fails to address, though, is that the plaintext form of passwords and keys will still be present in process memory on running systems, and in many cases these systems are configured with swap partitions or files which are themselves not encrypted, so it's still possible for the vulnerable secrets to end up on disk if they get paged out for any reason. Encrypted swap can help here, but at that point why not just encrypt the entire disk and be done with it? Measures to fully isolate secrets from storage on a system that uses them are rarely worth the operational costs. As a security professional, when presented with requirements like these my initial recommendation is to think hard about what exactly you're protecting and from whom. Essentially, "define your risks." To make sense, the cost of the solution should not exceed its value. If you can't define the value of a security measure, it probably has none in actuality. If you can't estimate the cost and likelihood of a risk being exploited, then it's likely not worth mitigating. If you can't answer these questions, then first find someone who can. -- Jeremy Stanley
On 7/29/25 4:58 PM, Jeremy Stanley wrote:
On 2025-07-29 21:27:51 +0100 (+0100), Sean Mooney wrote: [...]
`in general if someone can acess the host filesystem and read the nova.conf your security evnolope has already been breached. [...]
It's not uncommon in naïve security policies that are primarily concerned with "data at rest" to require all sensitive information on disk be encrypted. Their authors are typically focused on what happens if a decommissioned hard drive or stray backup tape falls into the wrong hands. As you note, there's a substantial Catch-22 here, in that the client keys for accessing the remote service represent a similar risk unless kept isolated, e.g. in an HSM. A cautionary comment here. A fair number of years ago I purchased a tape backup library system on eBay. It arrived and had a number of tapes still installed inside. For lark I looked to see if there was data on the tapes. ..... Well there was some. It was the database backup from an HMO in California. It included personal information. I did not look too deep but did verify that a few names still existed and they contact information was up to date.
So scrub those disks and other media before you junk them. In another case a company I was working with went bankrupt. The hardware was sold by the bankruptcy firm but it included the full Unix system V source code on some of the disks. The guy who bought the hardware apparently tried to ransom it back to AT&T but if my memory serves me correctly it did not go well for him. Clean up is important. -- Alvin Starr || land: (905)513-7688 Netvel Inc. || Cell: (416)806-0133 alvin@netvel.net ||
Just bringing my thoughts to this list; The OpenStack Summit Talk (https://youtu.be/6cjepMGQfrM?t=883) shows a proof-of-concept where they're able to use Barbican to protect glance.conf ( or any-other.conf ) passwords; but someone ask the winning question (I have linked to the exact time) which is "How does one then protect the Barbican or Vault?" The answer given ( I believe by Tin Lam?) pretty much hits the nail on the head - this leads to a chicken-and-egg issue - since Barbican or Vault must be accessible to the hosts/services which have the configuration files, then anyone who can see the configuration files' contents can then take the next logical step and ask the secret provider for the secret, using whatever privileges that host/service already has to request the secret. In their example, they suggest that the Barbican's credentials could be kept out of the configuration file by using environment variables, but that's just having the unencrypted credential in a different place; a privileged user on that host/service can access the service's environment variables, then request the key from Barbican. In the summit talk, they only go through one risk scenario which is what they are trying to address; which is when a developer reaches out to a vendor for support they might inadvertently send the entire configuration file, passwords and all, and that is the risk they're looking to mitigate. If you're using Kolla-ansible, your configuration files already don't have the passwords, as they're typically configuration overrides, supplementing the kolla-ansible j2 templates - unless you're pulling the fully compiled configuration files out of the running containers, and sending those. At the start of the Summit, they talk about some other avenues they did _not_ try because it introduced significant complexity; one of which was to encrypt the configuration files, or partially encrypt the contents of the configuration files so that the credentials were not stored in plain text at rest. This would have mitigated AT&T's requirement of developers sending full configuration files to vendors, but is less cloud-native and proposes new technological challenges such as what to do when you need to rotate those credentials. It may or may not afford you greater security than using Barbican or Vault, as at some point the service accessing that configuration file needs the keys to decrypt the contents of the configuration file; so we're back to square one - a user with privileged access to read the configuration files presumably has enough privileged access to access to configuration file's decryption key. The trade off is a non-linear scale of "Complexity and Greater Security <--------> Convenience but Lesser Security". Everyone's risk appetite for where they land on that scale is different, and a mature organisation will be able to shift around on that scale as they become more competent in overcoming complexity, and can then benefit from the greater security. So, depending on your risk profile, and what risks you are trying to mitigate, using Barbican or Hashicorp Vault to store secrets, and have the services collect these secrets as needed might not address your particular risks. Presuming you follow a Hashicorp Vault service, or a similar service, to protect your keys, and you dialled up the security using ACLs and Policies, you could greatly increase the secrecy of your credentials, but at a significant increase to your risk of service availability/interruption (i.e. What if the Vault goes away/down? What if you lose the trust relationship with your vault/policy - how do you recover?). If the increased secrecy mitigates some significant risks, that's great - you just want to make sure that you aren't introducing new, potentially far more impactful risks to your setup. One way to go very hard to the "Complexity and Greater Security" scale, you could consider re-building and changing every kolla container's runtime user/group and privileges, so that even using docker privilege escalation tricks can't access the kolla service container, but this requires a lot of tooling, and has far reaching consequences for things like troubleshooting and applying updates. I would encourage anyone who is looking to enhance their security position to do a genuine threat and risk assessment - really put down on paper (or a spreadsheet, or your risk analysis tooling if you're lucky enough to have some) to determine whether or not the steps you're taking to address a security concern have merit, or whether the effort could be more effectively used elsewhere. Kind Regards, Joel McLean – Micron21 Pty Ltd -----Original Message----- From: elnazcloud@gmail.com <elnazcloud@gmail.com> Sent: Tuesday, 29 July 2025 5:21 PM To: openstack-discuss@lists.openstack.org Subject: RE: [kolla][barbican] Secure password management in Kolla-Ansible using Castellan Yes, we are using Ansible Vault to encrypt our passwords.yml, and all sensitive values are stored encrypted in that file. However, our concern is that—even though the secrets are encrypted at deployment time—they still end up as plaintext in the final config files inside the containers (e.g., nova.conf). As highlighted in the OpenStack Summit talk, there's an initiative to improve secret handling by avoiding storing passwords directly in service configuration files, even on disk. Instead, passwords would be referenced or pulled dynamically from a secure external source, such as Barbican or HashiCorp Vault, via Castellan integration. So while encrypting passwords.yml with Ansible Vault is a good first step, we're interested in going further — specifically, preventing plaintext secrets from appearing in config files altogether, as part of a more secure design. Thanks in advance!
Hello, everyone. I was reading all of the arguments and consolidating my thoughts based on the ones that were presented. Here it goes my humble contribution. Answering the last question if application credentials meet this need, I'd say potentially YES. Considering all the features that AppCreds (for short) bring to the game, such as: 1. Enforcing Least Privilege. 2. Providing Revocation and Rotation. 3. Enabling Improved Auditability. 4. And addressing Service Authentication. I'd say it helps a lot in this sense. Remembering that AppCreds are an authentication mechanism, not a secrets storage solution... In this topic (protection of secrets), the argument that the privilege escalation to root on a specific system/host undermines the need to protect sensitive data, such as secrets, is absolutely obsolete. We have to have a strong way to protect the keys that encrypt the secrets to avoid that an attacker who successfully escalates privileges to root on an owned system is able to access such sensitive data, and potentially compromise other systems. Just to mention two de-facto standards that touch this point, we have NIST Special Publication 800-53 [1] and ENISA NIS 2 directive [2]. Defense in Depth must always come to the discussion no matter what. Best regards, Mauricio [1] https://csrc.nist.gov/pubs/sp/800/53/r5/final [2] https://www.enisa.europa.eu/publications/nis2-technical-implementation-guida...
On 2025-08-05 11:15:35 -0000 (-0000), mharley@redhat.com wrote: [...]
In this topic (protection of secrets), the argument that the privilege escalation to root on a specific system/host undermines the need to protect sensitive data, such as secrets, is absolutely obsolete. We have to have a strong way to protect the keys that encrypt the secrets to avoid that an attacker who successfully escalates privileges to root on an owned system is able to access such sensitive data, and potentially compromise other systems. [...]
Sadly, asserting that it "has to exist" doesn't make it happen. The only common (and I use the term loosely) approach I'm aware of is to lock them in dedicated hardware, something like an HSM, so that it can be asked to decrypt credentials or sign authentication requests and return the results while the key can't itself be accessed by the operating system kernel. And even then, someone with root access to the server can still make use of the keys in the HSM, this really only addresses the data-at-rest problem. Also at the very least keep swap encrypted (unlocked by the hardware at boot, not with an encryption key that's just lying around on the rootfs), because any plaintext secrets in process memory can get paged out. Python is really not designed for secure handling of data in memory, and short of only accessing it with special C extensions or employing dodgy memset hacks there's no way I'm aware of to actually wipe the contents of strings and similar immutable data structures without terminating the process (always handling secrets with mutable types and then avoiding dynamic resizing can help). These are not cheap solutions, neither in terms of special hardware costs nor operational complexity, and so the risks posed by having credentials on disk need to outweigh the costs of mitigation for such approaches to make sense. -- Jeremy Stanley
participants (7)
-
Alvin Starr
-
elnazcloud＠gmail.com
-
Jeremy Stanley
-
Joel McLean
-
mharley＠redhat.com
-
Satish Patel
-
Sean Mooney