[tripleo] Configuration Management without Puppet
(First of all: thanks to Alex Schultz who initiated the discussion during the last PTG).
Context: With the containerization of the services in TripleO, our usage of Puppet is pretty much limited to laying down parameters into configuration files. Our long term goal is to reduce our number of tools in TripleO and converge toward more Ansible and we are currently investigating how to make it happen for Configuration Management, in a backward compatible and non-disruptive way.
Problems: - Our current interface to configure things is very tight to Puppet and Hiera, which would make it complicated to bind it with Hiera. Some of us have tried (to create some Hiera module in Ansible) but clearly this is not the path we want to take as far I know. - We don't use the logic (generally) in the Puppet modules and again only care about the configuration providers (in puppet-openstacklib and for some services, templates files), so the Puppet modules now do too much for what we need in TripleO.
Proposals: - Create an interface in THT that will define what the configuration looks like for each service. The format will be simple and easy to consume from any tool (Puppet to start for backward compatibility but also Ansible and why not etcd one day).
Example with Glance in deployment/glance/glance-api-container-puppet.yaml: param_config: map_merge: - glance_api_config: DEFAULT: enable_v2_api: true log_dir: /var/log/glance oslo_middleware: enable_proxy_headers_parsing: true - glance_image_import_config: image_import_opts: image_import_plugins: {get_param: GlanceImageImportPlugins}
https://review.opendev.org/#/c/660791/29/deployment/glance/glance-api-contai...
It's not tight to Hiera and it's generating param_config which is JSON and consumable by Ansible or etcd later. Here, glance_api_config and glance_image_import_config are the Puppet providers which configure a specific file for Glance but we could imagine a binding for Ansible like: glance_api_config: /etc/glance/glance-api.conf
- Move the services to use this new interface and I think it'll take multiple cycles to do that due to our amount of services.
Things I haven't figured out (yet): - I'm still working on figuring out how we will handle ExtraConfig, service_config_settings and all the Hiera datafiles that we support in puppet/role.role.j2.yaml (ideas welcome). I can only imagine something like: ExtraConfig: - glance_api_config: DEFAULT: log_dir: /var/log/glance-2
Which somehow would override a previous hash. The blocker I found is that map_merge doesn't do deep merges, so if you merge glance_api_config, you overrides all keys... which is problematic if you only want to update one key. I'll investigate that further.
- Backward compatibility for things like "ExtraConfig: glance::api::workers". Since we wouldn't rely on the classes anymore, our users calling the parameters from the classes will be broken. This also needs to be investigated.
- non Puppet OpenStack modules like mysql, apache, etc, mostly use erb templates. This is, for now, out of scope and the plan is to look at OpenStack services first. But ideas welcome here as well. Of course we would prefer to consume upstream roles if they are well tested & maintained.
If you want to review the ongoing work, here are the links: https://review.opendev.org/#/c/660726 https://review.opendev.org/#/c/661377 https://review.opendev.org/#/c/661093 https://review.opendev.org/#/c/660791
Thanks for reading that far, feel free to comment and contribute. I plan to continue this work and evaluate if all of this is actually worth it.
This looks very promising, thank you. One concern I have is that we maintain some of the troubleshooting ability that the hieradata files give us today in the long term.
The hieradata files on overcloud nodes make it convenient to glean facts about the deployed host. They also act as a canary for when the overcloud Heat stacks are out of sync with the deployed hosts.
Once we make the param_config consumable by Ansible or etcd, please keep supportability and troubleshooting in mind. Having local config data in a convenient place on the deployed host is important.
On Mon, May 27, 2019 at 4:36 PM Emilien Macchi emilien@redhat.com wrote:
(First of all: thanks to Alex Schultz who initiated the discussion during the last PTG).
Context: With the containerization of the services in TripleO, our usage of Puppet is pretty much limited to laying down parameters into configuration files. Our long term goal is to reduce our number of tools in TripleO and converge toward more Ansible and we are currently investigating how to make it happen for Configuration Management, in a backward compatible and non-disruptive way.
Problems:
- Our current interface to configure things is very tight to Puppet and
Hiera, which would make it complicated to bind it with Hiera. Some of us have tried (to create some Hiera module in Ansible) but clearly this is not the path we want to take as far I know.
- We don't use the logic (generally) in the Puppet modules and again only
care about the configuration providers (in puppet-openstacklib and for some services, templates files), so the Puppet modules now do too much for what we need in TripleO.
Proposals:
- Create an interface in THT that will define what the configuration looks
like for each service. The format will be simple and easy to consume from any tool (Puppet to start for backward compatibility but also Ansible and why not etcd one day).
Example with Glance in deployment/glance/glance-api-container-puppet.yaml: param_config: map_merge: - glance_api_config: DEFAULT: enable_v2_api: true log_dir: /var/log/glance oslo_middleware: enable_proxy_headers_parsing: true - glance_image_import_config: image_import_opts: image_import_plugins: {get_param: GlanceImageImportPlugins}
https://review.opendev.org/#/c/660791/29/deployment/glance/glance-api-contai...
It's not tight to Hiera and it's generating param_config which is JSON and consumable by Ansible or etcd later. Here, glance_api_config and glance_image_import_config are the Puppet providers which configure a specific file for Glance but we could imagine a binding for Ansible like: glance_api_config: /etc/glance/glance-api.conf
- Move the services to use this new interface and I think it'll take
multiple cycles to do that due to our amount of services.
Things I haven't figured out (yet):
- I'm still working on figuring out how we will handle ExtraConfig,
service_config_settings and all the Hiera datafiles that we support in puppet/role.role.j2.yaml (ideas welcome). I can only imagine something like: ExtraConfig: - glance_api_config: DEFAULT: log_dir: /var/log/glance-2
Which somehow would override a previous hash. The blocker I found is that map_merge doesn't do deep merges, so if you merge glance_api_config, you overrides all keys... which is problematic if you only want to update one key. I'll investigate that further.
- Backward compatibility for things like "ExtraConfig:
glance::api::workers". Since we wouldn't rely on the classes anymore, our users calling the parameters from the classes will be broken. This also needs to be investigated.
- non Puppet OpenStack modules like mysql, apache, etc, mostly use erb
templates. This is, for now, out of scope and the plan is to look at OpenStack services first. But ideas welcome here as well. Of course we would prefer to consume upstream roles if they are well tested & maintained.
If you want to review the ongoing work, here are the links: https://review.opendev.org/#/c/660726 https://review.opendev.org/#/c/661377 https://review.opendev.org/#/c/661093 https://review.opendev.org/#/c/660791
Thanks for reading that far, feel free to comment and contribute. I plan to continue this work and evaluate if all of this is actually worth it. -- Emilien Macchi
On Mon, May 27, 2019 at 7:49 PM Dan Sneddon dsneddon@redhat.com wrote:
This looks very promising, thank you. One concern I have is that we maintain some of the troubleshooting ability that the hieradata files give us today in the long term.
The hieradata files on overcloud nodes make it convenient to glean facts about the deployed host. They also act as a canary for when the overcloud Heat stacks are out of sync with the deployed hosts.
Once we make the param_config consumable by Ansible or etcd, please keep supportability and troubleshooting in mind. Having local config data in a convenient place on the deployed host is important.
You made very good points and I think we'll maintain an optional puppet step where our operators can apply any Hiera like before; at least until we think our operators have switched to the new interface at some point.
For debugging, we could think of the same kind of usage, where you have your custom-params.yaml like :
--- glance_api_config: DEFAULT: enable_v2_api: true log_dir: /var/log/glance
We would merge the hashes and lay down the config (with some hierarchy like before with Hiera datafiles). Also a note on etcd; even if the data would be consumed from etcd by services (currently not supported by oslo-config AFIK), the data would be available in config-download when generating the playbooks & data, so we could make it so you can modify the data and run the playbooks to change any config.
Thanks for your feedback,
On 28.05.2019 1:35, Emilien Macchi wrote:
(First of all: thanks to Alex Schultz who initiated the discussion during the last PTG).
Context: With the containerization of the services in TripleO, our usage of Puppet is pretty much limited to laying down parameters into configuration files. Our long term goal is to reduce our number of tools in TripleO and converge toward more Ansible and we are currently investigating how to make it happen for Configuration Management, in a backward compatible and non-disruptive way.
Problems:
- Our current interface to configure things is very tight to Puppet and
Hiera, which would make it complicated to bind it with Hiera. Some of us have tried (to create some Hiera module in Ansible) but clearly this is not the path we want to take as far I know.
- We don't use the logic (generally) in the Puppet modules and again
only care about the configuration providers (in puppet-openstacklib and for some services, templates files), so the Puppet modules now do too much for what we need in TripleO.
Proposals:
- Create an interface in THT that will define what the configuration
looks like for each service. The format will be simple and easy to consume from any tool (Puppet to start for backward compatibility but also Ansible and why not etcd one day).
Example with Glance in deployment/glance/glance-api-container-puppet.yaml: param_config: map_merge: - glance_api_config: DEFAULT: enable_v2_api: true log_dir: /var/log/glance oslo_middleware: enable_proxy_headers_parsing: true - glance_image_import_config: image_import_opts: image_import_plugins: {get_param: GlanceImageImportPlugins}
https://review.opendev.org/#/c/660791/29/deployment/glance/glance-api-contai...
It's not tight to Hiera and it's generating param_config which is JSON and consumable by Ansible or etcd later. Here, glance_api_config and glance_image_import_config are the Puppet providers which configure a specific file for Glance but we could imagine a binding for Ansible like: glance_api_config: /etc/glance/glance-api.conf
- Move the services to use this new interface and I think it'll take
multiple cycles to do that due to our amount of services.
Things I haven't figured out (yet):
- I'm still working on figuring out how we will handle ExtraConfig,
service_config_settings and all the Hiera datafiles that we support in puppet/role.role.j2.yaml (ideas welcome). I can only imagine something like: ExtraConfig: - glance_api_config: DEFAULT: log_dir: /var/log/glance-2
Which somehow would override a previous hash. The blocker I found is that map_merge doesn't do deep merges, so if you merge glance_api_config, you overrides all keys... which is problematic if you only want to update one key. I'll investigate that further.
- Backward compatibility for things like "ExtraConfig:
glance::api::workers". Since we wouldn't rely on the classes anymore, our users calling the parameters from the classes will be broken. This also needs to be investigated.
- non Puppet OpenStack modules like mysql, apache, etc, mostly use erb
templates. This is, for now, out of scope and the plan is to look at OpenStack services first. But ideas welcome here as well. Of course we would prefer to consume upstream roles if they are well tested & maintained.
I believe we should "translate" hiera & puppet configurations we have in t-h-t directly into k8s-native (or at least looking-like) YAML structures, to have those consumable either as k8s config-maps or ansible vars, or etcd keys transparently and interchangeable. That would allow us to omit any additional translation work and intermediate data abstractions in future, when/if we'd decided to convert t-h-t to manage container pods and/or deploy it via k8s operators framework.
I know there had been the translation work done for adapting ceph-ansible data structures (right, not really puppet...) into Rook [0], an operator for Kubernetes clusters. So perhaps we should add the ceph clusters deployment automation folks and teams in the loop and leverage their experience with the subject.
[0] https://github.com/rook/rook
If you want to review the ongoing work, here are the links: https://review.opendev.org/#/c/660726 https://review.opendev.org/#/c/661377 https://review.opendev.org/#/c/661093 https://review.opendev.org/#/c/660791
Thanks for reading that far, feel free to comment and contribute. I plan to continue this work and evaluate if all of this is actually worth it. -- Emilien Macchi
On Mon, May 27, 2019, at 4:36 PM, Emilien Macchi wrote:
(First of all: thanks to Alex Schultz who initiated the discussion during the last PTG).
Context: With the containerization of the services in TripleO, our usage of Puppet is pretty much limited to laying down parameters into configuration files. Our long term goal is to reduce our number of tools in TripleO and converge toward more Ansible and we are currently investigating how to make it happen for Configuration Management, in a backward compatible and non-disruptive way.
Problems:
- Our current interface to configure things is very tight to Puppet and
Hiera, which would make it complicated to bind it with Hiera. Some of us have tried (to create some Hiera module in Ansible) but clearly this is not the path we want to take as far I know.
- We don't use the logic (generally) in the Puppet modules and again
only care about the configuration providers (in puppet-openstacklib and for some services, templates files), so the Puppet modules now do too much for what we need in TripleO.
snip
I'm not sure if this is useful but what the Infra team did was to transplant all of its hiera data into /etc/ansible/hosts/host_vars and /etc/ansible/hosts/group_vars. Then we updated our puppetry to pull hiera data out of there. This means that puppet and ansible read the same data sources which has made transitioning things easy for us.
Though I think the ansible-role-puppet role copies hiera data sources from /etc/ansible/hosts to the appropriate puppet hiera location on the remote source. But you mostly don't have to think about that and there is no double accounting from an ops perspective.
Clark
Wouldn't it be easier just to use OSA once things get past laying down the OS and the relevant configs for the network interfaces. It seems like building it all in ansible has already been done, and much of the work could be used. I understand that both have different opinions on how to deploy openstack, but it would lower the lift in getting an ansible based deployment operational much sooner.
On Tue, May 28, 2019 at 12:17 PM Clark Boylan cboylan@sapwetik.org wrote:
On Mon, May 27, 2019, at 4:36 PM, Emilien Macchi wrote:
(First of all: thanks to Alex Schultz who initiated the discussion during the last PTG).
Context: With the containerization of the services in TripleO, our usage of Puppet is pretty much limited to laying down parameters into configuration files. Our long term goal is to reduce our number of tools in TripleO and converge toward more Ansible and we are currently investigating how to make it happen for Configuration Management, in a backward compatible and non-disruptive way.
Problems:
- Our current interface to configure things is very tight to Puppet and
Hiera, which would make it complicated to bind it with Hiera. Some of us have tried (to create some Hiera module in Ansible) but clearly this is not the path we want to take as far I know.
- We don't use the logic (generally) in the Puppet modules and again
only care about the configuration providers (in puppet-openstacklib and for some services, templates files), so the Puppet modules now do too much for what we need in TripleO.
snip
I'm not sure if this is useful but what the Infra team did was to transplant all of its hiera data into /etc/ansible/hosts/host_vars and /etc/ansible/hosts/group_vars. Then we updated our puppetry to pull hiera data out of there. This means that puppet and ansible read the same data sources which has made transitioning things easy for us.
Though I think the ansible-role-puppet role copies hiera data sources from /etc/ansible/hosts to the appropriate puppet hiera location on the remote source. But you mostly don't have to think about that and there is no double accounting from an ops perspective.
Clark
On Tue, May 28, 2019, at 9:40 AM, Donny Davis wrote:
Wouldn't it be easier just to use OSA once things get past laying down the OS and the relevant configs for the network interfaces. It seems like building it all in ansible has already been done, and much of the work could be used. I understand that both have different opinions on how to deploy openstack, but it would lower the lift in getting an ansible based deployment operational much sooner.
Note sure if this was directed to my comments about what Infra has done or if you mean to suggest this for Tripleo. In the Infra case we do not deploy openstack. We deploy a ton of software on top of OpenStack so OSA isn't relevant for our uses.
On Tue, May 28, 2019 at 12:17 PM Clark Boylan cboylan@sapwetik.org wrote:
On Mon, May 27, 2019, at 4:36 PM, Emilien Macchi wrote:
(First of all: thanks to Alex Schultz who initiated the discussion during the last PTG).
Context: With the containerization of the services in TripleO, our usage of Puppet is pretty much limited to laying down parameters into configuration files. Our long term goal is to reduce our number of tools in TripleO and converge toward more Ansible and we are currently investigating how to make it happen for Configuration Management, in a backward compatible and non-disruptive way.
Problems:
- Our current interface to configure things is very tight to Puppet and
Hiera, which would make it complicated to bind it with Hiera. Some of us have tried (to create some Hiera module in Ansible) but clearly this is not the path we want to take as far I know.
- We don't use the logic (generally) in the Puppet modules and again
only care about the configuration providers (in puppet-openstacklib and for some services, templates files), so the Puppet modules now do too much for what we need in TripleO.
snip
I'm not sure if this is useful but what the Infra team did was to transplant all of its hiera data into /etc/ansible/hosts/host_vars and /etc/ansible/hosts/group_vars. Then we updated our puppetry to pull hiera data out of there. This means that puppet and ansible read the same data sources which has made transitioning things easy for us.
Though I think the ansible-role-puppet role copies hiera data sources from /etc/ansible/hosts to the appropriate puppet hiera location on the remote source. But you mostly don't have to think about that and there is no double accounting from an ops perspective.
Clark
I was directing this at 3O, but didn't want to jump into the middle of the thread. Thanks for helping to clarify.
~/D
On Tue, May 28, 2019 at 12:48 PM Clark Boylan cboylan@sapwetik.org wrote:
On Tue, May 28, 2019, at 9:40 AM, Donny Davis wrote:
Wouldn't it be easier just to use OSA once things get past laying down the OS and the relevant configs for the network interfaces. It seems like building it all in ansible has already been done, and much of the work could be used. I understand that both have different opinions on how to deploy openstack, but it would lower the lift in getting an ansible based deployment operational much sooner.
Note sure if this was directed to my comments about what Infra has done or if you mean to suggest this for Tripleo. In the Infra case we do not deploy openstack. We deploy a ton of software on top of OpenStack so OSA isn't relevant for our uses.
On Tue, May 28, 2019 at 12:17 PM Clark Boylan cboylan@sapwetik.org
wrote:
On Mon, May 27, 2019, at 4:36 PM, Emilien Macchi wrote:
(First of all: thanks to Alex Schultz who initiated the discussion during the last PTG).
Context: With the containerization of the services in TripleO, our usage of Puppet is pretty much limited to laying down parameters into configuration files. Our long term goal is to reduce our number of tools in TripleO and converge toward more Ansible and we are currently investigating how
to
make it happen for Configuration Management, in a backward
compatible
and non-disruptive way.
Problems:
- Our current interface to configure things is very tight to Puppet
and
Hiera, which would make it complicated to bind it with Hiera. Some
of
us have tried (to create some Hiera module in Ansible) but clearly
this
is not the path we want to take as far I know.
- We don't use the logic (generally) in the Puppet modules and
again
only care about the configuration providers (in puppet-openstacklib
and
for some services, templates files), so the Puppet modules now do
too
much for what we need in TripleO.
snip
I'm not sure if this is useful but what the Infra team did was to
transplant all of its hiera data into /etc/ansible/hosts/host_vars and /etc/ansible/hosts/group_vars. Then we updated our puppetry to pull hiera data out of there. This means that puppet and ansible read the same data sources which has made transitioning things easy for us.
Though I think the ansible-role-puppet role copies hiera data sources
from /etc/ansible/hosts to the appropriate puppet hiera location on the remote source. But you mostly don't have to think about that and there is no double accounting from an ops perspective.
Clark
On Tue, May 28, 2019 at 10:50 AM Donny Davis donny@fortnebula.com wrote:
Wouldn't it be easier just to use OSA once things get past laying down the OS and the relevant configs for the network interfaces. It seems like building it all in ansible has already been done, and much of the work could be used. I understand that both have different opinions on how to deploy openstack, but it would lower the lift in getting an ansible based deployment operational much sooner.
It's not easier because we then have to translate our current puppet/hiera configuration into their ansible vars. We're trying to come up with a generic construct that still allows end users to override specific configuration items without having to continue to use puppet to lay down the configuration. The proposed format maps more to what oslo.config needs and less about the implementation of what is ultimately doing the configuration writing. We still also need to support backwards compatibility to a certain extent. We unfortunately cannot just drop in OSA because we have our own opinionated assumptions about how we do containers/orchestration so for this conversation it's just about simplifying or reorganizing the configuration bits. In the longer term if we decided that we wanted to stop using configuration files and switch to "something else", we wouldn't then need to figure out how to rip out OSA (if they don't support the "something else").
Thanks, -Alex
On Tue, May 28, 2019 at 12:17 PM Clark Boylan cboylan@sapwetik.org wrote:
On Mon, May 27, 2019, at 4:36 PM, Emilien Macchi wrote:
(First of all: thanks to Alex Schultz who initiated the discussion during the last PTG).
Context: With the containerization of the services in TripleO, our usage of Puppet is pretty much limited to laying down parameters into configuration files. Our long term goal is to reduce our number of tools in TripleO and converge toward more Ansible and we are currently investigating how to make it happen for Configuration Management, in a backward compatible and non-disruptive way.
Problems:
- Our current interface to configure things is very tight to Puppet and
Hiera, which would make it complicated to bind it with Hiera. Some of us have tried (to create some Hiera module in Ansible) but clearly this is not the path we want to take as far I know.
- We don't use the logic (generally) in the Puppet modules and again
only care about the configuration providers (in puppet-openstacklib and for some services, templates files), so the Puppet modules now do too much for what we need in TripleO.
snip
I'm not sure if this is useful but what the Infra team did was to transplant all of its hiera data into /etc/ansible/hosts/host_vars and /etc/ansible/hosts/group_vars. Then we updated our puppetry to pull hiera data out of there. This means that puppet and ansible read the same data sources which has made transitioning things easy for us.
Though I think the ansible-role-puppet role copies hiera data sources from /etc/ansible/hosts to the appropriate puppet hiera location on the remote source. But you mostly don't have to think about that and there is no double accounting from an ops perspective.
Clark
On Tue, May 28, 2019 at 12:24 PM Clark Boylan cboylan@sapwetik.org wrote: [...]
I'm not sure if this is useful but what the Infra team did was to transplant all of its hiera data into /etc/ansible/hosts/host_vars and /etc/ansible/hosts/group_vars. Then we updated our puppetry to pull hiera data out of there. This means that puppet and ansible read the same data sources which has made transitioning things easy for us.
Yes this is useful to know, thanks for sharing. In our case, we are not so worried about the source of data (it can be anywhere really); but our challenge is how we manage the data in our Heat templates and how the way it is exposed to our end-users will remain backward compatible.
Example: Right now, if you want to override parameter DEFAULT/foo in Glance API config file you would do:
ControllerExtraConfig: glance::config::glance_api_config: DEFAULT/foo: value: bar
With the new interface proposal it would be:
ControllerExtraConfig: glance_api_config: DEFAULT: foo: bar
And the second challenge here is to maintain the hierarchies that we have between layers of configuration (per service, per role, per nodes, etc), knowing that Heat doesn't have a hash deep_merge function (yet) AFIK (happy to be corrected here). I think we can try to keep hiera now (which provides a nice way to do parameters hierarchy) and keep the focus on stop using the Puppet OpenStack modules logic, just build a new interface which directly lays down the configuration (with Puppet now but something else later).
participants (6)
-
Alex Schultz
-
Bogdan Dobrelya
-
Clark Boylan
-
Dan Sneddon
-
Donny Davis
-
Emilien Macchi