On 8/16/20 11:42 PM, Adrian Turjak wrote:

Hey OpenStackers!

I'm hoping to add CONFspirator to openstack/requirements as I'm using it Adjutant: https://review.opendev.org/#/c/746436/

The library has been in Adjutant for a while but I didn't add it to openstack/requirements, so I'm trying to remedy that now. I think it is different enough from oslo.config and I think the features/differences are ones that are unlikely to ever make sense in oslo.config without breaking it for people who do use it as it is, or adding too much complexity.

I wanted to use oslo.config but quickly found that the way I was currently doing config in Adjutant was heavily dependent on yaml, and the ability to nest things. I was in a bind because I didn't have a declarative config system like oslo.config, and the config for Adjutant was a mess to maintain and understand (even for me, and I wrote it) with random parts of the code pulling config that may or may not have been set/declared.

After finding oslo.config was not suitable for my rather weird needs, I took oslo.config as a starting point and ended up writing another library specific to my requirements in Adjutant, and rather than keeping it internal to Adjutant, moved it to an external library.

CONFspirator was built for a weird and complex edge case, because I have plugins that need to dynamically load config on startup, which then has to be lazy_loaded. I also have weird overlay logic for defaults that can be overridden, and building it into the library made Adjutant simpler. I also have nested config groups that need to be named dynamically to allow plugin classes to be extended without subclasses sharing the same config group name. I built something specific to my needs, that just so happens to also be a potentially useful library for people wanting something like oslo.config but that is targeted towards yaml and toml, and the ability to nest groups.

The docs are here: https://confspirator.readthedocs.io/ The code is here: https://gitlab.com/catalyst-cloud/confspirator

And for those interested in how I use it in Adjutant here are some places of interest (be warned, it may be a rabbit hole): https://opendev.org/openstack/adjutant/src/branch/master/adjutant/config https://opendev.org/openstack/adjutant/src/branch/master/adjutant/feature_se...

https://opendev.org/openstack/adjutant/src/branch/master/adjutant/core.py https://opendev.org/openstack/adjutant/src/branch/master/adjutant/api/v1/ope...

https://opendev.org/openstack/adjutant/src/branch/master/adjutant/actions/v1...

https://opendev.org/openstack/adjutant/src/branch/master/adjutant/actions/v1...

https://opendev.org/openstack/adjutant/src/branch/master/adjutant/tasks/v1/b...

https://opendev.org/openstack/adjutant/src/branch/master/adjutant/tasks/v1/b...

If there are strong opinions about working to add this to oslo.config, let's chat, as I'm not against merging this into it somehow if we find a way that make sense, but while some aspects where similar, I felt that this was cleaner without being part of oslo.config because the mindset I was building towards seemed different and oslo.config didn't need my complexity.

Okay, I'll take a crack at discussing this from the Oslo side. First, we've tried to avoid adding YAML support to oslo.config for a couple of reasons: 1) consistency of configs across services. We didn't want to end up with a mix of ini and yaml files. 2) As you discovered, the oslo.config model isn't conducive to nested YAML layouts, so most of the benefits are lost anyway. There are exceptions, of course. Just within oslo, oslo.policy uses YAML configs, but it gives up most of the oslo.config niceties to do so. Policy had to reimplement things like deprecation handling because it's dealing with raw YAML instead of a config object. I believe there are other examples where services had to refer to a YAML file for their complex config opts. With all that said, I'm pretty sure a motivated person could write a YAML driver for oslo.config. It would introduce a layer of indirection - the service would refer to a .conf file containing just the driver config, which would then point to a separate .yaml file. I'm not sure you could implement nesting this way, but I haven't dug into the code to find out for sure. In general, given the complexity of what you're talking about I think a driver plugin would be the way to go, as opposed to trying to fit this all in with the core oslo.config functionality (assuming you/we decide to pursue integrating at all). There were a few other things you mentioned as features of the library. The following are some off-the-cuff thoughts without having looked too closely at the library, so if they're nonsense that's my excuse. ;-) "because I have plugins that need to dynamically load config on startup, which then has to be lazy_loaded" Something like this could probably be done. I believe this is kind of how the castellan driver in oslo.config works. Config values are looked up and cached on-demand, as opposed to all at once. "I also have weird overlay logic for defaults that can be overridden" My knee-jerk reaction to this is that oslo.config already supports overriding defaults, so I assume there's something about your use case that didn't mesh with that functionality? Or is this part of oslo.config that you reused? "I also have nested config groups that need to be named dynamically to allow plugin classes to be extended without subclasses sharing the same config group name." Minus the nesting part, this is also something being done with oslo.config today. The config driver feature actually uses dynamically named groups, and I believe at least Cinder is using them too. They do cause a bit of grief for some config tools, like the validator, but it is possible to do something like this. Now, as to the question of whether we should try to integrate this library with oslo.config: I don't know. Helpful, right? ;-) I think answering that question definitively would take a deeper dive into what the new library is doing differently than I can commit to. As I noted above, I don't think the things you're doing are so far out in left field that it would be unreasonable to consider integrating into oslo.config, but the devil is in the details and there are a lot of details here that I don't know anything about. For example, will the oslo.config data model even accommodate nested groups? I suspect it doesn't now, but could it? Probably, but I can't say how difficult/disruptive it would be. If someone wanted to make incremental progress toward integration, I think writing a basic YAML driver for oslo.config would be a good start. It would be generally useful even if this work doesn't go anywhere, and it would provide a basis for a hypothetical future YAML-based driver providing CONFspirator functionality. From there we could start looking to integrate more advanced features one at a time. Apologies for the wall of text. I hope you got something out of this before your eyes glazed over. :-) -Ben