On 21/08/20 9:16 am, Ben Nemec wrote:

In general, given the complexity of what you're talking about I think a driver plugin would be the way to go, as opposed to trying to fit this all in with the core oslo.config functionality (assuming you/we decide to pursue integrating at all).

There were a few other things you mentioned as features of the library. The following are some off-the-cuff thoughts without having looked too closely at the library, so if they're nonsense that's my excuse.

"because I have plugins that need to dynamically load config on startup, which then has to be lazy_loaded"

Something like this could probably be done. I believe this is kind of how the castellan driver in oslo.config works. Config values are looked up and cached on-demand, as opposed to all at once.

This one is a little weird, but essentially the way this works this in Adjutant:

I load the config so I can start the app and go through base logic and loading plugins, but the config groups that are pulled from plugins aren't added to my config group tree until AFTER the config has already been loaded. So part of the config is usable, but some hasn't yet fully been loaded until after the plugins are done, and then that subtree will lazy_load itself when first accessed. It means that until a given lazy_loaded group is actually accessed as config, the config tree underneath it can still have config options added. It's likely not too crazy to do this in oslo, and have groups only read from the cached source (loaded file dict) when first accessed.

"I also have weird overlay logic for defaults that can be overridden"

My knee-jerk reaction to this is that oslo.config already supports overriding defaults, so I assume there's something about your use case that didn't mesh with that functionality? Or is this part of oslo.config that you reused?

Sooo, this one is a little special because what this feature lets you do is take any group in the config tree once loaded, and call the overlay function on it with either a dict, or another group. The returned value will be a deep copy of the config tree with the values present in the given dict/group overlaid on that original group. As in a depth first dict update, where only keys that exist on the overriding dict will be updated in the copy of the original dict.

I need to write the docs for this... with a sane example, but here is my unit test for it:
https://gitlab.com/catalyst-cloud/confspirator/-/blob/master/confspirator/tests/test_configs.py#L211

I use this in Adjutant by having some config groups which are a default for something, and another place where many things can override that default via another group in the config, so I create an overlay copy and pass that to the place that needs the config it it's most specific case when the code actually pulls values from the conf group I passed it.
See: https://opendev.org/openstack/adjutant/src/branch/master/adjutant/actions/v1/base.py#L146-L167

I hope that helps, because it is in my mind a little odd to explain, but it allows some useful things when reusing actions in different tasks.

"I also have nested config groups that need to be named dynamically to allow plugin classes to be extended without subclasses sharing the same config group name."

Minus the nesting part, this is also something being done with oslo.config today. The config driver feature actually uses dynamically named groups, and I believe at least Cinder is using them too. They do cause a bit of grief for some config tools, like the validator, but it is possible to do something like this.

Cool! yeah I implemented that feature in my code because I ran into a case of confliciting namespaces and needed to find a better way to handle subclasses that needed to have different dynamic names for their config groups.

Now, as to the question of whether we should try to integrate this library with oslo.config: I don't know. Helpful, right?

That's mostly where we got to last time I asked in #openstack-oslo!

I think answering that question definitively would take a deeper dive into what the new library is doing differently than I can commit to. As I noted above, I don't think the things you're doing are so far out in left field that it would be unreasonable to consider integrating into oslo.config, but the devil is in the details and there are a lot of details here that I don't know anything about. For example, will the oslo.config data model even accommodate nested groups? I suspect it doesn't now, but could it? Probably, but I can't say how difficult/disruptive it would be.

I looked into it briefly, and to do what I wanted, while also maintaining oslo.config how it was... ended up a bit messy, so I gave up because it would take too long and the politics of trying to get it merged/reviewed wouldn't be worth the effort most likely.

If someone wanted to make incremental progress toward integration, I think writing a basic YAML driver for oslo.config would be a good start. It would be generally useful even if this work doesn't go anywhere, and it would provide a basis for a hypothetical future YAML-based driver providing CONFspirator functionality. From there we could start looking to integrate more advanced features one at a time.

Apologies for the wall of text. I hope you got something out of this before your eyes glazed over.

-Ben

Thanks for the wall of text! It was useful!

I think ultimately it may be safer just maintaining my own separate library. For people who don't want to use .ini and prefer yaml/toml it's simpler, and for people who prefer .ini and don't need nesting etc, it's safer to keep oslo.config as it is. If someone does find anything I've done that makes sense in oslo.config I'd be happy to work porting it over, but I don't want to make structural changes to it. I'll always keep an eye on oslo.config, and may occasionally steal the odd idea if you add something cool, but I think other than my use of types.py and some of the Opt classes as a base, my code has diverged quite a bit.