Open Stack

On 21/08/20 9:16 am, Ben Nemec wrote:
>
> In general, given the complexity of what you're talking about I think 
> a driver plugin would be the way to go, as opposed to trying to fit 
> this all in with the core oslo.config functionality (assuming you/we 
> decide to pursue integrating at all).
>
> There were a few other things you mentioned as features of the 
> library. The following are some off-the-cuff thoughts without having 
> looked too closely at the library, so if they're nonsense that's my 
> excuse.
>
> "because I have plugins that need to dynamically load config on 
> startup, which then has to be lazy_loaded"
>
> Something like this could probably be done. I believe this is kind of 
> how the castellan driver in oslo.config works. Config values are 
> looked up and cached on-demand, as opposed to all at once.

This one is a little weird, but essentially the way this works this in 
Adjutant:

I load the config so I can start the app and go through base logic and 
loading plugins, but the config groups that are pulled from plugins 
aren't added to my config group tree until AFTER the config has already 
been loaded. So part of the config is usable, but some hasn't yet fully 
been loaded until after the plugins are done, and then that subtree will 
lazy_load itself when first accessed. It means that until a given 
lazy_loaded group is actually accessed as config, the config tree 
underneath it can still have config options added. It's likely not too 
crazy to do this in oslo, and have groups only read from the cached 
source (loaded file dict) when first accessed.

>
> "I also have weird overlay logic for defaults that can be overridden"
>
> My knee-jerk reaction to this is that oslo.config already supports 
> overriding defaults, so I assume there's something about your use case 
> that didn't mesh with that functionality? Or is this part of 
> oslo.config that you reused?

Sooo, this one is a little special because what this feature lets you do 
is take any group in the config tree once loaded, and call the overlay 
function on it with either a dict, or another group. The returned value 
will be a deep copy of the config tree with the values present in the 
given dict/group overlaid on that original group. As in a depth first 
dict update, where only keys that exist on the overriding dict will be 
updated in the copy of the original dict.

I need to write the docs for this... with a sane example, but here is my 
unit test for it:
https://gitlab.com/catalyst-cloud/confspirator/-/blob/master/confspirator/tests/test_configs.py#L211 


I use this in Adjutant by having some config groups which are a default 
for something, and another place where many things can override that 
default via another group in the config, so I create an overlay copy and 
pass that to the place that needs the config it it's most specific case 
when the code actually pulls values from the conf group I passed it.
See: 
https://opendev.org/openstack/adjutant/src/branch/master/adjutant/actions/v1/base.py#L146-L167 


I hope that helps, because it is in my mind a little odd to explain, but 
it allows some useful things when reusing actions in different tasks.

> "I also have nested config groups that need to be named dynamically to 
> allow plugin classes to be extended without subclasses sharing the 
> same config group name."
>
> Minus the nesting part, this is also something being done with 
> oslo.config today. The config driver feature actually uses dynamically 
> named groups, and I believe at least Cinder is using them too. They do 
> cause a bit of grief for some config tools, like the validator, but it 
> is possible to do something like this.
Cool! yeah I implemented that feature in my code because I ran into a 
case of confliciting namespaces and needed to find a better way to 
handle subclasses that needed to have different dynamic names for their 
config groups.
>
> Now, as to the question of whether we should try to integrate this 
> library with oslo.config: I don't know. Helpful, right?
That's mostly where we got to last time I asked in #openstack-oslo!
>
> I think answering that question definitively would take a deeper dive 
> into what the new library is doing differently than I can commit to. 
> As I noted above, I don't think the things you're doing are so far out 
> in left field that it would be unreasonable to consider integrating 
> into oslo.config, but the devil is in the details and there are a lot 
> of details here that I don't know anything about. For example, will 
> the oslo.config data model even accommodate nested groups? I suspect 
> it doesn't now, but could it? Probably, but I can't say how 
> difficult/disruptive it would be.
I looked into it briefly, and to do what I wanted, while also 
maintaining oslo.config how it was... ended up a bit messy, so I gave up 
because it would take too long and the politics of trying to get it 
merged/reviewed wouldn't be worth the effort most likely.
>
> If someone wanted to make incremental progress toward integration, I 
> think writing a basic YAML driver for oslo.config would be a good 
> start. It would be generally useful even if this work doesn't go 
> anywhere, and it would provide a basis for a hypothetical future 
> YAML-based driver providing CONFspirator functionality. From there we 
> could start looking to integrate more advanced features one at a time.
>
> Apologies for the wall of text. I hope you got something out of this 
> before your eyes glazed over.
>
> -Ben
>
Thanks for the wall of text! It was useful!

I think ultimately it may be safer just maintaining my own separate 
library. For people who don't want to use .ini and prefer yaml/toml it's 
simpler, and for people who prefer .ini and don't need nesting etc, it's 
safer to keep oslo.config as it is. If someone does find anything I've 
done that makes sense in oslo.config I'd be happy to work porting it 
over, but I don't want to make structural changes to it. I'll always 
keep an eye on oslo.config, and may occasionally steal the odd idea if 
you add something cool, but I think other than my use of types.py and 
some of the Opt classes as a base, my code has diverged quite a bit.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20200821/f61de1e7/attachment.html>