<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-text-flowed" style="font-family: -moz-fixed;
font-size: 12px;" lang="x-unicode">On 21/08/20 9:16 am, Ben Nemec
wrote:
<br>
<blockquote type="cite" style="color: #007cff;">
<br>
In general, given the complexity of what you're talking about I
think a driver plugin would be the way to go, as opposed to
trying to fit this all in with the core oslo.config
functionality (assuming you/we decide to pursue integrating at
all).
<br>
<br>
There were a few other things you mentioned as features of the
library. The following are some off-the-cuff thoughts without
having looked too closely at the library, so if they're nonsense
that's my excuse. <span class="moz-smiley-s3" title=";-)"></span>
<br>
<br>
"because I have plugins that need to dynamically load config on
startup, which then has to be lazy_loaded"
<br>
<br>
Something like this could probably be done. I believe this is
kind of how the castellan driver in oslo.config works. Config
values are looked up and cached on-demand, as opposed to all at
once.
<br>
</blockquote>
<br>
This one is a little weird, but essentially the way this works
this in Adjutant:
<br>
<br>
I load the config so I can start the app and go through base logic
and loading plugins, but the config groups that are pulled from
plugins aren't added to my config group tree until AFTER the
config has already been loaded. So part of the config is usable,
but some hasn't yet fully been loaded until after the plugins are
done, and then that subtree will lazy_load itself when first
accessed. It means that until a given lazy_loaded group is
actually accessed as config, the config tree underneath it can
still have config options added. It's likely not too crazy to do
this in oslo, and have groups only read from the cached source
(loaded file dict) when first accessed.
<br>
<br>
<blockquote type="cite" style="color: #007cff;">
<br>
"I also have weird overlay logic for defaults that can be
overridden"
<br>
<br>
My knee-jerk reaction to this is that oslo.config already
supports overriding defaults, so I assume there's something
about your use case that didn't mesh with that functionality? Or
is this part of oslo.config that you reused?
<br>
</blockquote>
<br>
Sooo, this one is a little special because what this feature lets
you do is take any group in the config tree once loaded, and call
the overlay function on it with either a dict, or another group.
The returned value will be a deep copy of the config tree with the
values present in the given dict/group overlaid on that original
group. As in a depth first dict update, where only keys that exist
on the overriding dict will be updated in the copy of the original
dict.
<br>
<br>
I need to write the docs for this... with a sane example, but here
is my unit test for it:
<br>
<a class="moz-txt-link-freetext"
href="https://gitlab.com/catalyst-cloud/confspirator/-/blob/master/confspirator/tests/test_configs.py#L211">https://gitlab.com/catalyst-cloud/confspirator/-/blob/master/confspirator/tests/test_configs.py#L211</a>
<br>
<br>
I use this in Adjutant by having some config groups which are a
default for something, and another place where many things can
override that default via another group in the config, so I create
an overlay copy and pass that to the place that needs the config
it it's most specific case when the code actually pulls values
from the conf group I passed it.
<br>
See: <a class="moz-txt-link-freetext"
href="https://opendev.org/openstack/adjutant/src/branch/master/adjutant/actions/v1/base.py#L146-L167">https://opendev.org/openstack/adjutant/src/branch/master/adjutant/actions/v1/base.py#L146-L167</a>
<br>
<br>
I hope that helps, because it is in my mind a little odd to
explain, but it allows some useful things when reusing actions in
different tasks.
<br>
<br>
<blockquote type="cite" style="color: #007cff;">"I also have
nested config groups that need to be named dynamically to allow
plugin classes to be extended without subclasses sharing the
same config group name."
<br>
<br>
Minus the nesting part, this is also something being done with
oslo.config today. The config driver feature actually uses
dynamically named groups, and I believe at least Cinder is using
them too. They do cause a bit of grief for some config tools,
like the validator, but it is possible to do something like
this.
<br>
</blockquote>
Cool! yeah I implemented that feature in my code because I ran
into a case of confliciting namespaces and needed to find a better
way to handle subclasses that needed to have different dynamic
names for their config groups.
<br>
<blockquote type="cite" style="color: #007cff;">
<br>
Now, as to the question of whether we should try to integrate
this library with oslo.config: I don't know. Helpful, right? <span
class="moz-smiley-s3" title=";-)"></span>
<br>
</blockquote>
That's mostly where we got to last time I asked in
#openstack-oslo! <span class="moz-smiley-s3" title=";)"></span>
<br>
<blockquote type="cite" style="color: #007cff;">
<br>
I think answering that question definitively would take a deeper
dive into what the new library is doing differently than I can
commit to. As I noted above, I don't think the things you're
doing are so far out in left field that it would be unreasonable
to consider integrating into oslo.config, but the devil is in
the details and there are a lot of details here that I don't
know anything about. For example, will the oslo.config data
model even accommodate nested groups? I suspect it doesn't now,
but could it? Probably, but I can't say how difficult/disruptive
it would be.
<br>
</blockquote>
I looked into it briefly, and to do what I wanted, while also
maintaining oslo.config how it was... ended up a bit messy, so I
gave up because it would take too long and the politics of trying
to get it merged/reviewed wouldn't be worth the effort most
likely. <span class="moz-smiley-s2" title=":("></span>
<br>
<blockquote type="cite" style="color: #007cff;">
<br>
If someone wanted to make incremental progress toward
integration, I think writing a basic YAML driver for oslo.config
would be a good start. It would be generally useful even if this
work doesn't go anywhere, and it would provide a basis for a
hypothetical future YAML-based driver providing CONFspirator
functionality. From there we could start looking to integrate
more advanced features one at a time.
<br>
<br>
Apologies for the wall of text. I hope you got something out of
this before your eyes glazed over. <span class="moz-smiley-s1"
title=":-)"></span>
<br>
<br>
-Ben
<br>
<br>
</blockquote>
Thanks for the wall of text! It was useful!
<br>
<br>
I think ultimately it may be safer just maintaining my own
separate library. For people who don't want to use .ini and prefer
yaml/toml it's simpler, and for people who prefer .ini and don't
need nesting etc, it's safer to keep oslo.config as it is. If
someone does find anything I've done that makes sense in
oslo.config I'd be happy to work porting it over, but I don't want
to make structural changes to it. I'll always keep an eye on
oslo.config, and may occasionally steal the odd idea if you add
something cool, but I think other than my use of types.py and some
of the Opt classes as a base, my code has diverged quite a bit.
<br>
</div>
</body>
</html>