[requirements][oslo] Inclusion of CONFspirator in openstack/requirements
openstack at nemebean.com
Thu Aug 20 21:16:44 UTC 2020
On 8/16/20 11:42 PM, Adrian Turjak wrote:
> Hey OpenStackers!
> I'm hoping to add CONFspirator to openstack/requirements as I'm using it
> The library has been in Adjutant for a while but I didn't add it to
> openstack/requirements, so I'm trying to remedy that now. I think it is
> different enough from oslo.config and I think the features/differences
> are ones that are unlikely to ever make sense in oslo.config without
> breaking it for people who do use it as it is, or adding too much
> I wanted to use oslo.config but quickly found that the way I was
> currently doing config in Adjutant was heavily dependent on yaml, and
> the ability to nest things. I was in a bind because I didn't have a
> declarative config system like oslo.config, and the config for Adjutant
> was a mess to maintain and understand (even for me, and I wrote it) with
> random parts of the code pulling config that may or may not have been
> After finding oslo.config was not suitable for my rather weird needs, I
> took oslo.config as a starting point and ended up writing another
> library specific to my requirements in Adjutant, and rather than keeping
> it internal to Adjutant, moved it to an external library.
> CONFspirator was built for a weird and complex edge case, because I have
> plugins that need to dynamically load config on startup, which then has
> to be lazy_loaded. I also have weird overlay logic for defaults that can
> be overridden, and building it into the library made Adjutant simpler. I
> also have nested config groups that need to be named dynamically to
> allow plugin classes to be extended without subclasses sharing the same
> config group name. I built something specific to my needs, that just so
> happens to also be a potentially useful library for people wanting
> something like oslo.config but that is targeted towards yaml and toml,
> and the ability to nest groups.
> The docs are here: https://confspirator.readthedocs.io/
> The code is here: https://gitlab.com/catalyst-cloud/confspirator
> And for those interested in how I use it in Adjutant here are some
> places of interest (be warned, it may be a rabbit hole):
> If there are strong opinions about working to add this to oslo.config,
> let's chat, as I'm not against merging this into it somehow if we find a
> way that make sense, but while some aspects where similar, I felt that
> this was cleaner without being part of oslo.config because the mindset I
> was building towards seemed different and oslo.config didn't need my
Okay, I'll take a crack at discussing this from the Oslo side.
First, we've tried to avoid adding YAML support to oslo.config for a
couple of reasons:
1) consistency of configs across services. We didn't want to end up with
a mix of ini and yaml files.
2) As you discovered, the oslo.config model isn't conducive to nested
YAML layouts, so most of the benefits are lost anyway.
There are exceptions, of course. Just within oslo, oslo.policy uses YAML
configs, but it gives up most of the oslo.config niceties to do so.
Policy had to reimplement things like deprecation handling because it's
dealing with raw YAML instead of a config object. I believe there are
other examples where services had to refer to a YAML file for their
complex config opts.
With all that said, I'm pretty sure a motivated person could write a
YAML driver for oslo.config. It would introduce a layer of indirection -
the service would refer to a .conf file containing just the driver
config, which would then point to a separate .yaml file. I'm not sure
you could implement nesting this way, but I haven't dug into the code to
find out for sure.
In general, given the complexity of what you're talking about I think a
driver plugin would be the way to go, as opposed to trying to fit this
all in with the core oslo.config functionality (assuming you/we decide
to pursue integrating at all).
There were a few other things you mentioned as features of the library.
The following are some off-the-cuff thoughts without having looked too
closely at the library, so if they're nonsense that's my excuse. ;-)
"because I have plugins that need to dynamically load config on startup,
which then has to be lazy_loaded"
Something like this could probably be done. I believe this is kind of
how the castellan driver in oslo.config works. Config values are looked
up and cached on-demand, as opposed to all at once.
"I also have weird overlay logic for defaults that can be overridden"
My knee-jerk reaction to this is that oslo.config already supports
overriding defaults, so I assume there's something about your use case
that didn't mesh with that functionality? Or is this part of oslo.config
that you reused?
"I also have nested config groups that need to be named dynamically to
allow plugin classes to be extended without subclasses sharing the same
config group name."
Minus the nesting part, this is also something being done with
oslo.config today. The config driver feature actually uses dynamically
named groups, and I believe at least Cinder is using them too. They do
cause a bit of grief for some config tools, like the validator, but it
is possible to do something like this.
Now, as to the question of whether we should try to integrate this
library with oslo.config: I don't know. Helpful, right? ;-)
I think answering that question definitively would take a deeper dive
into what the new library is doing differently than I can commit to. As
I noted above, I don't think the things you're doing are so far out in
left field that it would be unreasonable to consider integrating into
oslo.config, but the devil is in the details and there are a lot of
details here that I don't know anything about. For example, will the
oslo.config data model even accommodate nested groups? I suspect it
doesn't now, but could it? Probably, but I can't say how
difficult/disruptive it would be.
If someone wanted to make incremental progress toward integration, I
think writing a basic YAML driver for oslo.config would be a good start.
It would be generally useful even if this work doesn't go anywhere, and
it would provide a basis for a hypothetical future YAML-based driver
providing CONFspirator functionality. From there we could start looking
to integrate more advanced features one at a time.
Apologies for the wall of text. I hope you got something out of this
before your eyes glazed over. :-)
More information about the openstack-discuss