Open Stack

On Thu, Apr 3, 2014 at 5:42 PM, Zane Bitter <zbitter at redhat.com> wrote:
> On 03/04/14 08:48, Doug Hellmann wrote:
>>
>> On Wed, Apr 2, 2014 at 9:55 PM, Zane Bitter <zbitter at redhat.com> wrote:
>>>
>>> We have an issue in Heat where the sample config generator from Oslo is
>>> currently broken (see bug #1288586). Unfortunately it turns out that
>>> there
>>> is no fix to the generator script itself that can do the Right Thing for
>>> both Heat and Nova.
>>>
>>> A brief recap on how the sample config generator works: it goes through
>>> all
>>> of the files specified and finds all the ConfigOpt objects at the top
>>> level.
>>> It then searches for them in the registered options, and returns the name
>>> of
>>> the group in which they are registered. Previously it looked for the
>>> identical object being registered, but now it just looks for any
>>> equivalent
>>> ones. When you register two or more equivalent options, the second and
>>> subsequent ones are just ignored by oslo.config.
>>>
>>> The situation in Heat is that we have a bunch of equivalent options
>>> registered in multiple groups. This is because we have a set of options
>>> for
>>> each client library (i.e. python-novaclient, python-cinderclient, &c.),
>>> with
>>> each set containing equivalent options (e.g. every client has an
>>> "endpoint_type" option for looking up the keystone catalog). This used to
>>> work, but now that equivalent options (and not just identical options)
>>> match
>>> when searching for them in a group, we just end up with multiple copies
>>> of
>>> each option in the first group to be searched, and none in any of the
>>> other
>>> groups, in the generated sample config.
>>>
>>> Nova, on the other hand, has the opposite problem (see bug #1262148).
>>> Nova
>>> adds the auth middleware from python-keystoneclient to its list of files
>>> to
>>> search for options. That middleware imports a file from oslo-incubator
>>> that
>>> registers the option in the default group - a registration that is *not*
>>> wanted by the keystone middleware, because it registers an equivalent
>>> option
>>> in a different group instead (or, as it turns out, as well). Just to make
>>> it
>>> interesting, Nova uses the same oslo-incubator module and relies on the
>>> option being registered in the default group. Of course, oslo-incubator
>>> is
>>> not a real library, so it gets registered a second time but ignored
>>> (since
>>> an equivalent one is already present). Crucially, the oslo-incubator file
>>> from python-keystoneclient is not on the list of extra modules to search
>>> in
>>> Nova, so when the generator script was looking for options identical to
>>> the
>>> ones it found in modules, it didn't see this option at all. Hence the
>>> change
>>> to looking for equivalent options, which broke Heat.
>>>
>>> Neither comparing for equivalence nor for identity in the generator
>>> script
>>> can solve both use cases. It's hard to see what Heat could or should be
>>> doing differently. I think it follows that the fix needs to be in either
>>> Nova or python-keystoneclient in the first instance.
>>>
>>> One option I suggested was for the auth middleware to immediately
>>> deregister
>>> the extra option that had accidentally been registered upon importing a
>>> module from oslo-incubator. I put up patches to do this, but it seemed to
>>> be
>>> generally agreed by Oslo folks that this was a Bad Idea.
>>>
>>> Another option would be to specifically include the relevant module from
>>> keystoneclient.openstack.common when generating the sample config. This
>>> seems quite brittle to me.
>>>
>>> We could fix it by splitting the oslo-incubator module into one that
>>> provides the code needed by the auth middleware and one that does the
>>> registration of options, but this will likely result in cascading changes
>>> to
>>> a whole bunch of projects.
>>>
>>> Does anybody have any thoughts on what the right fix looks like here?
>>> Currently, verification of the sample config is disabled in the Heat gate
>>> because of this issue, so it would be good to get it resolved.
>>>
>>> cheers,
>>> Zane.
>>
>>
>> We've seen some similar issues in other projects where the "guessing"
>> done by the generator is not matching the newer ways we use
>> configuration options. In those cases, I suggested that projects use
>> the new entry-point feature that allows them to explicitly list
>> options within groups, instead of scanning a set of files. This
>> feature was originally added so apps can include the options from
>> libraries that use oslo.config (such as oslo.messaging), but it can be
>> used for options define by the applications as well.
>>
>> To define an option discovery entry point, create a function that
>> returns a sequence of (group name, option list) pairs. For an example,
>> see list_opts() in oslo.messaging [1]. Then define the entry point in
>> your setup.cfg under the "oslo.config.opts" namespace [2]. If you need
>> more than one function, register them separately.
>>
>> Then change the way generate_sample.sh is called for your project so
>> it passes the -l option [3] once for each name you have given to the
>> entry points. So if you have just "heat" you would pass "-l heat" and
>> if you have "heat-core" and "heat-some-driver" you would pass "-l
>> heat-core -l heat-some-driver".
>>
>> For application options, you shouldn't mix the -l option with the file
>> scanner, since you will end up with duplicate options.
>>
>> Doug
>>
>> [1]
>> http://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo/messaging/opts.py#n56
>> [2]
>> http://git.openstack.org/cgit/openstack/oslo.messaging/tree/setup.cfg#n53
>> [3]
>> http://git.openstack.org/cgit/openstack/oslo-incubator/tree/tools/config/generate_sample.sh#n25
>
>
> I gave this a go last week when the issue first came up, but couldn't get it
> working. The patch[1] was a mess because it required an update from
> oslo-incubator first, and the mechanism doesn't seem designed for using
> exclusively - I never figured out how to turn off the file scanner and rely
> purely on the 'library' load points.

The latest version of the config generator allows you to specify only
libraries and not modules to scan. You will probably need to adjust
the way it is called in tools/config/check_uptodate.sh.

> At that point I realised that things
> weren't actually broken in the gate yet, and I could just get things moving
> again for Icehouse by committing the diff that the gate check was asking
> for, so I did.
>
> Now that we're in Juno the gate is definitely broken, however. And this
> doesn't sound like a long term fix unless everything is moving over to it

Both mechanisms are useful. Several projects are using options in ways
that were not anticipated by the original generator, including listing
the same option in more than one "section". Simpler projects may
continue to use the code scanner, but for cases like yours it seems
like being explicit is going to work out better.

> (which I would support) - Heat is actually doing the Right Thing already;
> Nova is unintentionally importing an unwanted option and then ignoring it.
> Don't we need a solution for that?

We should stop registering options when modules are imported. I don't
know if that would fix the issue you mention, though.

Doug

>
> cheers,
> Zane.
>
> [1] http://paste.fedoraproject.org/91147/13964840
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev