Open Stack

It appears that the patch related to this discussion were rushed through
rather quickly, and without appropriate updates to the documentation. The
documentation of the library no longer matches the actual functionality,
and will need to be updated before we can release a new version of
oslo_middleware.

Mehdi, if you would please update the documentation in oslo_middleware as
quickly as possible? Some of us have features that we're trying to land in
liberty, which we cannot do without a new release of this library.

To the rest of the Oslo team, I'd like to propose that if we do not have
appropriate updated documentation by the end of this week, we revert
Medhi's patches so that we can unblock the library.

In case you're curious, the related patch is here:
https://review.openstack.org/#/c/209817/3

Michael Krotscheck

On Sun, Aug 9, 2015 at 7:58 PM Jamie Lennox <jamielennox at redhat.com> wrote:

>
>
> ----- Original Message -----
> > From: "Jamie Lennox" <jamielennox at redhat.com>
> > To: "OpenStack Development Mailing List (not for usage questions)" <
> openstack-dev at lists.openstack.org>
> > Sent: Monday, August 10, 2015 12:36:14 PM
> > Subject: Re: [openstack-dev] [oslo][keystone] oslo_config and wsgi
> middlewares
> >
> >
> >
> > ----- Original Message -----
> > > From: "Mehdi Abaakouk" <sileht at sileht.net>
> > > To: openstack-dev at lists.openstack.org
> > > Sent: Friday, August 7, 2015 1:57:54 AM
> > > Subject: [openstack-dev] [oslo][keystone] oslo_config and wsgi
> middlewares
> > >
> > > Hi,
> > >
> > > I want to share with you some problems I have recently encountered with
> > > openstack middlewares and oslo.config.
> > >
> > > The issues
> > > ----------
> > >
> > > In project Gnocchi, I would use oslo.middleware.cors, I have expected
> to
> > > just put the name of the middleware to the wsgi pipeline, but I can't.
> > > The middlewares only works if you pass the oslo_config.cfg.ConfigOpts()
> > > object or via 'paste-deploy'... Gnocchi doesn't use paste-deploy, so
> > > I have to modify the code to load it...
> > > (For the keystonemiddleware, Gnocchi already have a special
> > > handling/hack to load it [1] and [2]).
> > > I don't want to write the same hack for each openstack middlewares.
> > >
> > >
> > > In project Aodh (ceilometer-alarm), we recently got an issue with
> > > keystonemiddleware since we remove the usage of the global object
> > > oslo_config.cfg.CONF. The middleware doesn't load its options from the
> > > config file of aodh anymore. Our authentication is broken.
> > > We can still pass them through paste-deploy configuration but this
> looks
> > > a method of the past. I still don't want to write a hack for each
> > > openstack middlewares.
> > >
> > >
> > > Then I have digged into other middlewares and applications to see how
> > > they handle their conf.
> > >
> > > oslo_middlewarre.sizelimit and oslo_middlewarre.ssl take options only
> > > via the global oslo_config.cfg.CONF. So they are unusable for
> application
> > > that doesn't use this global object.
> > >
> > > oslo_middleware.healthcheck take options as dict like any other python
> > > middleware. This is suitable for 'paste-deploy'. But doesn't allow
> > > configuration via oslo.config, doesn't have a strong config options
> > > type checking and co.
> > >
> > > Zaqar seems got same kind of issue about keystonemiddleware, and just
> > > write a hack to workaround the issue (monkeypatch the cfg.CONF of
> > > keystonemiddleware with their local version of the object [3] and then
> > > transform the loaded options into a dict to pass them via the legacy
> > > middleware dict options [4]) .
> > >
> > > Most applications, just still use the global object for the
> > > configuration and don't, yet, see those issues.
> > >
> > >
> > > All of that is really not consistent.
> > >
> > > This is confusing for developer to have some middlewares that need
> > > pre-setup,
> > > enforce them to rely on global python object, and some others not.
> > > This is confusing for deployer their can't do the configuration of
> > > middlewares in the same way for each middlewares and each projects.
> > >
> > > But keystonemiddleware, oslo.middleware.cors,... are supposed to be
> wsgi
> > > middlewares, something that is independant of the app.
> > > And this is not really the case.
> > >
> > > From my point of view and what wsgi looks like generally in python, the
> > > middleware object should be just MyMiddleware(app, options_as_dict),
> > > if the middleware want to rely to another configuration system it
> should
> > > do the setup/initialisation itself.
> > >
> > >
> > >
> > > So, how to solve that ?
> > > ------------------------
> > >
> > > Do you agree:
> > >
> > > * all openstack middlewares should load their options with oslo.config
> ?
> > >   this permits type checking and all other features it provides, it's
> cool
> > >   :)
> > >   configuration in paste-deploy conf is thing of past
> > >
> > > * we must support local AND global oslo.config object ?
> > >   This is an application choice not something enforced by middleware.
> > >   The deployer experience should be the same in both case.
> > >
> > > * the middleware must be responsible of the section name in the
> oslo.config
> > > ?
> > >   Gnocchi/Zaqar hack have to hardcode the section name in their code,
> > >   this doesn't looks good.
> > >
> > > * we must support legacy python signature for WSGI object,
> > >   MyMiddleware(app, options_as_dict) ? To be able to use paste for
> > >   application/deployer that want it and not break already deployed
> things.
> > >
> > >
> > > I really think all our middlewares should be consistent:
> > >
> > > * to be usable by all applications without enforcing them to write crap
> > > around them.
> > > * and to made the deployer life easier.
> > >
> > >
> > > Possible solution:
> > > ------------------
> > >
> > > I have already started to work on something that do all of that for all
> > > middlewares [5], [6]
> > >
> > > The idea is, the middleware should create a
> oslo_config.cfg.ConfigOpts()
> > > (instead of rely on the global one) and load the configuration file of
> the
> > > application in. oslo.config will discover the file location just with
> the
> > > name of application as usual.
> > >
> > > So the middleware can now be loaded like this:
> > >
> > > code example:
> > >
> > >    app = MyMiddleware(app, {"oslo_config_project": "aodh"})
> > >
> > > paste-deploy example:
> > >
> > >    [filter:foobar]
> > >    paste.filter_factory = foobar:MyMiddleware.filter_factory
> > >    oslo_config_project = aodh
> > >
> > > oslo_config.cfg.ConfigOpts() will easly find the /etc/aodh/aodh.conf,
> > > This cut the hidden links between middleware and the application
> > > (through the global object).
> > >
> > > And of course if oslo_config_project is not provided, the middleware
> > > fallback the global oslo.config object.
> > > The middleware options can still be passed via the legacy dict.
> > > The backward compatibility is conserved.
> > >
> > > I have already tested that in Gnocchi and Aodh, and that solves all of
> > > my issues. Remove all hacks, the application doesn't need special pre
> > > setup. All our middleware become normal middleware but still can use
> > > oslo.config.
> > >
> > >
> > > WDYT ?
> >
> > So i agree on most of your points, the middlewares are certainly
> > inconsistent, i would like to make things easier for deployers and I
> don't
> > like global CONF objects (but i've now had that argument so many times
> i'm
> > done).
> >
> > Also to support some of the newer services that don't use paste i think
> we
> > should absolutely make it so that the CONF object is passed to middleware
> > rather than sourced globally. I think gnochhi and zaqar both fit into
> this
> > case.
> >
> > The problem i see with what you are saying is that it is mixing
> deployment
> > methodologies in a way that is unintended. Paste is designed to allow
> > deployers to add and remove middleware independent of configuring the
> > service. This means that at paste time there is no CONF object unless
> it's
> > globally registered and this is why most middlewares allow taking options
> > from paste.ini because if you don't have global CONF then it's the only
> way
> > to actually get options into the middleware.
> >
> > The proposed solution is then to provide enough options (via paste) to
> load a
> > CONF object local to the middleware that will be an exact copy of what
> the
> > service will have already loaded. In your test so far this has only
> required
> > project_name which might be enough because i don't know how often
> services
> > specify their own default_config_files[1] and it looks like CONF will use
> > sys.argv[1:] by default [2] for parsing CLI arguments. This conf loading
> > code would then have to be replicated into all middleware that wants to
> use
> > oslo.config for its options.
> >
> > Fixing this problem is always where i loose enthusiasm for removing
> global
> > CONF files.
> >
> > From a developer perspective I feel the solution is for us to reconsider
> how
> > we deploy and configure middleware. If you are using paste those pieces
> of
> > middleware should each be able to be configured without any integration
> with
> > the service underneath it. Otherwise if your service needs a piece of
> > middleware like auth_token middleware to operate or relies on oslo.config
> > options like cors then that is not something that should be controlled by
> > paste.
> >
> > From a deployer perspective there is no great answer i just want
> everything
> > to be in oslo.config.
> >
> > Equally from a deployer perspective this wasn't an issue until aodh (et
> al)
> > decided to remove the global CONF object which developers from all the
> > projects hate but live with. I don't mean to imply we shouldn't look at
> > better ways to do things, but if you're main concern is deployers the
> > easiest thing we can do for consistency is add back the global CONF
> object
> > or remove paste from aodh. :)
> >
> > Jamie
>
> On re-reading this i realize the :) doesn't prevent this coming acrross a
> bit snarky or aggressive. That's not my intent and i apologize.
>
> I would love to figure out ways we can make all this more consistent, but
> i'm not a fan of having to load and handle CONF objects in multiple places,
> especially once per middleware object.
>
>
> >
> >
> > [1]
> >
> https://git.openstack.org/cgit/openstack/oslo.config/tree/oslo_config/cfg.py#n1832
> > [2]
> >
> https://git.openstack.org/cgit/openstack/oslo.config/tree/oslo_config/cfg.py#n1880
> >
> > >
> > > Cheers,
> > >
> > > [1]
> > >
> https://github.com/openstack/gnocchi/blob/master/gnocchi/rest/app.py#L140
> > > [2]
> > >
> https://github.com/openstack/gnocchi/blob/master/gnocchi/service.py#L64-L73
> > > [3]
> > >
> https://github.com/openstack/zaqar/blob/87fd1aa93dafb64097f731dbd416c2eeb697d403/zaqar/transport/auth.py#L63
> > > [4]
> > >
> https://github.com/openstack/zaqar/blob/87fd1aa93dafb64097f731dbd416c2eeb697d403/zaqar/transport/auth.py#L70
> > > [5] https://review.openstack.org/#/c/208965/
> > > [6] https://review.openstack.org/#/c/209817/
> > >
> > >
> > > --
> > > Mehdi Abaakouk
> > > mail: sileht at sileht.net
> > > irc: sileht
> > >
> > >
> __________________________________________________________________________
> > > OpenStack Development Mailing List (not for usage questions)
> > > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150810/cd1b50f1/attachment.html>