Open Stack

Clark Boylan <cboylan at sapwetik.org> Wrotr :

> On Wed, Sep 17, 2014, at 06:48 PM, Clark Boylan wrote:
> > On Wed, Sep 17, 2014, at 06:37 PM, Matt Riedemann wrote:
> > >
> > >
> > > On 9/17/2014 7:59 PM, Ian Wienand wrote:
> > > > On 09/18/2014 09:49 AM, Clark Boylan wrote:
> > > >> Recent sampling of test run times shows that our tempest jobs run
> > > >> against clouds using PostgreSQL are significantly slower than jobs
> run
> > > >> against clouds using MySQL.
> > > >
> > > > FYI There is a possibly relevant review out for max_connections
> limits
> > > > [1], although it seems to have some issues with shmem usage
> > > >
> > > > -i
> > > >
> > > > [1] https://review.openstack.org/#/c/121952/
> > > >
> > > > _______________________________________________
> > > > OpenStack-dev mailing list
> > > > OpenStack-dev at lists.openstack.org <javascript:;>
> > > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > > >
> > >
> > > That's a backport of a fix from master where we were hitting fatal
> > > errors due to too many DB connections which was brought on by the
> > > changes to cinder and glance to run as many workers as there were CPUs
> > > available.  So I don't think it probably plays here...
> > >
> > > The errors pointed out in another part of the thread have been around
> > > for awhile, I think they are due to negative tests where we're hitting
> > > unique constraints because of the negative tests, so they are expected.
> > >
> > > We should also note that the postgresql jobs run with the nova metadata
> > > API service, I'm not sure how much of a factor that would have here.
> > >
> > > Is there anything else unique about those jobs from the MySQL ones?
> > >
> > Good question. There are apparently other differences. The postgres job
> > runs Keystone under eventlet instead of via apache mod_wsgi. It also
> > sets FORCE_CONFIGDRIVE=False instead of always. And the final difference
> > I can find is the one you point out, nova api metadata service is run as
> > an independent thing.
> >
> > Could these things be related? It would be relatively simple to push a
> > change or two to devstack-gate to test this but there are enough options
> > here that I probably won't do that until we think at least one of these
> > options is at fault.
> I am starting to feel bad that I picked on PostgreSQL and completely
> forgot that there were other items in play here. I went ahead and
> uploaded [0] to run all devstack jobs without keystone wsgi services
> (eventlet) and [1] to run all devstack job with keystone wsgi services
> and the initial results are pretty telling.
>
> It appears that keystone eventlet is the source of the slowness in this
> job. With keystone eventlet all of the devstack jobs are slower and with
> keystone wsgi all of the jobs are quicker. Probably need to collect a
> bit more data but this doesn't look good for keystone eventlet.
>
> Thank you Matt for pointing me in this direction.
>
> [0] https://review.openstack.org/#/c/122299/
> [1] https://review.openstack.org/#/c/122300/
> > >
> > > --
> > >
> > > Thanks,
> > >
> > > Matt Riedemann
> > >
>

I've kicked off a test[1] as well to check into some tunable options
(eventlet workers) for improving keystone eventlet performance. I'll circle
back with the infra team once we have a little more data on both fronts.
The Keystone team will use this data to figure out the best way to approach
this issue.

--Morgan

[1] https://review.openstack.org/#/c/122308/
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140917/9ab50cd8/attachment.html>