Open Stack

On Wed, Sep 17, 2014, at 06:48 PM, Clark Boylan wrote:
> On Wed, Sep 17, 2014, at 06:37 PM, Matt Riedemann wrote:
> > 
> > 
> > On 9/17/2014 7:59 PM, Ian Wienand wrote:
> > > On 09/18/2014 09:49 AM, Clark Boylan wrote:
> > >> Recent sampling of test run times shows that our tempest jobs run
> > >> against clouds using PostgreSQL are significantly slower than jobs run
> > >> against clouds using MySQL.
> > >
> > > FYI There is a possibly relevant review out for max_connections limits
> > > [1], although it seems to have some issues with shmem usage
> > >
> > > -i
> > >
> > > [1] https://review.openstack.org/#/c/121952/
> > >
> > > _______________________________________________
> > > OpenStack-dev mailing list
> > > OpenStack-dev at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> > >
> > 
> > That's a backport of a fix from master where we were hitting fatal 
> > errors due to too many DB connections which was brought on by the 
> > changes to cinder and glance to run as many workers as there were CPUs 
> > available.  So I don't think it probably plays here...
> > 
> > The errors pointed out in another part of the thread have been around 
> > for awhile, I think they are due to negative tests where we're hitting 
> > unique constraints because of the negative tests, so they are expected.
> > 
> > We should also note that the postgresql jobs run with the nova metadata 
> > API service, I'm not sure how much of a factor that would have here.
> > 
> > Is there anything else unique about those jobs from the MySQL ones?
> >
> Good question. There are apparently other differences. The postgres job
> runs Keystone under eventlet instead of via apache mod_wsgi. It also
> sets FORCE_CONFIGDRIVE=False instead of always. And the final difference
> I can find is the one you point out, nova api metadata service is run as
> an independent thing.
> 
> Could these things be related? It would be relatively simple to push a
> change or two to devstack-gate to test this but there are enough options
> here that I probably won't do that until we think at least one of these
> options is at fault.
I am starting to feel bad that I picked on PostgreSQL and completely
forgot that there were other items in play here. I went ahead and
uploaded [0] to run all devstack jobs without keystone wsgi services
(eventlet) and [1] to run all devstack job with keystone wsgi services
and the initial results are pretty telling.

It appears that keystone eventlet is the source of the slowness in this
job. With keystone eventlet all of the devstack jobs are slower and with
keystone wsgi all of the jobs are quicker. Probably need to collect a
bit more data but this doesn't look good for keystone eventlet.

Thank you Matt for pointing me in this direction.

[0] https://review.openstack.org/#/c/122299/
[1] https://review.openstack.org/#/c/122300/
> > 
> > -- 
> > 
> > Thanks,
> > 
> > Matt Riedemann
> > 
> > 
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev