Open Stack

Hi Aubreu,

So, having this in a development environment with max ~2 concurrent users and 2 VM`s is an issue?

keystone=# select count(*) from token;
 count
-------
 17360
(1 row)


I`m having about 300k of rows in another environment as well.

Thanks for info.
Nick


On 08/15, Aubrey Wells wrote:
> We have the same thing and found that the keystone tokens table had
> hundreds of thousands of expired tokens in it so the SELECT that gets done
> during the auth phase of API operations was taking ages to return. Wrote a
> script to clean up expired tokens and it hasn't recurred. A quick and dirty
> version to clean it up by hand would be 'delete from token where expires <
> NOW();' but you might want something a little safer in an automated script.
> 
> ------------------
> Aubrey Wells
> Director | Network Services
> VocalCloud
> 888.305.3850
> support at vocalcloud.com
> www.vocalcloud.com
> 
> 
> On Thu, Aug 15, 2013 at 10:45 AM, Jonathan Proulx <jon at jonproulx.com> wrote:
> 
> > Hi All,
> >
> > I have a single controller node 60 compute node cloud on Ubuntu 12.04 /
> > cloud archive and after upgrade to grizzly  everything seem painfully slow.
> >
> > I've had 'nova list' take on the order of one minute to return (there's 65
> > non-deleted instances and a total of just under 500k instances in the
> > instances table but that was true before upgrade as well)
> >
> > The controller node is 4x busier with this tiny load of a single user and
> > a few VMs as it has averaged in production with 1,500 VMs dozens of users
> > and VMs starting every 6sec on average.
> >
> > This has me a little worried but the system is so over spec'ed that I
> > can't see it as my current problem as the previous average was 5% CPU
> > utilization so now I'm only at 20%.  All the databases fit comfortably in
> > memory with plenty of room for caching so my disk I/0 is virtually nothing.
> >
> > Not quite sure where to start.  I'd like to blame conductor for
> > serializing database access, but I really hope any service could handle at
> > least one rack of servers before you needed to scale out...but besides the
> > poor user experience of sluggish response I'm also getting timeouts if I
> > try and start some number of 10's of servers, the usual work flow around
> > here often involves 100's.
> >
> > Anyone had similar problems and/or have suggestions of where else to look
> > for bottle necks.
> >
> > -Jon
> >
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
> >

> _______________________________________________
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators