Open Stack

My production environment stays at about 18k non-expired tokens (expiration
set to 24h) mostly from internal openstack stuff I assume and it hums along
just fine. The 300k is a problem though.

------------------
Aubrey Wells
Director | Network Services
VocalCloud
888.305.3850
support at vocalcloud.com
www.vocalcloud.com


On Mon, Aug 19, 2013 at 2:04 PM, Nick Maslov <azpekt at gmail.com> wrote:

> Hi Aubreu,
>
> So, having this in a development environment with max ~2 concurrent users
> and 2 VM`s is an issue?
>
> keystone=# select count(*) from token;
>  count
> -------
>  17360
> (1 row)
>
>
> I`m having about 300k of rows in another environment as well.
>
> Thanks for info.
> Nick
>
>
> On 08/15, Aubrey Wells wrote:
> > We have the same thing and found that the keystone tokens table had
> > hundreds of thousands of expired tokens in it so the SELECT that gets
> done
> > during the auth phase of API operations was taking ages to return. Wrote
> a
> > script to clean up expired tokens and it hasn't recurred. A quick and
> dirty
> > version to clean it up by hand would be 'delete from token where expires
> <
> > NOW();' but you might want something a little safer in an automated
> script.
> >
> > ------------------
> > Aubrey Wells
> > Director | Network Services
> > VocalCloud
> > 888.305.3850
> > support at vocalcloud.com
> > www.vocalcloud.com
> >
> >
> > On Thu, Aug 15, 2013 at 10:45 AM, Jonathan Proulx <jon at jonproulx.com>
> wrote:
> >
> > > Hi All,
> > >
> > > I have a single controller node 60 compute node cloud on Ubuntu 12.04 /
> > > cloud archive and after upgrade to grizzly  everything seem painfully
> slow.
> > >
> > > I've had 'nova list' take on the order of one minute to return
> (there's 65
> > > non-deleted instances and a total of just under 500k instances in the
> > > instances table but that was true before upgrade as well)
> > >
> > > The controller node is 4x busier with this tiny load of a single user
> and
> > > a few VMs as it has averaged in production with 1,500 VMs dozens of
> users
> > > and VMs starting every 6sec on average.
> > >
> > > This has me a little worried but the system is so over spec'ed that I
> > > can't see it as my current problem as the previous average was 5% CPU
> > > utilization so now I'm only at 20%.  All the databases fit comfortably
> in
> > > memory with plenty of room for caching so my disk I/0 is virtually
> nothing.
> > >
> > > Not quite sure where to start.  I'd like to blame conductor for
> > > serializing database access, but I really hope any service could
> handle at
> > > least one rack of servers before you needed to scale out...but besides
> the
> > > poor user experience of sluggish response I'm also getting timeouts if
> I
> > > try and start some number of 10's of servers, the usual work flow
> around
> > > here often involves 100's.
> > >
> > > Anyone had similar problems and/or have suggestions of where else to
> look
> > > for bottle necks.
> > >
> > > -Jon
> > >
> > > _______________________________________________
> > > OpenStack-operators mailing list
> > > OpenStack-operators at lists.openstack.org
> > >
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> > >
> > >
>
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20130819/6dd2ef14/attachment.html>