Open Stack

Hi Abel,

I'm petty confident that the delays are coming from the the openstack
server components. For example, Keystone. So when a client tries to
authenticate, Keystone is not marking a memcached server as dead and will
continue to try it and time out. At least that was my impression when I
tested removing the dead node from the config files.

I fully agree about everything going back to normal after a short time. If
the timeout delay didn't exist, I'd still try to implement some sort of
replication just because, as an end-user, being randomly logged out is
really annoying.

Thanks,
Joe


On Thu, Aug 14, 2014 at 10:23 AM, Abel Lopez <alopgeek at gmail.com> wrote:

> I played with this in the lab for a little bit, it didn’t seem to be a
> huge deal.
> My setup was three ‘controller’ nodes each running Multi-master Percona
> DB, rabbit-ha cluster, and memcached. Killing DB, no worries as it’s behind
> LB, killing rabbit? Usually not a problem especially after 3.2.x.
> Memcached? I noticed a brief period of invalid tokens, but within a few
> seconds, everything goes back to normal. I think the client software has a
> built-in retry when auth fails, or at least that was the observation. This
> was on Havana 1
>
> On Aug 14, 2014, at 9:09 AM, Joe Topjian <joe at topjian.net> wrote:
>
> > Hello,
> >
> > I have an OpenStack cloud with two HA cloud controllers. Each controller
> runs the standard controller components: glance, keystone, nova minus
> compute and network, cinder, horizon, mysql, rabbitmq, and memcached.
> >
> > Everything except memcached is accessed through haproxy and everything
> is working great (well, rabbit can be finicky ... I might post about that
> if it continues).
> >
> > The problem I currently have is how to effectively work with memcached
> in this environment. Since all components are load balanced, they need
> access to the same memcached servers. That's solved by the ability to
> specify multiple memcached servers in the various openstack config files.
> >
> > But if I take a server down for maintenance, I notice a 2-3 second delay
> in all requests. I've confirmed it's memcached by editing the list of
> memcached servers in the config files and the delay goes away.
> >
> > I'm wondering how people deploy memcached in environments like this? Are
> you using some type of memcached replication between servers? Or if a
> memcached server goes offline are you reconfiguring OpenStack to remove the
> offline memcached server?
> >
> > Thanks,
> > Joe
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140814/027fdbbe/attachment.html>