Open Stack

On Fri, Feb 10, 2017 at 12:27 PM, Anna Taraday
<akamyshnikova at mirantis.com> wrote:
> Hello everyone!
>
> In Juno in Neutron was implemented L3 HA feature based on Keepalived (VRRP).
> During next cycles it was improved, we performed scale testing [1] to find
> weak places and tried to fix them. The only alternative for L3 HA with VRRP
> is router rescheduling performed by Neutron server, but it is significantly
> slower and depends on control plane.
>
> What issues we experienced with L3 HA VRRP?
>
> Bugs in Keepalived (bad versions) [2]
> Split brain [3]
> Complex structure (ha networks, ha interfaces) - which actually cause races
> that we were fixing during Liberty, Mitaka and Newton.
>
> This all is not critical, but this is a bad experience and not everyone
> ready (or want) to use Keepalived approach.
>
> I think we can make things more flexible. For example, we can allow user to
> use external services like etcd instead of Keepalived to synchronize current
> HA state across agents. I've done several experiments and I've got failover
> time comparable to L3 HA with VRRP. Tooz [4] can be used to abstract from
> concrete backend. For example, it can allow us to use Zookeeper, Redis and
> other backends to store HA state.
>
> What I want to propose?
>
> I want to bring up idea that Neutron should have some general classes for L3
> HA which will allow to use not only Keepalived but also other backends for
> HA state. This at least will make it easier to try some other approaches and
> compare them with existing ones.
>
> Does this sound reasonable?

I understand that the intention is to add pluggability upstream so
that you could examine the viability of alternative solutions. I'd
advise instead to do the research locally, and if you find concrete
benefits to an alternative solution, come back, show your work and
have a discussion about it then. Merging extra complexity in the form
of a plug point without knowing if we're actually going to need it
seems risky.

On another note, after years of work the stability issues have largely
been resolved and L3 HA is in a good state with modern releases of
OpenStack. It's not a authoritative solution in the sense that it
doesn't cover every possible failure mode, but it covers the major
ones and in that sense better than not having any form of HA, and as
you pointed out the existing alternatives are not in a better state.
The subtext in your email is that now L3 HA is technically where we
want it, but some users are resisting adoption because of bad PR or a
bad past experience, but not for technical reasons. If that is the
case, then perhaps some good PR would be a more cost effective
investment than investigating, implementing, stabilizing and
maintaining a different backend that will likely take at least a cycle
to get merged and another 1 to 2 cycles to iron out kinks. Would you
have a critical mass of developers ready to support a pluggable L3 HA
now and in the long term?

Finally, I can share that L3 HA has been the default in RDO-land for a
few cycles now and is being used widely and successfully, in some
cases at significant scale.

>
> [1] -
> http://docs.openstack.org/developer/performance-docs/test_results/neutron_features/index.html
> [2] - https://bugs.launchpad.net/neutron/+bug/1497272
> https://bugs.launchpad.net/neutron/+bug/1433172
> [3] - https://bugs.launchpad.net/neutron/+bug/1375625
> [4] - http://docs.openstack.org/developer/tooz/
>
>
>
>
> --
> Regards,
> Ann Taraday
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>