Open Stack

If I propose some concrete solution that will be discussion about one
solution not about making things flexible.
At first I wanted to propose some PoC for other approach, but during my
experiments I understood that we may have different approaches, but for all
of them we need pluggable HA router in Neutron.

The thing that bothers me about L3 HA - it is complex. Yes, we fixed bunch
of races and John did significant refactor, but it is still too complex. In
the end we want to use L3 HA + DVR but DVR is pretty complex by itself. We
would like to try to offload this complexity to external service to replace
management of keepalived instances and networks withing Neutron. Router
rescheduling is not really an alternative for L3 HA.

RDO with L3 HA is a great example of success, but we want to have ability
to try something else that can suit other OpenStack deployments better.

I wrote this email to understand whether community have interest in
something like this, so that it will be worth doing.

On Tue, Feb 14, 2017 at 10:20 PM Assaf Muller <assaf at redhat.com> wrote:

On Fri, Feb 10, 2017 at 12:27 PM, Anna Taraday
<akamyshnikova at mirantis.com> wrote:
> Hello everyone!
>
> In Juno in Neutron was implemented L3 HA feature based on Keepalived
(VRRP).
> During next cycles it was improved, we performed scale testing [1] to find
> weak places and tried to fix them. The only alternative for L3 HA with
VRRP
> is router rescheduling performed by Neutron server, but it is
significantly
> slower and depends on control plane.
>
> What issues we experienced with L3 HA VRRP?
>
> Bugs in Keepalived (bad versions) [2]
> Split brain [3]
> Complex structure (ha networks, ha interfaces) - which actually cause
races
> that we were fixing during Liberty, Mitaka and Newton.
>
> This all is not critical, but this is a bad experience and not everyone
> ready (or want) to use Keepalived approach.
>
> I think we can make things more flexible. For example, we can allow user
to
> use external services like etcd instead of Keepalived to synchronize
current
> HA state across agents. I've done several experiments and I've got
failover
> time comparable to L3 HA with VRRP. Tooz [4] can be used to abstract from
> concrete backend. For example, it can allow us to use Zookeeper, Redis and
> other backends to store HA state.
>
> What I want to propose?
>
> I want to bring up idea that Neutron should have some general classes for
L3
> HA which will allow to use not only Keepalived but also other backends for
> HA state. This at least will make it easier to try some other approaches
and
> compare them with existing ones.
>
> Does this sound reasonable?

I understand that the intention is to add pluggability upstream so
that you could examine the viability of alternative solutions. I'd
advise instead to do the research locally, and if you find concrete
benefits to an alternative solution, come back, show your work and
have a discussion about it then. Merging extra complexity in the form
of a plug point without knowing if we're actually going to need it
seems risky.

On another note, after years of work the stability issues have largely
been resolved and L3 HA is in a good state with modern releases of
OpenStack. It's not a authoritative solution in the sense that it
doesn't cover every possible failure mode, but it covers the major
ones and in that sense better than not having any form of HA, and as
you pointed out the existing alternatives are not in a better state.
The subtext in your email is that now L3 HA is technically where we
want it, but some users are resisting adoption because of bad PR or a
bad past experience, but not for technical reasons. If that is the
case, then perhaps some good PR would be a more cost effective
investment than investigating, implementing, stabilizing and
maintaining a different backend that will likely take at least a cycle
to get merged and another 1 to 2 cycles to iron out kinks. Would you
have a critical mass of developers ready to support a pluggable L3 HA
now and in the long term?

Finally, I can share that L3 HA has been the default in RDO-land for a
few cycles now and is being used widely and successfully, in some
cases at significant scale.

>
> [1] -
>
http://docs.openstack.org/developer/performance-docs/test_results/neutron_features/index.html
> [2] - https://bugs.launchpad.net/neutron/+bug/1497272
> https://bugs.launchpad.net/neutron/+bug/1433172
> [3] - https://bugs.launchpad.net/neutron/+bug/1375625
> [4] - http://docs.openstack.org/developer/tooz/
>
>
>
>
> --
> Regards,
> Ann Taraday
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-- 
Regards,
Ann Taraday
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170215/e12a5811/attachment.html>