Open Stack

About success of RDO we need to remember that this deployment utilizes Peacemaker and when I was working on this feature and even I spoke with Assaf this external application was doing everything to make this solution working.
Peacemaker was responsible for checking external and internal connectivity. To detect split brain. Elect master, even keepalived was running but Peacemaker was automatically killing all services and moving FIP.
Assaf - is there any change in this implementation in RDO? Or you’re still doing everything outside of Neutron?

Because if RDO success is build on Peacemaker it means that yes, Neutron needs some solution which will be available for more than RH deployments.

Lubosz

On Feb 15, 2017, at 3:22 AM, Anna Taraday <akamyshnikova at mirantis.com<mailto:akamyshnikova at mirantis.com>> wrote:

If I propose some concrete solution that will be discussion about one solution not about making things flexible.
At first I wanted to propose some PoC for other approach, but during my experiments I understood that we may have different approaches, but for all of them we need pluggable HA router in Neutron.

The thing that bothers me about L3 HA - it is complex. Yes, we fixed bunch of races and John did significant refactor, but it is still too complex. In the end we want to use L3 HA + DVR but DVR is pretty complex by itself. We would like to try to offload this complexity to external service to replace management of keepalived instances and networks withing Neutron. Router rescheduling is not really an alternative for L3 HA.

RDO with L3 HA is a great example of success, but we want to have ability to try something else that can suit other OpenStack deployments better.

I wrote this email to understand whether community have interest in something like this, so that it will be worth doing.

On Tue, Feb 14, 2017 at 10:20 PM Assaf Muller <assaf at redhat.com<mailto:assaf at redhat.com>> wrote:
On Fri, Feb 10, 2017 at 12:27 PM, Anna Taraday
<akamyshnikova at mirantis.com<mailto:akamyshnikova at mirantis.com>> wrote:
> Hello everyone!
>
> In Juno in Neutron was implemented L3 HA feature based on Keepalived (VRRP).
> During next cycles it was improved, we performed scale testing [1] to find
> weak places and tried to fix them. The only alternative for L3 HA with VRRP
> is router rescheduling performed by Neutron server, but it is significantly
> slower and depends on control plane.
>
> What issues we experienced with L3 HA VRRP?
>
> Bugs in Keepalived (bad versions) [2]
> Split brain [3]
> Complex structure (ha networks, ha interfaces) - which actually cause races
> that we were fixing during Liberty, Mitaka and Newton.
>
> This all is not critical, but this is a bad experience and not everyone
> ready (or want) to use Keepalived approach.
>
> I think we can make things more flexible. For example, we can allow user to
> use external services like etcd instead of Keepalived to synchronize current
> HA state across agents. I've done several experiments and I've got failover
> time comparable to L3 HA with VRRP. Tooz [4] can be used to abstract from
> concrete backend. For example, it can allow us to use Zookeeper, Redis and
> other backends to store HA state.
>
> What I want to propose?
>
> I want to bring up idea that Neutron should have some general classes for L3
> HA which will allow to use not only Keepalived but also other backends for
> HA state. This at least will make it easier to try some other approaches and
> compare them with existing ones.
>
> Does this sound reasonable?

I understand that the intention is to add pluggability upstream so
that you could examine the viability of alternative solutions. I'd
advise instead to do the research locally, and if you find concrete
benefits to an alternative solution, come back, show your work and
have a discussion about it then. Merging extra complexity in the form
of a plug point without knowing if we're actually going to need it
seems risky.

On another note, after years of work the stability issues have largely
been resolved and L3 HA is in a good state with modern releases of
OpenStack. It's not a authoritative solution in the sense that it
doesn't cover every possible failure mode, but it covers the major
ones and in that sense better than not having any form of HA, and as
you pointed out the existing alternatives are not in a better state.
The subtext in your email is that now L3 HA is technically where we
want it, but some users are resisting adoption because of bad PR or a
bad past experience, but not for technical reasons. If that is the
case, then perhaps some good PR would be a more cost effective
investment than investigating, implementing, stabilizing and
maintaining a different backend that will likely take at least a cycle
to get merged and another 1 to 2 cycles to iron out kinks. Would you
have a critical mass of developers ready to support a pluggable L3 HA
now and in the long term?

Finally, I can share that L3 HA has been the default in RDO-land for a
few cycles now and is being used widely and successfully, in some
cases at significant scale.

>
> [1] -
> http://docs.openstack.org/developer/performance-docs/test_results/neutron_features/index.html
> [2] - https://bugs.launchpad.net/neutron/+bug/1497272
> https://bugs.launchpad.net/neutron/+bug/1433172
> [3] - https://bugs.launchpad.net/neutron/+bug/1375625
> [4] - http://docs.openstack.org/developer/tooz/
>
>
>
>
> --
> Regards,
> Ann Taraday
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe<http://OpenStack-dev-request@lists.openstack.org/?subject:unsubscribe>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
--
Regards,
Ann Taraday
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org<mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170215/7091dbc2/attachment.html>