<div dir="ltr">Hi Keshava,<div><br></div><div>For the purposes of Octavia, it's going to be service VMs (or containers or what have you). However, service VM or tenant VM the concept is roughly similar: We need some kind of layer-3 routing capability which works something like Neutron floating IPs (though not just a static NAT in this case) but which can distribute traffic to a set of back-end VMs running on a Neutron network according to some predictable algorithm (probably a distributed hash).</div><div><br></div><div>The idea behind ACTIVE-ACTIVE is that you have many service VMs (we call them amphorae) which service the same "public" IP in some way-- this allows for horizontal scaling of services which need it (ie. anything which does TLS termination with a significant amount of load).</div><div><br></div><div>Does this make sense to you?</div><div><br></div><div>Thanks,</div><div>Stephen</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Dec 8, 2014 at 9:56 PM, A, Keshava <span dir="ltr"><<a href="mailto:keshava.a@hp.com" target="_blank">keshava.a@hp.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal">Stephen,<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="text-indent:.5in">Interesting to know what is “ACTIVE-ACTIVE topology of load balancing VMs”.<u></u><u></u></p>
<p class="MsoNormal" style="text-indent:.5in">What is the scenario is it Service-VM (of NFV) or Tennant VM ?<u></u><u></u></p>
<p class="MsoNormal" style="text-indent:.5in">Curious to know the background of this thoughts .<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">keshava<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Stephen Balukoff [mailto:<a href="mailto:sbalukoff@bluebox.net" target="_blank">sbalukoff@bluebox.net</a>]
<br>
<b>Sent:</b> Tuesday, December 09, 2014 7:18 AM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)</span></p><div><div class="h5"><br>
<b>Subject:</b> Re: [openstack-dev] [Neutron] [RFC] Floating IP idea solicitation and collaboration<u></u><u></u></div></div><p></p><div><div class="h5">
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">For what it's worth, I know that the Octavia project will need something which can do more advanced layer-3 networking in order to deliver and ACTIVE-ACTIVE topology of load balancing VMs / containers / machines. That's still a "down the
road" feature for us, but it would be great to be able to do more advanced layer-3 networking in earlier releases of Octavia as well. (Without this, we might have to go through back doors to get Neutron to do what we need it to, and I'd rather avoid that.)<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm definitely up for learning more about your proposal for this project, though I've not had any practical experience with Ryu yet. I would also like to see whether it's possible to do the sort of advanced layer-3 networking you've described
without using OVS. (We have found that OVS tends to be not quite mature / stable enough for our needs and have moved most of our clouds to use ML2 / standard linux bridging.)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Carl: I'll also take a look at the two gerrit reviews you've linked. Is this week's L3 meeting not happening then? (And man-- I wish it were an hour or two later in the day. Coming at y'all from PST timezone here.)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Stephen<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Mon, Dec 8, 2014 at 11:57 AM, Carl Baldwin <<a href="mailto:carl@ecbaldwin.net" target="_blank">carl@ecbaldwin.net</a>> wrote:<u></u><u></u></p>
<blockquote style="border:none;border-left:solid #cccccc 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<p class="MsoNormal">Ryan,<br>
<br>
I'll be traveling around the time of the L3 meeting this week. My<br>
flight leaves 40 minutes after the meeting and I might have trouble<br>
attending. It might be best to put it off a week or to plan another<br>
time -- maybe Friday -- when we could discuss it in IRC or in a<br>
Hangout.<br>
<span style="color:#888888"><br>
<span>Carl</span></span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><br>
On Mon, Dec 8, 2014 at 8:43 AM, Ryan Clevenger<br>
<<a href="mailto:ryan.clevenger@rackspace.com" target="_blank">ryan.clevenger@rackspace.com</a>> wrote:<br>
> Thanks for getting back Carl. I think we may be able to make this weeks<br>
> meeting. Jason Kölker is the engineer doing all of the lifting on this side.<br>
> Let me get with him to review what you all have so far and check our<br>
> availability.<br>
><br>
> ________________________________________<br>
><br>
> Ryan Clevenger<br>
> Manager, Cloud Engineering - US<br>
> m: <a href="tel:678.548.7261" target="_blank">678.548.7261</a><br>
> e: <a href="mailto:ryan.clevenger@rackspace.com" target="_blank">ryan.clevenger@rackspace.com</a><br>
><br>
> ________________________________<br>
> From: Carl Baldwin [<a href="mailto:carl@ecbaldwin.net" target="_blank">carl@ecbaldwin.net</a>]<br>
> Sent: Sunday, December 07, 2014 4:04 PM<br>
> To: OpenStack Development Mailing List<br>
> Subject: Re: [openstack-dev] [Neutron] [RFC] Floating IP idea solicitation<br>
> and collaboration<br>
><br>
> Ryan,<br>
><br>
> I have been working with the L3 sub team in this direction. Progress has<br>
> been slow because of other priorities but we have made some. I have written<br>
> a blueprint detailing some changes needed to the code to enable the<br>
> flexibility to one day run glaring ups on an l3 routed network [1]. Jaime<br>
> has been working on one that integrates ryu (or other speakers) with neutron<br>
> [2]. Dvr was also a step in this direction.<br>
><br>
> I'd like to invite you to the l3 weekly meeting [3] to discuss further. I'm<br>
> very happy to see interest in this area and have someone new to collaborate.<br>
><br>
> Carl<br>
><br>
> [1] <a href="https://review.openstack.org/#/c/88619/" target="_blank">https://review.openstack.org/#/c/88619/</a><br>
> [2] <a href="https://review.openstack.org/#/c/125401/" target="_blank">https://review.openstack.org/#/c/125401/</a><br>
> [3] <a href="https://wiki.openstack.org/wiki/Meetings/Neutron-L3-Subteam" target="_blank">
https://wiki.openstack.org/wiki/Meetings/Neutron-L3-Subteam</a><br>
><br>
> On Dec 3, 2014 4:04 PM, "Ryan Clevenger" <<a href="mailto:ryan.clevenger@rackspace.com" target="_blank">ryan.clevenger@rackspace.com</a>><br>
> wrote:<br>
>><br>
>> Hi,<br>
>><br>
>> At Rackspace, we have a need to create a higher level networking service<br>
>> primarily for the purpose of creating a Floating IP solution in our<br>
>> environment. The current solutions for Floating IPs, being tied to plugin<br>
>> implementations, does not meet our needs at scale for the following reasons:<br>
>><br>
>> 1. Limited endpoint H/A mainly targeting failover only and not<br>
>> multi-active endpoints,<br>
>> 2. Lack of noisy neighbor and DDOS mitigation,<br>
>> 3. IP fragmentation (with cells, public connectivity is terminated inside<br>
>> each cell leading to fragmentation and IP stranding when cell CPU/Memory use<br>
>> doesn't line up with allocated IP blocks. Abstracting public connectivity<br>
>> away from nova installations allows for much more efficient use of those<br>
>> precious IPv4 blocks).<br>
>> 4. Diversity in transit (multiple encapsulation and transit types on a per<br>
>> floating ip basis).<br>
>><br>
>> We realize that network infrastructures are often unique and such a<br>
>> solution would likely diverge from provider to provider. However, we would<br>
>> love to collaborate with the community to see if such a project could be<br>
>> built that would meet the needs of providers at scale. We believe that, at<br>
>> its core, this solution would boil down to terminating north<->south traffic<br>
>> temporarily at a massively horizontally scalable centralized core and then<br>
>> encapsulating traffic east<->west to a specific host based on the<br>
>> association setup via the current L3 router's extension's 'floatingips'<br>
>> resource.<br>
>><br>
>> Our current idea, involves using Open vSwitch for header rewriting and<br>
>> tunnel encapsulation combined with a set of Ryu applications for management:<br>
>><br>
>> <a href="https://i.imgur.com/bivSdcC.png" target="_blank">https://i.imgur.com/bivSdcC.png</a><br>
>><br>
>> The Ryu application uses Ryu's BGP support to announce up to the Public<br>
>> Routing layer individual floating ips (/32's or /128's) which are then<br>
>> summarized and announced to the rest of the datacenter. If a particular<br>
>> floating ip is experiencing unusually large traffic (DDOS, slashdot effect,<br>
>> etc.), the Ryu application could change the announcements up to the Public<br>
>> layer to shift that traffic to dedicated hosts setup for that purpose. It<br>
>> also announces a single /32 "Tunnel Endpoint" ip downstream to the TunnelNet<br>
>> Routing system which provides transit to and from the cells and their<br>
>> hypervisors. Since traffic from either direction can then end up on any of<br>
>> the FLIP hosts, a simple flow table to modify the MAC and IP in either the<br>
>> SRC or DST fields (depending on traffic direction) allows the system to be<br>
>> completely stateless. We have proven this out (with static routing and<br>
>> flows) to work reliably in a small lab setup.<br>
>><br>
>> On the hypervisor side, we currently plumb networks into separate OVS<br>
>> bridges. Another Ryu application would control the bridge that handles<br>
>> overlay networking to selectively divert traffic destined for the default<br>
>> gateway up to the FLIP NAT systems, taking into account any configured<br>
>> logical routing and local L2 traffic to pass out into the existing overlay<br>
>> fabric undisturbed.<br>
>><br>
>> Adding in support for L2VPN EVPN<br>
>> (<a href="https://tools.ietf.org/html/draft-ietf-l2vpn-evpn-11" target="_blank">https://tools.ietf.org/html/draft-ietf-l2vpn-evpn-11</a>) and L2VPN EVPN<br>
>> Overlay (<a href="https://tools.ietf.org/html/draft-sd-l2vpn-evpn-overlay-03" target="_blank">https://tools.ietf.org/html/draft-sd-l2vpn-evpn-overlay-03</a>) to the<br>
>> Ryu BGP speaker will allow the hypervisor side Ryu application to advertise<br>
>> up to the FLIP system reachability information to take into account VM<br>
>> failover, live-migrate, and supported encapsulation types. We believe that<br>
>> decoupling the tunnel endpoint discovery from the control plane<br>
>> (Nova/Neutron) will provide for a more robust solution as well as allow for<br>
>> use outside of openstack if desired.<br>
>><br>
>> ________________________________________<br>
>><br>
>> Ryan Clevenger<br>
>> Manager, Cloud Engineering - US<br>
>> m: <a href="tel:678.548.7261" target="_blank">678.548.7261</a><br>
>> e: <a href="mailto:ryan.clevenger@rackspace.com" target="_blank">ryan.clevenger@rackspace.com</a><br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><u></u><u></u></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br>
<br clear="all">
<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal">-- <u></u><u></u></p>
<div>
<p class="MsoNormal">Stephen Balukoff <br>
Blue Box Group, LLC <br>
<a href="tel:%28800%29613-4305%20x807" value="+18006134305" target="_blank">(800)613-4305 x807</a> <u></u><u></u></p>
</div>
</div>
</div></div></div>
</div>
</blockquote></div><br><br clear="all"><div><br></div>-- <br><div class="gmail_signature"><span></span>Stephen Balukoff
<br>Blue Box Group, LLC
<br>(800)613-4305 x807
</div>
</div>