<div dir="ltr"><div>Hey Sergey,<br><br></div>In-line comments below:<br><div class="gmail_extra"><br><div class="gmail_quote">On Sun, Jun 5, 2016 at 8:07 AM, Sergey Guenender <span dir="ltr"><<a href="mailto:sgserg@gmail.com" target="_blank">sgserg@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
Hi Stephen, please find my reply next to your points below.<br>
<br>
Thank you,<br>
-Sergey.<span class=""><br>
<br>
<br>
On 01/06/2016 20:23, Stephen Balukoff wrote:<br>
> Hey Sergey--<br>
><br>
> Apologies for the delay in my response. I'm still wrapping my head<br>
> around your option 2 suggestion and the implications it might have for<br>
> the code base moving forward. I think, though, that I'm against your<br>
> option 2 proposal and in favor of option 1 (which, yes, is more work<br>
> initially) for the following reasons:<br>
><br>
> A. We have a precedent in the code tree with how the stand-alone and<br>
> active-standby topologies are currently being handled. Yes, this does<br>
> entail various conditionals and branches in tasks and flows-- which is<br>
> not really that ideal, as it means the controller worker needs to have<br>
> more specific information on how topologies work than I think any of us<br>
> would like, and this adds some rigidity to the implementation (meaning<br>
> 3rd party vendors may have more trouble interfacing at that level)...<br>
> but it's actually "not that bad" in many ways, especially given we don't<br>
> anticipate supporting a large or variable number of topologies.<br>
> (stand-alone, active-standby, active-active... and then what? We've been<br>
> doing this for a number of years and nobody has mentioned any radically<br>
> new topologies they would like in their load balancing. Things like<br>
> auto-scale are just a specific case of active-active).<br>
<br></span>
Just as you say, two topologies are being handled as of now by only one set of flows. Option two goes along the same lines, instead of adding new flows for active-active it suggests that minor adjustments to existing flows can also satisfy active-active.<span class=""><br></span></blockquote><div><br></div><div>My point was that I think the distributor and amphora roles are different enough that they ought to have separate drivers, separate flows, etc. almost entirely. There's not much difference between a stand-alone amphora and an amphora in an active-standby topology. However, there's a huge difference between both of these and a distributor (which will have its own back-end API, for example).<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
> B. If anything Option 2 builds more less-obvious rigidity into the<br>
> implementation than option 1. For example, it makes the assumption that<br>
> the distributor is necessarily an amphora or service VM, whereas we have<br>
> already heard that some will implement the distributor as a pure network<br>
> routing function that isn't going to be managed the same way other<br>
> amphorae are.<br>
<br></span>
This is a good point. By looking at the code, I see there are comments mentioning the intent to share amphora between several load balancers. Although probably not straightforward to implement, it might be a good idea one day, but the fact is it looks like amphora has not been shared between load balancers for a few years.<br>
<br>
Personally, when developing something complex, I believe in taking baby steps. If the virtual, non-shared distributor (which is promised by the AA blueprint anyway) is the smallest step towards a working active-active, then I guess it should be considered taking first.<br></blockquote><div><br></div><div>The AA blueprint has yet to be approved (and there appear to be a *lot* of comments on the latest revision). But yes-- in general you need to walk before you can run. But instead of torturing analogies, let me say this: Assumptions about design are reflected in the code. So, I generally like to do my best getting the design right... and then any baby steps taken should be evaluated against that end design to ensure they don't introduce assumptions that will make it difficult to get there.<br></div><div><br> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Unless of course, it precludes implementing the following, more complex topologies.<br>
<br>
My belief is it doesn't have to. The proposed change alone (splitting amphorae into sub-clusters to be used by the many for-loops) doesn't force any special direction on its own. Any future topology may leave its "front-facing amphorae" set equal to its "back-facing amphorae" which brings it back to the current style of for-loops handling.<span class=""><br></span></blockquote><div><br></div><div>See, I disagree that an amphora and a distributor are even really similar. The idea that a distributor is just a front-facing amphora I think is fundamentally false-- Especially if distributors are implemented with a direct-return topology (as the blueprint under evaluation describes) then they're almost nothing alike. The distributor service VM will not be running haproxy and will be running its own unique API specifically because it's fulfilling a vastly different role in the topology than the amphorae fill.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
> C. Option 2 seems like it's going to have a lot more permutations that<br>
> would need testing to ensure that code changes don't break existing /<br>
> potentially supported functionality. Option 1 keeps the distributor and<br>
> amphorae management code separate, which means tests should be more<br>
> straight-forward, and any breaking changes which slip through<br>
> potentially break less stuff. Make sense?<br>
<br></span>
It certainly does.<br>
<br>
My intent is that the simplest active-active implementation promised by the blueprint can be achieved with only minor changes by existing code. If required changes are not in fact small, or if this simplistic approach in some way impedes future work, we can drop this option.<span class=""><br>
<br></span></blockquote><div><br></div><div>What you do with your time on this is your own matter, I suppose-- but I don't think Option 2 is less work than Option 1 by any stretch of the imagination given how dissimilar distributors and amphorae are.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
> Stephen<br>
><br>
><br>
> On Sun, May 29, 2016 at 7:12 AM, Sergey Guenender<br>
> <<a href="mailto:GUENEN@il.ibm.com" target="_blank">GUENEN@il.ibm.com</a><br></span><div><div class="h5">
> <mailto:<a href="mailto:GUENEN@il.ibm.com" target="_blank">GUENEN@il.ibm.com</a>>> wrote:<br>
><br>
> I'm working with the IBM team implementing the Active-Active N+1<br>
> topology [1].<br>
><br>
> I've been commissioned with the task to help integrate the code<br>
> supporting the new topology while a) making as few code changes and<br>
> b) reusing as much code as possible.<br>
><br>
> To make sure the changes to existing code are future-proof, I'd like<br>
> to implement them outside AA N+1, submit them on their own and let<br>
> the AA N+1 base itself on top of it.<br>
><br>
> --TL;DR--<br>
><br>
> what follows is a description of the challenges I'm facing and the<br>
> way I propose to solve them. Please skip down to the end of the<br>
> email to see the actual questions.<br>
><br>
> --The details--<br>
><br>
> I've been studying the code for a few weeks now to see where the<br>
> best places for minimal changes might be.<br>
><br>
> Currently I see two options:<br>
><br>
> 1. introduce a new kind of entity (the distributor) and make<br>
> sure it's being handled on any of the 6 levels of controller worker<br>
> code (endpoint, controller worker, *_flows, *_tasks, *_driver)<br>
><br>
> 2. leave most of the code layers intact by building on the fact<br>
> that distributor will inherit most of the controller worker logic of<br>
> amphora<br>
><br>
><br>
> In Active-Active topology, very much like in Active/StandBy:<br>
> * top level of distributors will have to run VRRP<br>
> * the distributors will have a Neutron port made on the VIP network<br>
> * the distributors' neutron ports on VIP network will need the same<br>
> security groups<br>
> * the amphorae facing the pool member networks still require<br>
> * ports on the pool member networks<br>
> * "peers" HAProxy configuration for real-time state exchange<br>
> * VIP network connections with the right security groups<br>
><br>
> The fact that existing topologies lack the notion of distributor and<br>
> inspecting the 30-or-so existing references to amphorae clusters,<br>
> swayed me towards the second option.<br>
><br>
> The easiest way to make use of existing code seems to be by<br>
> splitting load-balancer's amphorae into three overlapping sets:<br>
> 1. The front-facing - those connected to the VIP network<br>
> 2. The back-facing - subset of front-facing amphorae, also connected<br>
> to the pool members' networks<br>
> 3. The VRRP-running - subset of front-facing amphorae, making sure<br>
> the VIP routing remains highly available<br>
><br>
> At the code-changes level<br>
> * the three sets can be simply added as properties of<br>
> common.data_model.LoadBalancer<br>
> * the existing amphorae cluster references would switch to using one<br>
> of these properties, for example<br>
> * the VRRP sub-flow would loop over only the VRRP amphorae<br>
> * the network driver, when plugging the VIP, would loop over<br>
> the front-facing amphorae<br>
> * when connecting to the pool members' networks,<br>
> network_tasks.CalculateDelta would only loop over the back-facing<br>
> amphorae<br>
><br></div></div>
> *<span class="im HOEnZb"><br>
><br>
> In terms of backwards compatibility, Active-StandBy topology would<br>
> have the 3 sets equal and contain both of its amphorae.<br>
><br>
> An even more future-proof approach might be to implement the<br>
> sets-getters as selector methods, supporting operation on subsets of<br>
> each kind of amphorae. For instance when growing/shrinking<br>
> back-facing amphorae cluster, only the added/removed ones will need<br>
> to be processed.<br>
><br>
> Finally (thank you for your patience, dear reader), my question is:<br>
> if any of the above makes sense, and to facilitate the design/code<br>
> review, what would be the best way to move forward?<br>
><br>
> Should I create a mini-blueprint describing the changes and<br>
> implement it?<br>
> Should I just open a bug for it and supply a fix?<br>
><br>
> Thanks,<br>
> -Sergey.<br>
<br>
<br>
<br></span><div class="HOEnZb"><div class="h5">
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>