<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><font color="#ff0000" face="Calibri,sans-serif">My comments in red (sorry again)</font><span style="color: rgb(255, 0, 0); font-family: Calibri, sans-serif; font-size: medium; ">.</span></div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<br>
</div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Eugene Nikanorov <<a href="mailto:enikanorov@mirantis.com">enikanorov@mirantis.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Friday, May 2, 2014 5:08 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Neutron][LBaaS] Fulfilling Operator Requirements: Driver / Management API<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">Hi Adam,
<div><br>
</div>
<div>My comments inline:</div>
<div class="gmail_extra"><br>
<br>
<div class="gmail_quote">On Fri, May 2, 2014 at 1:33 AM, Adam Harwell <span dir="ltr">
<<a href="mailto:adam.harwell@rackspace.com" target="_blank">adam.harwell@rackspace.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>I am sending this now to gauge interest and get feedback on what I see as an impending necessity updating the existing "haproxy" driver, replacing it, or both.
</div>
</div>
</blockquote>
<div>I agree with Stephen's first point here.</div>
<div>For HAProxy driver to support advanced use cases like routed mode, it's agent should be severely changed and receive some capabilities of L3 agent.</div>
<div>In fact, I'd suggest making additional driver, not for haproxy in VMs, but for... dedicated haproxy nodes.</div>
<div>Dedicated haproxy node is a host (similar to compute) with L2 agent and lbaas (not necessarily existing) agent on it.</div>
<div><br>
</div>
<div>In fact, it's essentially the same model as used right now, but i think it has it's advantages over haproxy-in-vm, at least:</div>
<div>- plugin driver doesn't need to manage VM life cycle (no orchestration)</div>
<div>- immediate "natural" multitenant support with isolated networks</div>
<div>- instead of adding haproxy in VM, you add a process (which is both faster and more efficient); </div>
<div>more scaling is achieved by adding physical haproxy node; existing agent health reporting will make it available for loadbalancer scheduling automatically.</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<br>
</div>
<div><font color="#ff0000" face="Calibri,sans-serif">I</font><font color="#ff0000"><font face="Calibri,sans-serif"> think that driver sounds like a good idea I think we agree in essence, that there will need to be drivers to provide a variety of different
approaches. I guess the question becomes, is there a smart way to accomplish this?</font></font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div><b>HAProxy</b>: This references two things currently, and I feel this is a source of some misunderstanding. When I refer to HAProxy (capitalized), I will be referring to the official software package (found here: <a href="http://haproxy.1wt.eu/" target="_blank">http://haproxy.1wt.eu/</a> ),
and when I refer to "haproxy" (lowercase, and in quotes) I will be referring to the neutron-lbaas driver (found here: <a href="https://github.com/openstack/neutron/tree/master/neutron/services/loadbalancer/drivers/haproxy" target="_blank">https://github.com/openstack/neutron/tree/master/neutron/services/loadbalancer/drivers/haproxy</a> ).
The fact that the neutron-lbaas driver is named directly after the software package seems very unfortunate, and while it is not directly in the scope of what I'd like to discuss here, I would love to see it changed to more accurately reflect what it is --
one specific driver implementation that coincidentally uses HAProxy as a backend. More on this later.<br>
</div>
</div>
</blockquote>
<div>We also was referring existing driver as "haproxy-on-host".</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">Ok, I will use that term from now on (I just hadn't seen it anywhere, and you can understand how it is confusing to just see "haproxy" as the driver name).</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div></div>
<div><br>
</div>
<div><b>Operator Requirements</b>: The requirements that can be found on the wiki page here: <a href="https://wiki.openstack.org/wiki/Neutron/LBaaS/requirements#Operator_Requirements" target="_blank">https://wiki.openstack.org/wiki/Neutron/LBaaS/requirements#Operator_Requirements</a> and
focusing on (but not limited to) the following list:</div>
<div><span style="white-space:pre-wrap"></span>* Scalability</div>
<div><span style="white-space:pre-wrap"></span>* DDoS Mitigation</div>
<div><span style="white-space:pre-wrap"></span>* Diagnostics</div>
<div><span style="white-space:pre-wrap"></span>* Logging and Alerting</div>
<div><span style="white-space:pre-wrap"></span>* Recoverability</div>
<div><span style="white-space:pre-wrap"></span>* High Availability (this is in the User Requirements section, but will be largely up to the operator to handle, so I would include it when discussing Operator Requirements)</div>
</div>
</blockquote>
<div>Those requirements are of very different kinds and they are going to be addressed by quite different components of lbaas, not solely by the driver.</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div><br>
</div>
<div><b>Management API</b>: A restricted API containing resources that Cloud Operators could access, including most of the list of Operator Requirements (above).</div>
</div>
</blockquote>
<div>The work is being done on this front: we're designing a way for plugin drivers to expose their own API, that specifically is needed for operator API which might not be common between providers.</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">Ok, this sounds like what some other people mentioned, and does sound like essentially what we'd need to do for this to work in any real capacity. The question I have then is, do we still need to talk about this at all, or just agree
to make sure this method works, and then go our own ways implementing our Management APIs?</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> <br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div><br>
</div>
<div><b>Load Balancer (LB)</b>: I use this term very generically essentially a logical entity that represents one "use case". As used in the sentence: "I have a Load Balancer in front of my website." or "The Load Balancer I set up to offload SSL Decryption
is lowering my CPU load nicely."</div>
<div><br>
</div>
<div>
<div>----------------------------------</div>
<div>---- Overview</div>
<div>----------------------------------</div>
</div>
<div>What we've all been discussing for the past month or two (the API, Object Model, etc) is being directly driven by the User and Operator Requirements that have somewhat recently been enumerated (many thanks to everyone who has contributed to that discussion!).
With that in mind, it is hopefully apparent that the current API proposals don't directly address many (or really, any) of the Operator requirements! Where in either of our API proposals are logging, high availability, scalability, DDoS mitigation, etc? I
believe the answer is that none of these things can possibly be handled by the API, but are really implementation details at the driver level. Radware, NetScaler, Stingray, F5 and HAProxy of any flavour would all have very different ways of handling these
things (these are just some of the possible backends I can think of). At the end of the day, what we really have are the requirements for a driver, which may or may not use HAProxy, that we hope will satisfy all of our concerns. That said, we may also want
to have some form of "Management API" to expose these features in a common way.</div>
</div>
</blockquote>
<div>I'm not sure on the 'common way' here. I'd prefer to let vendors implement what is suitable for them and converge on similarities later.</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">Same as above I guess we don't really work together, we just make our own Management APIs? Is this a good design decision? Is there any alternative? Please note that I'm not saying you're necessarily wrong
</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>In this case, we really need to discuss two things: </div>
<ol>
<li>Whether to update the existing "haproxy" driver to accommodate these Operator Requirements, or whether to start from scratch with a new driver (possibly both).</li></ol>
</div>
</blockquote>
<div>See my comment on this above. I'd prefer to have drivers in both variants, however I'm not sure if such code/solution duplication is acceptable, most probably it is (as they will support different use cases). The problem is that existing solution (particularly,
haproxy namespace driver) can't support some important use cases, but it hardly makes sense to rework it for those cases. On the other hand, the new driver might not support the way existing driver works, but that might be fine.</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">Essentially: yes, that all sounds like what I was thinking as well.</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<ol>
<li>How to expose these Operator features at the (Management?) API level.</li></ol>
</div>
</blockquote>
<div>See above. There was a bp filed for this ( <a href="https://blueprints.launchpad.net/neutron/+spec/lbaas-extensions">
https://blueprints.launchpad.net/neutron/+spec/lbaas-extensions</a> ) and we also had a session at Icehouse summit (
<a href="https://etherpad.openstack.org/p/icehouse-neutron-vendor-extension">https://etherpad.openstack.org/p/icehouse-neutron-vendor-extension</a>) on how this could be implemented.</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">Thanks for the links, I'll attempt to acquaint myself with these.</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>----------------------------------</div>
<div>---- 1) Driver</div>
<div>----------------------------------</div>
<div>I believe the current "haproxy" driver serves a very specific purpose, and while it will need some incremental updates, it would be in the best interest of the community to also create and maintain a new driver (which it sounds like several groups have
already begun work on ack!) that could support a different approach. For instance, the current "haproxy" driver is implemented by initializing HAProxy processes on a set of shared hosts, whereas there has been some momentum behind creating individual Virtual
Machines (via Nova) for each Load Balancer created, similar to Libra's approach. Alternatively, we could use LXC or a similar technology to more effectively isolate LBs and assuage concerns about tenant cross-talk (real or imaginary, this has been an issue
for some customers). </div>
</div>
</blockquote>
<div>I think VM approach is also possible as additional third option (in addition to existing driver and dedicated host).</div>
<div>Please note that similar work on this is also on the way: <a href="https://review.openstack.org/#/c/88213/">https://review.openstack.org/#/c/88213/</a></div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>Either way, we'd probably need a brand new driver, to avoid breaking backwards compatibility with the existing driver (which does work perfectly fine in many cases). In fact, it's possible that when we begin discussing this as a broader community, we might
decide to create more than one additional driver (depending on which approaches people want to use and what features are most important to them). The only concern I have about that outcome is the necessary amount of code-reuse, and whether it would be possible
to share certain aspects of these drivers without too much copy/pasting.</div>
</div>
</blockquote>
<div>I generally agree with that. I'm only a little concerned about possible duplicate solutions.</div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">I think German had some ideas about this, with regard to splitting up the provider from the driver, possibly allowing for much less copy/pasting or duplication of effort. I'm hoping we can discuss more along those lines.</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div><br>
</div>
<div>An example of one possible new driver could be the following (just off the top of my head):</div>
<div><span style="white-space:pre-wrap"></span>* Use a pair of new Nova VMs for each LB (Scalability), configured to use a Shared IP (High Availability).</div>
<div><span style="white-space:pre-wrap"></span>* Log to Swift / Ceilometer (Logging / Alerting / Metering).</div>
<div><span style="white-space:pre-wrap"></span>* Provide calls that could be exposed via a Management API to show low level diagnostic details (Diagnostics).</div>
<div><span style="white-space:pre-wrap"></span>* Provide calls that could be exposed via a Management API to allow syncing/reloading existing LBs or moving them across clusters (Recoverability, DDoS Mitigation).</div>
<div>This new driver would be named to reflect what features it provides, or at least given a unique name that can be referenced without confusion (something like "OpenHA" or "NovaHA" would work if that's not taken).</div>
<div><br>
</div>
<div>
<div>----------------------------------</div>
<div>---- 2) Management API</div>
<div>----------------------------------</div>
</div>
<div>Going forward, it should then be required (can we enforce this?) that any mainline driver include support for calls to handle these named Operator Requirements, for example: obtaining logs (or log locations?), diagnostic information, and admin type actions
including rebuilding or migrating LB instances.</div>
</div>
</blockquote>
<div> </div>
<div>I think we should not put these requirements, at least from the beginning. It seems that operator's API might be even more complex then tenant's and that makes consensus on it much harder.</div>
<div><br>
</div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>So far we haven't really talked about any of these features in depth, though I believe the general need for a Management API was alluded to on several occasions. Should we shelve this discussion until after we have the User API specification locked down?
Should we begin defining a contract for this Management API at the summit, since it would be the main gateway to the Operator Requirements that we have all been stressing recently? </div>
<div><br>
</div>
<div>
<div>----------------------------------</div>
<div>---- Summary</div>
<div>----------------------------------</div>
</div>
<div>I would apologize for not having much concrete specification here, but I think it is better to validate my basic assumptions first, before jumping deeper down this rabbit hole. The type of comments I'm hoping to prompt are along the lines of:</div>
<div><span style="white-space:pre-wrap"></span>* "We should just focus on the existing haproxy driver."</div>
</div>
</blockquote>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div>* "We should definitely collaborate to make a new driver as a community."</div>
</div>
</blockquote>
<div>New driver(s) that use haproxy is totally fine, i think. </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div><span style="white-space:pre-wrap"></span>* "I don't think a Management API is necessary."</div>
</div>
</blockquote>
<div>It really is! </div>
</div>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">Good, I think so too. :)</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div>
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div><span style="white-space:pre-wrap"></span>* "This is definitely what I was thinking we'd need to do."</div>
</div>
</blockquote>
<div> </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div style="word-wrap:break-word;color:rgb(0,0,0);font-size:14px;font-family:Calibri,sans-serif">
<div> Anything specific implementation details I've mentioned are intended be taken as one possible example, not as a well thought out proposal. I am, as one might say, "speaking my mind". My hope is that some of this will simmer on the general subconscious.
I'd like to hear what the general consensus is on these topics, because these are some of the assumptions I've been operating under during the rest of our discussions, and if they're invalid, I may need to rebase my view on the API discussion as a whole.</div>
<div><br>
</div>
<div><span style="white-space:pre-wrap"></span>Thanks ya'll, I'm looking forward to getting some additional viewpoints!</div>
<div><span style="white-space:pre-wrap"></span>--Adam Harwell (rm_work)</div>
</div>
<br>
<br>
</blockquote>
<div><br>
</div>
<div>Thanks,</div>
<div>Eugene. </div>
</div>
<br>
</div>
</div>
</div>
</div>
</span>
</body>
</html>