<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
<div><font color="#ff0000"><font face="Calibri,sans-serif">My comments in red (I apologize for not being able to get my mail client to indent properly, I know this coloring doesn't show up right in some cases, but I don't know what else I can do).</font></font></div>
<div style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<br>
</div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Ed Hall <<a href="mailto:edhall@yahoo-inc.com">edhall@yahoo-inc.com</a>><br>
<span style="font-weight:bold">Reply-To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Date: </span>Friday, May 2, 2014 4:23 PM<br>
<span style="font-weight:bold">To: </span>"OpenStack Development Mailing List (not for usage questions)" <<a href="mailto:openstack-dev@lists.openstack.org">openstack-dev@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [openstack-dev] [Neutron][LBaaS] Fulfilling Operator Requirements: Driver / Management API<br>
</div>
<div><br>
</div>
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
Hi all,
<div><br>
</div>
<div>At Yahoo, load balancing is heavily used throughout our stack for both HA and</div>
<div>load distribution, even within the OpenStack control plane itself. This involves a</div>
<div>variety of technologies, depending on scale and other requirements. For large</div>
<div>scale + L7 we use Apache Traffic Server, while L3DSR is the mainstay of the</div>
<div>highest bandwidth applications and a variety of technologies are used for simple</div>
<div>HA and lighter loads. </div>
</div>
</div>
</span>
<div><font color="#ff0000">Yeah, we're looking at HAProxy primarily right now, but I don't think that's technically set in stone (it's just quite likely). I definitely see the merit in investigating other options, and the possibility that some providers might
prefer one over the other, or some combination of many different solutions.</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
<div>Each of these technologies has its own special operational requirements, and although</div>
<div>a single well-abstracted tenant-facing API to control all of them is much to be desired,</div>
<div>there can be no such luck for operators. A major concern for us is insuring that when a</div>
<div>tenant* has an operational issue they can communicate needs and concerns with</div>
<div>operators quickly and effectively. This means that any operator API must “speak the</div>
<div>same language” as the user API while exposing the necessary information and controls</div>
<div>for the underlying technology.</div>
</div>
</div>
</span>
<div><font color="#ff0000">Yeah, what we're really talking about for a management API would be a very basic contract for the most basic and important features, like diagnostic data retrieval, or remediation commands like migration or rebuilding, to recover
from DDoS or failures. There is obviously a lot to discuss here, about what would be included, and if it's even feasible to do this. I'd like to see some preliminary ideas about what specifically we would need, and what might be difficult to accomplish with
a single Management API. I'm even open to the possibility that there might be no agreement at all, and operators would be left to implement their own Management API for their specific environment — though that would be the worst case scenario, I think. We
just need to talk about it.</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div><br>
</div>
<div>*In this case a “tenant” might represent a publicly-exposed URL with tens of millions of</div>
<div>users or an unexposed service which could impact several such web destinations.</div>
<div><br>
</div>
<div> -Ed</div>
<div><br>
</div>
<div><br>
</div>
<div>
<div>
<div>On May 2, 2014, at 9:34 AM, Eichberger, German <<a href="mailto:german.eichberger@hp.com">german.eichberger@hp.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<blockquote type="cite">
<div lang="EN-US" link="blue" vlink="purple" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div class="WordSection1" style="page: WordSection1;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Hi Stephen + Adam,<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thanks Stephen and Adam for starting this discussion. I also see several different drivers. We at HP indeed use a pool of software load balancing appliances to replace
any failing one. However, we are also interested in a model where we have load balancers in hot standby…<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">My hope with this effort is that we can somehow reuse the haproxy implementation and deploy it different ways depending on the necessary scalability, availability needs.
Akin to creating a strategy which deploys the same haproxy control layer in a pool, on nova vm, etc.</span></div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">So, somehow separate the "provider" (HAProxy in the current design) from the "driver" (which is really the Orchestration Manager / Scheduler) so they could be mixed and matched? VM driver + HAProxy backend, LXC driver + HAProxy backend,
VM driver + nginx, etc? That is something I was churning in the back of my head, but I'm not sure if that's a crazy endeavor to attempt. I love the idea though, so if you have an idea about implementation, I'm listening!</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>
<div>
<blockquote type="cite">
<div lang="EN-US" link="blue" vlink="purple" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div class="WordSection1" style="page: WordSection1;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">German<o:p></o:p></span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div>
<div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;">
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span>Stephen Balukoff [<a href="mailto:sbalukoff@bluebox.net">mailto:sbalukoff@bluebox.net</a>]<span class="Apple-converted-space"> </span><br>
<b>Sent:</b><span class="Apple-converted-space"> </span>Thursday, May 01, 2014 7:44 PM<br>
<b>To:</b><span class="Apple-converted-space"> </span>OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b><span class="Apple-converted-space"> </span>Re: [openstack-dev] [Neutron][LBaaS] Fulfilling Operator Requirements: Driver / Management API<o:p></o:p></span></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
Hi Adam,<o:p></o:p></div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
Thank you very much for starting this discussion! In answer do your questions from my perspective:<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
1. I think that it makes sense to start at least one new driver that focuses on running software virtual appliances on Nova nodes (the NovaHA you referred to above). The existing haproxy driver should not go away as I think it solves problems for small to medium
size deployments, and does well for setting up, for example, a 'development' or 'QA' load balancer that won't need to scale, but needs to duplicate much of the functionality of the production load balancer(s).</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">Full agreement here.</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>
<div>
<blockquote type="cite">
<div lang="EN-US" link="blue" vlink="purple" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div class="WordSection1" style="page: WordSection1;">
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
On this note, we may want to actually create several different drivers depending on the appliance model that operators are using. From the discussion about HA that I started a couple weeks ago, it sounds like HP is using an HA model that concentrates on pulling
additional instances from a waiting pool. The stingray solution you're using sounds like "raid 5" redundancy for load balancing. And what we've been using is more like "raid 1" redundancy.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
It probably makes sense to collaborate on a new driver and model if we agree on the topologies we want to support at our individual organizations. Even if we can't agree on this, it still makes sense for us to collaborate on determining that "basic set of operator
features" that all drivers should support, from an operator perspective.</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">I'm very aware of the possibility that every operator may decide to make their own driver specific to their environment, and I guess that could be ok, but I agree that at least deciding on a base set of features would be good in that
case.</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>
<div>
<blockquote type="cite">
<div lang="EN-US" link="blue" vlink="purple" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div class="WordSection1" style="page: WordSection1;">
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
I think a management API is necessary-- operators and their support personnel need to be able to troubleshoot problems down to the device level, and I think it makes sense to do this through an OpenStack interface if possible. In order to accommodate each
vendor's differences here, though, this may only be possible if we allow for different drivers to expose "operator controls" in their own way.</div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">Yeah, it might essentially be up to the driver to tell the management API what it should be exposing. I'm not actually sure what this would look like yet, to be perfectly honest, but I'll be thinking about it over the next few weeks.</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>
<div>
<blockquote type="cite">
<div lang="EN-US" link="blue" vlink="purple" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div class="WordSection1" style="page: WordSection1;">
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
I do not think any of this should be exposed to the user API we have been discussing.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
I think it's going to be important to come to some kind of agreement on the user API and object model changes before it's going to be possible to start to really talk about how to do the management API.<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
I am completely on board with this! As I have said in a couple other places on this list, Blue Box actually wrote our own software appliance based load balancing system based on HAProxy, stunnel, corosync/pacemaker, and a series of glue scripts (mostly written
in perl, ruby, and shell) that provide a "back-end API" and whatnot. We've actually done this (almost) from scratch twice now, and have plans and some work underway to do it a third time-- this time to be compatible with OpenStack (and specifically the Neutron
LBaaS API, hopefully as a driver for the same). This will be completely open source, and hopefully compliant with OpenStack standards (equivalent licensing, everything written in python, etc.) So far, I've only had time to port over the back-end API and a
couple design docs, but if you want to see what we have in mind, here's the documentation on this so far:<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<a href="https://github.com/blueboxgroup/octavia/" style="color: purple; text-decoration: underline;">https://github.com/blueboxgroup/octavia/</a><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
In particular, probably the theory of operation document will give you the best overview of how it works:<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<a href="https://github.com/blueboxgroup/octavia/blob/master/doc/theory-of-operation.md" style="color: purple; text-decoration: underline;">https://github.com/blueboxgroup/octavia/blob/master/doc/theory-of-operation.md</a><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
And the virtual appliance API (as it was two months ago. Some things will definitely change based on discussions of the last couple months):<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<a href="https://github.com/blueboxgroup/octavia/blob/master/doc/virtual-appliance-api.md" style="color: purple; text-decoration: underline;">https://github.com/blueboxgroup/octavia/blob/master/doc/virtual-appliance-api.md</a><o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
</div>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
</span>
<div><font color="#ff0000">I'm making a note to look these over soon — I haven't had a chance to look yet, trying to get through a stack of responses! Thanks for providing these though!</font></div>
<span id="OLK_SRC_BODY_SECTION" style="font-family: Calibri, sans-serif; font-size: 14px; color: rgb(0, 0, 0); ">
<div>
<div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;">
<div>
<div>
<blockquote type="cite">
<div lang="EN-US" link="blue" vlink="purple" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">
<div class="WordSection1" style="page: WordSection1;">
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
Thanks,<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
Stephen<o:p></o:p></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
</div>
<div>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></p>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
On Thu, May 1, 2014 at 2:33 PM, Adam Harwell <<a href="mailto:adam.harwell@rackspace.com" target="_blank" style="color: purple; text-decoration: underline;">adam.harwell@rackspace.com</a>> wrote:<o:p></o:p></div>
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">I am sending this now to gauge interest and get feedback on what I see as an impending necessity — updating the existing "haproxy" driver, replacing it, or both. Though we're not there yet,
it is probably best to at least start the discussion now, to hopefully limit some fragmentation that may be starting around this concept already.<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">To begin with, I should probably define some terms. Following is a list of the major things I'll be referencing and what I mean by them, since I would like to avoid ambiguity as much as possible.<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">----------------------------------<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">---- Glossary<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">----------------------------------<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<b><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">HAProxy</span></b><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">: This references two things currently, and I feel this is a source of some misunderstanding. When I
refer to HAProxy (capitalized), I will be referring to the official software package (found here: <a href="http://haproxy.1wt.eu/" target="_blank" style="color: purple; text-decoration: underline;">http://haproxy.1wt.eu/</a> ), and when I refer to "haproxy"
(lowercase, and in quotes) I will be referring to the neutron-lbaas driver (found here: <a href="https://github.com/openstack/neutron/tree/master/neutron/services/loadbalancer/drivers/haproxy" target="_blank" style="color: purple; text-decoration: underline;">https://github.com/openstack/neutron/tree/master/neutron/services/loadbalancer/drivers/haproxy</a> ).
The fact that the neutron-lbaas driver is named directly after the software package seems very unfortunate, and while it is not directly in the scope of what I'd like to discuss here, I would love to see it changed to more accurately reflect what it is --
one specific driver implementation that coincidentally uses HAProxy as a backend. More on this later.<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<b><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">Operator Requirements</span></b><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">: The requirements that can be found on the wiki page here: <a href="https://wiki.openstack.org/wiki/Neutron/LBaaS/requirements#Operator_Requirements" target="_blank" style="color: purple; text-decoration: underline;">https://wiki.openstack.org/wiki/Neutron/LBaaS/requirements#Operator_Requirements</a> and
focusing on (but not limited to) the following list:<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* Scalability<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* DDoS Mitigation<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* Diagnostics<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* Logging and Alerting<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* Recoverability<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* High Availability (this is in the User Requirements section, but will be largely up to the operator to handle, so I would include it when discussing Operator Requirements)<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<b><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">Management API</span></b><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">: A restricted API containing resources that Cloud Operators could access, including most of
the list of Operator Requirements (above).<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<b><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">Load Balancer (LB)</span></b><span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">: I use this term very generically — essentially a logical entity that represents one "use
case". As used in the sentence: "I have a Load Balancer in front of my website." or "The Load Balancer I set up to offload SSL Decryption is lowering my CPU load nicely."<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">----------------------------------<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">---- Overview<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">----------------------------------<o:p></o:p></span></div>
</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">What we've all been discussing for the past month or two (the API, Object Model, etc) is being directly driven by the User and Operator Requirements that have somewhat recently been enumerated
(many thanks to everyone who has contributed to that discussion!). With that in mind, it is hopefully apparent that the current API proposals don't directly address many (or really, any) of the Operator requirements! Where in either of our API proposals are
logging, high availability, scalability, DDoS mitigation, etc? I believe the answer is that none of these things can possibly be handled by the API, but are really implementation details at the driver level. Radware, NetScaler, Stingray, F5 and HAProxy of
any flavour would all have very different ways of handling these things (these are just some of the possible backends I can think of). At the end of the day, what we really have are the requirements for a driver, which may or may not use HAProxy, that we hope
will satisfy all of our concerns. That said, we may also want to have some form of "Management API" to expose these features in a common way.<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">In this case, we really need to discuss two things: <o:p></o:p></span></div>
</div>
<ol start="1" type="1" style="margin-bottom: 0in;">
<li class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">Whether to update the existing "haproxy" driver to accommodate these Operator Requirements, or whether to start from scratch with a new driver (possibly both).<o:p></o:p></span></li><li class="MsoNormal" style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">How to expose these Operator features at the (Management?) API level.<o:p></o:p></span></li></ol>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">----------------------------------<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">---- 1) Driver<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">----------------------------------<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">I believe the current "haproxy" driver serves a very specific purpose, and while it will need some incremental updates, it would be in the best interest of the community to also create and
maintain a new driver (which it sounds like several groups have already begun work on — ack!) that could support a different approach. For instance, the current "haproxy" driver is implemented by initializing HAProxy processes on a set of shared hosts, whereas
there has been some momentum behind creating individual Virtual Machines (via Nova) for each Load Balancer created, similar to Libra's approach. Alternatively, we could use LXC or a similar technology to more effectively isolate LBs and assuage concerns about
tenant cross-talk (real or imaginary, this has been an issue for some customers). Either way, we'd probably need a brand new driver, to avoid breaking backwards compatibility with the existing driver (which does work perfectly fine in many cases). In fact,
it's possible that when we begin discussing this as a broader community, we might decide to create more than one additional driver (depending on which approaches people want to use and what features are most important to them). The only concern I have about
that outcome is the necessary amount of code-reuse, and whether it would be possible to share certain aspects of these drivers without too much copy/pasting.<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">An example of one possible new driver could be the following (just off the top of my head):<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* Use a pair of new Nova VMs for each LB (Scalability), configured to use a Shared IP (High Availability).<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* Log to Swift / Ceilometer (Logging / Alerting / Metering).<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* Provide calls that could be exposed via a Management API to show low level diagnostic details (Diagnostics).<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* Provide calls that could be exposed via a Management API to allow syncing/reloading existing LBs or moving them across clusters (Recoverability, DDoS Mitigation).<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">This new driver would be named to reflect what features it provides, or at least given a unique name that can be referenced without confusion (something like "OpenHA" or "NovaHA" would work
if that's not taken).<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">----------------------------------<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">---- 2) Management API<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">----------------------------------<o:p></o:p></span></div>
</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">Going forward, it should then be required (can we enforce this?) that any mainline driver include support for calls to handle these named Operator Requirements, for example: obtaining logs
(or log locations?), diagnostic information, and admin type actions including rebuilding or migrating LB instances. So far we haven't really talked about any of these features in depth, though I believe the general need for a Management API was alluded to
on several occasions. Should we shelve this discussion until after we have the User API specification locked down? Should we begin defining a contract for this Management API at the summit, since it would be the main gateway to the Operator Requirements that
we have all been stressing recently? <o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">----------------------------------<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">---- Summary<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">----------------------------------<o:p></o:p></span></div>
</div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">I would apologize for not having much concrete specification here, but I think it is better to validate my basic assumptions first, before jumping deeper down this rabbit hole. The type of
comments I'm hoping to prompt are along the lines of:<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* "We should just focus on the existing haproxy driver."<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* "We should definitely collaborate to make a new driver as a community."<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* "I don't think a Management API is necessary."<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">* "This is definitely what I was thinking we'd need to do."<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> Anything specific implementation details I've mentioned are intended be taken as one possible example, not as a well thought out proposal. I am, as one might say, "speaking my mind". My hope
is that some of this will simmer on the general subconscious. I'd like to hear what the general consensus is on these topics, because these are some of the assumptions I've been operating under during the rest of our discussions, and if they're invalid, I
may need to rebase my view on the API discussion as a whole.<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; "> </span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">Thanks ya'll, I'm looking forward to getting some additional viewpoints!<o:p></o:p></span></div>
</div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<span style="font-size: 10.5pt; font-family: Calibri, sans-serif; ">--Adam Harwell (rm_work)<o:p></o:p></span></div>
</div>
</div>
<p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" style="color: purple; text-decoration: underline;">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank" style="color: purple; text-decoration: underline;">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><o:p></o:p></p>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<br>
<br clear="all">
<o:p></o:p></div>
<div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
<o:p> </o:p></div>
</div>
<div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;">
--<span class="Apple-converted-space"> </span><br>
Stephen Balukoff<span class="Apple-converted-space"> </span><br>
Blue Box Group, LLC<span class="Apple-converted-space"> </span><br>
(800)613-4305 x807<o:p></o:p></div>
</div>
</div>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</span>
</body>
</html>