<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 12 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0in;
margin-right:0in;
margin-bottom:0in;
margin-left:.5in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:372729740;
mso-list-template-ids:1697432012;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1
{mso-list-id:530191709;
mso-list-type:hybrid;
mso-list-template-ids:-1641929000 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.0in;
text-indent:-.25in;
font-family:Symbol;}
@list l2
{mso-list-id:1330673511;
mso-list-template-ids:189576872;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:.5in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:1.0in;
mso-level-number-position:left;
text-indent:-.25in;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l3
{mso-list-id:1601378369;
mso-list-type:hybrid;
mso-list-template-ids:965791960 67698703 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l3:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:1.0in;
text-indent:-.25in;}
@list l4
{mso-list-id:1814909091;
mso-list-type:hybrid;
mso-list-template-ids:-1457471144 67698689 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l4:level1
{mso-level-number-format:bullet;
mso-level-text:;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-.25in;
font-family:Symbol;}
ol
{margin-bottom:0in;}
ul
{margin-bottom:0in;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Hi,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Stephen visited us today (the joy of spending some days in Seattle</span><span style="font-size:11.0pt;font-family:Wingdings;color:#1F497D">J</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">)
and we discussed that further (and sorry for using VM – not sure what won):<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l3 level1 lfo5">
<![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">We will only support one driver per controller, e.g. if you upgrade a driver you deploy a new controller with the new driver and either make him take
over existing VMs (minor change) or spin up new ones (major change) but keep the “old” controller in place until it doesn’t serve any VMs any longer<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l3 level1 lfo5">
<![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">If we render configuration files on the VM we only support one upgrade model (replacing the VM) which might simplify development as opposed to the
driver model where we need to write code to push out configuration changes to all VMs for minor changes + write code to failover VMs for major changes<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l3 level1 lfo5">
<![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">I am afraid that half baked drivers will break the controller and I feel it’s easier to shoot VMs with half baked renderers than the controllers.
<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l3 level1 lfo5">
<![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">4.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The main advantage by using an Octavia format to talk to VMs is that we can mix and match VMs with different properties (e.g. nginx, haproxy) on the
same controller because the implementation detail (which file to render) is hidden<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:1.0in;text-indent:-.25in;mso-list:l3 level1 lfo5">
<![if !supportLists]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><span style="mso-list:Ignore">5.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">The major difference in The API between Stephen and me would be that I would send json files which get rendered on the VM into a haproxy file whereas
he would send an haproxy file. We still need to develop an interface on the VM to report stats and health in Octavia format. It is conceivable with Stephen’s design that drivers would exist which would translate stats and health from a proprietary format into
the Octavia one. I am not sure how we would get the proprietary VMs to emit the UDP health packets… In any case a lot of logic could end up in a driver – and fanning that processing out to the VMs might allow for less controllers.
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Overall, if I don’t like to take advantage of the minor update model the main difference between me and Stephen is in the haproxy case to ship json instead
of haproxy config. I understand that the minor update capability is a make or break for Stephen though in my experience configuration changes without other updates are rare (and my experience might not be representative).
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">In any case there certainly is some advantage for appliances which not necessarily speak the Octavia protocol by allowing custom drivers. However, given our
plan to use an Ocatvia UDP package to emit health messages from the VMs to the controller and since controller provision VMs in Nova it might be a better integration point for appliances to have custom controllers. I am just not convinced that a custom driver
is sufficient for all cases –<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">German<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Stephen Balukoff [mailto:sbalukoff@bluebox.net]
<br>
<b>Sent:</b> Wednesday, September 03, 2014 10:01 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> [openstack-dev] [Octavia] Question about where to render haproxy configurations<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Hello!<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">At today's Octavia meeting, one of the questions briefly discussed was where, in the Octavia architecture, haproxy configuration files should be rendered. This discussion went long and we decided to take it to the mailing list for more
thorough discussion and to gather alternate ideas.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Anyway, the two main ideas are to render the configuration in the back-end (haproxy) driver, and push complete configs out to the VMs/containers running the haproxy software via API, or to use the back-end API to push out configuration
updates and have the VMs/containers render them themselves.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I'm in favor of rendering haproxy configs in the driver, and will present arguments in favor of this architecture. I understand German @ HP is the main proponent of rendering the configs on the back-end VMs/containers and will let him speak
to his own points there, and respond to mine below.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Why we should render haproxy configurations within the driver:<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<ul type="disc">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
This is analogous to how other back-end drivers for proprietary virtual machines will do it. This means our reference implementation is easier used as a model for 3rd party vendors who want to use Octavia for managing their own appliance images.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
This keeps the back-end API simpler, as the back-end Octavia VM / container doesn't need to know anything about how to render a configuration, it just needs to know how to run it.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Simpler back-end API means fewer failure scenarios to have to plan for. Either the config pushed out works or it doesn't.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Minor bugfixes, configuration-related security fixes, and minor feature improvements can be done centrally without having to update potentially 10's of thousands of back-end VMs/containers. (The alternative is to have to do this for even the smallest of updates
using the other model.)<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Major bugfixes and changes will still need to touch all back-end VMs/containers, but this is no different than the other model.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
If an operator wants to deliver service using another set of load balancing software (eg. nginx) this can be done either by writing a new driver for a new VM / container image which does this, or by "enhancing" the haproxy driver to be able to render both configs
and updating the image and back-end API to know how to deal with nginx configs. No real advantage or disadvantage here, IMO.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
This better follows the "centralize intelligence / decentralize workload" development philosophy for the project. Putting the rendering logic on the Octavia VMs / containers unnecessarily pushes a fair bit of intelligence out to what should be fairly "dumb"
workload-handling instances.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
It is far simpler for the driver, being part of the controller, to pull in extra data needed for a complete config (eg. TLS certificates) than for the back-end VM / container to do this directly. The back-end VM / container should not have privileged resource
access to get this kind of information directly since that could easily lead to security problems with the whole of OpenStack.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Since the Octavia VM / container image is "dumber" it's also simpler to write, test, and troubleshoot.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
Spinning up new instances is far simpler because the new back-end just gets a complete config from the driver / controller and runs with it. No need to try to guess state.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l0 level1 lfo1">
This design is, overall, more idempotent. (Something went wrong with that update to a pool member configuration? No worries-- the next update will get that change, too.)<o:p></o:p></li></ul>
<div>
<p class="MsoNormal">The only down-side I see to this is:<o:p></o:p></p>
</div>
</div>
<div>
<ul type="disc">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l2 level1 lfo2">
We will need to keep track of which VMs / containers are running which images, and the driver will need to know how to speak to them appropriately. However, I think this is a false down-side, since:<o:p></o:p></li></ul>
<ul type="disc">
<ul type="circle">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l2 level2 lfo2">
In a large installation, it's unreasonable to expect the potentially 10's of thousands of VMs to be running the exact same version of software. Forcing this requirement makes things very inflexible and a lot more risky for the operator, as far as maintenance
and upgrade schedules are concerned.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l2 level2 lfo2">
The above point is to say, having the ability to run with different versions of back-end VMs / containers is actually a good thing, and probably a requirement for large installations anyway.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l2 level2 lfo2">
If you really want the model where you have to update all your back-end VM / container images with each update, you can still do that with this topology. Again, with the alternate topology suggested you are forced to do that even for very minor updates.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l2 level2 lfo2">
API versioning (even for the back-end) is also a requirement of this project's philosophy. And if you're going to have a versioned API:<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l2 level2 lfo2">
It's pretty simple to write a 'status' API command which among other things lists the API version in use, as well as the versions of all the major utilities installed (eg. haproxy, openssl, nginx, etc.) It's then up to the controller to render configurations
appropriate to the software versions installed and/or to force an image to be retired in favor of spinning up a new one and/or return an error to the user trying to push out a feature request for an image that doesn't support it. Lots of options, all of which
can be intelligently handled within the driver / controller.<o:p></o:p></li></ul>
</ul>
</div>
<div>
<p class="MsoNormal">Thoughts?<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Stephen<o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <br>
Stephen Balukoff <br>
Blue Box Group, LLC <br>
(800)613-4305 x807 <o:p></o:p></p>
</div>
</div>
</div>
</body>
</html>