<div dir="ltr">Hi German,<div><br></div><div>Responses in-line:</div><div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Sep 5, 2014 at 2:31 PM, Eichberger, German <span dir="ltr"><<a href="mailto:german.eichberger@hp.com" target="_blank">german.eichberger@hp.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Hi Stephen,<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">I think this is a good discussion to have and will make it more clear why we chose a specific design. I also believe by having this discussion we will make
the design stronger. I am still a little bit confused what the driver/controller/amphora agent roles are. In my driver-less design we don’t have to worry about the driver which most likely in haproxy’s case will be split to some degree between controller
and amphora device.</span></p></div></div></blockquote><div><br></div><div>Yep, I agree that a good technical debate like this can help both to get many people's points of view and can help determine the technical merit of one design over another. I appreciate your vigorous participation in this process. :)</div><div><br></div><div>So, the purpose of the controller / driver / amphora and the responsibilities they have are somewhat laid out in the Octavia v0.5 component design document, but it's also possible that there weren't enough specifics in that document to answer the concerns brought up in this thread. So, to that end in my mind, I see things like the following:</div><div><br></div><div>The controller:</div><div>* Is responsible for concerns of the Octavia system as a whole, including the intelligence around interfacing with the networking, virtualization, and other layers necessary to set up the amphorae on the network and getting them configured.</div><div>* Will rarely, if ever, talk directly to the end-systems or -services (like Neutron, Nova, etc.). Instead it goes through a "clean" driver interface for each of these.</div><div>* The controller has direct access to the database where state is stored.</div><div>* Must load at least one driver, may load several drivers and choose between them based on configuration logic (ex. flavors, config file, etc.)</div><div><br></div><div>The driver:</div><div>* Handles all communication to or from the amphorae</div><div>* Is loaded by the controller (ie. its interface with the controller is a base class, associated methods, etc. It's objects and code, not a RESTful API.)</div><div>* Speaks amphora-specific protocols on the back-end. In the case of the reference "haproxy" amphora, this will most likely be in the form of a RESTful API with an agent on the amp, as well as (probably) HMAC-signed UDP health, status and stats messages from the amp to the driver.</div><div><br></div><div>The amphora:</div><div>* Does the actual load balancing</div><div>* Is managed by the controller through the driver.</div><div>* Should be as "dumb" as possible.</div><div>* Comes in different types, based on the software in the amphora image. (Though all amps of a given type should be managed by the same driver.) Types might include "haproxy," "nginx," "haproxy + nginx," "3rd party vendor X," etc.</div><div>* Should never have direct access to the Octavia database, and therefore attempt to be as stateless as possible, as far as configuration is concerned.</div><div><br></div><div>To be honest, our current product does not have a "driver" layer per se, since we only interface with one type of back-end. However, we still render our haproxy configs in the controller. :)</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">So let’s try to sum up what we want a controller to do:<u></u><u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>-<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Provision new amphora devices<u></u><u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>-<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Monitor/Manage health<u></u><u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>-<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Gather stats<u></u><u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>-<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Manage/Perform configuration changes<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The driver as described would be:<u></u><u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>-<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Render configuration changes in a specific format, e.g. haproxy<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Amphora Device:<u></u><u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>-<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Communicate with the driver/controller to make things happen<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">So as Doug pointed out I can make a very thin driver which basically passes everything through to the Amphora Device or on the other hand of the spectrum I
can make a very thick driver which manages all aspects from the amphora life cycle to whatever (aka kitchen sink). I know we are going for uttermost flexibility but I believe:</span></p></div></div></blockquote><div><br></div><div>So, I'm not sure it's fair to characterize the driver I'm suggesting as "very thick." If you get right down to it, I'm pretty sure the only major thing we disagree on here is where the haproxy configuration is rendered: Just before it's sent over the wire to the amphora, or just after it's JSON-equivalent is received over the wire from the controller.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>-<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">With building an haproxy centric controller we don’t really know which things should be controller/which thing should be driver. So my shortcut is
not to build a driver at all </span><span style="font-size:11.0pt;font-family:Wingdings;color:#1f497d">J</span></p></div></div></blockquote><div>So, I've become more convinced that having a driver layer there is going to be important if we want to support 3rd party vendors creating their own amphorae at all (which I think we do). It's also going to be important if we want to be able to support other versions of open-source amphorae (or experimental versions prior to pushing out to a wider user-base, etc.)</div><div><br></div><div>Also, I think: Making ourselves use a driver here also helps keep interfaces clean. This helps us avoid spaghetti code and makes things more maintainable in the long run.</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><span>-<span style="font:7.0pt "Times New Roman"">
</span></span></span><u></u><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">The more flexibility increases complexity and makes it confusing for people to develop components. Should this concern go into the controller, the
driver, or the amphora VM? Two of them? Three of them? Limiting choices makes it simpler to achieve that.</span></p></div></div></blockquote><div>"Centralize intelligence / decentralize workload." There will often be multiple ways we can solve certain problems, but if we try to follow this mantra, and use clean interfaces between components, it starts to become more clear which code strategies we should be following. Yes, it's sometimes hard to know the right way to do things-- which is why we end up having these wonderful debates. ;) But I don't think the answer is "this is hard, let's just lump everything together."</div><div><br></div><div>Also, rule of thumb (perhaps not stated in our constitution... yet): Try to architect things so the most frequently deployed elements see the fewest changes. (This is actually related to the "centralize intelligence / decentralize workload" mantra in a round-about way: Central intelligence elements will be both fewer in number and more frequently changed than "dumb" workload components.) This makes managing change for large deployments easier. (Again, it's both easier and less risky to update 100 controllers versus 10,000+ amphorae.)</div><div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div lang="EN-US" link="blue" vlink="purple"><div><p><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">HPs worry is that by creating the potential to run multiple (version of drivers) drivers, on multiple versions of controllers, on multiple versions of amphora
devices creates a headache for testing. For example does the version 4.1 haproxy driver work with the cersion 4.2 controller on an 4.0 amphora device? Which compatibility matrix do we need to build/test? Limiting one driver to one controller can help with
making that manageable.</span></p></div></div></blockquote><div><br></div><div>Ok, so, I think this is possibly where part of our misunderstanding comes from. I realize above that I said a single driver could talk to multiple versions of back-end amphorae via a couple methods, but let's ignore that for a minute and assume that we only test / assume drivers will be speaking with the latest version of the amphorae to which they correspond.</div><div><br></div><div>I should probably clarify something that I've been assuming but may not be obvious: I'm assuming that the "version" of the amphorae (drawn mostly from the version of the glue scripts, agent, and other code we write which lives on the amphora) is numbered separately and moves at a different rate than the version of the driver. Think of this like the version of the firmware and version of the driver used with your printer. Sometimes a major bugfix entails updating both the firmware and driver. However, it's also common for a bugfix / feature enhancement to involve only updating the printer driver version and not the printer firmware.</div><div><br></div><div>What I'm getting at here is that if we're doing the configuration rendering in the driver and not on the amphora, there will be some bugfixes / feature enhancements which only entail updating the driver because <i>there are literally no changes that need to be made to the amphora for the bugfix / feature</i>.</div><div><br></div><div>Does this actually happen? Yes! To give a concrete example drawn from our product history: On our existing load balancer product, which is powered by stunnel + haproxy a new OpenSSL vulnerability was discovered, the fix for which was to add a line to the stunnel configuration disabling a certain kind of SSL negotiation. Since we were rendering configurations centrally on our controllers, all we needed to do was update the configuration template on our controller and push out new configs for anyone using SSL termination. Took literally 10 minutes to implement once we understood the problem, and we didn't have to touch or otherwise update the software or scripts running on our appliances at all.</div><div><br></div><div>It's even easier for L7 feature enhancements: You don't even have to push anything out to the amphora, just update the controller / driver to expose the new feature and users can then start using it at will.</div><div><br></div><div>Are all feature enhancements / bugfixes this easy? No! How do you tell the difference between which changes are major and minor? Anything which touches the code running on the amphora is "major" (ie. like a firmware update). Anything which only touches the controller / driver is "minor" (ie. like a driver update).</div><div><br></div><div>It seems strange to me that we'd force even minor changes to configurations to be "major" updates for the sake of sending JSON-which-will-immediately-be-turned-into-haproxy.cfg over the wire instead of just the haproxy.cfg. :/</div><div><br></div><div>So with that in mind: Please understand that your model and mine do not have to differ in the slightest when it comes to how to manage 'major' updates, whether that be running a different driver / controller for the new amphora version (Ick!), or doing on-demand lazy upgrades of amphora as the driver discovers old, incompatible-versioned amphora it needs to update (probably smoothest way to handle this, possibly as a default action of the option 2 I mentioned above), or whether we force all amphora to be updated as soon as possible after a controller update (most risky and probably not the best way to handle this). We've yet to define exactly how this workflow should be handled, but it's actually somewhat secondary to the problem of where to render the configs. (Maybe we should have a conversation about this in another thread?)</div><div><br></div><div>And in any case, I'm not seeing a need to ensure the driver works with anything but the latest amphora image version to which it corresponds (again, keeping in mind that amphora image and driver should be allowed to change at different rates and are therefore versioned separately). :/ This is especially the case if we define the default action to be taken upon a failure to push out a new config to be to check the version of the amphora and upgrade as necessary (ie. lazy upgrading)...</div><div><br></div><div>Also, not that we can't revisit this of course: But the v0.5 component design entailing a "VM Driver" already went through gerrit review and was approved (by yourself even!) This discussion was originally about where to render the haproxy configs, but it really seems like y'all are against the idea of having an amphora driver interface at all. :/</div><div><br></div><div>Stephen</div><div><br></div></div><br clear="all"><div><br></div>-- <br><span></span>Stephen Balukoff
<br>Blue Box Group, LLC
<br><a href="tel:%28800%29613-4305%20x807" value="+18006134305" target="_blank">(800)613-4305 x807</a>
</div></div>