Open Stack

Hi Michael!

Just to give others some background on this: The current proposal (by me)
is to have each Listener object, (as defined in the Neutron LBaaS v2 code
base) correspond with one haproxy process on the Octavia VM in the
currently proposed Octavia design document. Michael's proposal is to have
each Loadbalancer object correspond with one haproxy process (which would
have multiple front-end sections in it to service each Listener on the
Loadbalancer).

Anyway, we thought it would be useful to discuss this on the mailing list
so that we could give others a chance to register their opinions, and
justify the same.

That being said, my responses to your points are in-line below, followed by
my reasoning for wanting 1 haproxy process = 1 listener in the
implementation:


On Wed, Aug 20, 2014 at 12:34 PM, Michael Johnson <johnsomor at gmail.com>
wrote:

> I am proposing that Octavia should support deployment models that
> enable multiple listeners to be configured inside the HAProxy
> instance.
>
> The model I am proposing is:
>
> 1. One or more VIP per Octavia VM (propose one VIP in 0.5 release)
> 2. One or more HAProxy instance per Octavia VM
> 3. One or more listeners on each HAProxy instance
>

This is where our proposals differ. I propose 1 listener per haproxy
instance.


> 4. Zero or more pools per listener (shared pools should be supported
> as a configuration render optimization, but propose support post 0.5
> release)
> 5. One or more members per pool
>

I would also propose zero or more members per pool. A pool with zero
members in it has been (is being) used by some of our customers to
blacklist certain client IP addresses. These customers want to respond to
the blacklisted IPs with an error 503 page (which can be done by haproxy)
instead of simply not responding to packets (if the blacklist were done at
the firewall).


> This provides flexibility to the operator to support multiple
> deployment models,  including active-active and hot standby Octavia
> VMs.  Without the flexibility to have multiple listeners per HAProxy
> instance we are limiting the operators deployment models.
>

I don't think your conclusion follows logically from your justification
here.  Specifically, active-active and hot standby Octavia VMs are equally
supported by a one-process-per-listener model. Further, for reasons I'll
get into below, I think the one-process-per-listener model actually
provides more flexibility to the operators and users in how services are
deployed. Therefore, the conclusion I come to is the exact opposite of
yours: By insisting that all listeners on a given loadbalancer share a
single haproxy process, we actually limit flexibility in deployment models
(as well as introduce some potential operational problems we otherwise
wouldn't encounter).


I am advocating for multiple listeners per HAProxy instance because I
> think it provides the following advantages:
>
> 1. It reduces memory overhead due to running multiple HAProxy
> instances on one Octavia VM.  Since the Octavia constitution states
> that Octavia is for large operators where this memory overhead could
> have a financial impact we should allow alternate deployment options.

2. It reduces host CPU overhead due to reduced context switching that
> would occur between HAProxy instances.  HAProxy is event driven and
> will mostly be idle waiting for traffic, where multiple instances of
> HAProxy will require context switching between the processes which
> increases the VM’s CPU load.  Since the Octavia constitution states
> that we are designing for large operators, anything we can do to
> reduce the host CPU load reduces the operator’s costs.
>

So these two points might be the only compelling reason I see to follow the
approach you suggest. However, I would like to see the savings here
justified via benchmarks. If benchmarks don't show a significant difference
in performance running multiple haproxy instances to service different
listeners over running a single haproxy instance servicing the same
listeners, then I don't think these points are sufficient justification. I
understand your team (HP) is going to be working on these, hopefully in
time for next week's Octavia meeting.

Please also understand that memory and CPU usage are just two factors in
determining overall cost of the solution. Slowing progress on delivering
features, increasing faults and other problems by having a more complicated
configuration, and making problems more difficult to isolate and
troubleshoot are also factors that affect the cost of a solution (though
they aren't as easy to quantify). Therefore it does not necessarily
logically follow that "anything" we can do to reduce CPU load decreases the
operator's costs.

Keep in mind, also, that for large operators the scaling strategy is to
ensure services can be scaled horizontally (meaning the CPU / memory
footprint of a single process isn't very important for a large load that
will be spread across many machines anyway), and any costs for delivering
the service are usually passed on to the user in a more-or-less direct
fashion.  If a simpler solution is 5% less efficient than the more
complicated one, but experiences 50% fewer problems...  as both an operator
and a user I'll pick the simpler solution.


> 3. Hosting multiple HAProxy instances on one Octavia VM will increase
> the load balancer build time because multiple configuration files,
> start/stop scripts, health monitors, and HAProxy Unix sockets will
> have to be created.  This could significantly impact operator
> topologies that use hot standby Octavia VMs for failover.
>

I disagree. I think you're making several assumptions about how Octavia VMs
will be configured once they are up and running which are not true.
Specifically:

* Yes, one-haproxy-per-listener does require multiple configuration files.
These typically take times measured in milliseconds to generate and deploy.
This does not significantly impact the time to deploy new services.
* I think you're assuming the start/stop scripts used to control the
haproxy processes will be the ones shipped with standard OS haproxy
packages. This will not be the case with Octavia VMs, and a single set of
scripts for this purpose will work for all haproxy instances on a single VM.
* Health monitors are built into haproxy itself. A multi-listener
loadbalancer configuration using multiple pools will use the same number of
health monitor threads no matter whether this is accomplished using
multiple haproxy instances or just one.
* HAproxy unix sockets are created at approximate the same rate that a
process can create a file. In other words, tens of thousands per second.
Plus using multiple processes just increases the number of unix sockets in
use by the number of processes used (typically one or two in most
configurations). I think you're really scraping for justification for your
idea if you seriously think this is going to be a performance problem. :/

One point:  It's true that if pools are shared between listeners there are
overall fewer healthmonitors that need to be set up for the same group of
back-end servers. But I don't think this is sufficient justification for
sharing between listeners because:

1. In nearly all the configurations in use in our customer environments,
there actually isn't any sharing of pools going on between listeners on a
single 'loadbalancer' (for us 'VIP', but it's the same thing):  Even in
situations where HTTP and HTTPS services point to the same group of
back-end servers, the HTTPS connections are usually being forwarded to a
different port than the HTTP connections, which means these are effectively
separate pools.
2. If protocols mismatch (ex. 'HTTP' listener versus 'TCP' listener versus
non-terminated 'HTTPS' listener), then pool sharing is not an option for
obvious technical reasons.
3. In the very few cases where pools could be shared between listeners it's
almost always between HTTP and TERMINATED_HTTPS. In other words, just two
listeners on a single loadbalancer (not the hypothetical and probably
non-existent 50 listeners on the same loadbalancer pointing at the same
pool).
4. In the very few cases where pools could be shared between listeners,
this means that the number of health checks going to the back-end is
increased by a multiple of the number of haproxy processes servicing all
the listeners. In other words, instead of getting one health-check per 5
seconds, each back-end server gets two health-checks per 5 seconds. If the
back-end server or network infrastructure can't handle this, then there's
something seriously wrong with the application being load balanced or
network infrastructure.



> 4. It reduces network traffic and health monitoring overhead because
> only one HAProxy instance per Octavia VM will need to be monitored.
> This again, saves the operator money and increases the scalability for
> large operators.
>

The health-check of the Octavia VM will happen over a (single) REST API
request-response cycle that will look essentially no different if multiple
haproxy instances are used or if just one is. Also, in your model above, if
a single Octavia VM is being used to service multiple loadbalancers, then
your model would also involve multiple haproxy processes on a single
Octavia VM.  So this point is invalid.


> 5. Multiple listeners per instance allows the sharing of backend pools
> which reduces the amount of health monitoring traffic required to the
> backend servers.  It also has the potential to share SSL certificates
> and keys.
>

I made my point about health monitors above...  But just to put things in
perspective:  A typical health check involves a single, simple HTTP request
response cycle every 5 seconds to each member of a given pool, and usually
entails at most a few kilobytes of data transfer (usually < 1KB). This is
on a system designed to handle gigabits per second of data throughput for
the production load. We're talking many orders of magnitude difference
between the health check traffic and the potential production load. Again,
I think you're really grasping at straws if you think this is a serious
point to consider in justifying your argument.

There's also no reason the one-haproxy-per-listener model can't share SSL
certificates and keys (though this is a fairly rare set-up among our
customer base. Almost always a single SSL certificate+key is going to be
used on one listener).


> 6. It allows customers to think of load balancers (floating IPs) as an
> application service, sharing the fate of multiple listeners and
> providing a consolidated log file.  This also provides a natural
> grouping of services (around the floating IP) for a defined
> performance floor.  With multiple instances per Octavia VM one
> instance could negatively impact all of the other instances which may
> or may not be related to the other floating IP(s).
>

Consolidated log files can be used with multiple haproxy instances as well.

The 'defined performance floor' or grouping of services around a floating
IP (loadbalancer as defined in Neutron LBaaS v2) happens at the Octavia VM
level in either your proposal or mine, so this point is invalid.

With multiple listeners per haproxy instance, any one overly-used listener
(TCP port) will negatively affect the listeners on other ports as well--
 so that's really no different than one haproxy instance per listener,
again, nullifying the point I think you were making.

I'm also not sure why you're bringing up other instances on other floating
IPs.  Keep in mind that 'floating IP' here is the same thing as
'loadbalancer' as defined in the Neutron LBaaS v2 discussion. The model you
propose would also use multiple haproxy instances (one per floating IP),
and if you're worried about a bad neighbor problem on one floating IP
affecting services on another floating IP, you're problem isn't the number
of haproxy instances you're running, it's the number of floating IPs
(loadbalancers) you're allowing per Octavia VM. Our models do not differ at
all with regard to how to schedule loadbalancer deployments to Octavia VMs
at all, therefore... once again this point is invalid.

But there is one difference between your model and mine raised above. You
said that multiple listeners per haproxy process allows them to "sharing
the fate of multiple listeners." I think that you mean that if a problem
happens with Listener A, then Listener B (on the same loadbalancer) is also
affected because it's the same "service."

Said another way, you're saying that in your model there is no fault
isolation between listeners on a given loadbalancer. I think in IRC
discussions you touted this as a feature. I would instead say it's an
anti-feature: Not having fault isolation between listeners is actually a
*bad* thing that is avoided in my model.

I also don't see any reason why having multiple listeners per haproxy
instance versus one listener per haproxy instance would affect the way the
customer thinks about the service at all:  After all, this information is
not (and should not be) exposed to the customer in any way.


> 7. Multiple listeners per instance reduces the number of TCP ports
> used on the Octavia VM, increasing the per-VM scalability.
>

Um...  no it doesn't. If you're serving the same load from the same clients
to the same back-end servers for the same front-end listeners... then all
things being equal, you're using the same number of TCP ports whether this
is accomplished using one haproxy process or multiple haproxy processes.
I'm confused as to why you would expect anything different.



> I don’t want us, by design, to limit the operator flexibility in
> deployment an topologies, especially when it potentially impacts the
> costs for large operators.  Having multiple listeners per HAProxy
> instance is a very common topology in the HAProxy community and I
> don’t think we should block that use case with Octavia deployments.
>

Right... so I think that we would actually be limiting operator flexibility
by insisting that a single haproxy instance be used to service multiple
listeners (ie. I come to the exact opposite conclusion using the same data
that you are).  In your back-of-the-envelope calculations of cost you're
ignoring costs associated with longer times to develop new features given a
more complicated back-end, increased costs due to higher faults,
troubleshooting, and support costs for a more complicated back-end, etc.

Having multiple listeners per haproxy instance is a common topology in the
HAProxy community because it's the one people are subtly pushed toward
using by reading the haproxy configuration guide, and because they're using
OS-packaged control scripts, which are usually designed to allow for just
one haproxy process, because most servers being deployed are not meant to
be highly scalable, programmable load balancers. However, I think that one
haproxy process per floating IP is actually not an optimal choice for a VM
appliance that is intended to be thus.

Anyway, now that I've spent time poking holes in your justifications for a
multiple-listeners-per-haproxy implementation, let me add a few more to
think about as positive reasons for using a one-listener-per-haproxy
implementation:

A. Despite what you imply above at several points, one-listener-per-haproxy
does not reduce the flexibility of the solution from either the user's or
operator's perspective.

B. One-listener-per-haproxy actually increases the flexibility of the
solution because:
i. If the operator wants to use different log files per listener, they can
easily do so (but are not forced to do so).
ii. Certain haproxy configuration keywords are more logically expressed in
a the 'defaults' configuration area in the configuration file (since the
'listen' section won't be used in our templates-- only 'defaults'
'frontend' and 'backend'). Using one-listener-per-haproxy configuration
files, this allows us to easily vary parameters that would go in this
'defaults' section. In other words, different listeners can have different
haproxy 'defaults' easily. (This allows for a simple way to implement, say
a non-zero keepalive timeout for SSL termination, where keepalive can make
a huge performance difference, and no keepalive for a HTTP listener on the
same loadbalancer, where there are other performance benefits for doing
this.)

C. One-listener-per-haproxy configuration templates will be much simpler
than multiple-listener-per-haproxy templates, which has implications for
troubleshooting, feature development, test writing, and code maintenance
and flexibility. While less tangible, these all reduce the overall cost and
increase the reliability of the solution.

D. Fault isolation is a good thing. If a problem happens with some random
custom TCP listener on port 8686, it's better that this does not take out
the production HTTP listener on port 80 at the same time. This is a good
feature to have, not a bug.

E. Not only do we get fault isolation with this design, but we isolate
service disruptions due to normal updates to listeners, pools, or ports.
The only way to alter a running haproxy configuration is to restart the
haproxy process with a new configuration file. Haproxy provides for a way
to do this gracefully, but it still resets usage counters and health
monitor states for all listeners covered by that haproxy configuration.
 And if there's a problem running the new configuration (such that an
attempted graceful restart brings the service down), all listeners are
affected. This is an unnecessary are unjustified risk since it's so easily
avoided. Why should my attempt to, for example, update a member
configuration on a pool associated with the HTTPS listener not only reset
the usage counters and potentially disrupt end-users currently using the
port 80 HTTP listener, but also risk breaking that service entirely if the
resultant configuration file doesn't run for some reason?

F. It's actually easier / simpler / more reliable to parse the usage
statistic data for a listener for haproxy if only one listener per haproxy
instance is used.

G. Usage statistics have more intuitive inherent meaning if pools can only
be shared within one listener's configuration. In other words, a pool's
usage statistics are naturally interpreted to mean 'within the context of
the parent listener'.  (And, yes, the distinction between listener and pool
usage data becomes meaningful when L7 rules are used.)

H. The above point is also true for operational status information.

I. Since minimum resource guarantees happen at the Octavia VM level, then
SLAs are essentially equivalent using either your model or mine.

J. Troubleshooting is actually much easier to do with
one-listener-per-haproxy. For example:
i. An operator logged into an Octavia VM can easily see the different
resource usage of each listener on that VM using standard unix tools like
'top', 'ps', etc. This is not possible if all the listeners are being
serviced by a single haproxy instance.
ii. An operator can start / stop / manually alter the configuration for one
listener without affecting any of the other listeners on the Octavia VM.
Again, this is not possible if all listeners are serviced by one process.
iii. If desired, an Operator can easily alter global logging configuration
(alter log file location, increase verbosity, etc.) for a single listener
(or all listeners), as well as easily filter out log lines for unaffected
listeners if we go with the one-listener-per-haproxy model. This, again, is
not trivial to do on an individual listener basis using the
multiple-listeners-per-haproxy model.
iv. The same goes for any other parameters that logically fall into the
'defaults' section of the haproxy configuration.

And... I think that's about it.

I also don't want us, by design,  to limit the operator flexibility in
deployment
and topologies, especially when it potentially impacts the costs for large
operators. I simply come to the exact opposite conclusion than you do when
using the above justifications. :)

Thanks,
Stephen

-- 
Stephen Balukoff
Blue Box Group, LLC
(800)613-4305 x807
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140820/feeb3b5d/attachment.html>