<div dir="ltr">Hi Eugene,<div><br></div><div>First, thanks again for your speedy response and entertaining my ideas, eh! And I apologize that it took me a few days to get back to you on this.</div><div><br></div><div>My thoughts are inline below:</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">
On Thu, Feb 13, 2014 at 4:07 AM, Eugene Nikanorov <span dir="ltr"><<a href="mailto:enikanorov@mirantis.com" target="_blank">enikanorov@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><br></div><div>I've been reading through the LoadBalancerInsance description as outlined here and have some feedback: <a href="https://wiki.openstack.org/wiki/Neutron/LBaaS/LoadbalancerInstance" target="_blank">https://wiki.openstack.org/wiki/Neutron/LBaaS/LoadbalancerInstance</a></div>
<div><br></div><div>First off, I agree that we need a container object and that the pool shouldn't be the object root. This container object is going to have some attributes associated with it which then will apply to all related objects further down on the chain. (I'm thinking, for example, that it may make sense for the loadbalancer to have 'network_id' as an attribute, and the associated VIPs, pools, etc. will inherit this from the container object.)</div>
</div></blockquote><div> </div></div><div>Particularly, network_id could be different for vip and a pool in case the balancer works in routed mode (e.g. connects to different networks)</div></div></div></div></blockquote>
<div><br></div><div>For VIP and pool subnets in routed mode, would these be associated with a neutron network? (It occurs to me that layer-3 routing means subnets in this category won't be associated with a "connected" layer-2 network, so much as a layer-3 routing rule, which sends all traffic destined for said subnets ultimately to some address on a layer-2 "connected" subnet.) I'm unfamiliar with how this is supposed to work with Neutron.</div>
<div><br></div><div>Also, do "pools" really need a network_id? (Wouldn't it be valid for the pool's members to come from several different networks, so long as the load balancer has a way (route, NAT, etc.) to communicate with them?) (Practically speaking, we almost never see this-- but there's no technical reason it couldn't be done.)</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div></div><div>One thing that was not clear to me just yet: Is the 'loadbalancer' object meant, at least eventually, to be associated with an actual load balancer device of some kind (be that the neutron node with haproxy, a vendor appliance or a software appliance)?</div>
</div></blockquote><div> </div></div><div>Yes, that is one of the proposed roles of 'loadbalancer' object. But not the only. Appliance is not the only representation of the balancer that we are working with, it could also be a process on the host that is controlled by the agent. So other types of associations also are necessary (like association between the agent and the 'loadbalancer')</div>
<div>
<div><br></div></div></div></div></div></blockquote><div><br></div><div>Yes, of course the local haproxy instance would fall under the same model-- but what I mean by this is that a 'loadbalancer' is meant to ultimately be associated with a "device" of some kind, even if that device is the special case of an haproxy instance running on the neutron network node.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div><div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
<div><br></div><div>If not, then I think we should use a name other than 'Loadbalancer' so we don't confuse people. I realize I might just be harping on one of the two truly difficult problems in software engineering (which are: Naming things, cache invalidation, and off-by-one errors). But if a 'loadbalancer' object isn't meant to actually be synonymous with a load balancer appliance of some kind, the object needs a new name.</div>
</div></blockquote><div> </div></div><div>I don't mind to have other name like 'instance', for example. But appliance (would it be a device or a process+agent) is really a synonym of what I am proposing. </div>
<div><div>
<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">
<div><br></div><div>If the object and the device are meant to essentially be synonymous, then I think we're starting off too simplistic here, and the model proposed is going to need another significant revision when we add additional features later on. I suspect we'll be painting ourselves into a corner with the LoadBalancerInstance as proposed. Specifically, I'm thinking about:</div>
<div><br></div><div><ul><li>Operational concerns around the life cycle of a physical piece of infrastructure. If we're going to replace a physical load balancer, it often makes sense to have both the old and new load balancer defined in the system at the same time during the transition. If you then swap all the VIPs from the old to the new, suddenly all the child objects have their loadbalancer_id changed, which will often wreak havoc on client application code (who really shouldn't be hard-coding things like loadbalancer_id, but will do so anyway. :P ) Such transitions are much easier accomplished if both load balancers can exist within an overarching container object (ie. "cluster" in my proposal) which will never need to be swapped out.<br>
</li></ul></div></div></blockquote></div><div>I'd like to understand that better. I guess no model is complex enough to describe each end every use case. What I'm trying to address with the lb instance is both simplistic cases that are only supported by the current code plus some more complex configurations like multiple pools (L7) and multiple vips. And at the same time we need to consider backward compatibility and plus we need to make some progress. The bigger is the change, the harder it is to make a progress. So we need to find an iterative way of increasing API and model complexity.</div>
</div></div></div></blockquote><div><br></div><div>No model is complex enough to show every use case, but the model should be complex enough to show the most common (anticipated) topologies or use cases. I believe these are: Single load balancer, HA (active-standby), and HA (n-node active-active). I understand the concern about not wanting to change too many things at once in an effort to maintain backward compatibility. I would add that it does nobody any good to over-engineer the system without real-world data and use scenarios dictating need. Keep it simple, stupid, right?<br>
<br>Having said this, I know we're already talking about breaking backward-compatible workflows in the other thread. If we're going to have to do this in order to support L7, and if we're too late to get these changes into Icehouse anyway, then why not consider the next obvious features to solve in Neutron load balancing once we have L7 and SSL: High Availability and scalability? If it's such a pain to break backward-compatible workflows, realize we're probably signing up for the same pain again when we tackle the HA and scalability problems. :/<br>
<br>Besides, I don't think it's actually too much of a change to introduce objects to the model which correspond to topology descriptions (ie. "cluster" and "load balancer", separate from VIP, listener, etc.) These can have very little effect on work-flow for the tenant as well, but can make things much easier for the cloud administrator who has to worry about operational considerations in delivering the load balancing service.<br>
<br>Anyway regarding the operational considerations you wanted to hear more about:</div><div><br></div><div>One often hears the benefit of the cloud from the application developer's perspective: You don't have to worry about "real" hardware anymore. Software defines networking. There's a perception of infinite capacity. And operational concerns are minimized as the system scales without having to make (many) concessions in the code for the limitations of the underlying hardware.</div>
<div><br></div><div>What one doesn't hear touted nearly as often is that the cloud also makes things easier in many ways for those whose job it is to handle those operational concerns. Suddenly maintenance schedules don't need to be coordinated nearly as much with tenant schedules. Hardware can fail willy-nilly and the operator just has to plug a new machine into the network and the cloud "takes care of" assimilating it and making it available for tenant use. Old hardware can be end-of-lifed gracefully and new hardware installed without disrupting tenant application availability or performance. Capacity planning can be done "in aggregate" across the whole cloud without having to worry too much about a single tenant's quirky cluster or needs. And especially: once the cloud os is advanced enough to deliver out-of-the-box high-availability and scalability, those 3:00am phone calls start to become less frequent.</div>
<div><br></div><div>At some level, *someone* needs to be aware of and concerned with physical hardware and operational concerns. In the OpenStack paradigm, I think this is the person or team acting as the "cloud administrator" (as opposed to the "tenant" which is usually a team of application engineers). Tenants generally don't want to be and shouldn't be exposed to anything that approaches "real" hardware (unless there are business reasons for doing so-- like the client having paid for premium load balancing services on hardware purchased from Vendor X). But cloud administrators *must* be exposed to actual hardware (or analogues thereto, like virtual appliances).</div>
<div><br></div><div>In any case, the cloud operating system needs to be aware of each component (hardware, virtual appliance, process, whatever) that makes it work. It's true that some complexity can be hidden behind a "driver" of some kind. But I think this cripples us, when it comes to developing standard features (let alone advanced features) that can be supported in the generic use case. For example, I believe that high availability should be a standard feature and not delegated to the driver level. Having a model which cannot inherently work with high availability and scalability features is going to be limiting, and is going to prevent the cloud from delivering its benefits to cloud administrators.<br>
</div><div><br></div><div>There is precedent for this in Openstack as well: Nova is aware of the hardware nodes that make up compute. Neutron is aware of each of its agents, as well as what physical nodes are delivering network services. It seems strange to me that we wouldn't consider giving Neutron LBaaS intelligence about how each of the hardware (or virtual hardware) components fulfilling the load-balancing role are actually laid out and configured. Further, if we deliver HA / scalability features only at the driver level, then vendors are likely to have incompatible implementations, which means that organizations who are vendor agnostic will nevertheless experience vendor lock-in.<br>
<br>So why the need for the "cluster" object in the model I proposed? In my experience, application developers tend not to care at all about how their load balancing works... until they do. That is to say, throughout the application life-cycle, the needs of what load balancing must do tend to change toward the very specific as dictated by the application developers and/or the business needs of the organization they're working for. (eg. client doesn't care about the load balancer topology until a hardware failure takes down the site for a half hour.) It's also quite common for clients to be very opinionated about which Vendor's load balancing product they use, and they'll sometimes have security concerns which might not make much sense in a cloud (eg. "Staging and production environments *must* be on different physical hardware. But to save costs, we'd like all these 15 development environments to be served from the same single load balancer."), but which nevertheless are real concerns that operators must conform to.</div>
<div><br></div><div>The "cluster" object therefore serves the role of being that container object that tenants can be aware of, that will never need to be upgraded or replaced in and of itself, and that application developers can write automation around. At the same time, tenants do not need to be aware of the "physical" (or virtual) "load balancer" components, that the cloud administrator must be aware of, and out of which the "cluster" is actually built. In other words, the "cluster" component serves as the conceptual bridge and demarkation point between what the application developer / tenant might need to worry about / write automation around / include in the design for business reasons, and what the cloud administrator must worry about by virtue of having to reliably deliver and maintain the "physical" layer.</div>
<div><br></div><div>Does this make sense?</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><ul><li>Having tenants know about loadbalancer_id (if it corresponds with physical hardware) feels inherently un-cloud-like to me. Better that said tenants know about the container object (which doesn't actually correspond with any single physical piece of infrastructure) and not concern themselves with physical hardware.<br>
</li></ul></div></div></blockquote></div><div>Having loadbalancer_id has nothing to do with the appliance or particular backend, so that even might not give any clue to a tenant about the backend type. However, tenant may want to know something about the backend and he/she may want to use the single appliance for their needs (due to quotas, billing or topology limitations), and that's where loadbalancer_id helps to envelop resources and group them to just one (some!) physical backend.</div>
</div></div></div></blockquote><div><br></div><div>See discussion of "cluster" above, and why this should be different from "load balancer instance." </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><ul><li><br></li><li>In an active-standby or active-active HA load balancer topology (ie. anything other than 'single device' topology), multiple load balancers will carry the same configuration, as far as VIPs, Pools, Members, etc. are concerned. Therefore, it doesn't make sense for the 'container' object to be synonymous with a single device. It might be possible to hide this complexity from the model by having HA features exist/exposed only within the driver, but this seems like really backward thinking to me: Why shouldn't we allow API-based configuration of load balancer cluster topology within our model, or force clients to talk to a driver directly for these features? (This is one of the hack-ish work-arounds I alluded to in my e-mail from Monday which is both annoying and corrected with a model which can accurately reflect the topology we're working with.)<br>
</li></ul></div></div></blockquote></div><div>HA is a valid question, and HA is definitely not represented by two different loadbalancer configurations. It is a property of one instance. Remember that loadbalancer and all its child objects are logical config, it could be deployed in HA or in single mode, depending on user's choice and driver capabilities. </div>
<div>And regarding allowing users to talk directly to a driver - that's a big question. I think it can be desirable in some cases.</div><div>You know, lb appliances nowadays can make cookies and fly to space and I'm not sure we want all that in the generic lb API, but that could be provided via specific extensions. Even in a public cloud case, where you want your tenants to be completely unaware of the backend, it's still possible that there are some tenants with specific demands (willing to pay) where more control could be useful. </div>
<div><br></div><div>And going back to HA, this is a common feature that needs to be introduced into generic API.</div><div><div> </div></div></div></div></div></blockquote><div><br></div><div>So, I'm thinking that the "cluster" in my design fills that need.<br>
<br>For example, if a cluster has a "cluster_type" of "HA-active-standby" and from that the cloud OS knows it needs to keep two load balancer nodes up and running (and that these need to be configured to be in an active-standby role, though such specific configuration will happen at the driver level). Again, I think the cloud OS should be aware of how many physical (or virtual) load balancer nodes make up a "cluster" and that this shouldn't be visible only to the driver.</div>
<div><br></div><div>HA and horizontal scalability do not seem like those "fly into space" vendor-specific features. They seem like core components we should be shooting to achieve in the general case.</div><div>
<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><ul><li>Side note: It's possible to still have drivers / load balancer appliances which do not support certain types of HA or auto-scaling topologies. In this case, it probably makes sense to add some kind of 'capabilities' list that the driver publishes to the lbaas daemon when it's loaded.<br>
</li></ul></div></div></blockquote></div><div> Yes, that makes sense and was discussed previously, but not implemented so far.</div></div></div></div></blockquote><div><br></div><div>Aah, cool! </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div><ul><li>
</li></ul></div><div>So, I won't elaborate on the model I would propose we use instead of the above, since I've already done so before. I'll just say that what we're trying to solve with the LoadBalancerInstance resource in this proposal can also be solved with the 'cluster' and 'loadbalancer' resources in the model I've proposed, and my proposal is also capable of supporting HA and auto-scaling topologies without further significant model changes.</div>
<div><br></div><div>Beyond this, other feedback I have:</div><div><br></div><div><ul><li>I don't recommend having loadbalancer_id added to the pool, member, and healthmonitor objects. I see no reason a pool (and its child objects) needs to be restricted to a single load balancer (and it may be very advantageous in large clusters for it not to be).<br>
</li></ul></div></div></blockquote></div><div>That is something we need to evaluate. Currently we already have that with the difference that pool is our 'loadbalancer object'.</div><div>Ability to share objects between appliances definitely makes sense, but I'm not sure we need to address it right now.</div>
</div></div></div></blockquote><div><br></div><div>Fair enough-- but I worry that if people write automation around determining loadbalancer_id ("cluster_id" in my model) based on the pool or member, then we'll break that automation if we ever add the ability for a single pool to be shared across load balancer (clusters).</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><ul><li>I general, I prefer DRY code, so if we can avoid adding the loadbalancer_id attribute to existing resources except for where it's really needed, that's what I'd recommend. (Do we expect significant savings by repeating this attribute in various locations and avoiding one SQL query? It seems to me we're inviting annoying bugs we'll have to work out by having that field essentially act as a cache for the authoritative source of information-- ie. the load balancer (cluster) object itself.)<br>
</li></ul></div></div></blockquote></div><div>Well, DRY principle is good, but save on sql queries is good also, because the more queries we do, the more are the chances of different races, not to say that it could just eat up a bit of performance if DB is large enough. </div>
</div></div></div></blockquote><div><br></div><div>Fair enough, eh. </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div><ul><li>
<br></li><li>Having loadbalancer_id (cluster_id in my model) an attribute of the VIP makes sense. I can't think of any reason a given VIP would be associated with multiple load balancer (clusters).<br></li></ul></div>
<div><div class="gmail_extra"><br></div><div class="gmail_extra">Thanks,</div><div class="gmail_extra">Stephen</div><div class="gmail_extra"><br></div></div></div></blockquote><div><br></div></div><div>One of the important properties of proposed loadbalancer object is not just another entity of API, but also a helper object for various coding problems that mostly don't affect API consumers.</div>
<div><br></div></div></div></div></blockquote><div><br></div><div>Yep-- I think I'm addressing this with my proposed "cluster" object, also being separate from the "loadbalancer" object, eh.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">
<div></div><div>Thanks,</div><div>Eugene.</div><div><br></div></div></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><span></span>Stephen Balukoff
<br>Blue Box Group, LLC
<br><a href="tel:%28800%29613-4305%20x807" value="+18006134305" target="_blank">(800)613-4305 x807</a>
</div></div>