<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=gb2312"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoPlainText><span style='color:#17375E;mso-style-textfill-fill-color:#17375E;mso-style-textfill-fill-alpha:100.0%'>Salvatore,<o:p></o:p></span></p><p class=MsoPlainText><span style='color:#17375E;mso-style-textfill-fill-color:#17375E;mso-style-textfill-fill-alpha:100.0%'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:#17375E;mso-style-textfill-fill-color:#17375E;mso-style-textfill-fill-alpha:100.0%'>Thanks for providing input on this thread.<o:p></o:p></span></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText style='margin-left:.5in'>> In a separate API (admin API), an admin can beforehand create service <o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>> types and register device types (drivers) against these <o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>> “service_types”, so service type “regular” can contain {“LB”: “ha_proxy”} for example.<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>><o:p> </o:p></p><p class=MsoPlainText style='margin-left:.5in'><o:p> </o:p></p><p class=MsoPlainText style='margin-left:.5in'>Yeah it's pretty much like that.<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>If you look at the patch the request body is as follows:<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'><o:p> </o:p></p><p class=MsoPlainText style='margin-left:.5in'>{ "service_type":<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> { "name": "xxx",<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> "description": "yyy",<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> "enabled": True,<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> "default": False,<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> "service_definitions": [<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> {"service": "LB",<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> "plugin": "abc",<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> "driver": "123"},<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> {"service": "wootwoot",<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> "plugin": "mootmoot",<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> "driver": "lotlot"}<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'> ]<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText style='margin-left:.5in'>> What is not clear currently to me, and I need Salvatore to clarify <o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>> this point is whether a “service type” can contain several entries for <o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>> the same category like “LB”, and whether the same service type can <o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>> contain a mixture of categories, so for example a service type “fast” <o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>> would be defined as<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>> follows: {“LB”: “vendor1”, “FW”: “vendor2”, “LB”: “vendor3”}. This at <o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>> least is what is suggested in the blueprint for service insertion here:<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>> <a href="http://wiki.openstack.org/Quantum/ServiceInsertion"><span style='color:windowtext;text-decoration:none'>http://wiki.openstack.org/Quantum/ServiceInsertion</span></a><o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'><o:p> </o:p></p><p class=MsoPlainText style='margin-left:.5in'>We can definitely have a mixture of categories; for instance a comprehensive service type that give you options for load balancing, firewall, and other services.<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>At this stage I am not proposing multiple solutions for the same kind of service in a single service type. This option would indeed necessarily imply that there is some sort of logic in the quantum service for discriminating between the two options, and I would like to avoid it; or at least I would like to keep it out of what is expected to be delivered for the Grizzly release.<o:p></o:p></p><p class=MsoPlainText style='margin-left:.5in'>I think that the goal of a service type is to specify a plugin and possibly a driver for serving a request; the driver is however optional. In that case, within the plugin layer 'scheduling' mechanism might select the appropriate driver according to parameters such as requested features and device capabilities.<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText><span style='color:#17375E;mso-style-textfill-fill-color:#17375E;mso-style-textfill-fill-alpha:100.0%'>Ok, so for now, we can assume as far as LBaaS plugin is concerned, a <i>service_type</i> selects a <i>device_type</i>. So creating a VIP and setting its service_type to “X” (or defaulting to the default service type), would mean the VIP creation will be handled by the driver that corresponds to that service _<i>type “X” (for the “LB” category). </i>A generic scheduler can only choose one of the devices of *<b>that</b>* device_type. The next item we have to tackle is the device management plugin and keeping the “device DB” up-to-date wrt. device availability, load, etc. (and we would need driver-specific interfaces for this).</span><span style='color:black'><o:p></o:p></span></p><p class=MsoPlainText><span style='color:#17375E;mso-style-textfill-fill-color:#17375E;mso-style-textfill-fill-alpha:100.0%'><o:p> </o:p></span></p><p class=MsoPlainText><span style='color:#17375E;mso-style-textfill-fill-color:#17375E;mso-style-textfill-fill-alpha:100.0%'>Thanks<o:p></o:p></span></p><p class=MsoPlainText><span style='color:#17375E;mso-style-textfill-fill-color:#17375E;mso-style-textfill-fill-alpha:100.0%'>Youcef<o:p></o:p></span></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> From: Mellquist, Peter [<a href="mailto:peter.mellquist@hp.com"><span style='color:windowtext;text-decoration:none'>mailto:peter.mellquist@hp.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> Sent: Tuesday, December 4, 2012 5:38 PM<o:p></o:p></p><p class=MsoPlainText>> To: OpenStack Development Mailing List<o:p></o:p></p><p class=MsoPlainText>> Subject: Re: [openstack-dev] [Quantum][LBaaS] Selecting an LBaaS <o:p></o:p></p><p class=MsoPlainText>> device given a service type<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Youcef,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Is the current proposal for the user /admin to select a device type <o:p></o:p></p><p class=MsoPlainText>> through the API? I am hoping that ‘devices’ are abstracted through the <o:p></o:p></p><p class=MsoPlainText>> APIs and instead the API allows selection of the LBaaS service <o:p></o:p></p><p class=MsoPlainText>> offerings ( regular, premium, etc ) with a default when not specified at all.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Peter.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> From: Eugene Nikanorov [<a href="mailto:enikanorov@mirantis.com"><span style='color:windowtext;text-decoration:none'>mailto:enikanorov@mirantis.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> Sent: Tuesday, December 04, 2012 2:16 PM<o:p></o:p></p><p class=MsoPlainText>> To: OpenStack Development Mailing List<o:p></o:p></p><p class=MsoPlainText>> Subject: Re: [openstack-dev] [Quantum][LBaaS] Selecting an LBaaS <o:p></o:p></p><p class=MsoPlainText>> device given a service type<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Youcef,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> see my comments inline:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On Wed, Dec 5, 2012 at 1:29 AM, Youcef Laribi <o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:Youcef.Laribi@eu.citrix.com"><span style='color:windowtext;text-decoration:none'>Youcef.Laribi@eu.citrix.com</span></a>><o:p></o:p></p><p class=MsoPlainText>> wrote:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> There is some misunderstandings going on in this thread (nothing <o:p></o:p></p><p class=MsoPlainText>> unusual hereJ), but since we have some code to play with, let’s use <o:p></o:p></p><p class=MsoPlainText>> this to be more specific by what we mean.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Looking at the code that Eugene sent, I’m a bit confused, because we <o:p></o:p></p><p class=MsoPlainText>> seem to be talking now not only about an “LB” scheduler, but a <o:p></o:p></p><p class=MsoPlainText>> scheduler framework for all service types (LB, firewall, etc.).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> get_device_for_resource(resource)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> return<o:p></o:p></p><p class=MsoPlainText>> scheduling_drivers[resource.service_type].get_device_for_resource(reso<o:p></o:p></p><p class=MsoPlainText>> urce)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> device_info = get_device_for_resource(resource)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> It seems that for each service_type, we suggest having a “scheduler”, <o:p></o:p></p><p class=MsoPlainText>> which is confusingly called "scheduling_driver" (I imagine this has <o:p></o:p></p><p class=MsoPlainText>> nothing to do with the vendor-specific LB drivers, right?).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> "Scheduling driver" is a term taken from nova. Probably, since we're <o:p></o:p></p><p class=MsoPlainText>> already using "driver" in other context, we may call it "pluggable <o:p></o:p></p><p class=MsoPlainText>> scheduling algorithm".<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> So, if a service type contains service definitions for LB and <o:p></o:p></p><p class=MsoPlainText>> Firewall, we will have the same “scheduler_driver”?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> In fact, that depends on how generic the scheduling algorithm can be. <o:p></o:p></p><p class=MsoPlainText>> If it can be even service type-agnostic - then yes (in fact, "chance scheduler"<o:p></o:p></p><p class=MsoPlainText>> which picks devices randomly could be such a candidate).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> But I think it would be overgeneralization.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> So, lets just think that each service type will have it's own set of <o:p></o:p></p><p class=MsoPlainText>> scheduling algorithms (one is configured at a time).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> This way we may write scheduling algorithm for LB without thinking <o:p></o:p></p><p class=MsoPlainText>> about other service types, and yet fully to add other schedulers for <o:p></o:p></p><p class=MsoPlainText>> other service types.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Even if I assume that my service type has been created so it only <o:p></o:p></p><p class=MsoPlainText>> contains one “LB” service definition (and no other service <o:p></o:p></p><p class=MsoPlainText>> definitions), and therefore it will nicely map to a scheduler that <o:p></o:p></p><p class=MsoPlainText>> only does “LB” scheduling like the one included<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> class LBScheduler:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> device_handlers = [list of classes for each device type, which can <o:p></o:p></p><p class=MsoPlainText>> match VIP requirements to device caps and status]<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> get_device_for_vip(vip)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> for device in devices:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> if device_handlers[device.type].is_good(device, vip):<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> return device<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> In the above code, the method get_device_for _vip(), uses a concept of <o:p></o:p></p><p class=MsoPlainText>> “device_handler” for each “device type” (driver). Isn’t this <o:p></o:p></p><p class=MsoPlainText>> device_handler(), vendor-specific code?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Exactly! That's where vendor plugs in his code. Decision is still made <o:p></o:p></p><p class=MsoPlainText>> at more generic level while involving device-specific code.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> If the above assumption is correct, then the vendor *is* actually <o:p></o:p></p><p class=MsoPlainText>> taking part in the scheduling decision, which is already an <o:p></o:p></p><p class=MsoPlainText>> improvement on the previous proposal.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Well, in fact, I meant that from the very beginning :)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> It's just that one vendor (driver) can't make decision by itself.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> But does it mean we are suggesting to have vendor-specific code <o:p></o:p></p><p class=MsoPlainText>> scattered in several places, some of it in the scheduler, some of it in the agent/driver?<o:p></o:p></p><p class=MsoPlainText>> I don’t like this. I thought that the LB plugin should be completely <o:p></o:p></p><p class=MsoPlainText>> vendor-agnostic, and all vendor-specific code should be grouped in the <o:p></o:p></p><p class=MsoPlainText>> agent/driver component.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I agree with you regarding code locations. But I didn't mean we need <o:p></o:p></p><p class=MsoPlainText>> to write device-specific code in scheduler.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Just let device_handler be a part of driver library.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> It's just a device-specific code, that resides in a driver but is used <o:p></o:p></p><p class=MsoPlainText>> within a scheduler component.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> That may look like the following in scheduler.conf:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> [LoadBalancer]<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> device_handlers = vendorXDriver.deviceTypeA.HandlerClass1,<o:p></o:p></p><p class=MsoPlainText>> vendorYDriver.deviceTypeB.HandlerClass2<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Where HandlerClass could be:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> class HaproxyHandlerClass<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> def get_device_type()<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> return "HAPROXY"<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> def is_good(device, vip):<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> return ...<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Scheduler then make use of it:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> def load_handlers()<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> for handler_class in handlers:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> class = import_class(handler_class)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> inst = class()<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> device_handlers[inst.get_device_type()] = inst<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> That's, in fact, the same way plugins are currently loaded in quantum.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hope the idea became clearer!<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Eugene.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Youcef<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> From: Eugene Nikanorov [<a href="mailto:enikanorov@mirantis.com"><span style='color:windowtext;text-decoration:none'>mailto:enikanorov@mirantis.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> Sent: Tuesday, December 4, 2012 5:08 AM<o:p></o:p></p><p class=MsoPlainText>> To: OpenStack Development Mailing List<o:p></o:p></p><p class=MsoPlainText>> Subject: Re: [openstack-dev] [Quantum][LBaaS] Selecting an LBaaS <o:p></o:p></p><p class=MsoPlainText>> device given a service type<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Salvatore,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks for detailed reply.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I'm going to explain my idea in more detain with pseudocode.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> See my comments inline.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On Tue, Dec 4, 2012 at 2:52 PM, Salvatore Orlando <o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:sorlando@nicira.com"><span style='color:windowtext;text-decoration:none'>sorlando@nicira.com</span></a>><o:p></o:p></p><p class=MsoPlainText>> wrote:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> My only remark is that in my opinion having a 'global' LB scheduler <o:p></o:p></p><p class=MsoPlainText>> that will work across all drivers is definitely valuable, but probably <o:p></o:p></p><p class=MsoPlainText>> not necessary. If I were to set the priority of this feature, I would <o:p></o:p></p><p class=MsoPlainText>> put it on "wishlist" for Grizzly. And this for the several reasons:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - as already said in this thread, it is not easy to model features and <o:p></o:p></p><p class=MsoPlainText>> device capabilities in an agnostic way.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Capabilities are not device-agnostic, for sure.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Some part of device database model is common and generic, and some may <o:p></o:p></p><p class=MsoPlainText>> be stored in "extra" fields which are used by device-specific code.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I'm trying to design scheduler as extensible device-agnostic (and, in <o:p></o:p></p><p class=MsoPlainText>> fact,<o:p></o:p></p><p class=MsoPlainText>> service-type-agnostic) framework, where we will implement scheduling <o:p></o:p></p><p class=MsoPlainText>> logic for LB and, going deeper, some device-specific aspects of LB scheduling.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Once again, I think that LB scheduling algorithm should be generic and <o:p></o:p></p><p class=MsoPlainText>> also, configurable (e.g. you may write your own algorithm and make <o:p></o:p></p><p class=MsoPlainText>> scheduler use it).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> This, first of all, framework will allow us to make some stub like <o:p></o:p></p><p class=MsoPlainText>> scheduling on first available device, e.g., something like this:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> get_device_for_vip(vip)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> for device in devices:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> if is_good(device, vip):<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> return device<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> When I say, service type and device agnostic framework, I mean that <o:p></o:p></p><p class=MsoPlainText>> all logic is hidden by just a few generic calls like the following:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> device_info = get_device_for_resource(resource)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> which could be implemented as:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> get_device_for_resource(resource)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> return<o:p></o:p></p><p class=MsoPlainText>> scheduling_drivers[resource.service_type].get_device_for_resource(reso<o:p></o:p></p><p class=MsoPlainText>> urce)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> In turn, scheduling_driver.get_device_for_resource(resource) may be <o:p></o:p></p><p class=MsoPlainText>> our LB<o:p></o:p></p><p class=MsoPlainText>> scheduling:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> class LBScheduler:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> device_handlers = [list of classes for each device type, which can <o:p></o:p></p><p class=MsoPlainText>> match VIP requirements to device caps and status]<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> get_device_for_vip(vip)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> for device in devices:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> if device_handlers[device.type].is_good(device, vip):<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> return device<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> You may notice that scheduler component itself is quite thin layer <o:p></o:p></p><p class=MsoPlainText>> which serves several purposes:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 1) extensible and configurable:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - you add service types like you add plugins to quantum<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - you add drivers to let generic algorithm have better <o:p></o:p></p><p class=MsoPlainText>> understanding of particular device of particular type.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 2) synchronous. Choosing device is one fast synch. operation.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 3) The code itself is just routing resource to corresponding logic.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I think these are both good features and easy to implement.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> That may save lots of refactoring and redesigning when it'll come to <o:p></o:p></p><p class=MsoPlainText>> other advanced services.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - drivers apparently will be more than a simple "actuator", but will <o:p></o:p></p><p class=MsoPlainText>> have their own logic. I can see for instance at least three different <o:p></o:p></p><p class=MsoPlainText>> drivers<o:p></o:p></p><p class=MsoPlainText>> families: i) hardware load balancers, ii) contextualized hardware load <o:p></o:p></p><p class=MsoPlainText>> balancers (hw appliances where you create virtual LB appliances), and <o:p></o:p></p><p class=MsoPlainText>> iii) virtualized load balancers, that could be spawn, for instance, using nova.<o:p></o:p></p><p class=MsoPlainText>> What would be the criteria for choosing a virtual appliance versus <o:p></o:p></p><p class=MsoPlainText>> allocating a VIP on a hardware one?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> That's what I'm trying to avoid: some drivers will be "simple <o:p></o:p></p><p class=MsoPlainText>> actuators", some will have their own logic.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Regarding the question about criteria: good question :) But it, in my <o:p></o:p></p><p class=MsoPlainText>> opinion, a bit unrelated to the scheduling architecture, e.g. it is <o:p></o:p></p><p class=MsoPlainText>> actual whichever choice we make.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - In this Grizzly release we won't probably have a huge amount of drivers.<o:p></o:p></p><p class=MsoPlainText>> Or probably we'll have the drivers, but Quantum LB service, being <o:p></o:p></p><p class=MsoPlainText>> experimental, will probably be deployed with no more than one or two <o:p></o:p></p><p class=MsoPlainText>> drivers.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Another interesting point in my opinion is that this scheduling logic <o:p></o:p></p><p class=MsoPlainText>> is part of the LB plugin we're implementing for Grizzly, not part of <o:p></o:p></p><p class=MsoPlainText>> the DB model supporting the tenant API. There will be, of course, <o:p></o:p></p><p class=MsoPlainText>> model classes for device management, but they (and all the logic for <o:p></o:p></p><p class=MsoPlainText>> managing them) should be separate from the modules which implement the API.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> That's for sure. In fact we've currently thinking of scheduling and <o:p></o:p></p><p class=MsoPlainText>> device management as separate mandatory plugin which will provide it's <o:p></o:p></p><p class=MsoPlainText>> functionality to other advanced service plugins.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> My argument here is not that we should not have a global scheduler; <o:p></o:p></p><p class=MsoPlainText>> I'm just saying I have the impression that there are some important <o:p></o:p></p><p class=MsoPlainText>> details which are not yet completely fleshed out.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I understand that, and i just want to make this details affect <o:p></o:p></p><p class=MsoPlainText>> particular code (drivers, algorithms), but not whole architecture.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Nevertheless, driver-level scheduling is valuable too, and probably <o:p></o:p></p><p class=MsoPlainText>> easier to implement. I wouldn't disregard, in the long run, having a <o:p></o:p></p><p class=MsoPlainText>> two-step<o:p></o:p></p><p class=MsoPlainText>> process:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Step 1 - Quantum LB plugin schedules drivers according either to <o:p></o:p></p><p class=MsoPlainText>> service_type required by the user or request features<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Step 2 - Driver selects device according to capabilities<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Questions here:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 1) If driver selects device, is it mandatory for all drivers to <o:p></o:p></p><p class=MsoPlainText>> provide such functionality?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 2) Where device database is stored?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> If it is mandatory for driver to be able to select a device, should <o:p></o:p></p><p class=MsoPlainText>> device database be driver-specific, e.g. each driver has it's own?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Will drivers access single database remotely? Remember we decided that <o:p></o:p></p><p class=MsoPlainText>> drivers run within agent, and there could be several agents running. <o:p></o:p></p><p class=MsoPlainText>> Driver of which agent instance should be responsible for scheduling?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> If it's not mandatory for driver, then some scheduling logic will be <o:p></o:p></p><p class=MsoPlainText>> in generic scheduler, some in drivers.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> In fact, any option of above brings tons of coding and testing <o:p></o:p></p><p class=MsoPlainText>> complexity when we start to answer these questions.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>>> but to Sam’s point, a common scheduler might not have enough <o:p></o:p></p><p class=MsoPlainText>>> visibility or understanding of device specifics/limitations in order <o:p></o:p></p><p class=MsoPlainText>>> to “correctly” pick the right device.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Saying this you assume that driver has such understanding, and even <o:p></o:p></p><p class=MsoPlainText>> that might not be the case.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> This boils down to defining what a driver is. If it has to be a simple <o:p></o:p></p><p class=MsoPlainText>> "actuator" (I don't remember the name it had in Atlas), then it makes <o:p></o:p></p><p class=MsoPlainText>> perfectly sense to do the scheduling in the service, as the driver <o:p></o:p></p><p class=MsoPlainText>> just executes the LB operation.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> In fact, we decided that once, that driver is simple and synchronous <o:p></o:p></p><p class=MsoPlainText>> that maps generic LB model to device-specific.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> But by saying that "driver may not have understanding" I mean that in <o:p></o:p></p><p class=MsoPlainText>> some cases we need an extended status of the device to know its <o:p></o:p></p><p class=MsoPlainText>> "rating" in scheduling. Example: N of deployed VIPs (may be hard to <o:p></o:p></p><p class=MsoPlainText>> find out for the driver), current connections, preconfigured device <o:p></o:p></p><p class=MsoPlainText>> limits; some of these could be got from device, some are known at device DB.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> In fact, in order to avoid that, scheduler should contain:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 1) all necessary logic to make a decision (logic may be <o:p></o:p></p><p class=MsoPlainText>> device-specific, e.g. different for different kinds of devices, or <o:p></o:p></p><p class=MsoPlainText>> even different instances of the same device type). In that case <o:p></o:p></p><p class=MsoPlainText>> scheduling becomes simple fast<o:p></o:p></p><p class=MsoPlainText>> operation: read data from DB - make choice - write to DB.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 2) active device monitoring: that is needed for "visibility and <o:p></o:p></p><p class=MsoPlainText>> understanding", it can be device-specific. It is performed by <o:p></o:p></p><p class=MsoPlainText>> scheduler using it's device database and using device-specific code <o:p></o:p></p><p class=MsoPlainText>> from the drivers (but code is running under scheduler process or plugin).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> The idea behind such scheme is the same as scheduling in nova. Unlike <o:p></o:p></p><p class=MsoPlainText>> nova we don't have devices reporting their status to the scheduler, so <o:p></o:p></p><p class=MsoPlainText>> we need to poll them proactively.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I am not sure I agree on this statement. Scheduling in nova is a <o:p></o:p></p><p class=MsoPlainText>> decision which takes into account a limited set of capabilities, and <o:p></o:p></p><p class=MsoPlainText>> then picks the first node with enough resources. It does not select <o:p></o:p></p><p class=MsoPlainText>> the "best" one - though I concede you can just replace the scheduling <o:p></o:p></p><p class=MsoPlainText>> algorithm with another that select the best node. However, it assumes <o:p></o:p></p><p class=MsoPlainText>> all nodes are identical. Instead here we're not distinguishing only on <o:p></o:p></p><p class=MsoPlainText>> capabilities but also on features. And the concept of capability too might be quite different across drivers.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Scheduling in nova is done by scheduling drivers (which in that <o:p></o:p></p><p class=MsoPlainText>> context means pluggable scheduling algorithms), you know. There is <o:p></o:p></p><p class=MsoPlainText>> "chance" driver that picks host randomly, there is "least_cost" driver <o:p></o:p></p><p class=MsoPlainText>> that looks into node's load and status and applies more complex <o:p></o:p></p><p class=MsoPlainText>> algorithm to make a decision.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> That is exactly what I'd like to see in our scheduler. Make framework <o:p></o:p></p><p class=MsoPlainText>> that will allow primitive implementation while leaving door open to <o:p></o:p></p><p class=MsoPlainText>> more complex ones.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> However, it's probably down to me not understanding how you are <o:p></o:p></p><p class=MsoPlainText>> planning to design this scheduler. For instance how it would select <o:p></o:p></p><p class=MsoPlainText>> between creating a VIP on a physical load balancer, or spanning a <o:p></o:p></p><p class=MsoPlainText>> virtual appliance and create the VIP on it?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> It's a good question. Regarding this particular choice: my idea is <o:p></o:p></p><p class=MsoPlainText>> that scheduler doesn't make such decision at all.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> It doesn't operate on yet non-existing devices. In order to insert VM <o:p></o:p></p><p class=MsoPlainText>> LB into consideration, user needs to launch it and register as a device.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> You may argue that it is additional actions user will need to take. <o:p></o:p></p><p class=MsoPlainText>> But what is on other hand?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> If we let scheduler to make such decision, that it should also be <o:p></o:p></p><p class=MsoPlainText>> capable of doing the following:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - consider if there is already a VM LB in tenant's network that may be <o:p></o:p></p><p class=MsoPlainText>> used<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - spawn an instance of VM LB (tenant should also provide image id to <o:p></o:p></p><p class=MsoPlainText>> do so, it should also has its reflection in tenant API, which we don't <o:p></o:p></p><p class=MsoPlainText>> have at the<o:p></o:p></p><p class=MsoPlainText>> moment)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Second point alone has two disadvantages:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - scheduling can't be synchronous operation. That will affect whole <o:p></o:p></p><p class=MsoPlainText>> architecture, complicating it.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - user will need to pass some device-management-specific info (image <o:p></o:p></p><p class=MsoPlainText>> id) to device-management unaware Tenant API.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Sorry for the long email, you're probably tired of reading it :)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> But I think this is important discussion and worth covering in <o:p></o:p></p><p class=MsoPlainText>> upcoming LBaaS meetings; may be it's worth to setup a meeting on irc <o:p></o:p></p><p class=MsoPlainText>> for that topic specifically.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Eugene.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> What do you think?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Eugene.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On Fri, Nov 30, 2012 at 7:45 PM, Ilya Shakhat <<a href="mailto:ishakhat@mirantis.com"><span style='color:windowtext;text-decoration:none'>ishakhat@mirantis.com</span></a>> wrote:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Sam, Youcef,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Your point makes sense. I tried to make "scheduler" common, but it <o:p></o:p></p><p class=MsoPlainText>> really looks like driver should participate in decision making.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Ilya<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 2012/11/30 Samuel Bercovici <<a href="mailto:SamuelB@radware.com"><span style='color:windowtext;text-decoration:none'>SamuelB@radware.com</span></a>><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Ilya,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I concur with Youcef.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> -Sam.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> From: Youcef Laribi [<a href="mailto:Youcef.Laribi@eu.citrix.com"><span style='color:windowtext;text-decoration:none'>mailto:Youcef.Laribi@eu.citrix.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> Sent: Friday, November 30, 2012 3:57 AM<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> To: OpenStack Development Mailing List<o:p></o:p></p><p class=MsoPlainText>> Subject: Re: [openstack-dev] [Quantum][LBaaS] Selecting an LBaaS <o:p></o:p></p><p class=MsoPlainText>> device given a service type<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Ilya,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Let’s first separate device on-boarding and management from the “scheduler”<o:p></o:p></p><p class=MsoPlainText>> discussion. These are separate functions in the system, and we’ll keep <o:p></o:p></p><p class=MsoPlainText>> scheduler as the component that picks the driver/device (and we can <o:p></o:p></p><p class=MsoPlainText>> argue separately and decide whether this is a common component to all <o:p></o:p></p><p class=MsoPlainText>> vendors or a vendor-specific component, whether it resides in the <o:p></o:p></p><p class=MsoPlainText>> plugin or in the driver, etc.).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Now to come back to the scheduler discussion, it might seem that a <o:p></o:p></p><p class=MsoPlainText>> scheduler can be common to all drivers would work fine, but to Sam’s <o:p></o:p></p><p class=MsoPlainText>> point, a common scheduler might not have enough visibility or <o:p></o:p></p><p class=MsoPlainText>> understanding of device specifics/limitations in order to “correctly” <o:p></o:p></p><p class=MsoPlainText>> pick the right device. For example, some vendors have a limit of vlans <o:p></o:p></p><p class=MsoPlainText>> per interface, or cannot support overlapping IPs, other vendor devices <o:p></o:p></p><p class=MsoPlainText>> are meshed together in a cluster or a pool and there are optimal ways <o:p></o:p></p><p class=MsoPlainText>> to distribute VIPs or networks in those setups, that a common <o:p></o:p></p><p class=MsoPlainText>> scheduler wouldn’t understand. That’s why I previously said that the <o:p></o:p></p><p class=MsoPlainText>> scheduler (“placement component”) should pick the driver, and let the driver pick a specific device, that way each vendor is responsible<o:p></o:p></p><p class=MsoPlainText>> for their own allocation strategy on their devices. Or at least the driver<o:p></o:p></p><p class=MsoPlainText>> should have an input into the scheduler decision, so the scheduler <o:p></o:p></p><p class=MsoPlainText>> doesn’t pick the wrong device.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On the admin/operator APIs used for device on-boarding and management, <o:p></o:p></p><p class=MsoPlainText>> we need to initiate a separate thread, and discuss whether this be <o:p></o:p></p><p class=MsoPlainText>> implemented as a separate plugin than the LBaaS plugin, or we extend <o:p></o:p></p><p class=MsoPlainText>> the LBaaS plugin to also support a provider/admin API? And what is the <o:p></o:p></p><p class=MsoPlainText>> role of LBaaS agent/driver in the device on-boarding process.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Youcef<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> From: Ilya Shakhat [<a href="mailto:ishakhat@mirantis.com"><span style='color:windowtext;text-decoration:none'>mailto:ishakhat@mirantis.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> Sent: Thursday, November 29, 2012 7:34 AM<o:p></o:p></p><p class=MsoPlainText>> To: OpenStack Development Mailing List<o:p></o:p></p><p class=MsoPlainText>> Subject: Re: [openstack-dev] [Quantum][LBaaS] Selecting an LBaaS <o:p></o:p></p><p class=MsoPlainText>> device given a service type<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Just a small summary of our discussion. We have the following components:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> *aaS plugins - do the logic related to services. Plugins know service <o:p></o:p></p><p class=MsoPlainText>> data model only and don't hold information about devices. When Plugin <o:p></o:p></p><p class=MsoPlainText>> needs to deploy any changes, it calls Scheduler.<o:p></o:p></p><p class=MsoPlainText>> Scheduler ("placement component") - binds services to devices. It has <o:p></o:p></p><p class=MsoPlainText>> API to manage devices (similar to provider api in old LBaaS). <o:p></o:p></p><p class=MsoPlainText>> Scheduler knows how to find device by service_type and has DB to store <o:p></o:p></p><p class=MsoPlainText>> them. When it gets request from Plugin, it finds corresponding device <o:p></o:p></p><p class=MsoPlainText>> and forwards request to Agent Agent - dispatches commands to drivers. <o:p></o:p></p><p class=MsoPlainText>> Agent holds collection of drivers and knows how to dispatch message to <o:p></o:p></p><p class=MsoPlainText>> them Drivers - translate service model to device-specific.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Both Scheduler and Agent are common for all types of services. The <o:p></o:p></p><p class=MsoPlainText>> logic related to load balancing is implemented as drivers.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Please see <o:p></o:p></p><p class=MsoPlainText>> <a href="http://wiki.openstack.org/Quantum/LBaaS/Architecture/Scheduler"><span style='color:windowtext;text-decoration:none'>http://wiki.openstack.org/Quantum/LBaaS/Architecture/Scheduler</span></a><o:p></o:p></p><p class=MsoPlainText>> for details on how components interact and what the typical workflow <o:p></o:p></p><p class=MsoPlainText>> will be. Comments are welcome :)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Ilya<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 2012/11/28 Eugene Nikanorov <<a href="mailto:enikanorov@mirantis.com"><span style='color:windowtext;text-decoration:none'>enikanorov@mirantis.com</span></a>><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Youcef,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Please see my comments inline.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On Wed, Nov 28, 2012 at 2:14 AM, Youcef Laribi <o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:Youcef.Laribi@eu.citrix.com"><span style='color:windowtext;text-decoration:none'>Youcef.Laribi@eu.citrix.com</span></a>><o:p></o:p></p><p class=MsoPlainText>> wrote:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Changing the subject line (was: Progress on lbaas-plugin-api-crud)…<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Eugene,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Let’s make sure we agree on the assumptions:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - LBaaS Plugin has a set of drivers (vendor-specific). Drivers run<o:p></o:p></p><p class=MsoPlainText>> in the LBaaS agent process.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Agreed.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - Each driver (provider in Salvatore’s terminology) is registered<o:p></o:p></p><p class=MsoPlainText>> against a service type (yes, service type can include LB drivers, <o:p></o:p></p><p class=MsoPlainText>> firewall drivers, etc.).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Agreed.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - There can be several LBaaS drivers registered against the same<o:p></o:p></p><p class=MsoPlainText>> service type (e.g. “high-performance LB” service type).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> That probably needs to be clarified in more detail, but it does make sense.<o:p></o:p></p><p class=MsoPlainText>> As far as I understand there is exactly 1 driver per service type, but <o:p></o:p></p><p class=MsoPlainText>> there could be several service types referencing the same driver (like <o:p></o:p></p><p class=MsoPlainText>> you mentioned, "high-perf-lb", "low-cost-lb", etc)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> If these assumptions are incorrect or need to be clarified further, <o:p></o:p></p><p class=MsoPlainText>> let’s start by doing this first J<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Now, let’s imagine we have a component in the system whose job is to <o:p></o:p></p><p class=MsoPlainText>> pick a driver/provider (device type) and a device<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> (device id) given a certain service type. We will call this component <o:p></o:p></p><p class=MsoPlainText>> the “placement component” (it’s not necessarily a separate process <o:p></o:p></p><p class=MsoPlainText>> like the scheduler, and can be part of the plugin, the agent or the <o:p></o:p></p><p class=MsoPlainText>> driver, it doesn’t matter for this discussion at this stage).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I'd still prefer to call it a scheduler even though it will be a part <o:p></o:p></p><p class=MsoPlainText>> of our plugin or separate component.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> The Placement Component needs to choose a device that can load-balance <o:p></o:p></p><p class=MsoPlainText>> traffic coming from network A (where the VIP is) to VMs residing on <o:p></o:p></p><p class=MsoPlainText>> Network B (pool’s network). In order to do this, the Placement <o:p></o:p></p><p class=MsoPlainText>> Component needs to be aware of the capabilities of each <o:p></o:p></p><p class=MsoPlainText>> driver/provider and can follow a certain strategy of device allocation <o:p></o:p></p><p class=MsoPlainText>> that might take into account some of the following constraints.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - Some device types are physical appliances, others are virtual <o:p></o:p></p><p class=MsoPlainText>> appliances running on Nova. The driver might prefer one or the other <o:p></o:p></p><p class=MsoPlainText>> if both satisfy the service type.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Agreed.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - Some device types have a fixed number of devices (e.g. physical <o:p></o:p></p><p class=MsoPlainText>> appliances), while other devices can be created at will whenever <o:p></o:p></p><p class=MsoPlainText>> needed (e.g. HA-Proxy VMs).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Agreed.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - Some device types can host a high number of VIPs, others can host <o:p></o:p></p><p class=MsoPlainText>> a smaller number.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Agreed. Typically such factors are accounted during scheduling process.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - Given a choice between multiple device types that satisfy the same <o:p></o:p></p><p class=MsoPlainText>> service type, preference could be given to a device that is already <o:p></o:p></p><p class=MsoPlainText>> wired to network A and network B.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Not sure that this is necessary, but that could be an option.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - Given a choice between several equivalent devices (possibly of <o:p></o:p></p><p class=MsoPlainText>> different device types), the least loaded one is chosen.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - A placement policy could be to group all VIPs belonging to the <o:p></o:p></p><p class=MsoPlainText>> same tenant on the same device whenever possible.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - A placement policy could be to group all VIPs belonging to the <o:p></o:p></p><p class=MsoPlainText>> same network on the same device.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> All these are legitimate placement strategies/algorithms, and our <o:p></o:p></p><p class=MsoPlainText>> placement component might be very basic or very sophisticated, but we <o:p></o:p></p><p class=MsoPlainText>> can hide this from the rest of the system.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Nova has different scheduling drivers for this. We can use same <o:p></o:p></p><p class=MsoPlainText>> approach as well.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Now let's assume that Placement component working through some <o:p></o:p></p><p class=MsoPlainText>> combination of these rules, has finally chosen a driver/provider (e.g. <o:p></o:p></p><p class=MsoPlainText>> HA-Proxy) and a specific device (HA-Proxy device 1) or it decided to <o:p></o:p></p><p class=MsoPlainText>> create a new device in a driver (spawned new HA-Proxy VM, which is now <o:p></o:p></p><p class=MsoPlainText>> HA-Proxy device 2). Now it needs to wire the chosen device to Quantum <o:p></o:p></p><p class=MsoPlainText>> Network A and Network B (if it's not already wired to these networks). <o:p></o:p></p><p class=MsoPlainText>> This requires the Placement Component to call Quantum to do the wiring <o:p></o:p></p><p class=MsoPlainText>> (we need to figure out the interface between the 2). If the device is <o:p></o:p></p><p class=MsoPlainText>> a Nova VM, then this is easy as it's done like for any other VM. If <o:p></o:p></p><p class=MsoPlainText>> the device is physical then this depends on the L2 switch technology used in the Quantum service (VLAN, Linux-Bridge, etc.):<o:p></o:p></p><p class=MsoPlainText>> the physical device (or a proxy of it) needs to run a Quantum L2 agent <o:p></o:p></p><p class=MsoPlainText>> in order to wire the device correctly.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Agreed.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> After all this is done, the device is ready to be configured with a <o:p></o:p></p><p class=MsoPlainText>> VIP. The Placement Component can return the driver, device_id (and <o:p></o:p></p><p class=MsoPlainText>> possibly other config data, like the address chosen for the VIP) to <o:p></o:p></p><p class=MsoPlainText>> the LBaaS plugin, which proceeds to call the LBaaS agent in order to create the VIP on this device.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Agreed.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> If we can understand what are the tasks of the “placement component” <o:p></o:p></p><p class=MsoPlainText>> and the interactions this component needs to have with other <o:p></o:p></p><p class=MsoPlainText>> components, then it’s easier to figure out where it should run.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Recently we discussed an idea of separate plugin performing device <o:p></o:p></p><p class=MsoPlainText>> management and scheduling which will be a utility plugin for other <o:p></o:p></p><p class=MsoPlainText>> service plugins (not only lbaas).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I think we'll need at least some simple form of this component within <o:p></o:p></p><p class=MsoPlainText>> our lbaas efforts.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Youcef<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> From: Eugene Nikanorov [<a href="mailto:enikanorov@mirantis.com"><span style='color:windowtext;text-decoration:none'>mailto:enikanorov@mirantis.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> Sent: Monday, November 26, 2012 10:11 PM<o:p></o:p></p><p class=MsoPlainText>> To: OpenStack Development Mailing List<o:p></o:p></p><p class=MsoPlainText>> Subject: Re: [openstack-dev] [Quantum][LBaaS] Progress on <o:p></o:p></p><p class=MsoPlainText>> lbaas-plugin-api-crud<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Youcef,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Driver doesn't "choose" device-specific info, driver is <o:p></o:p></p><p class=MsoPlainText>> device-specific itself.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> When we send request to the agent, we need to specify which device to use.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> So once the user have chosen device type via service_type on VIP <o:p></o:p></p><p class=MsoPlainText>> creation, Quantum not only should associate VIP with device type, but <o:p></o:p></p><p class=MsoPlainText>> also it should choose particular instance of that device type to deploy the VIP.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> The process of choosing the instance is called scheduling. Unlike nova <o:p></o:p></p><p class=MsoPlainText>> it's unreasonable for LBaaS to have separate scheduler service, thus <o:p></o:p></p><p class=MsoPlainText>> it makes sense to have them built in the plugin.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I think we should not do this on agent since it doesn't have (and <o:p></o:p></p><p class=MsoPlainText>> should not<o:p></o:p></p><p class=MsoPlainText>> have) device database.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Not should it access quantum's database directly.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> So overall workflow will look like the following:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 1. Add a device (type, physical info) to device registry (this is a <o:p></o:p></p><p class=MsoPlainText>> part of Provider API. Call to Quantum made by cloud provider in case <o:p></o:p></p><p class=MsoPlainText>> of shared devices, or by tenant in case of private VM balancers)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 2. Create a VIP, specifying service type (=device type) (call by <o:p></o:p></p><p class=MsoPlainText>> tenant),<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 3. Choose device of specified type, associate the VIP with the device <o:p></o:p></p><p class=MsoPlainText>> (made by Quantum/Plugin)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 4. Send message with (logical VIP info, device_type, physical device <o:p></o:p></p><p class=MsoPlainText>> info) to LBaaS Agent (made by Quantum/Plugin)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 5. Communicate with particular device using driver according to <o:p></o:p></p><p class=MsoPlainText>> device_type (LBaaS Agent)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Any CRUD request processed by Agent should be supplied by device type <o:p></o:p></p><p class=MsoPlainText>> and device parameters.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> You may think of alternative approach where device registry is held by <o:p></o:p></p><p class=MsoPlainText>> the Agent or even driver, but this approach has the next disadvantages:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - Scheduling goes to Agent or Driver and thus Agent/Driver should <o:p></o:p></p><p class=MsoPlainText>> store VIP-device association while VIP is a "foreign" object for the Agent/Driver.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - If we go with multiple agents for large deployments, we'll need to <o:p></o:p></p><p class=MsoPlainText>> sync their device databases<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - Device locking will be complicated.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - If Agents will have non-intersecting sets of devices in their <o:p></o:p></p><p class=MsoPlainText>> registries than scheduling will be complicated or not possible.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Please share you thoughts on this.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Eugene.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On Tue, Nov 27, 2012 at 3:38 AM, Youcef Laribi <o:p></o:p></p><p class=MsoPlainText>> <<a href="mailto:Youcef.Laribi@eu.citrix.com"><span style='color:windowtext;text-decoration:none'>Youcef.Laribi@eu.citrix.com</span></a>><o:p></o:p></p><p class=MsoPlainText>> wrote:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Eugene, Leon,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Could we have the LBaaS plugin choose the “driver” based on <o:p></o:p></p><p class=MsoPlainText>> service_type info, and then it’s the driver which choose the “device”? <o:p></o:p></p><p class=MsoPlainText>> The driver can obviously have its own DB model where it stores device-specific info.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Youcef<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> From: Dan Wendlandt [<a href="mailto:dan@nicira.com"><span style='color:windowtext;text-decoration:none'>mailto:dan@nicira.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> Sent: Monday, November 26, 2012 9:13 AM<o:p></o:p></p><p class=MsoPlainText>> To: Leon Cui<o:p></o:p></p><p class=MsoPlainText>> Cc: OpenStack Development Mailing List; Salvatore Orlando<o:p></o:p></p><p class=MsoPlainText>> Subject: Re: [openstack-dev] <span style='font-family:SimSun'>答复</span>: <span style='font-family:SimSun'>答复</span>: <span style='font-family:SimSun'>答复</span>: <span style='font-family:SimSun'>答复</span>: <span style='font-family:SimSun'>答复</span>: Progress on <o:p></o:p></p><p class=MsoPlainText>> lbaas-plugin-api-crud<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On Mon, Nov 26, 2012 at 9:03 AM, Leon Cui <<a href="mailto:lcui@vmware.com"><span style='color:windowtext;text-decoration:none'>lcui@vmware.com</span></a>> wrote:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Eugene,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> When did you change get merged into master? I did rebase on last <o:p></o:p></p><p class=MsoPlainText>> Friday which supposed to be your latest code, but anyway I’m planning <o:p></o:p></p><p class=MsoPlainText>> to do it again today.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks for your reminder that I need to include device mgmt. into DB model.<o:p></o:p></p><p class=MsoPlainText>> Need to look at Salvatore’s change on ServiceType.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> It seems to me that each LB plugin should be able to define its own DB <o:p></o:p></p><p class=MsoPlainText>> models for "device mgmt" (e.g., device address/credentials/etc.), as <o:p></o:p></p><p class=MsoPlainText>> different plugins may have different strategies for how they manage devices.<o:p></o:p></p><p class=MsoPlainText>> The usual model is that plugins can define additional models/tables to <o:p></o:p></p><p class=MsoPlainText>> manage entities that are specific to that plugin. This is similar to <o:p></o:p></p><p class=MsoPlainText>> how we didn't back the notion of a "vlan" into the DB model for "core <o:p></o:p></p><p class=MsoPlainText>> plugins", since not all plugins will use vlans. If you don't go down <o:p></o:p></p><p class=MsoPlainText>> this route, you end up with a messy DB model as everyone keeps adding <o:p></o:p></p><p class=MsoPlainText>> columns for items that only a particular plugin needs to track.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Dan<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Leon<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>发件人</span>: Eugene Nikanorov [<a href="mailto:enikanorov@mirantis.com"><span style='color:windowtext;text-decoration:none'>mailto:enikanorov@mirantis.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>发送时间</span>: 2012<span style='font-family:SimSun'>年</span>11<span style='font-family:SimSun'>月</span>26<span style='font-family:SimSun'>日</span> 4:29<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>收件人</span>: Leon Cui<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>抄送</span>: Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando; <o:p></o:p></p><p class=MsoPlainText>> Dan Wendlandt<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>主题</span>: Re: <span style='font-family:SimSun'>答复</span>: <span style='font-family:SimSun'>答复</span>: <span style='font-family:SimSun'>答复</span>: <span style='font-family:SimSun'>答复</span>: Progress on lbaas-plugin-api-crud<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Leon,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks for sending me the patch.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I've looked at it briefly, there is one major thing I was able to identify:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> In order to couple things together (plugin, agent, drivers), we need <o:p></o:p></p><p class=MsoPlainText>> to add device management at least to DB model.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> In particular, each vip should have a reference to the device (which <o:p></o:p></p><p class=MsoPlainText>> has a type and address/credentials).<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> This information is passed in each agent notification message.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> This part is missing in current design blueprints but i think we need <o:p></o:p></p><p class=MsoPlainText>> to add it before we put the code on review.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Probably it will also depend on Salvatore's ServiceTypes part.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Also I see that your patch is based on some of my outdated patches.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> My code was recently merged into the master so you can rebase on <o:p></o:p></p><p class=MsoPlainText>> master using only Oleg's patch.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Eugene.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On Fri, Nov 23, 2012 at 2:40 PM, Leon Cui <<a href="mailto:lcui@vmware.com"><span style='color:windowtext;text-decoration:none'>lcui@vmware.com</span></a>> wrote:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Eugene,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I’m still waiting for approval as openstack contributor. For now I <o:p></o:p></p><p class=MsoPlainText>> simply attached the patch file that you might want to take a look <o:p></o:p></p><p class=MsoPlainText>> first. Once I got the approval, I’ll try to post the view asap.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Leon<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>发件人</span>: Eugene Nikanorov [<a href="mailto:enikanorov@mirantis.com"><span style='color:windowtext;text-decoration:none'>mailto:enikanorov@mirantis.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>发送时间</span>: 2012<span style='font-family:SimSun'>年</span>11<span style='font-family:SimSun'>月</span>20<span style='font-family:SimSun'>日</span> 22:57<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>收件人</span>: Leon Cui<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>抄送</span>: Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando; <o:p></o:p></p><p class=MsoPlainText>> Dan Wendlandt<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>主题</span>: Re: <span style='font-family:SimSun'>答复</span>: <span style='font-family:SimSun'>答复</span>: <span style='font-family:SimSun'>答复</span>: Progress on lbaas-plugin-api-crud<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Leon,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I'll take agent and rpc parts.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I have registered<o:p></o:p></p><p class=MsoPlainText>> <a href="https://blueprints.launchpad.net/quantum/+spec/lbaas-agent-and-rpc"><span style='color:windowtext;text-decoration:none'>https://blueprints.launchpad.net/quantum/+spec/lbaas-agent-and-rpc</span></a> to <o:p></o:p></p><p class=MsoPlainText>> track this.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Eugene.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On Tue, Nov 20, 2012 at 2:16 PM, Leon Cui <<a href="mailto:lcui@vmware.com"><span style='color:windowtext;text-decoration:none'>lcui@vmware.com</span></a>> wrote:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Eugene,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks for your suggestion. It looks good to me. I’ll work out the <o:p></o:p></p><p class=MsoPlainText>> UT first, and then align the class model to the diagram as you suggested.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Leon<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>发件人</span>: Eugene Nikanorov [<a href="mailto:enikanorov@mirantis.com"><span style='color:windowtext;text-decoration:none'>mailto:enikanorov@mirantis.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>发送时间</span>: 2012<span style='font-family:SimSun'>年</span>11<span style='font-family:SimSun'>月</span>20<span style='font-family:SimSun'>日</span> 17:32<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>收件人</span>: Leon Cui<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>抄送</span>: Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>主题</span>: Re: <span style='font-family:SimSun'>答复</span>: <span style='font-family:SimSun'>答复</span>: Progress on lbaas-plugin-api-crud<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> replying to all...<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Leon,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I think tests/unit/test_db_plugin.py is right code to refer when <o:p></o:p></p><p class=MsoPlainText>> writing unit tests for db code. The only thing is that unit tests <o:p></o:p></p><p class=MsoPlainText>> written in test_db_plugin.py are a bit generic, e.g. the backend <o:p></o:p></p><p class=MsoPlainText>> plugin is specified in particular plugin's UTs which inherit from <o:p></o:p></p><p class=MsoPlainText>> QuantumDbPluginV2TestCase. I think UTs for balancer plugin may be more <o:p></o:p></p><p class=MsoPlainText>> specific, testing LoadbalancerPluginDb class.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Since you need dababase utility methods from QuantumDbPluginV2 then <o:p></o:p></p><p class=MsoPlainText>> it's LoadbalancerPluginDb which should inherit from such QuantumDBBase <o:p></o:p></p><p class=MsoPlainText>> (or whatever you call it), so overall diagram will look like:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> ServicePluginBase<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> |<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> LoadBalancerPluginBase<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> |<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> | QuantumDBBase<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> | |<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> LoadBalancerPlugin <---------- LoadBalancerPluginDb<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Eugene.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On Tue, Nov 20, 2012 at 1:04 PM, Leon Cui <<a href="mailto:lcui@vmware.com"><span style='color:windowtext;text-decoration:none'>lcui@vmware.com</span></a>> wrote:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Eugene,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks for your suggestion. Please see my comments inline.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> One more question: I’m writing the unit test, mainly to verify the <o:p></o:p></p><p class=MsoPlainText>> database functionalities for LB CRUD. Do you think <o:p></o:p></p><p class=MsoPlainText>> tests/unit/test_db_plugin.py is the right test code that I should <o:p></o:p></p><p class=MsoPlainText>> refer to? Any good suggestions on this front?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Leon<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>发件人</span>: Eugene Nikanorov [<a href="mailto:enikanorov@mirantis.com"><span style='color:windowtext;text-decoration:none'>mailto:enikanorov@mirantis.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>发送时间</span>: 2012<span style='font-family:SimSun'>年</span>11<span style='font-family:SimSun'>月</span>20<span style='font-family:SimSun'>日</span> 16:44<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>收件人</span>: Leon Cui<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>抄送</span>: Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>主题</span>: Re: <span style='font-family:SimSun'>答复</span>: Progress on lbaas-plugin-api-crud<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Leon,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> A few thoughts on your diagram.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Please consider the following:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 1) If you want something from QuantumDbPluginV2 and you feel it may be <o:p></o:p></p><p class=MsoPlainText>> common plugin functionality - you need to extract it to a separate <o:p></o:p></p><p class=MsoPlainText>> class, something like QuantumPluginBase, and inherit QuantumDBPluginV2 <o:p></o:p></p><p class=MsoPlainText>> from this class, ServicePluginBase should inherit from that class as well.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> [Leon] I need some dababase utility methods from QuantumDbPluginV2.<o:p></o:p></p><p class=MsoPlainText>> Abstract to a separate class could be a good idea. But I’m not sure if <o:p></o:p></p><p class=MsoPlainText>> it’s a good idea to let ServicePluginBase to inherit from this class.<o:p></o:p></p><p class=MsoPlainText>> ServicePluginBase is an abstract class for service plugin service <o:p></o:p></p><p class=MsoPlainText>> (quantum<o:p></o:p></p><p class=MsoPlainText>> manager) to use.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 2) LoadBalancerPluginBase imho should inherit from ServicePluginBase<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> [Leon] Why it needs to inherit from ServicePluginBase?<o:p></o:p></p><p class=MsoPlainText>> LoadBalancerPluginBase defines the loadbalancer extension APIs. I <o:p></o:p></p><p class=MsoPlainText>> think we just make sure LoadbalancerPlugin inherits from both classes as below:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> ServicePluginBase QuantumPluginDbBase LoadbalancerPluginBase<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> | | |<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> ----------------------------------------------<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> |<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> LoadbalancerPlugin ------ LoadbalancerPluginDb<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> LoadbalancerPlugin will contain the LoadbalancerPluginDb instance for <o:p></o:p></p><p class=MsoPlainText>> database access.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> 3) Depending on what you need from <o:p></o:p></p><p class=MsoPlainText>> QuantumDbPluginV2/QuantumPluginBase, this may lead to the following inheritance sequence:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> QuantumPluginBase<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> |<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> ServicePluginBase<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> |<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> LoadBalancerPluginBase<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> |<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> LoadBalancerPluginDb<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> |<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> LoadBalancerPlugin<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Also, I think that LoadBalancerPlugin should not inherit <o:p></o:p></p><p class=MsoPlainText>> LoadBalancerPluginDb.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Unlike core plugins where it could make sense, I'd prefer to see <o:p></o:p></p><p class=MsoPlainText>> LoadBalancerPluginDb to be a part of LoadBalancerPlugin.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I mean LoadBalancerPlugin implements "has a" LoadBalancerPluginDb <o:p></o:p></p><p class=MsoPlainText>> instead of "is a" relation.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> The reason for this is that LoadBalancerPlugin provides CRUD <o:p></o:p></p><p class=MsoPlainText>> implementation which doesn't directly map to DB operations implemented <o:p></o:p></p><p class=MsoPlainText>> in LoadBalancerPluginDb.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> E.g. my idea is:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> LoadBalancerPlugin - CRUD, validation, calling LoadBalancerPluginDb, <o:p></o:p></p><p class=MsoPlainText>> sending/receiving messages to agent<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> LoadBalancerPluginDb - DB access.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Eugene.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> On Tue, Nov 20, 2012 at 6:54 AM, Leon Cui <<a href="mailto:lcui@vmware.com"><span style='color:windowtext;text-decoration:none'>lcui@vmware.com</span></a>> wrote:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Ilya,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Right now I took Eugene’s change under review<o:p></o:p></p><p class=MsoPlainText>> (<a href="https://review.openstack.org/#/c/15733/"><span style='color:windowtext;text-decoration:none'>https://review.openstack.org/#/c/15733/</span></a>) and am developing the <o:p></o:p></p><p class=MsoPlainText>> database access logic and plugin skeleton based on that service plugin <o:p></o:p></p><p class=MsoPlainText>> mechanism. The class model is illustrated in the below diagram:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> LoadBalancerPlugin module is the main body of loadbalancer plugin <o:p></o:p></p><p class=MsoPlainText>> which inherits from multiple classes:<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - ServicePluginBase: defines the abstract methods that a service<o:p></o:p></p><p class=MsoPlainText>> plugin should implemented.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - QuantumDbPluginV2: contains a set of generic quantum database<o:p></o:p></p><p class=MsoPlainText>> access methods. I’m not sure if we really want to inherit from this <o:p></o:p></p><p class=MsoPlainText>> class but I’d like to leverage the methods defined in this class.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> - LoadBalancerPluginDb: This the main part I’m coding on which wrap<o:p></o:p></p><p class=MsoPlainText>> the Lbaas database model and CRUD operation against the database.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> My thought is that LoadBalancerPlugin will control the LBaaS CRUD API flow.<o:p></o:p></p><p class=MsoPlainText>> For instance, “create_vip” method should first validate the input, <o:p></o:p></p><p class=MsoPlainText>> update the database, send message to the LbAgent over AMQP channel, <o:p></o:p></p><p class=MsoPlainText>> than update the database by setting the status from PENDING_CREATE to ACTIVE.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I’m trying to write unit tests against the database access now which <o:p></o:p></p><p class=MsoPlainText>> will take a while to complete. Meanwhile it would be great to have <o:p></o:p></p><p class=MsoPlainText>> your help on coding the RPC interaction between plugin and agent.<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> I don’t like blocking your part. What’s the best practice to <o:p></o:p></p><p class=MsoPlainText>> collaborate with you on this? Maybe I can shelve my change to you somehow?<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Leon<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>发件人</span>: Ilya Shakhat [<a href="mailto:ishakhat@mirantis.com"><span style='color:windowtext;text-decoration:none'>mailto:ishakhat@mirantis.com</span></a>]<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>发送时间</span>: 2012<span style='font-family:SimSun'>年</span>11<span style='font-family:SimSun'>月</span>19<span style='font-family:SimSun'>日</span> 22:08<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>收件人</span>: Sachin Thakkar; Leon Cui<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>抄送</span>: Eugene Nikanorov; Oleg Bondarev<o:p></o:p></p><p class=MsoPlainText>> <span style='font-family:SimSun'>主题</span>: Progress on lbaas-plugin-api-crud<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Hi Sachin, Leo,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Recently there was a thread related to LBaaS architecture <o:p></o:p></p><p class=MsoPlainText>> (<a href="http://lists.openstack.org/pipermail/openstack-dev/2012-November/002646.html"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/pipermail/openstack-dev/2012-November/002646.html</span></a>).<o:p></o:p></p><p class=MsoPlainText>> How good is it aligned with your implementation? Do you need help in coding?<o:p></o:p></p><p class=MsoPlainText>> (we may take Agent part)<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Thanks,<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Ilya<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> --<o:p></o:p></p><p class=MsoPlainText>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~<o:p></o:p></p><p class=MsoPlainText>> Dan Wendlandt<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> Nicira, Inc: <a href="http://www.nicira.com"><span style='color:windowtext;text-decoration:none'>www.nicira.com</span></a><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> twitter: danwendlandt<o:p></o:p></p><p class=MsoPlainText>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~<o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> _______________________________________________<o:p></o:p></p><p class=MsoPlainText>> OpenStack-dev mailing list<o:p></o:p></p><p class=MsoPlainText>> <a href="mailto:OpenStack-dev@lists.openstack.org"><span style='color:windowtext;text-decoration:none'>OpenStack-dev@lists.openstack.org</span></a><o:p></o:p></p><p class=MsoPlainText>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> _______________________________________________<o:p></o:p></p><p class=MsoPlainText>> OpenStack-dev mailing list<o:p></o:p></p><p class=MsoPlainText>> <a href="mailto:OpenStack-dev@lists.openstack.org"><span style='color:windowtext;text-decoration:none'>OpenStack-dev@lists.openstack.org</span></a><o:p></o:p></p><p class=MsoPlainText>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> _______________________________________________<o:p></o:p></p><p class=MsoPlainText>> OpenStack-dev mailing list<o:p></o:p></p><p class=MsoPlainText>> <a href="mailto:OpenStack-dev@lists.openstack.org"><span style='color:windowtext;text-decoration:none'>OpenStack-dev@lists.openstack.org</span></a><o:p></o:p></p><p class=MsoPlainText>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> _______________________________________________<o:p></o:p></p><p class=MsoPlainText>> OpenStack-dev mailing list<o:p></o:p></p><p class=MsoPlainText>> <a href="mailto:OpenStack-dev@lists.openstack.org"><span style='color:windowtext;text-decoration:none'>OpenStack-dev@lists.openstack.org</span></a><o:p></o:p></p><p class=MsoPlainText>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> _______________________________________________<o:p></o:p></p><p class=MsoPlainText>> OpenStack-dev mailing list<o:p></o:p></p><p class=MsoPlainText>> <a href="mailto:OpenStack-dev@lists.openstack.org"><span style='color:windowtext;text-decoration:none'>OpenStack-dev@lists.openstack.org</span></a><o:p></o:p></p><p class=MsoPlainText>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> _______________________________________________<o:p></o:p></p><p class=MsoPlainText>> OpenStack-dev mailing list<o:p></o:p></p><p class=MsoPlainText>> <a href="mailto:OpenStack-dev@lists.openstack.org"><span style='color:windowtext;text-decoration:none'>OpenStack-dev@lists.openstack.org</span></a><o:p></o:p></p><p class=MsoPlainText>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> _______________________________________________<o:p></o:p></p><p class=MsoPlainText>> OpenStack-dev mailing list<o:p></o:p></p><p class=MsoPlainText>> <a href="mailto:OpenStack-dev@lists.openstack.org"><span style='color:windowtext;text-decoration:none'>OpenStack-dev@lists.openstack.org</span></a><o:p></o:p></p><p class=MsoPlainText>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> _______________________________________________<o:p></o:p></p><p class=MsoPlainText>> OpenStack-dev mailing list<o:p></o:p></p><p class=MsoPlainText>> <a href="mailto:OpenStack-dev@lists.openstack.org"><span style='color:windowtext;text-decoration:none'>OpenStack-dev@lists.openstack.org</span></a><o:p></o:p></p><p class=MsoPlainText>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText>> _______________________________________________<o:p></o:p></p><p class=MsoPlainText>> OpenStack-dev mailing list<o:p></o:p></p><p class=MsoPlainText>> <a href="mailto:OpenStack-dev@lists.openstack.org"><span style='color:windowtext;text-decoration:none'>OpenStack-dev@lists.openstack.org</span></a><o:p></o:p></p><p class=MsoPlainText>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></p><p class=MsoPlainText>><o:p> </o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>_______________________________________________<o:p></o:p></p><p class=MsoPlainText>OpenStack-dev mailing list<o:p></o:p></p><p class=MsoPlainText><a href="mailto:OpenStack-dev@lists.openstack.org"><span style='color:windowtext;text-decoration:none'>OpenStack-dev@lists.openstack.org</span></a><o:p></o:p></p><p class=MsoPlainText><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev"><span style='color:windowtext;text-decoration:none'>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</span></a><o:p></o:p></p></div></body></html>