Open Stack

Hi,

Just a small summary of our discussion. We have the following components:

   -  *aaS plugins - do the logic related to services. Plugins know service
   data model only and don't hold information about devices. When Plugin needs
   to deploy any changes, it calls Scheduler.
   - Scheduler ("placement component") - binds services to devices. It has
   API to manage devices (similar to provider api in old LBaaS). Scheduler
   knows how to find device by service_type and has DB to store them. When it
   gets request from Plugin, it finds corresponding device and forwards
   request to Agent
   - Agent - dispatches commands to drivers. Agent holds collection of
   drivers and knows how to dispatch message to them
   - Drivers - translate service model to device-specific.

Both Scheduler and Agent are common for all types of services. The logic
related to load balancing is implemented as drivers.

Please see http://wiki.openstack.org/Quantum/LBaaS/Architecture/Schedulerfor
details on how components interact and what the typical workflow will
be. Comments are welcome :)

Thanks,
Ilya

2012/11/28 Eugene Nikanorov <enikanorov at mirantis.com>

> Hi Youcef,
>
> Please see my comments inline.
>
> On Wed, Nov 28, 2012 at 2:14 AM, Youcef Laribi <
> Youcef.Laribi at eu.citrix.com> wrote:
>
>> Changing the subject line (was: Progress on lbaas-plugin-api-crud)…****
>>
>> ** **
>>
>> Hi Eugene,****
>>
>> ** **
>>
>> Let’s make sure we agree on the assumptions:****
>>
>> ** **
>>
>> **-          **LBaaS Plugin has a set of drivers (vendor-specific).
>>  Drivers run in the LBaaS agent process.
>>
> Agreed.
>
>> **-          **Each driver (provider in Salvatore’s terminology) is
>> registered against a service type (yes, service type can include LB
>> drivers, firewall drivers, etc.).
>>
> Agreed.
>
>> **-          **There can be several LBaaS drivers registered against the
>> same service type (e.g. “high-performance LB” service type).
>>
> That probably needs to be clarified in more detail, but it does make
> sense. As far as I understand there is exactly 1 driver per service type,
> but there could be several service types referencing the same driver (like
> you mentioned, "high-perf-lb", "low-cost-lb", etc)
>
>
>> If these assumptions are incorrect or need to be clarified further, let’s
>> start by doing this first J****
>>
>> ** **
>>
>> Now, let’s imagine we have a component in the system whose job is to pick
>> a driver/provider (device type) and a device
>>
> (device id) given a certain service type. We will call this component the
>> “placement component” (it’s not necessarily a  separate process like the
>> scheduler, and can be part of the plugin, the agent or the driver, it
>> doesn’t matter for this discussion at this stage).
>>
> I'd still prefer to call it a scheduler even though it will be a part of
> our plugin or separate component.
>
>> ****
>>
>> ** **
>>
>> The Placement Component needs to choose a device that can load-balance
>> traffic coming from network A (where the VIP is) to VMs residing on Network
>> B (pool’s network). In order to do this, the Placement Component needs to
>> be aware of the capabilities of each driver/provider and can follow a
>> certain strategy of device allocation that might take into account some of
>> the following constraints. ****
>>
>> ** **
>>
>>   - Some device types are physical appliances, others are virtual
>> appliances running on Nova. The driver might prefer one or the other if
>> both satisfy the service type.
>>
> Agreed.
>
>
>>   - Some device types have a fixed number of devices (e.g. physical
>> appliances), while other devices can be created at will whenever needed
>> (e.g. HA-Proxy VMs).
>>
> Agreed.
>
>
>>   - Some device types can host a high number of VIPs, others can host a
>> smaller number.
>>
> Agreed. Typically such factors are accounted during scheduling process.
>
> ****
>>
>> ** **
>>
>>   - Given a choice between multiple device types that satisfy the same
>> service type, preference could be given to a device that is already wired
>> to network A and network B.
>>
> Not sure that this is necessary, but that could be an option.
>
>
>>  ****
>>
>>                 ****
>>
>>   - Given a choice between several equivalent devices (possibly of
>> different device types), the least loaded one is chosen.****
>>
>>                 ****
>>
>>   - A placement policy could be to group all VIPs belonging to the same
>> tenant on the same device whenever possible.****
>>
>>   ****
>>
>>   - A placement policy could be to group all VIPs belonging to the same
>> network on the same device.****
>>
>> ** **
>>
>> All these are legitimate placement strategies/algorithms, and our
>> placement component might be very basic or very sophisticated, but we can
>> hide this from the rest of the system.
>>
> Nova has different scheduling drivers for this. We can use same approach
> as well.
>
> Now let's assume that Placement component working through some combination
>> of these rules, has finally chosen a driver/provider (e.g. HA-Proxy) and a
>> specific device (HA-Proxy device 1) or it decided to create a new device in
>> a driver (spawned new HA-Proxy VM, which is now HA-Proxy device 2). Now it
>> needs to wire the chosen device to Quantum Network A and Network B (if it's
>> not already wired to these networks).  This requires the Placement
>> Component to call Quantum to do the wiring (we need to figure out the
>> interface between the 2). If the device is a Nova VM, then this is easy as
>> it's done like for any other VM. If the device is physical then this
>> depends on the L2 switch technology used in the Quantum service (VLAN,
>> Linux-Bridge, etc.): the physical device (or a proxy of it) needs to run a
>> Quantum L2 agent in order to wire the device correctly.
>>
> Agreed.
>
>> ****
>>
>> ** **
>>
>> After all this is done, the device is ready to be configured with a VIP.
>> The Placement Component can return the driver, device_id (and possibly
>> other config data, like the address chosen for the VIP) to the LBaaS
>> plugin, which proceeds to call the LBaaS agent in order to create the VIP
>> on this device.
>>
> Agreed.
>
>>
>>
> **
>>
>> If we can understand what are the tasks of the “placement component” and
>> the interactions this component needs to have with other components, then
>> it’s easier to figure out where it should run.
>>
> Recently we discussed an idea of separate plugin performing device
> management and scheduling which will be a utility plugin for other service
> plugins (not only lbaas).
> I think we'll need at least some simple form of this component within our
> lbaas efforts.
>
>> ****
>>
>> ** **
>>
>> Youcef****
>>
>> ** **
>>
>> *From:* Eugene Nikanorov [mailto:enikanorov at mirantis.com]
>> *Sent:* Monday, November 26, 2012 10:11 PM
>> *To:* OpenStack Development Mailing List
>> *Subject:* Re: [openstack-dev] [Quantum][LBaaS] Progress on
>> lbaas-plugin-api-crud****
>>
>> ** **
>>
>> Hi Youcef,****
>>
>> ** **
>>
>> Driver doesn't "choose" device-specific info, driver is device-specific
>> itself. ****
>>
>> When we send request to the agent, we need to specify which device to use.
>> ****
>>
>> ** **
>>
>> So once the user have chosen device type via service_type on VIP
>> creation, Quantum not only should associate VIP with device type, but also
>> it should choose particular instance of that device type to deploy the VIP.
>> ****
>>
>> The process of choosing the instance is called scheduling. Unlike nova
>> it's unreasonable for LBaaS to have separate scheduler service, thus it
>> makes sense to have them built in the plugin.****
>>
>> I think we should not do this on agent since it doesn't have (and should
>> not have) device database.****
>>
>> Not should it access quantum's database directly.****
>>
>> ** **
>>
>> So overall workflow will look like the following:****
>>
>> 1. Add a device (type, physical info) to device registry (this is a part
>> of Provider API. Call to Quantum made by cloud provider in case of shared
>> devices, or by tenant in case of private VM balancers) ****
>>
>> 2. Create a VIP, specifying service type (=device type) (call by tenant),
>> ****
>>
>> 3. Choose device of specified type, associate the VIP with the device
>> (made by Quantum/Plugin)****
>>
>> 4. Send message with (logical VIP info, device_type, physical device
>> info) to LBaaS Agent (made by Quantum/Plugin)****
>>
>> 5. Communicate with particular device using driver according to
>> device_type (LBaaS Agent)****
>>
>> ** **
>>
>> Any CRUD request processed by Agent should be supplied by device type and
>> device parameters.****
>>
>> ** **
>>
>> You may think of alternative approach where device registry is held by
>> the Agent or even driver, but this approach has the next disadvantages:**
>> **
>>
>> - Scheduling goes to Agent or Driver and thus Agent/Driver should store
>> VIP-device association while VIP is a "foreign" object for the Agent/Driver.
>> ****
>>
>> - If we go with multiple agents for large deployments, we'll need to sync
>> their device databases****
>>
>> - Device locking will be complicated.****
>>
>> - If Agents will have non-intersecting sets of devices in their
>> registries than scheduling will be complicated or not possible.****
>>
>> ** **
>>
>> Please share you thoughts on this.****
>>
>> ** **
>>
>> Thanks,****
>>
>> Eugene.****
>>
>> ** **
>>
>> ** **
>>
>> On Tue, Nov 27, 2012 at 3:38 AM, Youcef Laribi <
>> Youcef.Laribi at eu.citrix.com> wrote:****
>>
>> Hi Eugene, Leon,****
>>
>>  ****
>>
>> Could we have the LBaaS plugin choose the “driver” based on service_type
>> info, and then it’s the driver which choose the “device”? The driver can
>> obviously have its own DB model where it stores device-specific info.****
>>
>>  ****
>>
>> Youcef****
>>
>>  ****
>>
>> *From:* Dan Wendlandt [mailto:dan at nicira.com]
>> *Sent:* Monday, November 26, 2012 9:13 AM
>> *To:* Leon Cui
>> *Cc:* OpenStack Development Mailing List; Salvatore Orlando
>> *Subject:* Re: [openstack-dev] 答复: 答复: 答复: 答复: 答复: Progress on
>> lbaas-plugin-api-crud****
>>
>>  ****
>>
>>  ****
>>
>> On Mon, Nov 26, 2012 at 9:03 AM, Leon Cui <lcui at vmware.com> wrote:****
>>
>> Hi Eugene,****
>>
>> When did you change get merged into master? I did rebase on last Friday
>> which supposed to be your latest code, but anyway I’m planning to do it
>> again today. ****
>>
>>  ****
>>
>> Thanks for your reminder that I need to include device mgmt. into DB
>> model.  Need to look at Salvatore’s change on ServiceType.****
>>
>>  ****
>>
>> It seems to me that each LB plugin should be able to define its own DB
>> models for "device mgmt" (e.g., device address/credentials/etc.), as
>> different plugins may have different strategies for how they manage
>> devices.  The usual model is that plugins can define additional
>> models/tables to manage entities that are specific to that plugin.  This is
>> similar to how we didn't back the notion of a "vlan" into the DB model for
>> "core plugins", since not all plugins will use vlans.  If you don't go down
>> this route, you end up with a messy DB model as everyone keeps adding
>> columns for items that only a particular plugin needs to track.  ****
>>
>>  ****
>>
>> Dan****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>> Thanks****
>>
>> Leon****
>>
>> *发件人**:* Eugene Nikanorov [mailto:enikanorov at mirantis.com]
>> *发送时间:* 2012年11月26日 4:29****
>>
>>
>> *收件人:* Leon Cui
>> *抄送:* Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando;
>> Dan Wendlandt****
>>
>> *主题:* Re: 答复: 答复: 答复: 答复: Progress on lbaas-plugin-api-crud****
>>
>>  ****
>>
>> Hi Leon,****
>>
>>  ****
>>
>> Thanks for sending me the patch.****
>>
>> I've looked at it briefly, there is one major thing I was able to
>> identify:****
>>
>> In order to couple things together (plugin, agent, drivers), we need to
>> add device management at least to DB model.****
>>
>> In particular, each vip should have a reference to the device (which has
>> a type and address/credentials). ****
>>
>> This information is passed in each agent notification message.****
>>
>> This part is missing in current design blueprints but i think we need to
>> add it before we put the code on review.****
>>
>> Probably it will also depend on Salvatore's ServiceTypes part.****
>>
>>  ****
>>
>> Also I see that your patch is based on some of my outdated patches. ****
>>
>> My code was recently merged into the master so you can rebase on master
>> using only Oleg's patch.****
>>
>>  ****
>>
>> Thanks,****
>>
>> Eugene.****
>>
>> On Fri, Nov 23, 2012 at 2:40 PM, Leon Cui <lcui at vmware.com> wrote:****
>>
>> Hi Eugene,****
>>
>> I’m still waiting for approval as openstack contributor.  For now I
>> simply attached the patch file that you might want to take a look first.
>> Once I got the approval, I’ll try to post the view asap.****
>>
>>  ****
>>
>> Thanks****
>>
>> Leon****
>>
>> *发件人**:* Eugene Nikanorov [mailto:enikanorov at mirantis.com]
>> *发送时间:* 2012年11月20日 22:57
>> *收件人:* Leon Cui
>> *抄送:* Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando;
>> Dan Wendlandt
>> *主题:* Re: 答复: 答复: 答复: Progress on lbaas-plugin-api-crud****
>>
>>  ****
>>
>> Leon,****
>>
>>  ****
>>
>> I'll take agent and rpc parts.****
>>
>> I have registered
>> https://blueprints.launchpad.net/quantum/+spec/lbaas-agent-and-rpc to
>> track this.****
>>
>>  ****
>>
>> Thanks,****
>>
>> Eugene.****
>>
>> On Tue, Nov 20, 2012 at 2:16 PM, Leon Cui <lcui at vmware.com> wrote:****
>>
>> Hi Eugene,****
>>
>> Thanks for your suggestion.  It looks good to me.  I’ll work out the UT
>> first, and then align the class model to the diagram as you suggested.***
>> *
>>
>>  ****
>>
>> Thanks****
>>
>> Leon****
>>
>> *发件人**:* Eugene Nikanorov [mailto:enikanorov at mirantis.com]
>> *发送时间:* 2012年11月20日 17:32****
>>
>>
>> *收件人:* Leon Cui
>> *抄送:* Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando****
>>
>> *主题:* Re: 答复: 答复: Progress on lbaas-plugin-api-crud****
>>
>>  ****
>>
>> replying to all...****
>>
>>  ****
>>
>> Leon,****
>>
>>  ****
>>
>> I think tests/unit/test_db_plugin.py  is right code to refer when
>> writing unit tests for db code. The only thing is that unit tests written
>> in test_db_plugin.py are a bit generic, e.g. the backend plugin is
>> specified in particular plugin's UTs which inherit
>> from QuantumDbPluginV2TestCase. I think UTs for balancer plugin may be more
>> specific, testing LoadbalancerPluginDb class.****
>>
>>  ****
>>
>> Since you need dababase utility methods from QuantumDbPluginV2 then
>> it's LoadbalancerPluginDb which should inherit from such QuantumDBBase (or
>> whatever you call it), so overall diagram will look like:****
>>
>>  ****
>>
>> ServicePluginBase****
>>
>>     |****
>>
>> LoadBalancerPluginBase ****
>>
>>     |****
>>
>>     |                                       QuantumDBBase ****
>>
>>     |                                            |****
>>
>> LoadBalancerPlugin  <---------- LoadBalancerPluginDb****
>>
>>  ****
>>
>> Thanks,****
>>
>> Eugene.****
>>
>>  ****
>>
>> On Tue, Nov 20, 2012 at 1:04 PM, Leon Cui <lcui at vmware.com> wrote:****
>>
>> Hi Eugene,****
>>
>> Thanks for your suggestion.  Please see my comments inline.****
>>
>>  ****
>>
>> One more question: I’m writing the unit test, mainly to verify the
>> database functionalities for LB CRUD.  Do you think
>> tests/unit/test_db_plugin.py is the right test code that I should refer
>> to?  Any good suggestions on this front?****
>>
>>  ****
>>
>> Thanks****
>>
>> Leon****
>>
>> *发件人**:* Eugene Nikanorov [mailto:enikanorov at mirantis.com]
>> *发送时间:* 2012年11月20日 16:44
>> *收件人:* Leon Cui
>> *抄送:* Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando
>> *主题:* Re: 答复: Progress on lbaas-plugin-api-crud****
>>
>>  ****
>>
>> Hi Leon,****
>>
>>  ****
>>
>> A few thoughts on your diagram.****
>>
>>  ****
>>
>> Please consider the following:****
>>
>> 1) If you want something from QuantumDbPluginV2 and you feel it may be
>> common plugin functionality - you need to extract it to a separate class,
>> something like QuantumPluginBase, and inherit QuantumDBPluginV2 from this
>> class, ServicePluginBase should inherit from that class as well.****
>>
>> [Leon] I need some dababase utility methods from QuantumDbPluginV2.
>> Abstract to a separate class could be a good idea. But I’m not sure if it
>> ’s a good idea to let ServicePluginBase to inherit from this class.
>> ServicePluginBase is an abstract class for service plugin service (quantum
>> manager) to use. ****
>>
>>  ****
>>
>> 2) LoadBalancerPluginBase imho should inherit from ServicePluginBase****
>>
>> [Leon] Why it needs to inherit from ServicePluginBase?
>> LoadBalancerPluginBase defines the loadbalancer extension APIs.  I think we
>> just make sure LoadbalancerPlugin inherits from both classes as below:***
>> *
>>
>> ServicePluginBase    QuantumPluginDbBase   LoadbalancerPluginBase****
>>
>>        |                       |                      |****
>>
>>         ----------------------------------------------****
>>
>>                                |****
>>
>>                         LoadbalancerPlugin    ------ LoadbalancerPluginDb
>> ****
>>
>>               ****
>>
>> LoadbalancerPlugin will contain the LoadbalancerPluginDb instance for
>> database access.****
>>
>>  ****
>>
>>  ****
>>
>> 3) Depending on what you need from QuantumDbPluginV2/QuantumPluginBase,
>> this may lead to the following inheritance sequence:****
>>
>> QuantumPluginBase****
>>
>>     |****
>>
>> ServicePluginBase****
>>
>>     |****
>>
>> LoadBalancerPluginBase ****
>>
>>     |****
>>
>> LoadBalancerPluginDb****
>>
>>     |****
>>
>> LoadBalancerPlugin****
>>
>>  ****
>>
>> Also, I think that LoadBalancerPlugin should not
>> inherit LoadBalancerPluginDb. ****
>>
>> Unlike core plugins where it could make sense, I'd prefer to
>> see LoadBalancerPluginDb to be a part of LoadBalancerPlugin.****
>>
>> I mean LoadBalancerPlugin implements "has a" LoadBalancerPluginDb
>> instead of "is a" relation. ****
>>
>> The reason for this is that LoadBalancerPlugin provides CRUD
>> implementation which doesn't directly map to DB operations implemented
>> in LoadBalancerPluginDb. ****
>>
>> E.g. my idea is:****
>>
>> LoadBalancerPlugin - CRUD, validation, calling LoadBalancerPluginDb,
>> sending/receiving messages to agent****
>>
>> LoadBalancerPluginDb - DB access.****
>>
>>  ****
>>
>> Thanks,****
>>
>> Eugene.****
>>
>>  ****
>>
>> On Tue, Nov 20, 2012 at 6:54 AM, Leon Cui <lcui at vmware.com> wrote:****
>>
>> Hi Ilya,****
>>
>> Right now I took Eugene’s change under review (
>> https://review.openstack.org/#/c/15733/) and am developing the database
>> access logic and plugin skeleton based on that service plugin mechanism. The
>> class model is illustrated in the below diagram:****
>>
>> [image: ServicePluginBase (quantum.plugins.services.service_base)
>> ,QuantumDbPluginV2 (quantum.db. db_base_plugin_v2),LoadBalancerPluginDb
>> (quantum.plugins.services.loadbalancer.loadbalancer_db),LoadBalancerPluginBase
>> (quantum.extensions.loadbalancer)]****
>>
>> [image: LoadBalancerPlugin
>> (quantum.plugins.services.loadbalancer.loadbalancerPlugin)]****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>
>> ****
>>
>> LoadBalancerPlugin module is the main body of loadbalancer plugin which
>> inherits from multiple classes:****
>>
>> -          ServicePluginBase: defines the abstract methods that a
>> service plugin should implemented.****
>>
>> -          QuantumDbPluginV2: contains a set of generic quantum database
>> access methods. I’m not sure if we really want to inherit from this
>> class but I’d like to leverage the methods defined in this class.****
>>
>> -          LoadBalancerPluginDb: This the main part I’m coding on which
>> wrap the Lbaas database model and CRUD operation against the database.***
>> *
>>
>>  ****
>>
>> My thought is that LoadBalancerPlugin will control the LBaaS CRUD API
>> flow. For instance, “create_vip” method should first validate the input,
>> update the database, send message to the LbAgent over AMQP channel, than
>> update the database by setting the status from PENDING_CREATE to ACTIVE.*
>> ***
>>
>>  ****
>>
>> I’m trying to write unit tests against the database access now which will
>> take a while to complete. Meanwhile it would be great to have your help on
>> coding the RPC interaction between plugin and agent.****
>>
>>  ****
>>
>> I don’t like blocking your part. What’s the best practice to collaborate
>> with you on this? Maybe I can shelve my change to you somehow?****
>>
>>  ****
>>
>> Thanks****
>>
>> Leon****
>>
>> *发件人**:* Ilya Shakhat [mailto:ishakhat at mirantis.com]
>> *发送时间:* 2012年11月19日 22:08
>> *收件人:* Sachin Thakkar; Leon Cui
>> *抄送:* Eugene Nikanorov; Oleg Bondarev
>> *主题:* Progress on lbaas-plugin-api-crud****
>>
>>  ****
>>
>> Hi Sachin, Leo, ****
>>
>>  ****
>>
>> Recently there was a thread related to LBaaS architecture (
>> http://lists.openstack.org/pipermail/openstack-dev/2012-November/002646.html).
>> How good is it aligned with your implementation? Do you need help in
>> coding? (we may take Agent part) ****
>>
>>  ****
>>
>> Thanks,****
>>
>> Ilya****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>  ****
>>
>>
>>
>> ****
>>
>>  ****
>>
>> --
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>> Dan Wendlandt ****
>>
>> Nicira, Inc: www.nicira.com****
>>
>> twitter: danwendlandt
>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~****
>>
>>  ****
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev****
>>
>> ** **
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121129/1f19c19b/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 13351 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121129/1f19c19b/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1745 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121129/1f19c19b/attachment-0001.png>