Open Stack

Eugene,

Please find my inlined reply.

Thanks
Anand

From: Eugene Nikanorov <enikanorov at mirantis.com<mailto:enikanorov at mirantis.com>>
Reply-To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Tue, 27 Nov 2012 12:48:26 +0400
To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [Quantum][LBaaS] Progress on lbaas-plugin-api-crud

Anand,

1. Service Type is generic set of types not limited to balancers (VendorX firewall, VendorY balancer, etc).
Salvatore will correct me if I'm wrong.
As far as I understand, service type applies to router, while device type describes load balancing device.
Therefore Service Type may map directly to balancer device type - it's detail of how Salvatore implements it.
That's why I said "device type via service_type"
[Anand] I do not expect that Tenant will specify the vendor product where the VIP should be created.
It is up-to the scheduler  to decide based on the available capability (it could be a extension capability also) and place the VIP appropriate LB.

2. I'm not sure what do you mean by multiple versions of the same device type.
Do you mean multiple versions of device driver?
I don't see why we should have multiple versions of device drivers loaded simultaneously.
[Anand] We have to run more than 1 software version per LB to meet operational requirements like upgrades,backward compatibility etc.
               For example, we will run Ver10 and Ver11 for LB Vendor Product1 and Ver8 & Ver9 for another product.
               Scheduler should pick the right driver to deal with the devices interaction.

3. I tried to cover it in this blueprint: http://wiki.openstack.org/Quantum/LBaaS/Agent
Feel free to comment
[Anand] The locking mechanism for handling the concurrency is considered and looks good to me. The DB approach does not seems to be scaling for large scale infrastructure. We used "Zoo Keeper" in eBay to meet the SLA. JFYI
                But we need to make the decision where we want to lock the entire device or VIP level. Also, some use cases make the changes to standby node and sync it over to Primary for safe guard the misconfiguration issues.

4. What do you mean by saying "LBaaS should handle its scheduling instead of the plugin."?
LBaaS is a plugin within the quantum.
[Anand] I meant the same. Driver is not going to schedule the LB operations. Also, we need a mechanism to track all these Operations in LBaaS. For example, when we lock the LB for stopping the concurrent modifications, the subsequent request will get queued and the tenant might interested in knowing the status of the LB changes that he requested. It would be better to report back the errors for a particular execution failure.

Thanks,
Eugene.

On Tue, Nov 27, 2012 at 10:41 AM, Palanisamy, Anand <apalanisamy at paypal.com<mailto:apalanisamy at paypal.com>> wrote:
Eugene,

We need to get more details about following before we conclude the implementation details.

  1.  Possible values for "device type via service_type"?
  2.  How does the multiple versions for each device type is handled? Agent or Plugin?
  3.  How is the concurrency config update issues and sync issues solved?
  4.  I think that LBaaS should handle its scheduling instead of the plugin.

Thanks
Anand

From: Eugene Nikanorov <enikanorov at mirantis.com<mailto:enikanorov at mirantis.com>>
Reply-To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Date: Tue, 27 Nov 2012 10:10:57 +0400
To: OpenStack Development Mailing List <openstack-dev at lists.openstack.org<mailto:openstack-dev at lists.openstack.org>>
Subject: Re: [openstack-dev] [Quantum][LBaaS] Progress on lbaas-plugin-api-crud

Hi Youcef,

Driver doesn't "choose" device-specific info, driver is device-specific itself.
When we send request to the agent, we need to specify which device to use.

So once the user have chosen device type via service_type on VIP creation, Quantum not only should associate VIP with device type, but also it should choose particular instance of that device type to deploy the VIP.
The process of choosing the instance is called scheduling. Unlike nova it's unreasonable for LBaaS to have separate scheduler service, thus it makes sense to have them built in the plugin.
I think we should not do this on agent since it doesn't have (and should not have) device database.
Not should it access quantum's database directly.

So overall workflow will look like the following:
1. Add a device (type, physical info) to device registry (this is a part of Provider API. Call to Quantum made by cloud provider in case of shared devices, or by tenant in case of private VM balancers)
2. Create a VIP, specifying service type (=device type) (call by tenant),
3. Choose device of specified type, associate the VIP with the device (made by Quantum/Plugin)
4. Send message with (logical VIP info, device_type, physical device info) to LBaaS Agent (made by Quantum/Plugin)
5. Communicate with particular device using driver according to device_type (LBaaS Agent)

Any CRUD request processed by Agent should be supplied by device type and device parameters.

You may think of alternative approach where device registry is held by the Agent or even driver, but this approach has the next disadvantages:
- Scheduling goes to Agent or Driver and thus Agent/Driver should store VIP-device association while VIP is a "foreign" object for the Agent/Driver.
- If we go with multiple agents for large deployments, we'll need to sync their device databases
- Device locking will be complicated.
- If Agents will have non-intersecting sets of devices in their registries than scheduling will be complicated or not possible.

Please share you thoughts on this.

Thanks,
Eugene.


On Tue, Nov 27, 2012 at 3:38 AM, Youcef Laribi <Youcef.Laribi at eu.citrix.com<mailto:Youcef.Laribi at eu.citrix.com>> wrote:
Hi Eugene, Leon,

Could we have the LBaaS plugin choose the “driver” based on service_type info, and then it’s the driver which choose the “device”? The driver can obviously have its own DB model where it stores device-specific info.

Youcef

From: Dan Wendlandt [mailto:dan at nicira.com<mailto:dan at nicira.com>]
Sent: Monday, November 26, 2012 9:13 AM
To: Leon Cui
Cc: OpenStack Development Mailing List; Salvatore Orlando
Subject: Re: [openstack-dev] 答复: 答复: 答复: 答复: 答复: Progress on lbaas-plugin-api-crud


On Mon, Nov 26, 2012 at 9:03 AM, Leon Cui <lcui at vmware.com<mailto:lcui at vmware.com>> wrote:
Hi Eugene,
When did you change get merged into master? I did rebase on last Friday which supposed to be your latest code, but anyway I’m planning to do it again today.

Thanks for your reminder that I need to include device mgmt. into DB model.  Need to look at Salvatore’s change on ServiceType.

It seems to me that each LB plugin should be able to define its own DB models for "device mgmt" (e.g., device address/credentials/etc.), as different plugins may have different strategies for how they manage devices.  The usual model is that plugins can define additional models/tables to manage entities that are specific to that plugin.  This is similar to how we didn't back the notion of a "vlan" into the DB model for "core plugins", since not all plugins will use vlans.  If you don't go down this route, you end up with a messy DB model as everyone keeps adding columns for items that only a particular plugin needs to track.

Dan



Thanks
Leon
发件人: Eugene Nikanorov [mailto:enikanorov at mirantis.com<mailto:enikanorov at mirantis.com>]
发送时间: 2012年11月26日 4:29

收件人: Leon Cui
抄送: Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando; Dan Wendlandt
主题: Re: 答复: 答复: 答复: 答复: Progress on lbaas-plugin-api-crud

Hi Leon,

Thanks for sending me the patch.
I've looked at it briefly, there is one major thing I was able to identify:
In order to couple things together (plugin, agent, drivers), we need to add device management at least to DB model.
In particular, each vip should have a reference to the device (which has a type and address/credentials).
This information is passed in each agent notification message.
This part is missing in current design blueprints but i think we need to add it before we put the code on review.
Probably it will also depend on Salvatore's ServiceTypes part.

Also I see that your patch is based on some of my outdated patches.
My code was recently merged into the master so you can rebase on master using only Oleg's patch.

Thanks,
Eugene.
On Fri, Nov 23, 2012 at 2:40 PM, Leon Cui <lcui at vmware.com<mailto:lcui at vmware.com>> wrote:
Hi Eugene,
I’m still waiting for approval as openstack contributor.  For now I simply attached the patch file that you might want to take a look first.  Once I got the approval, I’ll try to post the view asap.

Thanks
Leon
发件人: Eugene Nikanorov [mailto:enikanorov at mirantis.com<mailto:enikanorov at mirantis.com>]
发送时间: 2012年11月20日 22:57
收件人: Leon Cui
抄送: Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando; Dan Wendlandt
主题: Re: 答复: 答复: 答复: Progress on lbaas-plugin-api-crud

Leon,

I'll take agent and rpc parts.
I have registered https://blueprints.launchpad.net/quantum/+spec/lbaas-agent-and-rpc to track this.

Thanks,
Eugene.
On Tue, Nov 20, 2012 at 2:16 PM, Leon Cui <lcui at vmware.com<mailto:lcui at vmware.com>> wrote:
Hi Eugene,
Thanks for your suggestion.  It looks good to me.  I’ll work out the UT first, and then align the class model to the diagram as you suggested.

Thanks
Leon
发件人: Eugene Nikanorov [mailto:enikanorov at mirantis.com<mailto:enikanorov at mirantis.com>]
发送时间: 2012年11月20日 17:32

收件人: Leon Cui
抄送: Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando
主题: Re: 答复: 答复: Progress on lbaas-plugin-api-crud

replying to all...

Leon,

I think tests/unit/test_db_plugin.py  is right code to refer when writing unit tests for db code. The only thing is that unit tests written in test_db_plugin.py are a bit generic, e.g. the backend plugin is specified in particular plugin's UTs which inherit from QuantumDbPluginV2TestCase. I think UTs for balancer plugin may be more specific, testing LoadbalancerPluginDb class.

Since you need dababase utility methods from QuantumDbPluginV2 then it's LoadbalancerPluginDb which should inherit from such QuantumDBBase (or whatever you call it), so overall diagram will look like:

ServicePluginBase
    |
LoadBalancerPluginBase
    |
    |                                       QuantumDBBase
    |                                            |
LoadBalancerPlugin  <---------- LoadBalancerPluginDb

Thanks,
Eugene.

On Tue, Nov 20, 2012 at 1:04 PM, Leon Cui <lcui at vmware.com<mailto:lcui at vmware.com>> wrote:
Hi Eugene,
Thanks for your suggestion.  Please see my comments inline.

One more question: I’m writing the unit test, mainly to verify the database functionalities for LB CRUD.  Do you think tests/unit/test_db_plugin.py is the right test code that I should refer to?  Any good suggestions on this front?

Thanks
Leon
发件人: Eugene Nikanorov [mailto:enikanorov at mirantis.com<mailto:enikanorov at mirantis.com>]
发送时间: 2012年11月20日 16:44
收件人: Leon Cui
抄送: Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando
主题: Re: 答复: Progress on lbaas-plugin-api-crud

Hi Leon,

A few thoughts on your diagram.

Please consider the following:
1) If you want something from QuantumDbPluginV2 and you feel it may be common plugin functionality - you need to extract it to a separate class, something like QuantumPluginBase, and inherit QuantumDBPluginV2 from this class, ServicePluginBase should inherit from that class as well.
[Leon] I need some dababase utility methods from QuantumDbPluginV2.  Abstract to a separate class could be a good idea. But I’m not sure if it’s a good idea to let ServicePluginBase to inherit from this class. ServicePluginBase is an abstract class for service plugin service (quantum manager) to use.

2) LoadBalancerPluginBase imho should inherit from ServicePluginBase
[Leon] Why it needs to inherit from ServicePluginBase?   LoadBalancerPluginBase defines the loadbalancer extension APIs.  I think we just make sure LoadbalancerPlugin inherits from both classes as below:
ServicePluginBase    QuantumPluginDbBase   LoadbalancerPluginBase
       |                       |                      |
        ----------------------------------------------
                               |
                        LoadbalancerPlugin    ------ LoadbalancerPluginDb

LoadbalancerPlugin will contain the LoadbalancerPluginDb instance for database access.


3) Depending on what you need from QuantumDbPluginV2/QuantumPluginBase, this may lead to the following inheritance sequence:
QuantumPluginBase
    |
ServicePluginBase
    |
LoadBalancerPluginBase
    |
LoadBalancerPluginDb
    |
LoadBalancerPlugin

Also, I think that LoadBalancerPlugin should not inherit LoadBalancerPluginDb.
Unlike core plugins where it could make sense, I'd prefer to see LoadBalancerPluginDb to be a part of LoadBalancerPlugin.
I mean LoadBalancerPlugin implements "has a" LoadBalancerPluginDb  instead of "is a" relation.
The reason for this is that LoadBalancerPlugin provides CRUD implementation which doesn't directly map to DB operations implemented in LoadBalancerPluginDb.
E.g. my idea is:
LoadBalancerPlugin - CRUD, validation, calling LoadBalancerPluginDb, sending/receiving messages to agent
LoadBalancerPluginDb - DB access.

Thanks,
Eugene.

On Tue, Nov 20, 2012 at 6:54 AM, Leon Cui <lcui at vmware.com<mailto:lcui at vmware.com>> wrote:
Hi Ilya,
Right now I took Eugene’s change under review (https://review.openstack.org/#/c/15733/) and am developing the database access logic and plugin skeleton based on that service plugin mechanism.The class model is illustrated in the below diagram:


[ServicePluginBase  (quantum.plugins.services.service_base)  ,QuantumDbPluginV2 (quantum.db. db_base_plugin_v2),LoadBalancerPluginDb (quantum.plugins.services.loadbalancer.loadbalancer_db),LoadBalancerPluginBase (quantum.extensions.loadbalancer)]



[LoadBalancerPlugin (quantum.plugins.services.loadbalancer.loadbalancerPlugin)]



















LoadBalancerPlugin module is the main body of loadbalancer plugin which inherits from multiple classes:

-          ServicePluginBase: defines the abstract methods that a service plugin should implemented.

-          QuantumDbPluginV2: contains a set of generic quantum database access methods. I’m not sure if we really want to inherit from this class but I’d like to leverage the methods defined in this class.

-          LoadBalancerPluginDb: This the main part I’m coding on which wrap the Lbaas database model and CRUD operation against the database.

My thought is that LoadBalancerPlugin will control the LBaaS CRUD API flow. For instance, “create_vip” method should first validate the input, update the database, send message to the LbAgent over AMQP channel, than update the database by setting the status from PENDING_CREATE to ACTIVE.

I’m trying to write unit tests against the database access now which will take a while to complete. Meanwhile it would be great to have your help on coding the RPC interaction between plugin and agent.

I don’t like blocking your part. What’s the best practice to collaborate with you on this? Maybe I can shelve my change to you somehow?

Thanks
Leon
发件人: Ilya Shakhat [mailto:ishakhat at mirantis.com<mailto:ishakhat at mirantis.com>]
发送时间: 2012年11月19日 22:08
收件人: Sachin Thakkar; Leon Cui
抄送: Eugene Nikanorov; Oleg Bondarev
主题: Progress on lbaas-plugin-api-crud

Hi Sachin, Leo,

Recently there was a thread related to LBaaS architecture (http://lists.openstack.org/pipermail/openstack-dev/2012-November/002646.html). How good is it aligned with your implementation? Do you need help in coding? (we may take Agent part)

Thanks,
Ilya







--
~~~~~~~~~~~~~~~~~~~~~~~~~~~
Dan Wendlandt
Nicira, Inc: www.nicira.com<http://www.nicira.com>
twitter: danwendlandt
~~~~~~~~~~~~~~~~~~~~~~~~~~~


_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________ OpenStack-dev mailing list OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org>
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev


_______________________________________________ OpenStack-dev mailing list OpenStack-dev at lists.openstack.org<mailto:OpenStack-dev at lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121128/3202c40f/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 13351 bytes
Desc: image001.png
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121128/3202c40f/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1745 bytes
Desc: image002.png
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121128/3202c40f/attachment-0001.png>