Open Stack

Hi Sam,

Why do you think scheduler should be vendor-specific?
Typically it's a mechanism to select a device for a service.
We may implement scheduling methods to be configurable (say, dependent on
service type) but why should it be vendor-specific?

Thanks,
Eugene.

On Thu, Nov 29, 2012 at 11:44 PM, Samuel Bercovici <SamuelB at radware.com>wrote:

>  Hi,****
>
> ** **
>
> *The Scheduler might be different for different vendors.*
>
> *The assumption that this is a one size fits all is wrong.*
>
> I think that the way to handle this could be:****
>
>    -  *aaS plugins - do the logic related to services. Plugins know
>    service data model only and don't hold information about devices. When
>    Plugin needs to deploy any changes, it calls Scheduler.****
>    - Agent - dispatches asynchronously commands to drivers. ****
>    - Driver can implement a Scheduler or any other logic to do it¡¯s stuff*
>    ***
>    - Optionally - Scheduler ("placement component")  is an internal
>    component for the Driver. ¨C we might have a reference implementation
>    if a vendor wishes to use the default logic.****
>
> Regards,****
>
>                 -Sam.****
>
> ** **
>
> ** **
>
> *From:* Ilya Shakhat [mailto:ishakhat at mirantis.com]
> *Sent:* Thursday, November 29, 2012 5:34 PM
>
> *To:* OpenStack Development Mailing List
> *Subject:* Re: [openstack-dev] [Quantum][LBaaS] Selecting an LBaaS device
> given a service type****
>
>  ** **
>
> Hi,****
>
> ** **
>
> Just a small summary of our discussion. We have the following components:*
> ***
>
>    -  *aaS plugins - do the logic related to services. Plugins know
>    service data model only and don't hold information about devices. When
>    Plugin needs to deploy any changes, it calls Scheduler.****
>    - Scheduler ("placement component") - binds services to devices. It
>    has API to manage devices (similar to provider api in old LBaaS). Scheduler
>    knows how to find device by service_type and has DB to store them. When it
>    gets request from Plugin, it finds corresponding device and forwards
>    request to Agent****
>    - Agent - dispatches commands to drivers. Agent holds collection of
>    drivers and knows how to dispatch message to them****
>    - Drivers - translate service model to device-specific.****
>
> Both Scheduler and Agent are common for all types of services. The logic
> related to load balancing is implemented as drivers. ****
>
> ** **
>
> Please see http://wiki.openstack.org/Quantum/LBaaS/Architecture/Schedulerfor details on how components interact and what the typical workflow will
> be. Comments are welcome :)****
>
> ** **
>
> Thanks,****
>
> Ilya****
>
> ** **
>
> 2012/11/28 Eugene Nikanorov <enikanorov at mirantis.com>****
>
> Hi Youcef,****
>
> ** **
>
> Please see my comments inline.****
>
> On Wed, Nov 28, 2012 at 2:14 AM, Youcef Laribi <
> Youcef.Laribi at eu.citrix.com> wrote:****
>
> Changing the subject line (was: Progress on lbaas-plugin-api-crud)¡­****
>
>  ****
>
> Hi Eugene,****
>
>  ****
>
> Let¡¯s make sure we agree on the assumptions:****
>
>  ****
>
> -          LBaaS Plugin has a set of drivers (vendor-specific).  Drivers
> run in the LBaaS agent process.****
>
> Agreed. ****
>
>  -          Each driver (provider in Salvatore¡¯s terminology) is
> registered against a service type (yes, service type can include LB
> drivers, firewall drivers, etc.).****
>
>  Agreed. ****
>
>  -          There can be several LBaaS drivers registered against the
> same service type (e.g. ¡°high-performance LB¡± service type).****
>
>  That probably needs to be clarified in more detail, but it does make
> sense. As far as I understand there is exactly 1 driver per service type,
> but there could be several service types referencing the same driver (like
> you mentioned, "high-perf-lb", "low-cost-lb", etc)****
>
>  ****
>
>  If these assumptions are incorrect or need to be clarified further,
> let¡¯s start by doing this first J****
>
>  ****
>
> Now, let¡¯s imagine we have a component in the system whose job is to pick
> a driver/provider (device type) and a device ****
>
>  (device id) given a certain service type. We will call this component
> the ¡°placement component¡± (it¡¯s not necessarily a  separate process like
> the scheduler, and can be part of the plugin, the agent or the driver, it
> doesn¡¯t matter for this discussion at this stage).****
>
>  I'd still prefer to call it a scheduler even though it will be a part of
> our plugin or separate component. ****
>
>   ****
>
> The Placement Component needs to choose a device that can load-balance
> traffic coming from network A (where the VIP is) to VMs residing on Network
> B (pool¡¯s network). In order to do this, the Placement Component needs to
> be aware of the capabilities of each driver/provider and can follow a
> certain strategy of device allocation that might take into account some of
> the following constraints. ****
>
>  ****
>
>   - Some device types are physical appliances, others are virtual
> appliances running on Nova. The driver might prefer one or the other if
> both satisfy the service type.****
>
>  Agreed.****
>
>  ****
>
>    - Some device types have a fixed number of devices (e.g. physical
> appliances), while other devices can be created at will whenever needed
> (e.g. HA-Proxy VMs).****
>
>  Agreed.****
>
>  ****
>
>    - Some device types can host a high number of VIPs, others can host a
> smaller number.****
>
>  Agreed. Typically such factors are accounted during scheduling process.**
> **
>
> ** **
>
>   ****
>
>   - Given a choice between multiple device types that satisfy the same
> service type, preference could be given to a device that is already wired
> to network A and network B. ****
>
>  Not sure that this is necessary, but that could be an option.****
>
>  ****
>
>                  ****
>
>   - Given a choice between several equivalent devices (possibly of
> different device types), the least loaded one is chosen.****
>
>                 ****
>
>   - A placement policy could be to group all VIPs belonging to the same
> tenant on the same device whenever possible.****
>
>   ****
>
>   - A placement policy could be to group all VIPs belonging to the same
> network on the same device.****
>
>  ****
>
> All these are legitimate placement strategies/algorithms, and our
> placement component might be very basic or very sophisticated, but we can
> hide this from the rest of the system.  ****
>
>  Nova has different scheduling drivers for this. We can use same approach
> as well.****
>
> ** **
>
>  Now let's assume that Placement component working through some
> combination of these rules, has finally chosen a driver/provider (e.g.
> HA-Proxy) and a specific device (HA-Proxy device 1) or it decided to create
> a new device in a driver (spawned new HA-Proxy VM, which is now HA-Proxy
> device 2). Now it needs to wire the chosen device to Quantum Network A and
> Network B (if it's not already wired to these networks).  This requires the
> Placement Component to call Quantum to do the wiring (we need to figure out
> the interface between the 2). If the device is a Nova VM, then this is easy
> as it's done like for any other VM. If the device is physical then this
> depends on the L2 switch technology used in the Quantum service (VLAN,
> Linux-Bridge, etc.): the physical device (or a proxy of it) needs to run a
> Quantum L2 agent in order to wire the device correctly.****
>
>  Agreed. ****
>
>   ****
>
> After all this is done, the device is ready to be configured with a VIP.
> The Placement Component can return the driver, device_id (and possibly
> other config data, like the address chosen for the VIP) to the LBaaS
> plugin, which proceeds to call the LBaaS agent in order to create the VIP
> on this device.****
>
>  Agreed. ****
>
>   ****
>
>  If we can understand what are the tasks of the ¡°placement component¡± and
> the interactions this component needs to have with other components, then
> it¡¯s easier to figure out where it should run.****
>
>  Recently we discussed an idea of separate plugin performing device
> management and scheduling which will be a utility plugin for other service
> plugins (not only lbaas). ****
>
> I think we'll need at least some simple form of this component within our
> lbaas efforts.****
>
>   ****
>
> Youcef****
>
>  ****
>
> *From:* Eugene Nikanorov [mailto:enikanorov at mirantis.com]
> *Sent:* Monday, November 26, 2012 10:11 PM
> *To:* OpenStack Development Mailing List
> *Subject:* Re: [openstack-dev] [Quantum][LBaaS] Progress on
> lbaas-plugin-api-crud****
>
>  ****
>
> Hi Youcef,****
>
>  ****
>
> Driver doesn't "choose" device-specific info, driver is device-specific
> itself. ****
>
> When we send request to the agent, we need to specify which device to use.
> ****
>
>  ****
>
> So once the user have chosen device type via service_type on VIP
> creation, Quantum not only should associate VIP with device type, but also
> it should choose particular instance of that device type to deploy the VIP.
> ****
>
> The process of choosing the instance is called scheduling. Unlike nova
> it's unreasonable for LBaaS to have separate scheduler service, thus it
> makes sense to have them built in the plugin.****
>
> I think we should not do this on agent since it doesn't have (and should
> not have) device database.****
>
> Not should it access quantum's database directly.****
>
>  ****
>
> So overall workflow will look like the following:****
>
> 1. Add a device (type, physical info) to device registry (this is a part
> of Provider API. Call to Quantum made by cloud provider in case of shared
> devices, or by tenant in case of private VM balancers) ****
>
> 2. Create a VIP, specifying service type (=device type) (call by tenant),
> ****
>
> 3. Choose device of specified type, associate the VIP with the device
> (made by Quantum/Plugin)****
>
> 4. Send message with (logical VIP info, device_type, physical device info)
> to LBaaS Agent (made by Quantum/Plugin)****
>
> 5. Communicate with particular device using driver according to
> device_type (LBaaS Agent)****
>
>  ****
>
> Any CRUD request processed by Agent should be supplied by device type and
> device parameters.****
>
>  ****
>
> You may think of alternative approach where device registry is held by the
> Agent or even driver, but this approach has the next disadvantages:****
>
> - Scheduling goes to Agent or Driver and thus Agent/Driver should store
> VIP-device association while VIP is a "foreign" object for the Agent/Driver.
> ****
>
> - If we go with multiple agents for large deployments, we'll need to sync
> their device databases****
>
> - Device locking will be complicated.****
>
> - If Agents will have non-intersecting sets of devices in their registries
> than scheduling will be complicated or not possible.****
>
>  ****
>
> Please share you thoughts on this.****
>
>  ****
>
> Thanks,****
>
> Eugene.****
>
>  ****
>
>  ****
>
> On Tue, Nov 27, 2012 at 3:38 AM, Youcef Laribi <
> Youcef.Laribi at eu.citrix.com> wrote:****
>
> Hi Eugene, Leon,****
>
>  ****
>
> Could we have the LBaaS plugin choose the ¡°driver¡± based on service_type
> info, and then it¡¯s the driver which choose the ¡°device¡±? The driver can
> obviously have its own DB model where it stores device-specific info.****
>
>  ****
>
> Youcef****
>
>  ****
>
> *From:* Dan Wendlandt [mailto:dan at nicira.com]
> *Sent:* Monday, November 26, 2012 9:13 AM
> *To:* Leon Cui
> *Cc:* OpenStack Development Mailing List; Salvatore Orlando
> *Subject:* Re: [openstack-dev] ´ð¸´: ´ð¸´: ´ð¸´: ´ð¸´: ´ð¸´: Progress on
> lbaas-plugin-api-crud****
>
>  ****
>
>  ****
>
> On Mon, Nov 26, 2012 at 9:03 AM, Leon Cui <lcui at vmware.com> wrote:****
>
> Hi Eugene,****
>
> When did you change get merged into master? I did rebase on last Friday
> which supposed to be your latest code, but anyway I¡¯m planning to do it
> again today. ****
>
>  ****
>
> Thanks for your reminder that I need to include device mgmt. into DB
> model.  Need to look at Salvatore¡¯s change on ServiceType.****
>
>  ****
>
> It seems to me that each LB plugin should be able to define its own DB
> models for "device mgmt" (e.g., device address/credentials/etc.), as
> different plugins may have different strategies for how they manage
> devices.  The usual model is that plugins can define additional
> models/tables to manage entities that are specific to that plugin.  This
> is similar to how we didn't back the notion of a "vlan" into the DB model
> for "core plugins", since not all plugins will use vlans.  If you don't
> go down this route, you end up with a messy DB model as everyone keeps
> adding columns for items that only a particular plugin needs to track.  **
> **
>
>  ****
>
> Dan****
>
>  ****
>
>  ****
>
>   ****
>
> Thanks****
>
> Leon****
>
> *·¢¼þÈË**:* Eugene Nikanorov [mailto:enikanorov at mirantis.com]
> *·¢ËÍʱ¼ä:* 2012Äê11ÔÂ26ÈÕ 4:29****
>
>
> *ÊÕ¼þÈË:* Leon Cui
> *³­ËÍ:* Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando; Dan
> Wendlandt****
>
> *Ö÷Ìâ**:* Re: ´ð¸´: ´ð¸´: ´ð¸´: ´ð¸´: Progress on lbaas-plugin-api-crud****
>
>  ****
>
> Hi Leon,****
>
>  ****
>
> Thanks for sending me the patch.****
>
> I've looked at it briefly, there is one major thing I was able to
> identify:****
>
> In order to couple things together (plugin, agent, drivers), we need to
> add device management at least to DB model.****
>
> In particular, each vip should have a reference to the device (which has a
> type and address/credentials). ****
>
> This information is passed in each agent notification message.****
>
> This part is missing in current design blueprints but i think we need to
> add it before we put the code on review.****
>
> Probably it will also depend on Salvatore's ServiceTypes part.****
>
>  ****
>
> Also I see that your patch is based on some of my outdated patches. ****
>
> My code was recently merged into the master so you can rebase on master
> using only Oleg's patch.****
>
>  ****
>
> Thanks,****
>
> Eugene.****
>
> On Fri, Nov 23, 2012 at 2:40 PM, Leon Cui <lcui at vmware.com> wrote:****
>
> Hi Eugene,****
>
> I¡¯m still waiting for approval as openstack contributor.  For now I simply
> attached the patch file that you might want to take a look first.  Once I
> got the approval, I¡¯ll try to post the view asap.****
>
>  ****
>
> Thanks****
>
> Leon****
>
> *·¢¼þÈË**:* Eugene Nikanorov [mailto:enikanorov at mirantis.com]
> *·¢ËÍʱ¼ä:* 2012Äê11ÔÂ20ÈÕ 22:57
> *ÊÕ¼þÈË:* Leon Cui
> *³­ËÍ:* Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando; Dan
> Wendlandt
> *Ö÷Ìâ:* Re: ´ð¸´: ´ð¸´: ´ð¸´: Progress on lbaas-plugin-api-crud****
>
>  ****
>
> Leon,****
>
>  ****
>
> I'll take agent and rpc parts.****
>
> I have registered
> https://blueprints.launchpad.net/quantum/+spec/lbaas-agent-and-rpc to
> track this.****
>
>  ****
>
> Thanks,****
>
> Eugene.****
>
> On Tue, Nov 20, 2012 at 2:16 PM, Leon Cui <lcui at vmware.com> wrote:****
>
> Hi Eugene,****
>
> Thanks for your suggestion.  It looks good to me.  I¡¯ll work out the UT
> first, and then align the class model to the diagram as you suggested.****
>
>  ****
>
> Thanks****
>
> Leon****
>
> *·¢¼þÈË**:* Eugene Nikanorov [mailto:enikanorov at mirantis.com]
> *·¢ËÍʱ¼ä:* 2012Äê11ÔÂ20ÈÕ 17:32****
>
>
> *ÊÕ¼þÈË:* Leon Cui
> *³­ËÍ:* Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando****
>
> *Ö÷Ìâ**:* Re: ´ð¸´: ´ð¸´: Progress on lbaas-plugin-api-crud****
>
>  ****
>
> replying to all...****
>
>  ****
>
> Leon,****
>
>  ****
>
> I think tests/unit/test_db_plugin.py  is right code to refer when writing
> unit tests for db code. The only thing is that unit tests written
> in test_db_plugin.py are a bit generic, e.g. the backend plugin is
> specified in particular plugin's UTs which inherit
> from QuantumDbPluginV2TestCase. I think UTs for balancer plugin may be more
> specific, testing LoadbalancerPluginDb class.****
>
>  ****
>
> Since you need dababase utility methods from QuantumDbPluginV2 then
> it's LoadbalancerPluginDb which should inherit from such QuantumDBBase (or
> whatever you call it), so overall diagram will look like:****
>
>  ****
>
> ServicePluginBase****
>
>     |****
>
> LoadBalancerPluginBase ****
>
>     |****
>
>     |                                       QuantumDBBase ****
>
>     |                                            |****
>
> LoadBalancerPlugin  <---------- LoadBalancerPluginDb****
>
>  ****
>
> Thanks,****
>
> Eugene.****
>
>  ****
>
> On Tue, Nov 20, 2012 at 1:04 PM, Leon Cui <lcui at vmware.com> wrote:****
>
> Hi Eugene,****
>
> Thanks for your suggestion.  Please see my comments inline.****
>
>  ****
>
> One more question: I¡¯m writing the unit test, mainly to verify the
> database functionalities for LB CRUD.  Do you think
> tests/unit/test_db_plugin.py is the right test code that I should refer
> to?  Any good suggestions on this front?****
>
>  ****
>
> Thanks****
>
> Leon****
>
> *·¢¼þÈË**:* Eugene Nikanorov [mailto:enikanorov at mirantis.com]
> *·¢ËÍʱ¼ä:* 2012Äê11ÔÂ20ÈÕ 16:44
> *ÊÕ¼þÈË:* Leon Cui
> *³­ËÍ:* Ilya Shakhat; Sachin Thakkar; Oleg Bondarev; Salvatore Orlando
> *Ö÷Ìâ:* Re: ´ð¸´: Progress on lbaas-plugin-api-crud****
>
>  ****
>
> Hi Leon,****
>
>  ****
>
> A few thoughts on your diagram.****
>
>  ****
>
> Please consider the following:****
>
> 1) If you want something from QuantumDbPluginV2 and you feel it may be
> common plugin functionality - you need to extract it to a separate class,
> something like QuantumPluginBase, and inherit QuantumDBPluginV2 from this
> class, ServicePluginBase should inherit from that class as well.****
>
> [Leon] I need some dababase utility methods from QuantumDbPluginV2. Abstract to a separate class could be a good idea. But I
> ¡¯m not sure if it¡¯s a good idea to let ServicePluginBase to inherit from
> this class. ServicePluginBase is an abstract class for service plugin
> service (quantum manager) to use. ****
>
>  ****
>
> 2) LoadBalancerPluginBase imho should inherit from ServicePluginBase****
>
> [Leon] Why it needs to inherit from ServicePluginBase?
> LoadBalancerPluginBase defines the loadbalancer extension APIs.  I think we
> just make sure LoadbalancerPlugin inherits from both classes as below:****
>
> ServicePluginBase    QuantumPluginDbBase   LoadbalancerPluginBase****
>
>        |                       |                      |****
>
>         ----------------------------------------------****
>
>                                |****
>
>                         LoadbalancerPlugin    ------ LoadbalancerPluginDb*
> ***
>
>               ****
>
> LoadbalancerPlugin will contain the LoadbalancerPluginDb instance for
> database access.****
>
>  ****
>
>  ****
>
> 3) Depending on what you need from QuantumDbPluginV2/QuantumPluginBase,
> this may lead to the following inheritance sequence:****
>
> QuantumPluginBase****
>
>     |****
>
> ServicePluginBase****
>
>     |****
>
> LoadBalancerPluginBase ****
>
>     |****
>
> LoadBalancerPluginDb****
>
>     |****
>
> LoadBalancerPlugin****
>
>  ****
>
> Also, I think that LoadBalancerPlugin should not inherit
> LoadBalancerPluginDb. ****
>
> Unlike core plugins where it could make sense, I'd prefer to see LoadBalancerPluginDb
> to be a part of LoadBalancerPlugin.****
>
> I mean LoadBalancerPlugin implements "has a" LoadBalancerPluginDb instead of "is a" relation.
>  ****
>
> The reason for this is that LoadBalancerPlugin provides CRUD
> implementation which doesn't directly map to DB operations implemented in LoadBalancerPluginDb.
> ****
>
> E.g. my idea is:****
>
> LoadBalancerPlugin - CRUD, validation, calling LoadBalancerPluginDb,
> sending/receiving messages to agent****
>
> LoadBalancerPluginDb - DB access.****
>
>  ****
>
> Thanks,****
>
> Eugene.****
>
>  ****
>
> On Tue, Nov 20, 2012 at 6:54 AM, Leon Cui <lcui at vmware.com> wrote:****
>
> Hi Ilya,****
>
> Right now I took Eugene¡¯s change under review (
> https://review.openstack.org/#/c/15733/) and am developing the database
> access logic and plugin skeleton based on that service plugin mechanism. The
> class model is illustrated in the below diagram:****
>
> [image: ServicePluginBase (quantum.plugins.services.service_base)
> ,QuantumDbPluginV2 (quantum.db. db_base_plugin_v2),LoadBalancerPluginDb
> (quantum.plugins.services.loadbalancer.loadbalancer_db),LoadBalancerPluginBase
> (quantum.extensions.loadbalancer)]****
>
> [image: LoadBalancerPlugin
> (quantum.plugins.services.loadbalancer.loadbalancerPlugin)]****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>
> ****
>
> LoadBalancerPlugin module is the main body of loadbalancer plugin which
> inherits from multiple classes:****
>
> -          ServicePluginBase: defines the abstract methods that a service
> plugin should implemented.****
>
> -          QuantumDbPluginV2: contains a set of generic quantum database
> access methods. I¡¯m not sure if we really want to inherit from this class
> but I¡¯d like to leverage the methods defined in this class.****
>
> -          LoadBalancerPluginDb: This the main part I¡¯m coding on which
> wrap the Lbaas database model and CRUD operation against the database.****
>
>  ****
>
> My thought is that LoadBalancerPlugin will control the LBaaS CRUD API
> flow. For instance, ¡°create_vip¡± method should first validate the input,
> update the database, send message to the LbAgent over AMQP channel, than
> update the database by setting the status from PENDING_CREATE to ACTIVE.**
> **
>
>  ****
>
> I¡¯m trying to write unit tests against the database access now which will
> take a while to complete. Meanwhile it would be great to have your help on
> coding the RPC interaction between plugin and agent.****
>
>  ****
>
> I don¡¯t like blocking your part. What¡¯s the best practice to collaborate
> with you on this? Maybe I can shelve my change to you somehow?****
>
>  ****
>
> Thanks****
>
> Leon****
>
> *·¢¼þÈË**:* Ilya Shakhat [mailto:ishakhat at mirantis.com]
> *·¢ËÍʱ¼ä:* 2012Äê11ÔÂ19ÈÕ 22:08
> *ÊÕ¼þÈË:* Sachin Thakkar; Leon Cui
> *³­ËÍ:* Eugene Nikanorov; Oleg Bondarev
> *Ö÷Ìâ:* Progress on lbaas-plugin-api-crud****
>
>  ****
>
> Hi Sachin, Leo, ****
>
>  ****
>
> Recently there was a thread related to LBaaS architecture (
> http://lists.openstack.org/pipermail/openstack-dev/2012-November/002646.html).
> How good is it aligned with your implementation? Do you need help in
> coding? (we may take Agent part) ****
>
>  ****
>
> Thanks,****
>
> Ilya****
>
>  ****
>
>  ****
>
>  ****
>
>  ****
>
>
>
> ****
>
>  ****
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt ****
>
> Nicira, Inc: www.nicira.com****
>
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~****
>
>  ****
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev****
>
>  ****
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev****
>
>   ** **
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev****
>
> ** **
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121129/e500211c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1745 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121129/e500211c/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 13351 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121129/e500211c/attachment-0001.png>