Open Stack

On Thu, Aug 7, 2014 at 12:08 PM, Kevin Benton <blak111 at gmail.com> wrote:

> >I mean't 'side stepping' why GBP allows for the comment you made
> previous, "With the latter, a mapping driver could determine that
> communication between these two hosts can be prevented by using an ACL on a
> router or a switch, which doesn't violate the user's intent and buys a
> performance improvement and works with ports that don't support security
> groups.".
>
> >Neutron's current API is a logical abstraction and enforcement can be
> done however one chooses to implement it. I'm really trying to understand
> at the network level why GBP allows for these optimizations and performance
> improvements you talked about.
>
> You absolutely cannot enforce security groups on a firewall/router that
> sits at the boundary between networks. If you try, you are lying to the
> end-user because it's not enforced at the port level. The current neutron
> APIs force you to decide where things like that are implemented.
>

The current neutron API's are just logical abstractions. Where and how
things are actually enforced are 100% an implementation detail of a vendors
system.  Anyways, moving the discussion to the etherpad...

>

> The higher level abstractions give you the freedom to move the enforcement
> by allowing the expression of broad connectivity requirements.
>
>Why are you bringing up logging connections?
>
> This was brought up as a feature proposal to FWaaS because this is a basic
> firewall feature missing from OpenStack. However, this does not preclude a
> FWaaS vendor from logging.
>
> >Personally, I think one could easily write up a very short document
> probably less than one page with examples showing/exampling how the current
> neutron API works even without a much networking background.
>
> The difficulty of the API for establishing basic connectivity isn't really
> the problem. It's when you have to compose a bunch of requirements and make
> sure nothing is violating auditing and connectivity constraints that it
> becomes a problem. We are arguing about the levels of abstraction. You
> could also write up a short document explaining to novice programmers how
> to use C to read and write database entries to an sqlite database, but that
> doesn't mean it's the best level of abstraction for what the users are
> trying to accomplish.
>
> I'll let someone else explain the current GBP API because I'm not working
> on that. I'm just trying to convince you of the value of declarative
> network configuration.
>
>
> On Thu, Aug 7, 2014 at 12:02 PM, Aaron Rosen <aaronorosen at gmail.com>
> wrote:
>
>>
>>
>>
>> On Thu, Aug 7, 2014 at 9:54 AM, Kevin Benton <blak111 at gmail.com> wrote:
>>
>>> You said you had no idea what group based policy was buying us so I
>>> tried to illustrate what the difference between declarative and imperative
>>> network configuration looks like. That's the major selling point of GBP so
>>> I'm not sure how that's 'side stepping' any points. It removes the need for
>>> the user to pick between implementation details like security
>>> groups/FWaaS/ACLs.
>>>
>>
>> I mean't 'side stepping' why GBP allows for the comment you made
>> previous, "With the latter, a mapping driver could determine that
>> communication between these two hosts can be prevented by using an ACL on a
>> router or a switch, which doesn't violate the user's intent and buys a
>> performance improvement and works with ports that don't support security
>> groups.".
>>
>> Neutron's current API is a logical abstraction and enforcement can be
>> done however one chooses to implement it. I'm really trying to understand
>> at the network level why GBP allows for these optimizations and performance
>> improvements you talked about.
>>
>>
>>
>>> >So are you saying that GBP allows someone to be able to configure an
>>> application that at the end of the day is equivalent  to
>>> networks/router/FWaaS rules without understanding networking concepts?
>>>
>>> It's one thing to understand the ports an application leverages and
>>> another to understand the differences between configuring VM firewalls,
>>> security groups, FWaaS, and router ACLs.
>>>
>>
>> Sure, but how does group based policy solve this. Security Groups and
>> FWaaS are just different places of enforcement. Say I want different
>> security enforcement on my router than on my instances. One still needs to
>> know enough to tell group based policy this right?  They need to know
>> enough that there are different enforcement points? How is doing this with
>> Group based policy make it easier?
>>
>>
>>
>>> > I'm also curious how this GBP is really less error prone than the
>>> model we have today as it seems the user will basically have to tell
>>> neutron the same information about how he wants his networking to function.
>>>
>>> With GBP, the user just gives the desired end result (e.g. allow
>>> connectivity between endpoint groups via TCP port 22 with all connections
>>> logged). Without it, the user has to do the following:
>>>
>>
>> Why are you bringing up logging connections? Neutron has no concept of
>> this at all today in it's code base. Is logging something related to GBP?
>>
>>>
>>>    1. create a network/subnet for each endpoint group
>>>    2. allow all traffic on the security groups since the logging would
>>>    need to be accomplished with FWaaS
>>>    3. create an FWaaS instance
>>>    4. attach the FWaaS to both networks
>>>
>>> Today FWaaS api is still incomplete as there is no real point of
>> enforcement in it's api (though it really seems that it should just be
>> router ports) it's just global on the router right  now.
>>
>>
>>>
>>>    1. add an FWaaS policy and the FWaaS rules to allow the correct
>>>    traffic
>>>
>>> I'd more or less agree with these steps. Would you mind also giving the
>> steps involved in group based policy so we can compare?
>>
>>
>>
>>> The declarative approach is less error prone because the user can give
>>> neutron the desired state of connectivity rather than a mentally compiled
>>> set of instructions describing how to configure a bunch of individual
>>> network components. How well do you think someone will handle the latter
>>> approach that got all of their networking knowledge from one college course
>>> 5 years ago?
>>>
>>
>> Personally, I think one could easily write up a very short document
>> probably less than one page with examples showing/exampling how the current
>> neutron API works even without a much networking background.  The funny
>> thing is I've been trying to completely understand the proposed group
>> policy api for a little while now and I'm still having trouble. Seems like
>> we're taking abstractions that are quite well known/understood and changing
>> them to a different model that requires one to know about this new
>> terminology:
>>
>>
>> Endpoint (EP): An L2/L3 addressable entity.
>> Endpoint Group (EPG): A collection of endpoints.
>> Contract: It defines how the application services provided by an EPG can
>> be accessed. In effect it specifies how an EPG communicates with other
>> EPGs. A Contract consists of Policy Rules.
>> Policy Rule: These are individual rules used to define the communication
>> criteria between EPGs. Each rule contains a Filter, Classifier, and Action.
>> Classifier: Characterizes the traffic that a particular Policy Rule acts
>> on. Corresponding action is taken on traffic that satisfies this
>> classification criteria.
>> Action: The action that is taken for a matching Policy Rule defined in a
>> Contract.
>> Filter: Provides a way to tag a Policy Rule with Capability and Role
>> labels.
>> Capability: It is a Policy Label that defines what part of a Contract a
>> particular EPG provides.
>> Role: It is a Policy Label that defines what part of a Contract an EPG
>> wants to consume.
>> Contract Scope: An EPG conveys its intent to provide or consume a
>> Contract (or its part) by defining a Contract Scope which references the
>> target Contract.
>> Selector: A Contract Scope can define additional constraints around
>> choosing the matching provider or consumer EPGs for a Contract via a
>> Selector.
>> Policy Labels: These are labels contained within a namespace hierarchy
>> and used to define Capability and Role tags used in Filters.
>> Bridge Domain: Used to define a L2 boundary and impose additional
>> constraints (such as no broadcast) within that L2 boundary.
>> Routing Domain: Used to define a non-overlapping IP address space.
>>
>>
>>
>>>
>>> IIRC one of the early nova parity requests was an API to automagically
>>> setup a neutron network and router so no extra work would be required to
>>> get instances connected to the Internet. That wasn't requested because
>>> people thought neutron networks were too easy to setup already. :-)
>>>
>>
>> I think the confusion why that comment was made is probably because the
>> nova-networks model doesn't have a concept of routers. Neutron can operate
>> in the same way though with flat and provider networks. Either way one
>> could easily write a nova-binding to the command they used previously to
>> create a router and uplink it hiding the fact that there is a router there.
>>
>>>
>>>
>>> On Thu, Aug 7, 2014 at 9:10 AM, Aaron Rosen <aaronorosen at gmail.com>
>>> wrote:
>>>
>>>> Hi Kevin,
>>>>
>>>> I feel as your latest response is completely side stepping the points
>>>> we have been trying to get to in the last series of emails. At the end of
>>>> the day I don't believe we are changing the laws of networking (or perhaps
>>>> we are?).  Thus I think it's important to actually get down to the
>>>> networking level to actually figure out why optimizations such as this one
>>>> are enabled via GBP and not the model we have:
>>>>
>>>>
>>>> "With the latter, a mapping driver could determine that communication
>>>> between these two hosts can be prevented by using an ACL on a router or a
>>>> switch, which doesn't violate the user's intent and buys a performance
>>>> improvement and works with ports that don't support security groups."
>>>>
>>>>
>>>>
>>>> On Wed, Aug 6, 2014 at 7:48 PM, Kevin Benton <blak111 at gmail.com> wrote:
>>>>
>>>>> Do you not see a difference between explicitly configuring networks, a
>>>>> router and FWaaS rules with logging and just stating that two groups of
>>>>> servers can only communicate via one TCP port with all connections logged?
>>>>> The first is very prone to errors for someone deploying an application
>>>>> without a strong networking background, and the latter is basically just
>>>>> stating the requirements and letting Neutron figure out how to implement
>>>>> it.
>>>>>
>>>>> So are you saying that GBP allows someone to be able to configure an
>>>> application that at the end of the day is equivalent  to
>>>> networks/router/FWaaS rules without understanding networking concepts? I'm
>>>> also curious how this GBP is really less error prone than the model we have
>>>> today as it seems the user will basically have to tell neutron the same
>>>> information about how he wants his networking to function.
>>>>
>>>>
>>>>
>>>>> Just stating requirements becomes even more important when something
>>>>> like the logging requirement comes from someone other than the app deployer
>>>>> (e.g. a security team). In the above example, someone could set everything
>>>>> up using security groups; however, when the logging requirement came in
>>>>> from the security team, they would have to undo all of that work and
>>>>> replace it with something like FWaaS that can centrally log all of the
>>>>> connections.
>>>>>
>>>>> It's the difference between using puppet and bash scripts. Sure you
>>>>> can write a script that uses awk/sed to ensure that an ini file has a
>>>>> particular setting and then restart a service if the setting is changed,
>>>>> but it's much easier and less error prone to just write a puppet manifest
>>>>> that uses the INI module with a pointer to the file, the section name, the
>>>>> key, and the value with a notification to restart the service.
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Aug 6, 2014 at 7:40 PM, Aaron Rosen <aaronorosen at gmail.com>
>>>>> wrote:
>>>>>
>>>>>>
>>>>>> On Wed, Aug 6, 2014 at 5:27 PM, Kevin Benton <blak111 at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Web tier can communicate with anything except for the DB.
>>>>>>> App tier can only communicate with Web and DB.
>>>>>>> DB can communicate with App.
>>>>>>>
>>>>>>> These high-level constraints can then be implemented as security
>>>>>>> groups like you showed, or network hardware ACLs like I had shown.
>>>>>>> But if you start with the security groups API, you are forcing it to
>>>>>>> be implemented there.
>>>>>>>
>>>>>>>
>>>>>> I still have no idea what group based policy is buying us then. It
>>>>>> seems to me that the key point we've identified going backing and forth
>>>>>> here is the difference between the current model and the GBP model is that
>>>>>> GBP constricts topology which allows us to write these types of enforcement
>>>>>> rules. The reason we want this is because it yields performance
>>>>>> optimizations (for some reason, which I don't think we've gotten into).
>>>>>> Would you agree this is accurate?
>>>>>>
>>>>>> Honestly, I know a lot of work has been put into this. I haven't said
>>>>>> I'm for or against it either. I'm really just trying to understand what is
>>>>>> the motivation for this and why does it make neutron better.
>>>>>>
>>>>>> Best,
>>>>>>
>>>>>> Aaron
>>>>>>
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Wed, Aug 6, 2014 at 6:06 PM, Aaron Rosen <aaronorosen at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Aug 6, 2014 at 4:46 PM, Kevin Benton <blak111 at gmail.com>
>>>>>>>> wrote:
>>>>>>>>
>>>>>>>>> That's the point. By using security groups, you are forcing a
>>>>>>>>> certain kind of enforcement that must be honored and might not be necessary
>>>>>>>>> if the original intent was just to isolate between groups. In the example
>>>>>>>>> you gave, it cannot be implemented on the router without violating the
>>>>>>>>> constraints of the security group.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Hi Kevin,
>>>>>>>>
>>>>>>>> Mind proposing an alternative example then. The only way I can see
>>>>>>>> this claim to be made is because Group Based policy is actually limiting
>>>>>>>> what a tenants desired topology can be?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>>
>>>>>>>> Aaron
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>>  On Wed, Aug 6, 2014 at 5:39 PM, Aaron Rosen <
>>>>>>>>> aaronorosen at gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Aug 6, 2014 at 4:18 PM, Kevin Benton <blak111 at gmail.com>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>>> >Given this information I don't see any reason why the backend
>>>>>>>>>>> system couldn't do enforcement at the logical router and if it did so
>>>>>>>>>>> neither parties would know.
>>>>>>>>>>>
>>>>>>>>>>> With security groups you are specifying that nothing can contact
>>>>>>>>>>> these devices on those ports unless they come from the allowed IP
>>>>>>>>>>> addresses. If you tried to enforce this at the router you would be
>>>>>>>>>>> violating that specification because devices in the same subnet would be
>>>>>>>>>>> able to communicate on those blocked ports.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Sure, though this is a problem of where you are doing your
>>>>>>>>>> enforcement. If the backend system chooses to implement this optimization
>>>>>>>>>> in this fashion (which was the example you gave previously [1]). Then, if
>>>>>>>>>> the topology changes, i.e adding a port to the same network with
>>>>>>>>>> conflicting security group rules, the backend system can no longer optimize
>>>>>>>>>> in this same fashion at the router level and a more fine grain filtering
>>>>>>>>>> will need to be done. How would this be any different with group based
>>>>>>>>>> policy?
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> [1] - With the latter, a mapping driver could determine that
>>>>>>>>>> communication between these two hosts can be prevented by using an ACL on a
>>>>>>>>>> router or a switch, which doesn't violate the user's intent and buys a
>>>>>>>>>> performance improvement and works with ports that don't support security
>>>>>>>>>> groups.
>>>>>>>>>>
>>>>>>>>>> states
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Aug 6, 2014 at 5:00 PM, Aaron Rosen <
>>>>>>>>>>> aaronorosen at gmail.com> wrote:
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Aug 6, 2014 at 3:35 PM, Kevin Benton <blak111 at gmail.com
>>>>>>>>>>>> > wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> By working at the port level you have already eliminated your
>>>>>>>>>>>>> ability to implement the filtering at different components of the network.
>>>>>>>>>>>>> They now need to be implemented in stateful rules at the port level and the
>>>>>>>>>>>>> device has to support security groups.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Lets take this example where we setup a 2 tier app with
>>>>>>>>>>>> web-servers and db-servers that are connected on two different networks
>>>>>>>>>>>> attached to a router. We add a security group rules such that web-servers
>>>>>>>>>>>> can talk to db-servers on tcp:3306 and a rule to allow tcp:80 into the
>>>>>>>>>>>> web-servers from anywhere.
>>>>>>>>>>>>
>>>>>>>>>>>> neutron net-create web_net
>>>>>>>>>>>> neutron subnet-create --name web_subnet web_net 10.0.0.0/24
>>>>>>>>>>>>
>>>>>>>>>>>> neutron net-create db_net
>>>>>>>>>>>> neutron subnet-create --name db_subnet db_net 10.2.0.0/24
>>>>>>>>>>>>
>>>>>>>>>>>> neutron router-create myrouter
>>>>>>>>>>>> neutron router-interface-add myrouter web_subnet
>>>>>>>>>>>> neutron router-interface-add myrouter db_subnet
>>>>>>>>>>>>
>>>>>>>>>>>> neutron security-group-create  web-servers;
>>>>>>>>>>>> neutron security-group-create db-servers;
>>>>>>>>>>>>
>>>>>>>>>>>> # add rule to allow web members to talk to the db-servers on
>>>>>>>>>>>> TCP 3306 for their db connection;
>>>>>>>>>>>> neutron security-group-rule-create --protocol TCP
>>>>>>>>>>>> --port-range-min 3306 --port-range-max 3306 --remote-group-id web-servers
>>>>>>>>>>>> db-servers
>>>>>>>>>>>>
>>>>>>>>>>>> # add rule to allow TCP 80 into the web-server sg
>>>>>>>>>>>> neutron security-group-rule-create --protocol TCP
>>>>>>>>>>>> --port-range-min 80 --port-range-max 80 web-servers db-servers
>>>>>>>>>>>>
>>>>>>>>>>>> # create some ports with desired security profiles.
>>>>>>>>>>>> neutron port-create  --security-group web-servers web_net
>>>>>>>>>>>> neutron port-create  --security-group web-servers web_net
>>>>>>>>>>>>
>>>>>>>>>>>> neutron port-create  --security-group db-servers db_net
>>>>>>>>>>>> neutron port-create  --security-group db-servers db_net
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Now to your point:
>>>>>>>>>>>>
>>>>>>>>>>>> By working at the port level you have already eliminated your
>>>>>>>>>>>>> ability to implement the filtering at different components of the network.
>>>>>>>>>>>>> They now need to be implemented in stateful rules at the port level and the
>>>>>>>>>>>>> device has to support security groups.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>> Given this information I don't see any reason why the backend
>>>>>>>>>>>> system couldn't do enforcement at the logical router and if it did so
>>>>>>>>>>>> neither parties would know. The backend system should have the full graph
>>>>>>>>>>>> of everything and be able to do enforcement optimizations where ever it
>>>>>>>>>>>> likes.
>>>>>>>>>>>>
>>>>>>>>>>>> btw: I say the enforcement could be done on the logical router
>>>>>>>>>>>> though the backend system could also do this on the physical fabic as well
>>>>>>>>>>>> if it wanted to as it should also know that graph. No?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Aug 6, 2014 at 4:03 PM, Aaron Rosen <
>>>>>>>>>>>>> aaronorosen at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Aug 6, 2014 at 12:46 PM, Kevin Benton <
>>>>>>>>>>>>>> blak111 at gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> >I believe the referential security group rules solve this
>>>>>>>>>>>>>>> problem (unless I'm not understanding):
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think the disconnect is that you are comparing the way to
>>>>>>>>>>>>>>> current mapping driver implements things for the reference implementation
>>>>>>>>>>>>>>> with the existing APIs. Under this light, it's not going to look like there
>>>>>>>>>>>>>>> is a point to this code being in Neutron since, as you said, the
>>>>>>>>>>>>>>> abstraction could happen at a client. However, this changes once new
>>>>>>>>>>>>>>> mapping drivers can be added that implement things differently.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Let's take the security groups example. Using the security
>>>>>>>>>>>>>>> groups API directly is imperative ("put a firewall rule on this port that
>>>>>>>>>>>>>>> blocks this IP") compared to a higher level declarative abstraction ("make
>>>>>>>>>>>>>>> sure these two endpoints cannot communicate"). With the former, the ports
>>>>>>>>>>>>>>> must support security groups and there is nowhere except for the firewall
>>>>>>>>>>>>>>> rules on that port to implement it without violating the user's
>>>>>>>>>>>>>>> expectation. With the latter, a mapping driver could determine that
>>>>>>>>>>>>>>> communication between these two hosts can be prevented by using an ACL on a
>>>>>>>>>>>>>>> router or a switch, which doesn't violate the user's intent and buys a
>>>>>>>>>>>>>>> performance improvement and works with ports that don't support security
>>>>>>>>>>>>>>> groups.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Group based policy is trying to move the requests into the
>>>>>>>>>>>>>>> declarative abstraction so optimizations like the one above can be made.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Kevin,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Interesting points. Though, let me ask this. Why do we need
>>>>>>>>>>>>>> to move to a declarative API abstraction in neutron in order to perform
>>>>>>>>>>>>>> this optimization on the backend? For example, In the current neutron model
>>>>>>>>>>>>>> say we want to create a port with a security group attached to it called
>>>>>>>>>>>>>> web that allows TCP:80 in and members who are in a security group called
>>>>>>>>>>>>>> database. From this mapping I fail to see how it's really any different
>>>>>>>>>>>>>> from the declarative model? The ports in neutron are logical abstractions
>>>>>>>>>>>>>> and the backend system could be implemented in order to determine that the
>>>>>>>>>>>>>> communication between these two hosts could be prevented by using an ACL on
>>>>>>>>>>>>>> a router or switch as well.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Best,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Aaron
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Kevin Benton
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>>>>>
>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>>>>
>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Kevin Benton
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Kevin Benton
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Kevin Benton
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Kevin Benton
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>
>>>
>>> --
>>> Kevin Benton
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
>
> --
> Kevin Benton
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140807/d048669b/attachment.html>