<div dir="ltr">You said you had no idea what group based policy was buying us so I tried to illustrate what the difference between declarative and imperative network configuration looks like. That's the major selling point of GBP so I'm not sure how that's 'side stepping' any points. It removes the need for the user to pick between implementation details like security groups/FWaaS/ACLs. <div>
<br></div><div>><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">So are you saying that GBP allows someone to be able to configure an application that at the end of the day is equivalent to networks/router/FWaaS rules without understanding networking concepts?</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">It's one thing to understand the ports an application leverages and another to understand the differences between configuring VM firewalls, security groups, FWaaS, and router ACLs. </span></div>
<div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">></span><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"> </span><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">I'm also curious how this GBP is really less error prone than the model we have today as it seems the user will basically have to tell neutron the same information about how he wants his networking to function.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br></span></div><div>With GBP, the user just gives the desired end result (e.g. allow connectivity between endpoint groups via TCP port 22 with all connections logged). Without it, the user has to do the following:</div>
<div><ol><li>create a network/subnet for each endpoint group</li><li>allow all traffic on the security groups since the logging would need to be accomplished with FWaaS<br></li><li>create an FWaaS instance</li><li>attach the FWaaS to both networks</li>
<li>add an FWaaS policy and the FWaaS rules to allow the correct traffic</li></ol><div>The declarative approach is less error prone because the user can give neutron the desired state of connectivity rather than a mentally compiled set of instructions describing how to configure a bunch of individual network components. How well do you think someone will handle the latter approach that got all of their networking knowledge from one college course 5 years ago?</div>
<div><br></div><div>IIRC one of the early nova parity requests was an API to automagically setup a neutron network and router so no extra work would be required to get instances connected to the Internet. That wasn't requested because people thought neutron networks were too easy to setup already. :-)</div>
</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, Aug 7, 2014 at 9:10 AM, Aaron Rosen <span dir="ltr"><<a href="mailto:aaronorosen@gmail.com" target="_blank">aaronorosen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra">Hi Kevin, </div><div class="gmail_extra"><br></div><div class="gmail_extra">I feel as your latest response is completely side stepping the points we have been trying to get to in the last series of emails. At the end of the day I don't believe we are changing the laws of networking (or perhaps we are?). Thus I think it's important to actually get down to the networking level to actually figure out why optimizations such as this one are enabled via GBP and not the model we have: <div class="">
<br>
<div style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.800000190734863px"><span style="font-size:12.666666984558105px"><br></span></div><div style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.800000190734863px">
<span style="font-size:12.666666984558105px">"With the latter, a mapping driver could determine that communication between these two hosts can be prevented by using an ACL on a router or a switch, which doesn't violate the user's intent and buys a performance improvement and works with ports that don't support security groups."</span><br>
</div><div style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.800000190734863px"><br></div><div style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.800000190734863px"><br></div><div><span style="font-size:12.666666984558105px"><br>
</span></div></div><div class="gmail_quote"><div class="">On Wed, Aug 6, 2014 at 7:48 PM, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div>Do you not see a difference between explicitly configuring networks, a router and FWaaS rules with logging and just stating that two groups of servers can only communicate via one TCP port with all connections logged? The first is very prone to errors for someone deploying an application without a strong networking background, and the latter is basically just stating the requirements and letting Neutron figure out how to implement it. </div>
<div><br></div></div></blockquote></div><div>So are you saying that GBP allows someone to be able to configure an application that at the end of the day is equivalent to networks/router/FWaaS rules without understanding networking concepts? I'm also curious how this GBP is really less error prone than the model we have today as it seems the user will basically have to tell neutron the same information about how he wants his networking to function.</div>
<div><div class="h5">
<div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div></div><div>
Just stating requirements becomes even more important when something like the logging requirement comes from someone other than the app deployer (e.g. a security team). In the above example, someone could set everything up using security groups; however, when the logging requirement came in from the security team, they would have to undo all of that work and replace it with something like FWaaS that can centrally log all of the connections.</div>
<div><br></div><div>It's the difference between using puppet and bash scripts. Sure you can write a script that uses awk/sed to ensure that an ini file has a particular setting and then restart a service if the setting is changed, but it's much easier and less error prone to just write a puppet manifest that uses the INI module with a pointer to the file, the section name, the key, and the value with a notification to restart the service.</div>
<div><br></div></div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Aug 6, 2014 at 7:40 PM, Aaron Rosen <span dir="ltr"><<a href="mailto:aaronorosen@gmail.com" target="_blank">aaronorosen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">
<div>On Wed, Aug 6, 2014 at 5:27 PM, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">Web tier can communicate with anything except for the DB.<div>
App tier can only communicate with Web and DB.</div><div>DB can communicate with App.</div><div><br></div><div>These high-level constraints can then be implemented as security groups like you showed, or network hardware ACLs like I had shown.</div>
<div>But if you start with the security groups API, you are forcing it to be implemented there.</div></div><div><div><div class="gmail_extra"><br></div></div></div></blockquote><div><br></div></div><div>I still have no idea what group based policy is buying us then. It seems to me that the key point we've identified going backing and forth here is the difference between the current model and the GBP model is that GBP constricts topology which allows us to write these types of enforcement rules. The reason we want this is because it yields performance optimizations (for some reason, which I don't think we've gotten into). Would you agree this is accurate? <br>
</div><div><br></div><div>Honestly, I know a lot of work has been put into this. I haven't said I'm for or against it either. I'm really just trying to understand what is the motivation for this and why does it make neutron better. </div>
<div><br></div><div>Best, </div><div><div><div><br></div><div>Aaron</div><div><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div>
<div class="gmail_extra"><br><div class="gmail_quote">On Wed, Aug 6, 2014 at 6:06 PM, Aaron Rosen <span dir="ltr"><<a href="mailto:aaronorosen@gmail.com" target="_blank">aaronorosen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">
<div>On Wed, Aug 6, 2014 at 4:46 PM, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">That's the point. By using security groups, you are forcing a certain kind of enforcement that must be honored and might not be necessary if the original intent was just to isolate between groups. In the example you gave, it cannot be implemented on the router without violating the constraints of the security group.</div>
<div><div>
<div class="gmail_extra"><br><br></div></div></div></blockquote></div><div>Hi Kevin, </div><div><br></div><div>Mind proposing an alternative example then. The only way I can see this claim to be made is because Group Based policy is actually limiting what a tenants desired topology can be? </div>
<div><br></div><div>Thanks,</div><div><br>Aaron</div><div><div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><div><div class="gmail_extra">
<div class="gmail_quote">On Wed, Aug 6, 2014 at 5:39 PM, Aaron Rosen <span dir="ltr"><<a href="mailto:aaronorosen@gmail.com" target="_blank">aaronorosen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">
On Wed, Aug 6, 2014 at 4:18 PM, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">Given this information I don't see any reason why the backend system couldn't do enforcement at the logical router and if it did so neither parties would know.</span><div>
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br></span></div></div><div><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">With security groups you are specifying that nothing can contact these devices on those ports unless they come from the allowed IP addresses. If you tried to enforce this at the router you would be violating that specification because devices in the same subnet would be able to communicate on those blocked ports.</span></div>
</div></blockquote><div><br></div><div>Sure, though this is a problem of where you are doing your enforcement. If the backend system chooses to implement this optimization in this fashion (which was the example you gave previously [1]). Then, if the topology changes, i.e adding a port to the same network with conflicting security group rules, the backend system can no longer optimize in this same fashion at the router level and a more fine grain filtering will need to be done. How would this be any different with group based policy? </div>
<div><br></div><div><br></div><div><span style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.666666984558105px">[1] - With the latter, a mapping driver could determine that communication between these two hosts can be prevented by using an ACL on a router or a switch, which doesn't violate the user's intent and buys a performance improvement and works with ports that don't support security groups.</span><br>
</div><div><span style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.666666984558105px"><br></span></div><div><font color="#500050" face="arial, sans-serif"><span style="font-size:12.666666984558105px">states</span></font></div>
<div><span style="color:rgb(80,0,80);font-family:arial,sans-serif;font-size:12.666666984558105px"><br></span></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr">
</div><div><div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Aug 6, 2014 at 5:00 PM, Aaron Rosen <span dir="ltr"><<a href="mailto:aaronorosen@gmail.com" target="_blank">aaronorosen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><br><div class="gmail_quote">
<div>On Wed, Aug 6, 2014 at 3:35 PM, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">By working at the port level you have already eliminated your ability to implement the filtering at different components of the network. They now need to be implemented in stateful rules at the port level and the device has to support security groups.</div>
</blockquote><div><br></div><div><br></div></div><div>Lets take this example where we setup a 2 tier app with web-servers and db-servers that are connected on two different networks attached to a router. We add a security group rules such that web-servers can talk to db-servers on tcp:3306 and a rule to allow tcp:80 into the web-servers from anywhere. </div>
<div><br></div><div>neutron net-create web_net</div><div>neutron subnet-create --name web_subnet web_net <a href="http://10.0.0.0/24" target="_blank">10.0.0.0/24</a></div><div><br></div><div>neutron net-create db_net</div>
<div>neutron subnet-create --name db_subnet db_net <a href="http://10.2.0.0/24" target="_blank">10.2.0.0/24</a></div>
<div><br></div><div>neutron router-create myrouter</div><div>neutron router-interface-add myrouter web_subnet<br></div><div>neutron router-interface-add myrouter db_subnet</div><div><br></div><div>neutron security-group-create web-servers; </div>
<div>neutron security-group-create db-servers; </div><div><br></div><div># add rule to allow web members to talk to the db-servers on TCP 3306 for their db connection;</div><div>neutron security-group-rule-create --protocol TCP --port-range-min 3306 --port-range-max 3306 --remote-group-id web-servers db-servers</div>
<div><br></div><div><div># add rule to allow TCP 80 into the web-server sg</div><div>neutron security-group-rule-create --protocol TCP --port-range-min 80 --port-range-max 80 web-servers db-servers</div></div><div><br></div>
<div># create some ports with desired security profiles. </div><div>neutron port-create --security-group web-servers web_net</div><div><div>neutron port-create --security-group web-servers web_net</div></div><div><br></div>
<div>neutron port-create --security-group db-servers db_net<br></div><div><div>neutron port-create --security-group db-servers db_net</div></div><div><br></div><div><br></div><div>Now to your point: </div><div>
<div><br></div>
<div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr">By working at the port level you have already eliminated your ability to implement the filtering at different components of the network. They now need to be implemented in stateful rules at the port level and the device has to support security groups.</div>
<div><br></div></blockquote></div><div><br></div></div><div>Given this information I don't see any reason why the backend system couldn't do enforcement at the logical router and if it did so neither parties would know. The backend system should have the full graph of everything and be able to do enforcement optimizations where ever it likes.</div>
<div><br></div><div>btw: I say the enforcement could be done on the logical router though the backend system could also do this on the physical fabic as well if it wanted to as it should also know that graph. No?</div><div>
<div><div>
<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div class="gmail_extra"><div><div><br>
<br><div class="gmail_quote">On Wed, Aug 6, 2014 at 4:03 PM, Aaron Rosen <span dir="ltr"><<a href="mailto:aaronorosen@gmail.com" target="_blank">aaronorosen@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">
<div><div>On Wed, Aug 6, 2014 at 12:46 PM, Kevin Benton <span dir="ltr"><<a href="mailto:blak111@gmail.com" target="_blank">blak111@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div dir="ltr"><div>><span style="font-family:arial,sans-serif;font-size:13.333333969116211px">I believe the referential security group rules solve this problem (unless I'm not understanding): </span><div>
<span style="font-family:arial,sans-serif;font-size:13.333333969116211px"><br>
</span></div></div><div>I think the disconnect is that you are comparing the way to current mapping driver implements things for the reference implementation with the existing APIs. Under this light, it's not going to look like there is a point to this code being in Neutron since, as you said, the abstraction could happen at a client. However, this changes once new mapping drivers can be added that implement things differently.</div>
<div><br></div><div>Let's take the security groups example. Using the security groups API directly is imperative ("put a firewall rule on this port that blocks this IP") compared to a higher level declarative abstraction ("make sure these two endpoints cannot communicate"). With the former, the ports must support security groups and there is nowhere except for the firewall rules on that port to implement it without violating the user's expectation. With the latter, a mapping driver could determine that communication between these two hosts can be prevented by using an ACL on a router or a switch, which doesn't violate the user's intent and buys a performance improvement and works with ports that don't support security groups.</div>
<div><br></div><div>Group based policy is trying to move the requests into the declarative abstraction so optimizations like the one above can be made.</div></div></blockquote><div><br></div></div></div><div>Hi Kevin, </div>
<div><br>
</div><div>Interesting points. Though, let me ask this. Why do we need to move to a declarative API abstraction in neutron in order to perform this optimization on the backend? For example, In the current neutron model say we want to create a port with a security group attached to it called web that allows TCP:80 in and members who are in a security group called database. >From this mapping I fail to see how it's really any different from the declarative model? The ports in neutron are logical abstractions and the backend system could be implemented in order to determine that the communication between these two hosts could be prevented by using an ACL on a router or switch as well.</div>
<div>
<div><br>Best, </div><div><br></div><div>Aaron</div><div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div></div></div><br></div></div><div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span><font color="#888888">-- <br><div>Kevin Benton</div>
</font></span></div><div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></div></blockquote></div></div></div><br></div></div><div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></div></blockquote></div><br><br clear="all"><span><font color="#888888"><div><br></div>-- <br><div>Kevin Benton</div>
</font></span></div>
</div></div><div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></div></blockquote></div><br></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Kevin Benton</div>
</div>
</div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div></div></div><br></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Kevin Benton</div>
</div>
</div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div></div></div><br></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Kevin Benton</div>
</div>
</div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div></div></div><br></div></div>
<br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Kevin Benton</div>
</div>