<div dir="ltr">><span style="font-family:arial,sans-serif;font-size:13px">Agree, but the reason the RIB needs the IP address is so that it can select from multiple possible routes based on availability.</span><div><span style="font-family:arial,sans-serif;font-size:13px"><br>
</span></div><div>That's only one possible reason. One of the big reasons that you aren't giving enough credit is flexibility and ease of configuration. Being able to specify an IP address is important if you ever want to change the device acting as the next-hop router. It's very closely modeled after the 'route' command on Linux, which, to my knowledge, doesn't even take MAC addresses for the GW argument.</div>
<div><br></div><div>><span style="font-family:arial,sans-serif;font-size:13px">I have to admit that I still don't really understand this concept of "not managed by OpenStack". This is a network completely defined in software... where even the Internet has its own UUID... how is it even possible that there are devices attached that the controller knows nothing about?</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">The networks aren't always completely defined in software (</span><font face="arial, sans-serif"><a href="http://developer.rackspace.com/blog/neutron-networking-vlan-provider-networks.html">http://developer.rackspace.com/blog/neutron-networking-vlan-provider-networks.html</a>)</font><span style="font-family:arial,sans-serif;font-size:13px">. In short, a network may map to a real VLAN on physical infrastructure shared with routers, servers, and other widgets not managed by openstack.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div>The rest of your email with the use case of the nova instance running a VPN service makes sense. Proposing a fix to the ExtraRoutes API to allow the addition and deletion of specific routes is already on my todo list. Once that's in place, I will get in contact with you on #heat again to look at how to implement these use-cases.</div>
<div><br></div><div><br></div><div>--</div><div>Kevin Benton</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Tue, Apr 8, 2014 at 10:35 AM, Zane Bitter <span dir="ltr"><<a href="mailto:zbitter@redhat.com" target="_blank">zbitter@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">On 07/04/14 21:52, Kevin Benton wrote:<br>
</div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">
I will just provide a few quick points of clarity.<br>
<br>
>Instinctively, I want a Subnet to be something like a virtual VLAN�<br>
<br>
That's what a network is. The network is the broadcast domain. That's<br>
why you attach ports to the network. The subnet is just blocks of IP<br>
addresses to use on this network. If you have two subnets on the same<br>
network, they will be sharing a broadcast domain. Since DHCP servers do<br>
not answer anonymous queries, there are no conflicts, which is where the<br></div>
subnet requirement comes in when creating a port.�<br>
</blockquote>
<br>
Oh! Thank you, I don't think I would ever have guessed that from the Network/Subnet terminology. Looking back at the documentation, I see that this is stated in the Glossary, so we can put this down to a case of Déformation professionnelle on my part.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">
To attach a port to a network and give it an IP from a specific subnet<br></div>
on that network, you would use the *--fixed-ip subnet_id *option.<div class=""><br>
Otherwise, the create port request will use the first subnet it finds<br>
attached to that network to allocate the port an IP address. This is why<br>
you are encountering the port-> subnet-> network chain. Subnets provide<br></div>
the addresses. Networks are the actual layer 2 boundaries.�<br>
</blockquote>
<br>
It sounds like maybe Subnets need to be created independently of Networks and then passed as a list to the Network when it is created. In Heat there's no way to even predict which Subnet will be "first" unless the user adds explicit "depends_on" annotations (and even then, a Subnet could have been created outside of the template already).<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
�if I can do a create call immediately followed by an update call then<br>
</blockquote><div class="">
the Neutron API can certainly do this internally<br>
<br>
Are you sure you can do that in an update_router call? There are<br>
separate methods to add and remove router interfaces, none of which seem<br></div>
to be referenced from the update_router method.�<br>
</blockquote>
<br>
You're right, I misinterpreted the docs because there's a line break in the middle of a URL fragment. That doesn't really change my argument though.<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
It's not exactly clear why the external gateway is special enough that<br>
</blockquote>
you can have only one interface of this type on a Router,�but not so<br>
special that it would be considered a separate thing.<br>
<br>
This is because it currently doubles as an indicator of the default<br>
route. If you had multiple external networks, you would need another<br>
method of specifying which to use for outbound traffic. The reason it's<br>
not a regular port is because�the port created for the external gateway<br>
cannot be owned by the tenant since it's attaching to a network that the<br>
tenant does not own. A special port is created for this gateway which<br>
the tenant does not have direct control over so they can't mess with the<br>
external network.<br>
</blockquote>
<br></div>
This makes sense, thanks. How does the user find out the UUID of the external network? Would it be just as easy to supply them with the UUID of the special port that's created for them?<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
>An extra route doesn't behave at all like a static RIB entry (with a<br>
weight and an administrative distance)<br>
<br>
You and I have discussed this at lengths, but I will document it here<br>
for the mailing list. :-)<br>
<br>
This allows you to create static routes, which most certainly may live<br>
in the RIB with IP addresses as the next hop. It's up to the neighbor<br>
(or adjacency) discovery components to translate this to an L2 address<br>
(or interface) when it's installed in the RIB. It is very rare to find a<br>
</blockquote></div>
(ZB) I assume you meant FIB here ^<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">
modern router that doesn't let you configure static routes with IP<br></div>
addresses.�<br>
</blockquote>
<br>
Agree, but the reason the RIB needs the IP address is so that it can select from multiple possible routes based on availability. So you can add a bunch of static routes for the same prefix but with different costs to the RIB, and the router will select the one with the lowest cost that it can resolve and insert only that *MAC* address as the nexthop in the FIB. (Of course the RIB also contains routes from other sources like e.g. RIP or OSPF, and generally you can specify the administrative distance of static routes to adjust their ranking with respect to other sources of routes.)<br>
<br>
ExtraRoutes doesn't do anything like that; there's no way to specify a cost (or an administrative distance). If your route appears in ExtraRoutes it _will_ be installed (if the nexthop is available, presumably - or does this generate an error? Hopefully traffic gets blackholed if the IP address isn't available... I wouldn't be a happy camper if my VPN traffic got sent to the Internet just because the VPN gateway was down), like a FIB but unlike a RIB.<br>
<br>
So, given that ExtraRoutes is not designed to do anything useful with the IP address (like select from multiple nexthops based on availability), why does it need an IP address as an argument?<div class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Correct me if I'm wrong, but the fundamental contention is that you<br>
think routes should never be allowed to point to anything not managed by<br>
OpenStack.<br>
</blockquote>
<br></div>
It's a little more subtle than that; I don't mind what it points to, as long as OpenStack knows how to reach it. So my position is that to the extent that routes depend on things that *are* managed by OpenStack, that dependency needs to be explicitly reflected in the API (or, failing that, at least in the Heat resource).<br>
<br>
I have to admit that I still don't really understand this concept of "not managed by OpenStack". This is a network completely defined in software... where even the Internet has its own UUID... how is it even possible that there are devices attached that the controller knows nothing about?<div class="">
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This constraint gives heat the ability to reference neutron<br>
port objects as next hops, which is very useful for resolving<br>
dependencies. However, this gain in dependency management comes at the<br>
cost of tenant routers never being allowed to use devices outside of<br>
neutron as next-hop devices. This may cover many of the use cases, but<br>
it is a breaking change due to the loss of generality.<br>
</blockquote>
<br></div>
I'll give an example of why I think this is a big deal. Let's say that you use some obscure VPN that doesn't have a client available as a VPNaaS in Neutron. You want to spin up a Nova server to connect to your corporate network. Let's also say that you now want to make a change to that server (maybe a software upgrade), which will mean replacing the Nova server with a newly-built one.<br>
<br>
For starters, let's assume that the ExtraRoute takes as an argument either a Port or a Server UUID, rather than an IP address, as a nexthop (in AWS, routes take a port, server or gateway as the nexthop).<br>
<br>
Router <---<br>
\<br>
Server <---- Route<br>
<br>
The good news is that just by changing the configuration of the existing Server in your template, Heat will automatically:<br>
1) Create a replacement<br>
2) Switch all references over to the new server<br>
3) Delete the original<br>
<br>
It does this by traversing the dependencies of the new template in create order, followed by the dependencies of the old template in the reverse (i.e. delete) order to clean up.<br>
<br>
Create Update Delete<br>
Server' <---- Route <---- Server<br>
<br>
So we just replaced our VPN gateway without interruption, and without doing anything special in the template or needing in any way to know how it works. We hooked everything up in the only way that we conceivably could, and Heat just did the Right Thing.<br>
<br>
If we look at the current plugin in /contrib, it takes an IP address and there is no dependency relationship:<br>
<br>
Router <---- Route<br>
<br>
Server<br>
<br>
Our first thought would probably be to give the server a static IP. We won't be able to perform this update, however, because I assume that Neutron won't allow us to assign the same static IP to two different servers, and even if it did that would probably cause chaos in the network. So we'll have to change the static IP, and therefore the Route, during the update. Here's what happens:<br>
<br>
Update<br>
Route<br>
<br>
Create Delete<br>
Server' <---- Server<br>
<br>
So the Route gets updated before the new server has even been created. I assume this doesn't cause an error, but it at least guarantees a disruption of service.<br>
<br>
There are ways around this, including using "get_attr" to retrieve the IP address of the nexthop from the Server (which creates a dependency). But they're not obvious, require a detailed level of knowledge of the system, and you'll most likely only find out that you got it wrong when you go to upgrade your VPN endpoint in production.<br>
<br>
Looking back at the history of your patch, I see that the hidden dependencies have been whittled down to two - the RouterGateway (which we have fixed by deprecating it), and the RouterInterfaces which, I have discovered in the process of researching this thread, suffer from exactly the same design problem and need to be deprecated in exactly the same way. So my objection to your patch on the grounds of hidden dependencies on RouterInterfaces was misplaced. The race conditions remain a concern, but I guess from a Heat perspective one that we can claim is Not Our Fault. I would still prefer to see some sort of interface that results in the dependencies mentioned above occurring naturally in the template - the alternative is to heavily document how to do this yourself using get_attr to get, e.g. the IP address of a Server or the peer address of a VPNaaS gateway... but if it's possible to do this manually it seems like it should also be possible to automatically retrieve the IP addresses in Heat.<br>
<br>
cheers,<br>
Zane.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="">
--<br>
Kevin Benton<br>
<br>
<br>
On Mon, Apr 7, 2014 at 5:28 PM, Zane Bitter <<a href="mailto:zbitter@redhat.com" target="_blank">zbitter@redhat.com</a><br></div><div><div class="h5">
<mailto:<a href="mailto:zbitter@redhat.com" target="_blank">zbitter@redhat.com</a>>> wrote:<br>
<br>
The Neutron API is a constant cause of pain for us as Heat<br>
developers, but afaik we've never attempted to bring up the issues<br>
we have found in a cross-project forum. I've recently been doing<br>
some more investigation and I want to document the exact ways in<br>
which the current Neutron API breaks orchestration, both in the hope<br>
that a future version of it might be better and as a guide for other<br>
API authors.<br>
<br>
BTW it's my contention that an API that is bad for orchestration is<br>
also hard to use for the ordinary user as well. When you're trying<br>
to figure out the order of operations you need to do, there are two<br>
times at which you could find out you've got it wrong:<br>
<br>
1) Before you run the command, when you realise you don't have all<br>
of the required data yet; or<br>
2) After you run the command, when you get a cryptic error message.<br>
<br>
Not only is (1) *mandatory* for a data-driven orchestration system<br>
like Heat, it offers orders-of-magnitude better user experience for<br>
everyone.<br>
<br>
I should say at the outset that I know next to nothing about<br>
Neutron, and one of the goals of this message is to find out which<br>
parts I am completely wrong about. I did know a little bit about<br>
traditional networking at one time, and even remember some of it ;)<br>
<br>
<br>
Neutron has a little documentation on workflow, so let's begin<br>
there:<br></div></div>
<a href="http://docs.openstack.org/api/__openstack-network/2.0/content/__Overview-d1e71.html#Theory" target="_blank">http://docs.openstack.org/api/<u></u>__openstack-network/2.0/<u></u>content/__Overview-d1e71.html#<u></u>Theory</a><div>
<div class="h5"><br>
<<a href="http://docs.openstack.org/api/openstack-network/2.0/content/Overview-d1e71.html#Theory" target="_blank">http://docs.openstack.org/<u></u>api/openstack-network/2.0/<u></u>content/Overview-d1e71.html#<u></u>Theory</a>><br>
<br>
(1) Create a network<br>
Instinctively, I want a Network to be something like a virtual VRF<br>
(VVRF?): a separate namespace with it's own route table, within<br>
which subnet prefixes are not overlapping, but which is completely<br>
independent of other Networks that may contain overlapping subnets.<br>
As far as I can tell, this basically seems to be the case. The<br>
difference, of course, is that instead of having to configure a VRF<br>
on every switch/router and make sure they're all in sync and<br>
connected up in the right ways, I just define it in one place<br>
globally and Neutron does the rest. I call this #winning. Nice work,<br>
Neutron.<br>
<br>
(2) Associate a subnet with the network<br>
Slightly odd choice of words, because you're actually creating a new<br>
Subnet (there's no such thing as a Subnet not associated with a<br>
Network), but this is probably just a minor documentation nit.<br>
Instinctively, I want a Subnet to be something like a virtual VLAN<br>
(VVLAN?): at its most basic level, just a group of ports that share<br>
a broadcast domain, but also having other properties (e.g. if L3 is<br>
in use, all IP addresses in the subnet should be in the same CIDR).<br>
This doesn't seem to be the case, though, it's just a CIDR prefix,<br>
which leaves me wondering how L2 traffic will be treated, as well as<br>
how I would do things like use both IPv4 and IPv6 on a single port<br>
(by assigning a port to multiple Subnets?). Looking at the docs,<br>
there is a much bigger emphasis on DHCP client settings than I<br>
expected - surely I might want to want to give two sets of ports in<br>
the same Subnet different DHCP configs? Still, this is not bad - the<br>
DHCP configuration is done by the time the Subnet is created, so<br>
there's no problem in connecting stuff to it immediately after.<br>
<br>
(3) Boot a VM and attach it to the network<br>
Here's where you completely lost me. I just created a Subnet - maybe<br>
a bunch of Subnets. I don't want to attach my VM just anywhere in<br>
the *Network*, I want to attach it to a *particular* Subnet. It's<br>
not at all obvious where my instance will get attached (at random?),<br>
because this API just plain takes the Wrong data type. As a user,<br>
I'm irritated and confused.<br>
<br>
The situation for orchestration, though, is much, much worse.<br>
Because the server takes a reference to a network, the dependency<br>
graph generated from my template will look like this:<br>
<br>
� �Network <---------- Subnet<br>
� � � � �^<br>
� � � � �\<br>
� � � � � ------------ Server<br>
<br>
And yet if the Server is created before the Subnet (as will happen<br>
~50% of the time), it will fail. And vice-versa on delete, where the<br>
server must be removed before the subnet. The dependency graph we<br>
needed to create was this:<br>
<br>
� �Network <---------- Subnet <---------- Server<br>
<br>
The solution used here was to jury-rig the resource types in Heat<br>
with a hidden dependency. We can't know which Subnet the server will<br>
end up attached to, so we create hidden dependencies on all of the<br>
ones defined in the same template. There's nothing we can do about<br>
Subnets defined in different templates (Heat allows a tree of<br>
templates to be instantiated with a single command) - I'm not sure,<br>
but it may be possible even now to create a tree of stacks that in<br>
practice could never be successfully deleted.<br>
<br>
The Neutron models in Heat are so riddled with these kinds of<br>
invisible special-case hacks that all of our public documentation<br>
about how Heat can be expected to respond to a particular template<br>
is rendered effectively meaningless with respect to Neutron.<br>
<br>
I should add that we can't blame Nova here, because explicitly<br>
creating a Port doesn't help - it too takes only a network argument,<br>
despite _requiring_ a Subnet that it will be attached to, presumably<br>
at random. In fact using a Port makes things even worse, because<br>
although there is an API for it Nova and Neutron seem to assume that<br>
nobody would ever use it, and therefore even if you create a port<br>
explicitly and pass it to Nova to connect a Server, when you<br>
disconnect the Server again the Port will be deleted at the same<br>
time as if you had let Nova create it implicitly for you. This issue<br>
is currently breaking stack updates because we tend to assume that<br>
once we've explicitly created something, it stays created.<br>
<br>
Evidently there is a mechanism for associating a Port with a Subnet,<br>
and that's by assigning a fixed IP - which is hardly ever what I<br>
want. There's no middle ground that I can find between specifying<br>
the exact, fixed IP for a port and just letting it end up somewhere<br>
- anywhere - on the network, entirely at random.<br>
<br>
<br>
Let's move on to the L3 extension, starting with Routers. There's<br>
kind of an inconsistency here, because Routers are virtual devices<br>
that I need to manage. Hitherto, the point of Neutron was to free me<br>
from managing individual devices and let me manage the network as a<br>
whole. Is there a reason I wouldn't want all of the Subnets in the<br>
Network to just do the Right Thing and make sure everywhere is<br>
reachable efficiently from everywhere else? If I want something<br>
separate, wouldn't I use a different Network? (It's not like I have<br>
any control over where in a Network ports get attached anyway.)<br>
<br>
Nonetheless, Routers exist and it appears I have to create one to<br>
route packets between Subnets. From an orchestration perspective,<br>
I'd like Router to take a list of Ports to attach to (and of course<br>
I'd like each Port to be explicitly associated with a Subnet!). I'd<br>
be out of luck though, because even though the Port list is a<br>
property of a Router, you can't set it at creation time, only<br>
through an update. This is by definition possible to do at creation<br>
time (if I can do a create call immediately followed by an update<br>
call then the Neutron API can certainly do this internally), so it's<br>
very strange to see it disallowed. Following this API led us to<br>
implement it wrong in Heat as well, leading to headaches with<br>
floating IPs, about which more later. We also mistakenly used a<br>
similar design for the Router's external gateway, but later<br>
corrected it by making it a property of the Router, as it is in the<br>
API (though we still have to live with a lengthy deprecation<br>
period). We'll probably end up doing the same with the interfaces.<br>
<br>
Of course it goes without saying that the router gateway is just a<br>
reference to another network and, once again, requires a hidden<br>
dependency on all of the Subnets in the hopes of picking up the<br>
right one. BTW I'm just assuming that the definition of the gateway<br>
is "interface to another Network over which I will do NAT"? I assume<br>
that because of the generic way in which Floating IPs are handled,<br>
with a reference to an external network (I guess the operator<br>
provides the user with the Network UUID for the Internet?) It's not<br>
exactly clear why the external gateway is special enough that you<br>
can have only one interface of this type on a Router, but not so<br>
special that it would be considered a separate thing. There is also<br>
a separate Network Gateway, and I have no idea what that is...<br>
<br>
The big problem with Floating IPs is that you can't create them<br>
until all the necessary hops in the internetwork have been set up.<br>
And, once again, there's nothing in the creation parameters that<br>
would naturally order them - you just pass a reference to the<br>
external network. We still have a bug open on this, but what we will<br>
have to do is create a hidden dependency on any RouterInterfaces<br>
that connect any Routers whose external gateway is the same network<br>
where the floating IP is allocated. That's about as horrible as it<br>
sounds. A Floating IP needs to take as an argument a reference to<br>
the Router/Gateway which does the NAT:<br>
<br>
External � � � External<br>
Network �<---- Subnet � <---- (gateway)<br>
� � � � � � � � � � � � � � �\<br>
� � � � � � � � � � � � � � Router <---- Floating IP<br>
Internal � � � � � � � � � � / � � � � � � � /<br>
Network �<---- Subnet <------<---- Port <----<br>
<br>
The bane of my existence during Icehouse development has been the<br>
ExtraRoutes table. First off, this is broken in a way completely<br>
unrelated to orchestration: you can't add, remove or change an entry<br>
in the table without rewriting the whole table, so the whole API is<br>
a giant race condition waiting to happen. (This can, and IMHO<br>
should, be fixed - at least for those using the official client -<br>
with an ETags header and the 409 return code.) Everything about this<br>
API, though, is strange. It's another one of those only-on-update<br>
properties of a Router, though in this case that's forced by the<br>
fact that you can't attach the Router to its Subnets during its<br>
creation. An extra route doesn't behave at all like a static RIB<br>
entry (with a weight and an administrative distance), but much like<br>
a FIB entry (i.e. it's for routes that have already been selected to<br>
be active). That part makes sense, but the next hop for a FIB entry<br>
is a layer 2 address and this takes an IP address. That makes no<br>
sense to me, since the IP address(es) assigned to the nexthop play<br>
no part in how packets are forwarded. And, of course, it creates<br>
massive dependency issues, because we don't know which ports are<br>
going to end up with the IP addresses required. This API should take<br>
a reference to a Port as the nexthop. I've been told we can't even<br>
simulate this in Heat at the moment because a VPN connection doesn't<br>
have a port associated with it. (If the API accepted _either_ a Port<br>
or a VPN connection, that would be fine by me though.) So far we've<br>
been unable to merge ExtraRoutes into Heat, except for a plugin in<br>
/contrib, for want of a way to make this reliably work in the<br>
correct dependency order without resorting to progressively worse hacks.<br>
<br>
I'm sure fresh horrors await in corners I have not yet dug into. I<br>
must say that the VPN Service, happily, is one that seems to have<br>
done things right. Firewall looks pretty good in itself, although<br>
the fact that it is completely disjoint from any other configuration<br>
- i.e. you can't even specify which network it applies to, let alone<br>
which gateway - is incomprehensible.<br>
<br>
<br>
Over the past couple of development cycles, we've seen a number of<br>
proposals to push orchestration-like features into Neutron itself.<br>
It is now clear to me why: because the Neutron API is illegible to<br>
external orchestration tools, this leads to people wanting to do an<br>
end run around it.<br>
<br>
I don't expect that the current API can be fixed without breaking<br>
backwards compatibility, but I hope that folks will take these<br>
concepts into account the next time the Neutron API gets revised. (I<br>
also hope we won't see any more proposals to effectively reimplement<br>
Heat behind the Neutron API ;) Please fell free to include [Heat] in<br>
any discussion along those lines, we'd be happy to give feedback on<br>
any given API designs. In exchange, if any Neutron folks are able to<br>
explain the exact ways in which my ideas about how the current<br>
Neutron API does and/or should work are wrong and/or crazy, I would<br>
be most appreciative :)<br>
<br>
cheers,<br>
Zane.<br>
<br></div></div>
______________________________<u></u>___________________<br>
OpenStack-dev mailing list<br>
OpenStack-dev@lists.openstack.<u></u>__org<br>
<mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.<u></u>openstack.org</a>><br>
<a href="http://lists.openstack.org/__cgi-bin/mailman/listinfo/__openstack-dev" target="_blank">http://lists.openstack.org/__<u></u>cgi-bin/mailman/listinfo/__<u></u>openstack-dev</a> <<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a>><div class="">
<br>
<br>
<br>
<br>
<br>
--<br>
Kevin Benton<br>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
<br>
</div></blockquote><div class="HOEnZb"><div class="h5">
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Kevin Benton</div>
</div>