<div dir="ltr"><br><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div>Vladimir Kozhukalov</div></div></div>
<br><div class="gmail_quote">On Tue, Dec 9, 2014 at 3:51 PM, Dmitry Tantsur <span dir="ltr"><<a href="mailto:dtantsur@redhat.com" target="_blank">dtantsur@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Hi folks,<br>
<br>
Thank you for additional explanation, it does clarify things a bit. I'd like to note, however, that you talk a lot about how _different_ Fuel Agent is from what Ironic does now. I'd like actually to know how well it's going to fit into what Ironic does (in additional to your specific use cases). Hence my comments inline:</blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div class="h5"><br>
<br>
On 12/09/2014 01:01 PM, Vladimir Kozhukalov wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
Just a short explanation of Fuel use case.<br>
<br>
Fuel use case is not a cloud. Fuel is a deployment tool. We install OS<br>
on bare metal servers and on VMs<br>
and then configure this OS using Puppet. We have been using Cobbler as<br>
our OS provisioning tool since the beginning of Fuel.<br>
However, Cobbler assumes using native OS installers (Anaconda and<br>
Debian-installer). For some reasons we decided to<br>
switch to image based approach for installing OS.<br>
<br>
One of Fuel features is the ability to provide advanced partitioning<br>
schemes (including software RAIDs, LVM).<br>
Native installers are quite difficult to customize in the field of<br>
partitioning<br>
(that was one of the reasons to switch to image based approach).<br>
Moreover, we'd like to implement even more<br>
flexible user experience. We'd like to allow user to choose which hard<br>
drives to use for root FS, for<br>
allocating DB. We'd like user to be able to put root FS over LV or MD<br>
device (including stripe, mirror, multipath).<br>
We'd like user to be able to choose which hard drives are bootable (if<br>
any), which options to use for mounting file systems.<br>
Many many various cases are possible. If you ask why we'd like to<br>
support all those cases, the answer is simple:<br>
because our users want us to support all those cases.<br>
Obviously, many of those cases can not be implemented as image<br>
internals, some cases can not be also implemented on<br>
configuration stage (placing root fs on lvm device).<br>
<br>
As far as those use cases were rejected to be implemented in term of<br>
IPA, we implemented so called Fuel Agent.<br>
Important Fuel Agent features are:<br>
<br>
* It does not have REST API<br>
</blockquote></div></div>
I would not call it a feature :-P<br>
<br>
Speaking seriously, if you agent is a long-running thing and it gets it's configuration from e.g. JSON file, how can Ironic notify it of any changes?<span class=""><br>
<br></span></blockquote><div>Fuel Agent is not long-running service. Currently there is no need to have REST API. If we deal with kind of keep alive stuff of inventory/discovery then we probably add API. Frankly, IPA REST API is not REST at all. However that is not a reason to not to call it a feature and through it away. It is a reason to work on it and improve. That is how I try to look at things (pragmatically). </div><div><br></div><div>Fuel Agent has executable entry point[s] like /usr/bin/provision. You can run this entry point with options (oslo.config) and point out where to find input json data. It is supposed Ironic will use ssh (currently in Fuel we use mcollective) connection and run this waiting for exit code. If exit code is equal to 0, provisioning is done. Extremely simple. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
* it has executable entry point[s]<br>
* It uses local json file as it's input<br>
* It is planned to implement ability to download input data via HTTP<br>
(kind of metadata service)<br>
* It is designed to be agnostic to input data format, not only Fuel<br>
format (data drivers)<br>
* It is designed to be agnostic to image format (tar images, file system<br>
images, disk images, currently fs images)<br>
* It is designed to be agnostic to image compression algorithm<br>
(currently gzip)<br>
* It is designed to be agnostic to image downloading protocol (currently<br>
local file and HTTP link)<br>
</blockquote></span>
Does it support Glance? I understand it's HTTP, but it requires authentication.<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
So, it is clear that being motivated by Fuel, Fuel Agent is quite<br>
independent and generic. And we are open for<br>
new use cases.<br>
</blockquote></span>
My favorite use case is hardware introspection (aka getting data required for scheduling from a node automatically). Any ideas on this? (It's not a priority for this discussion, just curious).</blockquote><div><br></div><div>That is exactly what we do in Fuel. Currently we use so called 'Default' pxelinux config and all nodes being powered on are supposed to boot with so called 'Bootstrap' ramdisk where Ohai based agent (not Fuel Agent) runs periodically and sends hardware report to Fuel master node.</div><div>User then is able to look at CPU, hard drive and network info and choose which nodes to use for controllers, which for computes, etc. That is what nova scheduler is supposed to do (look at hardware info and choose a suitable node).</div><div><br></div><div>Talking about future, we are planning to re-implement inventory/discovery stuff in terms of Fuel Agent (currently, this stuff is implemented as Ohai based independent script). Estimation for that is March 2015.</div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
According Fuel itself, our nearest plan is to get rid of Cobbler because<br>
in the case of image based approach it is huge overhead. The question is<br>
which tool we can use instead of Cobbler. We need power management,<br>
we need TFTP management, we need DHCP management. That is<br>
exactly what Ironic is able to do. Frankly, we can implement power/TFTP/DHCP<br>
management tool independently, but as Devananda said, we're all working<br>
on the same problems,<br>
so let's do it together. Power/TFTP/DHCP management is where we are<br>
working on the same problems,<br>
but IPA and Fuel Agent are about different use cases. This case is not<br>
just Fuel, any mature<br>
deployment case require advanced partition/fs management.<br>
</blockquote></span>
Taking into consideration that you're doing a generic OS installation tool... yeah, it starts to make some sense. For cloud advanced partition is definitely a "pet" case.</blockquote><div><br></div><div>Generic image based OS installation tool.</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class=""><br>
However, for<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
me it is OK, if it is easily possible<br>
to use Ironic with external drivers (not merged to Ironic and not tested<br>
on Ironic CI).<br>
<br>
AFAIU, this spec <a href="https://review.openstack.org/#/c/138115/" target="_blank">https://review.openstack.org/#<u></u>/c/138115/</a> does not<br>
assume changing Ironic API and core.<br>
Jim asked about how Fuel Agent will know about advanced disk<br>
partitioning scheme if API is not<br>
changed. The answer is simple: Ironic is supposed to send a link to<br>
metadata service (http or local file)<br>
where Fuel Agent can download input json data.<br>
</blockquote></span>
That's not about not changing Ironic. Changing Ironic is ok for reasonable use cases - we do a huge change right now to accommodate zapping, hardware introspection and RAID configuration.<br>
<br></blockquote><div>Minimal changes because we don't want to break anything. It is clear how difficult to convince people to do even minimal changes. Again it is just a pragmatic approach. We want to do things iteratively so as not to break Ironic as well as Fuel. We just can not change all at once. </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
I actually have problems with this particular statement. It does not sound like Fuel Agent will integrate enough with Ironic. This JSON file: who is going to generate it? In the most popular use case we're driven by Nova. Will Nova generate this file?<br>
<br>
If the answer is "generate it manually for every node", it's too much a "pet" case for me personally.<span class=""><br>
<br></span></blockquote><div>That is how this provision data look like right now <a href="https://github.com/stackforge/fuel-web/blob/master/fuel_agent_ci/samples/provision.json">https://github.com/stackforge/fuel-web/blob/master/fuel_agent_ci/samples/provision.json</a> Do you still think it is written manually? Currently Fuel Agent works as a part of Fuel ecosystem. We have a service which serializes provision data for us into json. Fuel Agent is agnostic to data format (data drivers). If someone wants to use another format, they are welcome to implement a driver. </div><div><br></div><div>We assume next step will be to put provision data (disk partition scheme, maybe other data) into driver_info and make Fuel Agent driver able to serialize those data (special format) and implement a corresponding data driver in Fuel Agent for this format. Again very simple. Maybe it is time to think of having Ironic metadata service (just maybe). </div><div><br></div><div>Another point is that currently Fuel stores hardware info in its own database but when it is possible to get those data from Ironic (when inventory stuff is implemented) we will be glad to use Ironic API for that. That is what I mean when I say 'to make Fuel stuff closer to Ironic abstractions'</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
As Roman said, we try to be pragmatic and suggest something which does<br>
not break anything. All changes<br>
are supposed to be encapsulated into a driver. No API and core changes.<br>
We have resources to support, test<br>
and improve this driver. This spec is just a zero step. Further steps<br>
are supposed to improve driver<br>
so as to make it closer to Ironic abstractions.<br>
</blockquote></span>
Honestly I think you should at least write a roadmap for it - see my comments above.<br></blockquote><div><br></div><div>Honestly, I think writing roadmap right now is not very rational as far as I am not even sure people are interested in widening Ironic use cases. Some of the comments were not even constructive like "I don't understand what your use case is, please use IPA". </div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
About testing and support: are you providing a 3rdparty CI for it? It would be a big plus as to me: we already have troubles with drivers broken accidentally.</blockquote><div><br></div><div>We are flexible here but I'm not ready to answer this question right now. We will try to fit Ironic requirements wherever it is possible. </div><div> <br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class=""><br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
For Ironic that means widening use cases and user community. But, as I<br>
already said,<br>
we are OK if Ironic does not need this feature.<br>
</blockquote></span>
I don't think we should through away your hardware provision use case, but I personally would like to see how well Fuel Agent is going to play with how Ironic and Nova operate.<br></blockquote><div><br></div><div>Nova is not our case. Fuel is totally about deployment. There is some in common </div><div><br></div><div>As I already explained, currently we need power/tftp/dhcp management Ironic capabilities. Again, it is not a problem to implement this stuff independently like it happened with Fuel Agent (because this use case was rejected several months ago). Our suggestion is not about "let's compete with IPA" it is totally about "let's work on the same problems together".</div><div> <br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><span class="">
<br>
Vladimir Kozhukalov<br>
<br>
On Tue, Dec 9, 2014 at 1:09 PM, Roman Prykhodchenko<br></span><div><div class="h5">
<<a href="mailto:rprikhodchenko@mirantis.com" target="_blank">rprikhodchenko@mirantis.com</a> <mailto:<a href="mailto:rprikhodchenko@mirantis.com" target="_blank">rprikhodchenko@<u></u>mirantis.com</a>>> wrote:<br>
<br>
It is true that IPA and FuelAgent share a lot of functionality in<br>
common. However there is a major difference between them which is<br>
that they are intended to be used to solve a different problem.<br>
<br>
IPA is a solution for provision-use-destroy-use_by_<u></u>different_user<br>
use-case and is really great for using it for providing BM nodes for<br>
other OS services or in services like Rackspace OnMetal. FuelAgent<br>
itself serves for provision-use-use-…-use use-case like Fuel or<br>
TripleO have.<br>
<br>
Those two use-cases require concentration on different details in<br>
first place. For instance for IPA proper decommissioning is more<br>
important than advanced disk management, but for FuelAgent<br>
priorities are opposite because of obvious reasons.<br>
<br>
Putting all functionality to a single driver and a single agent may<br>
cause conflicts in priorities and make a lot of mess inside both the<br>
driver and the agent. Actually previously changes to IPA were<br>
blocked right because of this conflict of priorities. Therefore<br>
replacing FuelAgent by IPA in where FuelAgent is used currently does<br>
not seem like a good option because come people (and I’m not talking<br>
about Mirantis) might loose required features because of different<br>
priorities.<br>
<br>
Having two separate drivers along with two separate agents for those<br>
different use-cases will allow to have two independent teams that<br>
are concentrated on what’s really important for a specific use-case.<br>
I don’t see any problem in overlapping functionality if it’s used<br>
differently.<br>
<br>
<br>
P. S.<br>
I realise that people may be also confused by the fact that<br>
FuelAgent is actually called like that and is used only in Fuel atm.<br>
Our point is to make it a simple, powerful and what’s more important<br>
a generic tool for provisioning. It is not bound to Fuel or Mirantis<br>
and if it will cause confusion in the future we will even be happy<br>
to give it a different and less confusing name.<br>
<br>
P. P. S.<br>
Some of the points of this integration do not look generic enough or<br>
nice enough. We look pragmatic on the stuff and are trying to<br>
implement what’s possible to implement as the first step. For sure<br>
this is going to have a lot more steps to make it better and more<br>
generic.<br>
<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div class="h5">
On 09 Dec 2014, at 01:46, Jim Rollenhagen <<a href="mailto:jim@jimrollenhagen.com" target="_blank">jim@jimrollenhagen.com</a><br></div></div><span class="">
<mailto:<a href="mailto:jim@jimrollenhagen.com" target="_blank">jim@jimrollenhagen.com</a><u></u>>> wrote:<br>
<br>
<br>
<br>
On December 8, 2014 2:23:58 PM PST, Devananda van der Veen<br></span><div><div class="h5">
<<a href="mailto:devananda.vdv@gmail.com" target="_blank">devananda.vdv@gmail.com</a> <mailto:<a href="mailto:devananda.vdv@gmail.com" target="_blank">devananda.vdv@gmail.<u></u>com</a>>> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
I'd like to raise this topic for a wider discussion outside of the<br>
hallway<br>
track and code reviews, where it has thus far mostly remained.<br>
<br>
In previous discussions, my understanding has been that the Fuel team<br>
sought to use Ironic to manage "pets" rather than "cattle" - and<br>
doing<br>
so<br>
required extending the API and the project's functionality in<br>
ways that<br>
no<br>
one else on the core team agreed with. Perhaps that understanding was<br>
wrong<br>
(or perhaps not), but in any case, there is now a proposal to add a<br>
FuelAgent driver to Ironic. The proposal claims this would meet that<br>
teams'<br>
needs without requiring changes to the core of Ironic.<br>
<br>
<a href="https://review.openstack.org/#/c/138115/" target="_blank">https://review.openstack.org/#<u></u>/c/138115/</a><br>
</blockquote>
<br>
I think it's clear from the review that I share the opinions<br>
expressed in this email.<br>
<br>
That said (and hopefully without derailing the thread too much),<br>
I'm curious how this driver could do software RAID or LVM without<br>
modifying Ironic's API or data model. How would the agent know how<br>
these should be built? How would an operator or user tell Ironic<br>
what the disk/partition/volume layout would look like?<br>
<br>
And before it's said - no, I don't think vendor passthru API calls<br>
are an appropriate answer here.<br>
<br>
// jim<br>
<br>
</div></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div class="h5">
<br>
The Problem Description section calls out four things, which have all<br>
been<br>
discussed previously (some are here [0]). I would like to address<br>
each<br>
one,<br>
invite discussion on whether or not these are, in fact, problems<br>
facing<br>
Ironic (not whether they are problems for someone, somewhere),<br>
and then<br>
ask<br>
why these necessitate a new driver be added to the project.<br>
<br>
<br>
They are, for reference:<br>
<br>
1. limited partition support<br>
<br>
2. no software RAID support<br>
<br>
3. no LVM support<br>
<br>
4. no support for hardware that lacks a BMC<br>
<br>
#1.<br>
<br>
When deploying a partition image (eg, QCOW format), Ironic's PXE<br>
deploy<br>
driver performs only the minimal partitioning necessary to<br>
fulfill its<br>
mission as an OpenStack service: respect the user's request for root,<br>
swap,<br>
and ephemeral partition sizes. When deploying a whole-disk image,<br>
Ironic<br>
does not perform any partitioning -- such is left up to the operator<br>
who<br>
created the disk image.<br>
<br>
Support for arbitrarily complex partition layouts is not required by,<br>
nor<br>
does it facilitate, the goal of provisioning physical servers via a<br>
common<br>
cloud API. Additionally, as with #3 below, nothing prevents a<br>
user from<br>
creating more partitions in unallocated disk space once they have<br>
access to<br>
their instance. Therefor, I don't see how Ironic's minimal<br>
support for<br>
partitioning is a problem for the project.<br>
<br>
#2.<br>
<br>
There is no support for defining a RAID in Ironic today, at all,<br>
whether<br>
software or hardware. Several proposals were floated last cycle;<br>
one is<br>
under review right now for DRAC support [1], and there are multiple<br>
call<br>
outs for RAID building in the state machine mega-spec [2]. Any such<br>
support<br>
for hardware RAID will necessarily be abstract enough to support<br>
multiple<br>
hardware vendor's driver implementations and both in-band<br>
creation (via<br>
IPA) and out-of-band creation (via vendor tools).<br>
<br>
Given the above, it may become possible to add software RAID<br>
support to<br>
IPA<br>
in the future, under the same abstraction. This would closely tie the<br>
deploy agent to the images it deploys (the latter image's kernel<br>
would<br>
be<br>
dependent upon a software RAID built by the former), but this would<br>
necessarily be true for the proposed FuelAgent as well.<br>
<br>
I don't see this as a compelling reason to add a new driver to the<br>
project.<br>
Instead, we should (plan to) add support for software RAID to the<br>
deploy<br>
agent which is already part of the project.<br>
<br>
#3.<br>
<br>
LVM volumes can easily be added by a user (after provisioning) within<br>
unallocated disk space for non-root partitions. I have not yet seen a<br>
compelling argument for doing this within the provisioning phase.<br>
<br>
#4.<br>
<br>
There are already in-tree drivers [3] [4] [5] which do not require a<br>
BMC.<br>
One of these uses SSH to connect and run pre-determined commands.<br>
Like<br>
the<br>
spec proposal, which states at line 122, "Control via SSH access<br>
feature<br>
intended only for experiments in non-production environment," the<br>
current<br>
SSHPowerDriver is only meant for testing environments. We could<br>
probably<br>
extend this driver to do what the FuelAgent spec proposes, as far as<br>
remote<br>
power control for cheap always-on hardware in testing<br>
environments with<br>
a<br>
pre-shared key.<br>
<br>
(And if anyone wonders about a use case for Ironic without external<br>
power<br>
control ... I can only think of one situation where I would<br>
rationally<br>
ever<br>
want to have a control-plane agent running inside a<br>
user-instance: I am<br>
both the operator and the only user of the cloud.)<br>
<br>
<br>
----------------<br>
<br>
In summary, as far as I can tell, all of the problem statements upon<br>
which<br>
the FuelAgent proposal are based are solvable through incremental<br>
changes<br>
in existing drivers, or out of scope for the project entirely. As<br>
another<br>
software-based deploy agent, FuelAgent would duplicate the<br>
majority of<br>
the<br>
functionality which ironic-python-agent has today.<br>
<br>
Ironic's driver ecosystem benefits from a diversity of<br>
hardware-enablement<br>
drivers. Today, we have two divergent software deployment drivers<br>
which<br>
approach image deployment differently: "agent" drivers use a local<br>
agent to<br>
prepare a system and download the image; "pxe" drivers use a remote<br>
agent<br>
and copy the image over iSCSI. I don't understand how a second driver<br>
which<br>
duplicates the functionality we already have, and shares the same<br>
goals<br>
as<br>
the drivers we already have, is beneficial to the project.<br>
<br>
Doing the same thing twice just increases the burden on the team;<br>
we're<br>
all<br>
working on the same problems, so let's do it together.<br>
<br>
-Devananda<br>
<br>
<br>
[0]<br>
<a href="https://blueprints.launchpad.net/ironic/+spec/ironic-python-agent-partition" target="_blank">https://blueprints.launchpad.<u></u>net/ironic/+spec/ironic-<u></u>python-agent-partition</a><br>
<br>
[1] <a href="https://review.openstack.org/#/c/107981/" target="_blank">https://review.openstack.org/#<u></u>/c/107981/</a><br>
<br>
[2]<br>
<a href="https://review.openstack.org/#/c/133828/11/specs/kilo/new-ironic-state-machine.rst" target="_blank">https://review.openstack.org/#<u></u>/c/133828/11/specs/kilo/new-<u></u>ironic-state-machine.rst</a><br>
<br>
<br>
[3]<br>
<a href="http://git.openstack.org/cgit/openstack/ironic/tree/ironic/drivers/modules/snmp.py" target="_blank">http://git.openstack.org/cgit/<u></u>openstack/ironic/tree/ironic/<u></u>drivers/modules/snmp.py</a><br>
<br>
[4]<br>
<a href="http://git.openstack.org/cgit/openstack/ironic/tree/ironic/drivers/modules/iboot.py" target="_blank">http://git.openstack.org/cgit/<u></u>openstack/ironic/tree/ironic/<u></u>drivers/modules/iboot.py</a><br>
<br>
[5]<br>
<a href="http://git.openstack.org/cgit/openstack/ironic/tree/ironic/drivers/modules/ssh.py" target="_blank">http://git.openstack.org/cgit/<u></u>openstack/ironic/tree/ironic/<u></u>drivers/modules/ssh.py</a><br>
<br>
<br>
------------------------------<u></u>------------------------------<u></u>------------<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br></div></div>
<mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.<u></u>openstack.org</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote><span class="">
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br></span>
<mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.<u></u>openstack.org</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</blockquote><span class="">
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br></span>
<mailto:<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.<u></u>openstack.org</a>><span class=""><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
<br>
</span></blockquote><div class=""><div class="h5">
<br>
<br>
______________________________<u></u>_________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.<u></u>org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-dev</a><br>
</div></div></blockquote></div><br></div></div>