<div dir="ltr">Hi,<div><br></div><div>On upcoming summit we will have a track on Fuel & Ironic (Ironic-inspector) integration, so you are welcome to participate.</div><div><br></div><div>Thursday 9:00</div><div><a href="https://etherpad.openstack.org/p/fuel-newton-summit-planning">https://etherpad.openstack.org/p/fuel-newton-summit-planning</a><br></div><div><br></div><div>Thanks,</div></div><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Mar 18, 2016 at 9:49 PM, Jim Rollenhagen <span dir="ltr"><<a href="mailto:jim@jimrollenhagen.com" target="_blank">jim@jimrollenhagen.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On Fri, Mar 18, 2016 at 07:26:03PM +0300, Vladimir Kozhukalov wrote:<br>
> >Well, there's a number of reasons. Ironic is not meant only for an<br>
> >"undercloud" (deploying OpenStack on ironic instances). There are both<br>
> >public and private cloud deployments of ironic in production today, that<br>
> >make bare metal instances available to users of the cloud. Those users<br>
> >may not want an agent running inside their instance, and more<br>
> >importantly, the operators of those clouds may not want to expose the<br>
> >ironic or inspector APIs to their users.<br>
><br>
> >I'm not sure ironic should say "no, that isn't allowed" but at a minimum<br>
> >it would need to be opt-in behavior.<br>
><br>
> For me it's absolutely clear why cloud case does assume running any kind of<br>
> agent<br>
> inside user instance. It is clear why cloud case does not assume exposing<br>
> API<br>
> to the user instance. But cloud is not the only case that exists.<br>
> Fuel is a deployment tool. Fuel case is not cloud. It is 'cattle' (cattle<br>
> vs. pets), but<br>
> it is not cloud in a sense that instances are 'user instances'.<br>
> Fuel 'user instances' are not even 'user' instances.<br>
> Fuel manages the content of instances throughout their whole life cycle.<br>
<br>
</span>To be clear, I'm not saying we shouldn't do it. I'm saying we should<br>
talk about it. Ironic can't assume there's an agent, but we sure could<br>
make it optional. I do realize Fuel is a valid use case, and I want to<br>
be able to support that use case.<br>
<span class="HOEnZb"><font color="#888888"><br>
// jim<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
> As you might remember we talked about this about two years ago (when we<br>
> tried to contribute lvm and md features to IPA). I don't know why this case<br>
> (deployment) was rejected again and again while it's still viable and<br>
> widely used.<br>
> And I don't know why it could not be implemented to be 'opt-in'.<br>
> Since that we have invented our own fuel-agent (that supports lvm, md) and<br>
> a driver for Ironic conductor that allows to use Ironic with fuel-agent.<br>
><br>
> >Is the fuel team having a summit session of some sort about integrating<br>
> >with ironic better? I'd be happy to come to that if it can be scheduled<br>
> >at a time that ironic doesn't have a session. Otherwise maybe we can<br>
> >catch up on Friday or something.<br>
><br>
> >I'm glad to see Fuel wanting to integrate better with Ironic.<br>
><br>
> We are still quite interested in closer integration with Ironic (we need<br>
> power<br>
> management features that Ironic provides). We'll be happy to schedule yet<br>
> another discussion on closer integration with Ironic.<br>
><br>
> BTW, about a year ago (in Grenoble) we agreed that it is not even<br>
> necessary to merge such custom things into Ironic tree. Happily, Ironic is<br>
> smart enough to consume drivers using stevedore. About ironic-inspector<br>
> the case is the same. Whether we are going to run it inside 'user instance'<br>
> or inside ramdisk it does not affect ironic-inspector itself. If Ironic<br>
> team is<br>
> open for merging "non-cloud" features (of course 'opt-in') we'll be happy<br>
> to contribute.<br>
><br>
> Vladimir Kozhukalov<br>
><br>
> On Fri, Mar 18, 2016 at 6:03 PM, Jim Rollenhagen <<a href="mailto:jim@jimrollenhagen.com">jim@jimrollenhagen.com</a>><br>
> wrote:<br>
><br>
> > On Fri, Mar 18, 2016 at 05:26:13PM +0300, Evgeniy L wrote:<br>
> > > On Thu, Mar 17, 2016 at 3:16 PM, Dmitry Tantsur <<a href="mailto:dtantsur@redhat.com">dtantsur@redhat.com</a>><br>
> > wrote:<br>
> > ><br>
> > > > On 03/16/2016 01:39 PM, Evgeniy L wrote:<br>
> > > ><br>
> > > >> Hi Dmitry,<br>
> > > >><br>
> > > >> I can try to provide you description on what current Nailgun agent is,<br>
> > > >> and what are potential requirements we may need from HW discovery<br>
> > system.<br>
> > > >><br>
> > > >> Nailgun agent is a one-file Ruby script [0] which is periodically run<br>
> > > >> under cron. It collects information about HW using ohai [1], plus it<br>
> > > >> does custom parsing, filtration, retrieval of HW information. After<br>
> > the<br>
> > > >> information is collected, it is sent to Nailgun, that is how node gets<br>
> > > >> discovered in Fuel.<br>
> > > >><br>
> > > ><br>
> > > > Quick clarification: does it run on user instances? or does it run on<br>
> > > > hardware while it's still not deployed to? The former is something that<br>
> > > > Ironic tries not to do. There is an interest in the latter.<br>
> > ><br>
> > ><br>
> > > Both, on user instances (with deployed OpenStack) and on instances which<br>
> > > are not deployed and in bootstrap.<br>
> > > What are the reasons Ironic tries not to do that (running HW discovery on<br>
> > > deployed node)?<br>
> ><br>
> > Well, there's a number of reasons. Ironic is not meant only for an<br>
> > "undercloud" (deploying OpenStack on ironic instances). There are both<br>
> > public and private cloud deployments of ironic in production today, that<br>
> > make bare metal instances available to users of the cloud. Those users<br>
> > may not want an agent running inside their instance, and more<br>
> > importantly, the operators of those clouds may not want to expose the<br>
> > ironic or inspector APIs to their users.<br>
> ><br>
> > I'm not sure ironic should say "no, that isn't allowed" but at a minimum<br>
> > it would need to be opt-in behavior.<br>
> ><br>
> > ><br>
> > ><br>
> > > ><br>
> > > ><br>
> > > >> To summarise entire process:<br>
> > > >> 1. After Fuel master node is installed, user restarts the nodes and<br>
> > they<br>
> > > >> get booted via PXE with bootstrap image.<br>
> > > >> 2. Inside of bootstrap image Nailgun agent is configured and<br>
> > installed.<br>
> > > >> 3. Cron runs Nailgun agent.<br>
> > > >> 3. Information is collected by Nailgun agent.<br>
> > > >> 4. Information is sent to Nailgun.<br>
> > > >> 5. Nailgun creates new node, for which user using UI can define<br>
> > > >> partitioning schema and networks allocation.<br>
> > > >> 6. After that provisioning/deployment can be run.<br>
> > > >><br>
> > > ><br>
> > > > So it looks quite similar to ironic-inspector + IPA, except<br>
> > introspection<br>
> > > > runs once. Rerunning it would not be impossible to implement, though it<br>
> > > > will require some changes to inspector, so that it can accept<br>
> > "updates" to<br>
> > > > a node after the introspection is finished.<br>
> > > ><br>
> > > ><br>
> > > >> Every time Nailgun agent sends a request, we submit information about<br>
> > > >> the time last request from agent was done, if there was no request for<br>
> > > >> time N, we mark the node as offline.<br>
> > > >><br>
> > > ><br>
> > > > This is similar to IPA heartbeating, I guess.<br>
> > > ><br>
> > > ><br>
> > > >> With current implementation we have several problems (not all of them<br>
> > > >> should be addressed by HW discovery system only):<br>
> > > >><br>
> > > >> 1. A lot of things are hardcoded on the agent's side, which does<br>
> > > >> additional filtration based on pre-hardcoded parameters [2], the less<br>
> > > >> hardcoded logic in agent we have the easier it's to do upgrades and<br>
> > > >> deliver fixes (upgrade one service is simpler than hundreds of<br>
> > agents).<br>
> > > >><br>
> > > ><br>
> > > > Oh, I hear it. In the inspector world we are moving away from<br>
> > processing<br>
> > > > things on the ramdisk side exactly for this reason: it's too hard to<br>
> > change.<br>
> > > ><br>
> > > ><br>
> > > >> 2. In order to get additional HW information user has to continue<br>
> > > >> hardcoding it right in Ruby code, as the result, there is no way for<br>
> > > >> Fuel plugin [3], to get additional hardware specific information, we<br>
> > > >> need data-driven mechanism to be able to describe, what/how/where<br>
> > > >> information to get from the node.<br>
> > > >><br>
> > > ><br>
> > > > Hmm, interesting. Right now we have a plugin mechanism for the<br>
> > ramdisk. We<br>
> > > > also have a plugin (extra-hardware) trying to collect as much<br>
> > information<br>
> > > > as it's feasible (based on <a href="https://github.com/redhat-cip/hardware" rel="noreferrer" target="_blank">https://github.com/redhat-cip/hardware</a>).<br>
> > > ><br>
> > ><br>
> > > Could you please provide a link where I can learn more on plugin<br>
> > mechanism<br>
> > > for ramdisk?<br>
> ><br>
> > When IPA does inspection, it sends the inventory as reported by the<br>
> > hardware managers. When building a ramdisk, you can include out-of-tree<br>
> > hardware managers, and each hardware manager is called to fetch<br>
> > inventory.<br>
> ><br>
> > Docs:<br>
> > <a href="http://docs.openstack.org/developer/ironic-python-agent/#hardware-inventory" rel="noreferrer" target="_blank">http://docs.openstack.org/developer/ironic-python-agent/#hardware-inventory</a><br>
> > Example out-of-tree hardware managers:<br>
> ><br>
> > <a href="https://github.com/openstack/proliantutils/tree/master/proliantutils/ipa_hw_manager" rel="noreferrer" target="_blank">https://github.com/openstack/proliantutils/tree/master/proliantutils/ipa_hw_manager</a><br>
> > <a href="https://github.com/rackerlabs/onmetal-ironic-hardware-manager" rel="noreferrer" target="_blank">https://github.com/rackerlabs/onmetal-ironic-hardware-manager</a><br>
> ><br>
> > ><br>
> > ><br>
> > > ><br>
> > > > On the other side, there is ongoing work to have an ansible-based<br>
> > deploy<br>
> > > > ramdisk in Ironic, maybe inspector could benefit from it too. Didn't<br>
> > think<br>
> > > > about it yet, would be interesting to discuss on the summit.<br>
> > ><br>
> > ><br>
> > > And here, I would appreciate if you have any link to get more context (I<br>
> > > was able to find only playbook for Ironic installation).<br>
> > > In Fuel we had an idea to implement tasks (abstract from specific<br>
> > > deployment tool) to make configuration and get information about specific<br>
> > > hardware.<br>
> ><br>
> > The spec is in review, from some Mirantis folks in fact:<br>
> > <a href="https://review.openstack.org/#/c/241946/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/241946/</a><br>
> ><br>
> > ><br>
> > ><br>
> > > ><br>
> > > ><br>
> > > ><br>
> > > >> 3. Hardware gets changed, we have cases when NICs, HDDs, Motherboards<br>
> > > >> are replaced/removed/added, as the result we should have a tool which<br>
> > > >> would allow us to see these changes and when they were performed,<br>
> > based<br>
> > > >> on that we want to be able to notify the user and provide suggestions<br>
> > > >> how to proceed with these changes.<br>
> > > >><br>
> > > ><br>
> > > > This could probably done with a new ironic-inspector plugin.<br>
> > > ><br>
> > > ><br>
> > > >> 4. With 3rd real-world cases, we have a problem of node<br>
> > identification,<br>
> > > >> when HW gets changed and automatic matching doesn't happen (when we<br>
> > > >> cannot say for sure that this is the node which we've already had),<br>
> > user<br>
> > > >> should be able to say, that new node X is in fact offline node Y.<br>
> > > >><br>
> > > ><br>
> > > > Very good question. Right now Inspector is using either BMC IP address<br>
> > or<br>
> > > > MAC's.<br>
> > > ><br>
> > > ><br>
> > > >> 5. Different source of HW information, we want to have a system which<br>
> > > >> would allow us to have hardware discovery from IPMI, CSV file,<br>
> > Cobbler,<br>
> > > >> CMDB, etc at the same time.<br>
> > > >><br>
> > > ><br>
> > > > Not sure something like that should live within Ironic to be honest.<br>
> > Also<br>
> > > > worth discussing in details.<br>
> > > ><br>
> > > ><br>
> > > >> 6. Not only hardware gets changed, but operating system (with kernel<br>
> > > >> versions), for example when we used CentOS as a bootstrap (in<br>
> > bootstrap<br>
> > > >> we do provisioning/partitioning + initial configuration) and Ubuntu<br>
> > for<br>
> > > >> running OpenStack, we were getting wide range of weird problems, from<br>
> > > >> NICs renaming to Disks' ids duplication and deduplication. There<br>
> > should<br>
> > > >> be a way to track these problems (3rd item), and we should be able to<br>
> > > >> add operating system specific system to get HW information.<br>
> > > >><br>
> > > >> 7. Cron + Agent based mechanism to define if node is offline is not<br>
> > the<br>
> > > >> best, since it adds race conditions and in fact it only says if there<br>
> > is<br>
> > > >> connectivity between Nailgun and Nailgun agent.<br>
> > > >><br>
> > > ><br>
> > > > We are thinking about using some DLM for that.. No specific plans<br>
> > though,<br>
> > > > again a topic for the summit.<br>
> > > ><br>
> > > ><br>
> > > >> Will be glad to answer any questions you have, if there are any.<br>
> ><br>
> > Is the fuel team having a summit session of some sort about integrating<br>
> > with ironic better? I'd be happy to come to that if it can be scheduled<br>
> > at a time that ironic doesn't have a session. Otherwise maybe we can<br>
> > catch up on Friday or something.<br>
> ><br>
> > I'm glad to see Fuel wanting to integrate better with Ironic.<br>
> ><br>
> > // jim<br>
> ><br>
> > > >><br>
> > > >> Thanks,<br>
> > > >><br>
> > > >> [0] <a href="https://github.com/openstack/fuel-nailgun-agent/blob/master/agent" rel="noreferrer" target="_blank">https://github.com/openstack/fuel-nailgun-agent/blob/master/agent</a><br>
> > > >> [1] <a href="https://docs.chef.io/ohai.html" rel="noreferrer" target="_blank">https://docs.chef.io/ohai.html</a><br>
> > > >> [2]<br>
> > > >><br>
> > <a href="https://github.com/openstack/fuel-nailgun-agent/blob/master/agent#L46-L51" rel="noreferrer" target="_blank">https://github.com/openstack/fuel-nailgun-agent/blob/master/agent#L46-L51</a><br>
> > > >> [3] <a href="https://wiki.openstack.org/wiki/Fuel/Plugins" rel="noreferrer" target="_blank">https://wiki.openstack.org/wiki/Fuel/Plugins</a><br>
> > > >><br>
> > > >><br>
> > > >> On Wed, Mar 16, 2016 at 1:39 PM, Dmitry Tantsur <<a href="mailto:dtantsur@redhat.com">dtantsur@redhat.com</a><br>
> > > >> <mailto:<a href="mailto:dtantsur@redhat.com">dtantsur@redhat.com</a>>> wrote:<br>
> > > >><br>
> > > >> On 03/15/2016 01:53 PM, Serge Kovaleff wrote:<br>
> > > >><br>
> > > >> Dear All,<br>
> > > >><br>
> > > >> Let's compare functional abilities of both solutions.<br>
> > > >><br>
> > > >> Till the recent Mitaka release Ironic-inspector had only<br>
> > > >> Introspection<br>
> > > >> ability.<br>
> > > >><br>
> > > >> Discovery part is proposed and implemented by Anton Arefiev.<br>
> > We<br>
> > > >> should<br>
> > > >> align expectations and current and future functionality.<br>
> > > >><br>
> > > >> Adding Tags to attract the Inspector community.<br>
> > > >><br>
> > > >><br>
> > > >> Hi!<br>
> > > >><br>
> > > >> It would be great to see what we can do to fit the nailgun use<br>
> > case.<br>
> > > >> Unfortunately, I don't know much about it right now. What are you<br>
> > > >> missing?<br>
> > > >><br>
> > > >><br>
> > > >> Cheers,<br>
> > > >> Serge Kovaleff<br>
> > > >> <a href="http://www.mirantis.com" rel="noreferrer" target="_blank">http://www.mirantis.com</a> <<a href="http://www.mirantis.com/" rel="noreferrer" target="_blank">http://www.mirantis.com/</a>><br>
> > > >> cell: +38 (063) 83-155-70<br>
> > > >><br>
> > > >> On Tue, Mar 15, 2016 at 2:07 PM, Alexander Saprykin<br>
> > > >> <<a href="mailto:asaprykin@mirantis.com">asaprykin@mirantis.com</a> <mailto:<a href="mailto:asaprykin@mirantis.com">asaprykin@mirantis.com</a>><br>
> > > >> <mailto:<a href="mailto:asaprykin@mirantis.com">asaprykin@mirantis.com</a> <mailto:<a href="mailto:asaprykin@mirantis.com">asaprykin@mirantis.com</a><br>
> > >>><br>
> > > >> wrote:<br>
> > > >><br>
> > > >> Dear all,<br>
> > > >><br>
> > > >> Thank you for the opinions about this problem.<br>
> > > >><br>
> > > >> I would agree with Roman, that it is always better to<br>
> > reuse<br>
> > > >> solutions than re-inventing the wheel. We should<br>
> > investigate<br>
> > > >> possibility of using ironic-inspector and integrating it<br>
> > > >> into fuel.<br>
> > > >><br>
> > > >> Best regards,<br>
> > > >> Alexander Saprykin<br>
> > > >><br>
> > > >> 2016-03-15 13:03 GMT+01:00 Sergii Golovatiuk<br>
> > > >> <<a href="mailto:sgolovatiuk@mirantis.com">sgolovatiuk@mirantis.com</a> <mailto:<br>
> > <a href="mailto:sgolovatiuk@mirantis.com">sgolovatiuk@mirantis.com</a>><br>
> > > >> <mailto:<a href="mailto:sgolovatiuk@mirantis.com">sgolovatiuk@mirantis.com</a><br>
> > > >> <mailto:<a href="mailto:sgolovatiuk@mirantis.com">sgolovatiuk@mirantis.com</a>>>>:<br>
> > > >><br>
> > > >> My strong +1 to drop off nailgun-agent completely in<br>
> > > >> favour of<br>
> > > >> ironic-inspector. Even taking into consideration<br>
> > we'lll<br>
> > > >> need to<br>
> > > >> extend ironic-inspector for fuel needs.<br>
> > > >><br>
> > > >> --<br>
> > > >> Best regards,<br>
> > > >> Sergii Golovatiuk,<br>
> > > >> Skype #golserge<br>
> > > >> IRC #holser<br>
> > > >><br>
> > > >> On Tue, Mar 15, 2016 at 11:06 AM, Roman Prykhodchenko<br>
> > > >> <<a href="mailto:me@romcheg.me">me@romcheg.me</a> <mailto:<a href="mailto:me@romcheg.me">me@romcheg.me</a>><br>
> > > >> <mailto:<a href="mailto:me@romcheg.me">me@romcheg.me</a> <mailto:<a href="mailto:me@romcheg.me">me@romcheg.me</a>>>> wrote:<br>
> > > >><br>
> > > >> My opition on this is that we have too many<br>
> > > >> re-invented<br>
> > > >> wheels in Fuel and it’s better think about<br>
> > > >> replacing them<br>
> > > >> with something we can re-use than re-inventing<br>
> > them<br>
> > > >> one more<br>
> > > >> time.<br>
> > > >><br>
> > > >> Let’s take a look at Ironic and try to figure out<br>
> > > >> how we can<br>
> > > >> use its features for the same purpose.<br>
> > > >><br>
> > > >><br>
> > > >> - romcheg<br>
> > > >> > 15 бер. 2016 р. о 10:38 Neil Jerram<br>
> > > >> <<a href="mailto:Neil.Jerram@metaswitch.com">Neil.Jerram@metaswitch.com</a><br>
> > > >> <mailto:<a href="mailto:Neil.Jerram@metaswitch.com">Neil.Jerram@metaswitch.com</a>><br>
> > > >> <mailto:<a href="mailto:Neil.Jerram@metaswitch.com">Neil.Jerram@metaswitch.com</a><br>
> > > >><br>
> > > >> <mailto:<a href="mailto:Neil.Jerram@metaswitch.com">Neil.Jerram@metaswitch.com</a>>>> написав(ла):<br>
> > > >><br>
> > > >> ><br>
> > > >> > On 15/03/16 07:11, Vladimir Kozhukalov wrote:<br>
> > > >> >> Alexander,<br>
> > > >> >><br>
> > > >> >> We have many other places where use Ruby<br>
> > > >> (astute, puppet<br>
> > > >> custom types,<br>
> > > >> >> etc.). I don't think it is a good reason to<br>
> > > >> re-write<br>
> > > >> something just<br>
> > > >> >> because it is written in Ruby. You are right<br>
> > > >> about<br>
> > > >> tests, about plugins,<br>
> > > >> >> but let's look around. Ironic community has<br>
> > > >> already<br>
> > > >> invented discovery<br>
> > > >> >> component (btw written in python) and I can't<br>
> > > >> see any<br>
> > > >> reason why we<br>
> > > >> >> should continue putting efforts in nailgun<br>
> > > >> agent and not<br>
> > > >> try to switch<br>
> > > >> >> to ironic-inspector.<br>
> > > >> ><br>
> > > >> > +1 in general terms. It's strange to me that<br>
> > > >> there are<br>
> > > >> so many<br>
> > > >> > OpenStack deployment systems that each do each<br>
> > > >> piece of<br>
> > > >> the puzzle in<br>
> > > >> > their own way (Fuel, Foreman, MAAS/Juju etc.)<br>
> > -<br>
> > > >> and which<br>
> > > >> also means<br>
> > > >> > that I need substantial separate learning in<br>
> > > >> order to use<br>
> > > >> all these<br>
> > > >> > systems. It would be great to see some<br>
> > > >> consolidation.<br>
> > > >> ><br>
> > > >> > Regards,<br>
> > > >> > Neil<br>
> > > >> ><br>
> > > >> ><br>
> > > >> ><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > __________________________________________________________________________<br>
> > > >> > OpenStack Development Mailing List (not for<br>
> > > >> usage questions)<br>
> > > >> > Unsubscribe:<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > >> <<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > > >> <<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > > >> ><br>
> > > >><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > __________________________________________________________________________<br>
> > > >> OpenStack Development Mailing List (not for usage<br>
> > > >> questions)<br>
> > > >> Unsubscribe:<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > >> <<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > > >> <<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > > >><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > __________________________________________________________________________<br>
> > > >> OpenStack Development Mailing List (not for usage<br>
> > > >> questions)<br>
> > > >> Unsubscribe:<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > >> <<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > > >><br>
> > > >> <<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > > >><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > __________________________________________________________________________<br>
> > > >> OpenStack Development Mailing List (not for usage<br>
> > questions)<br>
> > > >> Unsubscribe:<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > >> <<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > > >><br>
> > > >> <<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > > >><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > __________________________________________________________________________<br>
> > > >> OpenStack Development Mailing List (not for usage questions)<br>
> > > >> Unsubscribe:<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > >> <<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> > > >><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > __________________________________________________________________________<br>
> > > >> OpenStack Development Mailing List (not for usage questions)<br>
> > > >> Unsubscribe:<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > >> <<br>
> > <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > >> ><br>
> > > >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > > >><br>
> > __________________________________________________________________________<br>
> > > >> OpenStack Development Mailing List (not for usage questions)<br>
> > > >> Unsubscribe:<br>
> > > >> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > >> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > > >><br>
> > > >><br>
> > > ><br>
> > > ><br>
> > __________________________________________________________________________<br>
> > > > OpenStack Development Mailing List (not for usage questions)<br>
> > > > Unsubscribe:<br>
> > <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> > > ><br>
> ><br>
> > ><br>
> > __________________________________________________________________________<br>
> > > OpenStack Development Mailing List (not for usage questions)<br>
> > > Unsubscribe:<br>
> > <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
> ><br>
> > __________________________________________________________________________<br>
> > OpenStack Development Mailing List (not for usage questions)<br>
> > Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> > <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
> ><br>
<br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>