<div dir="ltr">Hi Karthik,<div><br></div><div>what do you mean that the plugin is incompatible with <a href="https://review.openstack.org/#/c/114393/">https://review.openstack.org/#/c/114393/</a>?</div><div>you're mentioning a rebase issue - but the patch in question appears to cleanly apply to master.</div>
<div><br></div><div>Is your probably because patch #114393 does not have in its log some changes you need to accommodate test_lib changes?</div><div>Are these changes you need already merged? In this case you might try to rebase the patch you're going to test on master before running devstack, which I think it's also what happens in the upstream gate.</div>
<div><br></div><div>Salvatore</div><div><br></div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On 26 August 2014 21:57, Karthik Natarajan <span dir="ltr"><<a href="mailto:natarajk@brocade.com" target="_blank">natarajk@brocade.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi Edgar,<br>
<br>
We are also facing CI issues when the neutron patch set is not rebased with latest changes.<br>
For e.g. CI audit that you posted today (<a href="https://review.openstack.org/#/c/114393/" target="_blank">https://review.openstack.org/#/c/114393/</a>) is not rebased with neutron test_lib related changes.<br>
We had refactored the Brocade Vyatta plugin unit tests to accommodate the test_lib related changes.<br>
But our plugin is not compatible with the patch you have posted. So CI is failing.<br>
<br>
I had a discussion with Dane Leblanc on this. We also need to post the SKIPPED status for such patch sets.<br>
We will also experiment with Kevin's suggestion.<br>
<br>
Thanks,<br>
Karthik<br>
<div class="HOEnZb"><div class="h5"><br>
-----Original Message-----<br>
From: Dane Leblanc (leblancd) [mailto:<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>]<br>
Sent: Monday, August 25, 2014 10:02 AM<br>
To: OpenStack Development Mailing List (not for usage questions)<br>
Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run<br>
<br>
Edgar, Kyle:<br>
<br>
Kevin's suggestion should work for me (still hashing out the implementation). I've added an item to the 3rd Party IRC agenda anyway to discuss this corner case.<br>
<br>
Thanks!<br>
Dane<br>
<br>
-----Original Message-----<br>
From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
Sent: Monday, August 25, 2014 12:44 PM<br>
To: OpenStack Development Mailing List (not for usage questions)<br>
Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run<br>
<br>
Dane,<br>
<br>
I will second Kyle's idea. Let's discuss this during today IRC meeting if Kevin's suggestion does not work for you.<br>
<br>
Thanks,<br>
<br>
Edgar<br>
<br>
On 8/25/14, 10:08 AM, "Kyle Mestery" <<a href="mailto:mestery@mestery.com">mestery@mestery.com</a>> wrote:<br>
<br>
>Dane, thanks for all the great work you're doing in the third-party CI<br>
>area. It's great to see you working to share this knowledge with others<br>
>as well!<br>
><br>
>Did Kevin's idea work for you to move past this issue? If not, I<br>
>suggest you put an item on the neutron meeting agenda today and we<br>
>cover this there. You could put the item on the third-party meeting<br>
>agenda as well.<br>
><br>
>Thanks!<br>
>Kyle<br>
><br>
>On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd)<br>
><<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>> wrote:<br>
>> Hi Kevin:<br>
>><br>
>><br>
>><br>
>> Thanks, this is a great idea! I may try just a slight variation of<br>
>>this concept. Maybe your idea could be the recommended way to create<br>
>>a 3rd party CI for plugins that are just being introduced and need to<br>
>>limit the scope of testing to a small set of plugin-related commits<br>
>>(or plugins blocked on a certain fix).<br>
>><br>
>><br>
>><br>
>> Thanks,<br>
>> Dane<br>
>><br>
>><br>
>><br>
>> From: Kevin Benton [mailto:<a href="mailto:blak111@gmail.com">blak111@gmail.com</a>]<br>
>> Sent: Saturday, August 23, 2014 5:47 AM<br>
>><br>
>><br>
>> To: OpenStack Development Mailing List (not for usage questions)<br>
>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are<br>
>>required to be run<br>
>><br>
>><br>
>><br>
>> Can you disable posting of results directly from your Jenkins/Zuul<br>
>>setup and have a script that just checks the log file for special<br>
>>markers to determine if the vote should be FAILED/PASSED/SKIPPED?<br>
>>Another advantage of this approach is that it gives you an<br>
>>opportunity to detect when a job just failed to setup due to<br>
>>infrastructure reasons and trigger a recheck without ever first<br>
>>posting a failure to gerrit.<br>
>><br>
>><br>
>><br>
>> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd)<br>
>> <<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>> wrote:<br>
>><br>
>> Thanks Edgar for updating the APIC status!!!<br>
>><br>
>> Edgar and Kyle: *****PLEASE NOTE******!!!! I need your understanding<br>
>>and advice on the following:<br>
>><br>
>> We are still stuck with a problem stemming from a design limitation<br>
>> of Jenkins that prevents us from being compliant with Neutron 3rd<br>
>> Party CI requirements for our DFA CI.<br>
>><br>
>> The issue is that Jenkins only allows our scripts to<br>
>>(programmatically) return either Success or Fail. There is no option<br>
>>to return "Aborted", "Not Tested", or "Skipped".<br>
>><br>
>> Why does this matter? The DFA plugin is just being introduced, and<br>
>>initial DFA-enabling change sets have not yet been merged. Therefore,<br>
>>all other change sets will fail our Tempest tests, since they are not<br>
>>DFA-enabled.<br>
>><br>
>> Similarly, we were recently blocked in our APIC CI with a critical<br>
>> bug, causing all change sets without this fix to fail on our APIC testbed.<br>
>><br>
>> In these cases, we would like to enter a "throttled" or "partially<br>
>>blocked"<br>
>> mode, where we would skip testing on change sets we know will fail,<br>
>>and (in an ideal world) signal this shortcoming to Gerrit e.g. by<br>
>>returning a "Skipped" status. Unfortunately, this option is not<br>
>>available in Jenkins scripts, as Jenkins is currently designed. The<br>
>>only options we have available is "Success" or all "Fail", which are<br>
>>both misleading. We would also incorrectly report success or fail on<br>
>>one of the following test<br>
>> commits:<br>
>> <a href="https://review.openstack.org/#/c/114393/" target="_blank">https://review.openstack.org/#/c/114393/</a><br>
>> <a href="https://review.openstack.org/#/c/40296/" target="_blank">https://review.openstack.org/#/c/40296/</a><br>
>><br>
>> I've brought this issue up on the openstack-infra IRC, and jeblair<br>
>>confirmed the Jenkins limitation, but asked me to get consensus from<br>
>>the Neutron community as to this being a problem/requirement. I've<br>
>>also sent out an e-mail on the Neutron ML trying to start a<br>
>>discussion on this problem (no traction). I plan on bringing this up<br>
>>in the 3rd Party CI IRC on Monday, assuming there is time permitted<br>
>>in the open discussion.<br>
>><br>
>> I'm also investigating<br>
>><br>
>> For the short term, I would like to propose the following:<br>
>> * We bring this up on the 3rd Party CI IRC on Monday to get a<br>
>>solution or workaround, if available. If a solution is available,<br>
>>let's consider including that as a hint when we come up with CI<br>
>>requirements for handling CIs bocked by some critical fix.<br>
>> * I'm also looking into using a REST API to cancel a Jenkins job<br>
>>programmatically.<br>
>> * If no solution or workaround is available, we work with infra team<br>
>>or with Jenkins team to create a solution.<br>
>> * Until a solution is available, for plugins which are blocked by a<br>
>>critical bug, we post a status/notes indicating the plugin's<br>
>>situation on our 3rd party CI status wiki, e.g.:<br>
>><br>
>> Vendor Plugin/Driver Name Contact Name<br>
>> Status Notes<br>
>> My Vendor Name My Plugin CI My Contact Person T<br>
>> Throttled / Partially blocked / Awaiting Intial Commits<br>
>><br>
>> The status/notes should be clear and understood by the Neutron team.<br>
>>The<br>
>> console logs for change sets where the tests were skipped should also<br>
>>contain a message that all testing is being skipped for that commit.<br>
>><br>
>> Note that when the DFA initial commits are merged, then this issue<br>
>>would go away for the DFA CI. However, this problem will reappear<br>
>>every time a blocking critical bug shows up for a 3rd party CI setup,<br>
>>or a new plugin is introduced and the hardware-enabling commits are<br>
>>not yet merged. (That is, until we have a solution for the Jenkins<br>
>>limitation).<br>
>><br>
>> Let me know what you think.<br>
>><br>
>><br>
>> Thanks,<br>
>> Dane<br>
>><br>
>> -----Original Message-----<br>
>> From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
>><br>
>> Sent: Friday, August 22, 2014 1:57 PM<br>
>> To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not<br>
>>for usage questions)<br>
>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are<br>
>>required to be run<br>
>><br>
>> Sorry my bad but I just changed.<br>
>><br>
>> Edgar<br>
>><br>
>> On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>><br>
>>wrote:<br>
>><br>
>>>Edgar:<br>
>>><br>
>>>I'm still seeing the comment "Results are not accurate. Needs<br>
>>>clarification..."<br>
>>><br>
>>>Dane<br>
>>><br>
>>>-----Original Message-----<br>
>>>From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
>>>Sent: Thursday, August 21, 2014 2:58 PM<br>
>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not<br>
>>>for usage questions)<br>
>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are<br>
>>>required to be run<br>
>>><br>
>>>Dane,<br>
>>><br>
>>>Wiki has been updated.<br>
>>><br>
>>>Thanks,<br>
>>><br>
>>>Edgar<br>
>>><br>
>>>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>><br>
>>>wrote:<br>
>>><br>
>>>>Edgar:<br>
>>>><br>
>>>>The status on the wiki page says "Results are not accurate. Needs<br>
>>>>clarification from Cisco".<br>
>>>>Can you please tell me what we are missing?<br>
>>>><br>
>>>>-Dane<br>
>>>><br>
>>>>-----Original Message-----<br>
>>>>From: Dane Leblanc (leblancd)<br>
>>>>Sent: Tuesday, August 19, 2014 3:05 PM<br>
>>>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for<br>
>>>>usage<br>
>>>>questions)<br>
>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are<br>
>>>>required to be run<br>
>>>><br>
>>>>The APIC CI did run tests against that commit (after some queue<br>
>>>>latency):<br>
>>>><br>
>>>><a href="http://128.107.233.28:8080/job/apic/1860/" target="_blank">http://128.107.233.28:8080/job/apic/1860/</a><br>
>>>><a href="http://cisco-neutron-ci.cisco.com/logs/apic/1860/" target="_blank">http://cisco-neutron-ci.cisco.com/logs/apic/1860/</a><br>
>>>><br>
>>>>But the review comments never showed up on Gerrit. This seems to be<br>
>>>>an intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered<br>
>>>>from this Jenkins/Gerrit server. Whenever we disable another one of<br>
>>>>our other Jenkins jobs (in this case, we disabled DFA for some<br>
>>>>rework), the review comments sometimes stop showing up on Gerrit.<br>
>>>><br>
>>>>-----Original Message-----<br>
>>>>From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
>>>>Sent: Tuesday, August 19, 2014 1:33 PM<br>
>>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not<br>
>>>>for usage questions)<br>
>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are<br>
>>>>required to be run<br>
>>>><br>
>>>>I was looking to one of the most recent Neutron commits:<br>
>>>><a href="https://review.openstack.org/#/c/115175/" target="_blank">https://review.openstack.org/#/c/115175/</a><br>
>>>><br>
>>>><br>
>>>>I could not find the APIC report.<br>
>>>><br>
>>>>Edgar<br>
>>>><br>
>>>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>><br>
>>>>wrote:<br>
>>>><br>
>>>>>From which commit is it missing?<br>
>>>>><a href="https://review.openstack.org/#/c/114629/" target="_blank">https://review.openstack.org/#/c/114629/</a><br>
>>>>><a href="https://review.openstack.org/#/c/114393/" target="_blank">https://review.openstack.org/#/c/114393/</a><br>
>>>>><br>
>>>>>-----Original Message-----<br>
>>>>>From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
>>>>>Sent: Tuesday, August 19, 2014 12:28 PM<br>
>>>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List<br>
>>>>>(not for usage questions)<br>
>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are<br>
>>>>>required to be run<br>
>>>>><br>
>>>>>Dane,<br>
>>>>><br>
>>>>>Are you sure about it?<br>
>>>>>I just went to this commit and I could not find the APIC tests.<br>
>>>>><br>
>>>>>Thanks,<br>
>>>>><br>
>>>>>Edgar<br>
>>>>><br>
>>>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>><br>
>>>>>wrote:<br>
>>>>><br>
>>>>>>Edgar:<br>
>>>>>><br>
>>>>>>The Cisco APIC should be reporting results for both APIC-related<br>
>>>>>>and non-APIC related changes now.<br>
>>>>>>(See <a href="http://cisco-neutron-ci.cisco.com/logs/apic/1738/" target="_blank">http://cisco-neutron-ci.cisco.com/logs/apic/1738/</a>).<br>
>>>>>><br>
>>>>>>Will you be updating the wiki page?<br>
>>>>>><br>
>>>>>>-Dane<br>
>>>>>><br>
>>>>>>-----Original Message-----<br>
>>>>>>From: Dane Leblanc (leblancd)<br>
>>>>>>Sent: Friday, August 15, 2014 8:18 PM<br>
>>>>>>To: OpenStack Development Mailing List (not for usage questions)<br>
>>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests<br>
>>>>>>are required to be run<br>
>>>>>><br>
>>>>>>Also, you can add me as a contact person for the Cisco VPNaaS driver.<br>
>>>>>><br>
>>>>>>-----Original Message-----<br>
>>>>>>From: Dane Leblanc (leblancd)<br>
>>>>>>Sent: Friday, August 15, 2014 8:14 PM<br>
>>>>>>To: OpenStack Development Mailing List (not for usage questions)<br>
>>>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests<br>
>>>>>>are required to be run<br>
>>>>>><br>
>>>>>>Edgar:<br>
>>>>>><br>
>>>>>>For the Notes for the Cisco APIC, can you change the comment<br>
>>>>>>"results are fake" to something like "results are only valid for<br>
>>>>>>APIC-related commits"? I think this more accurately represents our<br>
>>>>>>current results (for reasons we chatted about on another thread).<br>
>>>>>><br>
>>>>>>Thanks,<br>
>>>>>>Dane<br>
>>>>>><br>
>>>>>>-----Original Message-----<br>
>>>>>>From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
>>>>>>Sent: Friday, August 15, 2014 6:36 PM<br>
>>>>>>To: OpenStack Development Mailing List (not for usage questions)<br>
>>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests<br>
>>>>>>are required to be run<br>
>>>>>>Importance: High<br>
>>>>>><br>
>>>>>>Team,<br>
>>>>>><br>
>>>>>>I did a quick audit on the Neutron CI. Very sad results. Only few<br>
>>>>>>plugins and drivers are running properly and testing all Neutron<br>
>>>>>>commits.<br>
>>>>>>I created a report here:<br>
>>>>>><a href="https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existi" target="_blank">https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existi</a><br>
>>>>>>ng<br>
>>>>>>_<br>
>>>>>>P<br>
>>>>>>l<br>
>>>>>>ugi<br>
>>>>>>n<br>
>>>>>>_and_Drivers<br>
>>>>>><br>
>>>>>><br>
>>>>>>We will discuss the actions to take on the next Neutron IRC meeting.<br>
>>>>>>So please, reach me out to clarify what is the status of your CI.<br>
>>>>>>I had two commits to quickly verify the CI reliability:<br>
>>>>>><br>
>>>>>><a href="https://review.openstack.org/#/c/114393/" target="_blank">https://review.openstack.org/#/c/114393/</a><br>
>>>>>><br>
>>>>>><a href="https://review.openstack.org/#/c/40296/" target="_blank">https://review.openstack.org/#/c/40296/</a><br>
>>>>>><br>
>>>>>><br>
>>>>>>I would expect all plugins and drivers passing on the first one<br>
>>>>>>and failing for the second but I got so many surprises.<br>
>>>>>><br>
>>>>>>Neutron code quality and reliability is a top priority, if you<br>
>>>>>>ignore this report that plugin/driver will be candidate to be<br>
>>>>>>remove from Neutron tree.<br>
>>>>>><br>
>>>>>>Cheers,<br>
>>>>>><br>
>>>>>>Edgar<br>
>>>>>><br>
>>>>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the<br>
>>>>>>dirty job!<br>
>>>>>><br>
>>>>>><br>
>>>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <<a href="mailto:mestery@mestery.com">mestery@mestery.com</a>> wrote:<br>
>>>>>><br>
>>>>>>>Folks, I'm not sure if all CI accounts are running sufficient tests.<br>
>>>>>>>Per the requirements wiki page here [1], everyone needs to be<br>
>>>>>>>running more than just Tempest API tests, which I still see most<br>
>>>>>>>neutron third-party CI setups doing. I'd like to ask everyone who<br>
>>>>>>>operates a third-party CI account for Neutron to please look at<br>
>>>>>>>the link below and make sure you are running appropriate tests.<br>
>>>>>>>If you have questions, the weekly third-party meeting [2] is a<br>
>>>>>>>great place to ask questions.<br>
>>>>>>><br>
>>>>>>>Thanks,<br>
>>>>>>>Kyle<br>
>>>>>>><br>
>>>>>>>[1] <a href="https://wiki.openstack.org/wiki/NeutronThirdPartyTesting" target="_blank">https://wiki.openstack.org/wiki/NeutronThirdPartyTesting</a><br>
>>>>>>>[2] <a href="https://wiki.openstack.org/wiki/Meetings/ThirdParty" target="_blank">https://wiki.openstack.org/wiki/Meetings/ThirdParty</a><br>
>>>>>>><br>
>>>>>>>_______________________________________________<br>
>>>>>>>OpenStack-dev mailing list<br>
>>>>>>><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>>>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>>>><br>
>>>>>><br>
>>>>>>_______________________________________________<br>
>>>>>>OpenStack-dev mailing list<br>
>>>>>><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>>>><br>
>>>>>>_______________________________________________<br>
>>>>>>OpenStack-dev mailing list<br>
>>>>>><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>>><br>
>>>><br>
>>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> --<br>
>><br>
>> Kevin Benton<br>
>><br>
>><br>
>> _______________________________________________<br>
>> OpenStack-dev mailing list<br>
>> <a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>><br>
><br>
>_______________________________________________<br>
>OpenStack-dev mailing list<br>
><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>