<div dir="ltr">Can you disable posting of results directly from your Jenkins/Zuul setup and have a script that just checks the log file for special markers to determine if the vote should be FAILED/PASSED/SKIPPED? Another advantage of this approach is that it gives you an opportunity to detect when a job just failed to setup due to infrastructure reasons and trigger a recheck without ever first posting a failure to gerrit.</div>
<div class="gmail_extra"><br><br><div class="gmail_quote">On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) <span dir="ltr"><<a href="mailto:leblancd@cisco.com" target="_blank">leblancd@cisco.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks Edgar for updating the APIC status!!!<br>
<br>
Edgar and Kyle: *****PLEASE NOTE******!!!! I need your understanding and advice on the following:<br>
<br>
We are still stuck with a problem stemming from a design limitation of Jenkins that prevents us from being compliant with Neutron 3rd Party CI requirements for our DFA CI.<br>
<br>
The issue is that Jenkins only allows our scripts to (programmatically) return either Success or Fail. There is no option to return "Aborted", "Not Tested", or "Skipped".<br>
<br>
Why does this matter? The DFA plugin is just being introduced, and initial DFA-enabling change sets have not yet been merged. Therefore, all other change sets will fail our Tempest tests, since they are not DFA-enabled.<br>
<br>
Similarly, we were recently blocked in our APIC CI with a critical bug, causing all change sets without this fix to fail on our APIC testbed.<br>
<br>
In these cases, we would like to enter a "throttled" or "partially blocked" mode, where we would skip testing on change sets we know will fail, and (in an ideal world) signal this shortcoming to Gerrit e.g. by returning a "Skipped" status. Unfortunately, this option is not available in Jenkins scripts, as Jenkins is currently designed. The only options we have available is "Success" or all "Fail", which are both misleading. We would also incorrectly report success or fail on one of the following test commits:<br>
<a href="https://review.openstack.org/#/c/114393/" target="_blank">https://review.openstack.org/#/c/114393/</a><br>
<a href="https://review.openstack.org/#/c/40296/" target="_blank">https://review.openstack.org/#/c/40296/</a><br>
<br>
I've brought this issue up on the openstack-infra IRC, and jeblair confirmed the Jenkins limitation, but asked me to get consensus from the Neutron community as to this being a problem/requirement. I've also sent out an e-mail on the Neutron ML trying to start a discussion on this problem (no traction). I plan on bringing this up in the 3rd Party CI IRC on Monday, assuming there is time permitted in the open discussion.<br>
<br>
I'm also investigating<br>
<br>
For the short term, I would like to propose the following:<br>
* We bring this up on the 3rd Party CI IRC on Monday to get a solution or workaround, if available. If a solution is available, let's consider including that as a hint when we come up with CI requirements for handling CIs bocked by some critical fix.<br>
* I'm also looking into using a REST API to cancel a Jenkins job programmatically.<br>
* If no solution or workaround is available, we work with infra team or with Jenkins team to create a solution.<br>
* Until a solution is available, for plugins which are blocked by a critical bug, we post a status/notes indicating the plugin's situation on our 3rd party CI status wiki, e.g.:<br>
<br>
Vendor Plugin/Driver Name Contact Name Status Notes<br>
My Vendor Name My Plugin CI My Contact Person T Throttled / Partially blocked / Awaiting Intial Commits<br>
<br>
The status/notes should be clear and understood by the Neutron team. The console logs for change sets where the tests were skipped should also contain a message that all testing is being skipped for that commit.<br>
<br>
Note that when the DFA initial commits are merged, then this issue would go away for the DFA CI. However, this problem will reappear every time a blocking critical bug shows up for a 3rd party CI setup, or a new plugin is introduced and the hardware-enabling commits are not yet merged. (That is, until we have a solution for the Jenkins limitation).<br>
<br>
Let me know what you think.<br>
<div class="im HOEnZb"><br>
Thanks,<br>
Dane<br>
<br>
-----Original Message-----<br>
From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
</div><div class="HOEnZb"><div class="h5">Sent: Friday, August 22, 2014 1:57 PM<br>
To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for usage questions)<br>
Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run<br>
<br>
Sorry my bad but I just changed.<br>
<br>
Edgar<br>
<br>
On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>> wrote:<br>
<br>
>Edgar:<br>
><br>
>I'm still seeing the comment "Results are not accurate. Needs<br>
>clarification..."<br>
><br>
>Dane<br>
><br>
>-----Original Message-----<br>
>From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
>Sent: Thursday, August 21, 2014 2:58 PM<br>
>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not<br>
>for usage questions)<br>
>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are<br>
>required to be run<br>
><br>
>Dane,<br>
><br>
>Wiki has been updated.<br>
><br>
>Thanks,<br>
><br>
>Edgar<br>
><br>
>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>> wrote:<br>
><br>
>>Edgar:<br>
>><br>
>>The status on the wiki page says "Results are not accurate. Needs<br>
>>clarification from Cisco".<br>
>>Can you please tell me what we are missing?<br>
>><br>
>>-Dane<br>
>><br>
>>-----Original Message-----<br>
>>From: Dane Leblanc (leblancd)<br>
>>Sent: Tuesday, August 19, 2014 3:05 PM<br>
>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for usage<br>
>>questions)<br>
>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are<br>
>>required to be run<br>
>><br>
>>The APIC CI did run tests against that commit (after some queue latency):<br>
>><br>
>><a href="http://128.107.233.28:8080/job/apic/1860/" target="_blank">http://128.107.233.28:8080/job/apic/1860/</a><br>
>><a href="http://cisco-neutron-ci.cisco.com/logs/apic/1860/" target="_blank">http://cisco-neutron-ci.cisco.com/logs/apic/1860/</a><br>
>><br>
>>But the review comments never showed up on Gerrit. This seems to be an<br>
>>intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from<br>
>>this Jenkins/Gerrit server. Whenever we disable another one of our<br>
>>other Jenkins jobs (in this case, we disabled DFA for some rework),<br>
>>the review comments sometimes stop showing up on Gerrit.<br>
>><br>
>>-----Original Message-----<br>
>>From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
>>Sent: Tuesday, August 19, 2014 1:33 PM<br>
>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not<br>
>>for usage questions)<br>
>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are<br>
>>required to be run<br>
>><br>
>>I was looking to one of the most recent Neutron commits:<br>
>><a href="https://review.openstack.org/#/c/115175/" target="_blank">https://review.openstack.org/#/c/115175/</a><br>
>><br>
>><br>
>>I could not find the APIC report.<br>
>><br>
>>Edgar<br>
>><br>
>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>><br>
>>wrote:<br>
>><br>
>>>From which commit is it missing?<br>
>>><a href="https://review.openstack.org/#/c/114629/" target="_blank">https://review.openstack.org/#/c/114629/</a><br>
>>><a href="https://review.openstack.org/#/c/114393/" target="_blank">https://review.openstack.org/#/c/114393/</a><br>
>>><br>
>>>-----Original Message-----<br>
>>>From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
>>>Sent: Tuesday, August 19, 2014 12:28 PM<br>
>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not<br>
>>>for usage questions)<br>
>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are<br>
>>>required to be run<br>
>>><br>
>>>Dane,<br>
>>><br>
>>>Are you sure about it?<br>
>>>I just went to this commit and I could not find the APIC tests.<br>
>>><br>
>>>Thanks,<br>
>>><br>
>>>Edgar<br>
>>><br>
>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <<a href="mailto:leblancd@cisco.com">leblancd@cisco.com</a>><br>
>>>wrote:<br>
>>><br>
>>>>Edgar:<br>
>>>><br>
>>>>The Cisco APIC should be reporting results for both APIC-related and<br>
>>>>non-APIC related changes now.<br>
>>>>(See <a href="http://cisco-neutron-ci.cisco.com/logs/apic/1738/" target="_blank">http://cisco-neutron-ci.cisco.com/logs/apic/1738/</a>).<br>
>>>><br>
>>>>Will you be updating the wiki page?<br>
>>>><br>
>>>>-Dane<br>
>>>><br>
>>>>-----Original Message-----<br>
>>>>From: Dane Leblanc (leblancd)<br>
>>>>Sent: Friday, August 15, 2014 8:18 PM<br>
>>>>To: OpenStack Development Mailing List (not for usage questions)<br>
>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are<br>
>>>>required to be run<br>
>>>><br>
>>>>Also, you can add me as a contact person for the Cisco VPNaaS driver.<br>
>>>><br>
>>>>-----Original Message-----<br>
>>>>From: Dane Leblanc (leblancd)<br>
>>>>Sent: Friday, August 15, 2014 8:14 PM<br>
>>>>To: OpenStack Development Mailing List (not for usage questions)<br>
>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are<br>
>>>>required to be run<br>
>>>><br>
>>>>Edgar:<br>
>>>><br>
>>>>For the Notes for the Cisco APIC, can you change the comment<br>
>>>>"results are fake" to something like "results are only valid for<br>
>>>>APIC-related commits"? I think this more accurately represents our<br>
>>>>current results (for reasons we chatted about on another thread).<br>
>>>><br>
>>>>Thanks,<br>
>>>>Dane<br>
>>>><br>
>>>>-----Original Message-----<br>
>>>>From: Edgar Magana [mailto:<a href="mailto:edgar.magana@workday.com">edgar.magana@workday.com</a>]<br>
>>>>Sent: Friday, August 15, 2014 6:36 PM<br>
>>>>To: OpenStack Development Mailing List (not for usage questions)<br>
>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are<br>
>>>>required to be run<br>
>>>>Importance: High<br>
>>>><br>
>>>>Team,<br>
>>>><br>
>>>>I did a quick audit on the Neutron CI. Very sad results. Only few<br>
>>>>plugins and drivers are running properly and testing all Neutron<br>
>>>>commits.<br>
>>>>I created a report here:<br>
>>>><a href="https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing" target="_blank">https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing</a><br>
>>>>_<br>
>>>>P<br>
>>>>l<br>
>>>>ugi<br>
>>>>n<br>
>>>>_and_Drivers<br>
>>>><br>
>>>><br>
>>>>We will discuss the actions to take on the next Neutron IRC meeting.<br>
>>>>So please, reach me out to clarify what is the status of your CI.<br>
>>>>I had two commits to quickly verify the CI reliability:<br>
>>>><br>
>>>><a href="https://review.openstack.org/#/c/114393/" target="_blank">https://review.openstack.org/#/c/114393/</a><br>
>>>><br>
>>>><a href="https://review.openstack.org/#/c/40296/" target="_blank">https://review.openstack.org/#/c/40296/</a><br>
>>>><br>
>>>><br>
>>>>I would expect all plugins and drivers passing on the first one and<br>
>>>>failing for the second but I got so many surprises.<br>
>>>><br>
>>>>Neutron code quality and reliability is a top priority, if you<br>
>>>>ignore this report that plugin/driver will be candidate to be remove<br>
>>>>from Neutron tree.<br>
>>>><br>
>>>>Cheers,<br>
>>>><br>
>>>>Edgar<br>
>>>><br>
>>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the<br>
>>>>dirty job!<br>
>>>><br>
>>>><br>
>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <<a href="mailto:mestery@mestery.com">mestery@mestery.com</a>> wrote:<br>
>>>><br>
>>>>>Folks, I'm not sure if all CI accounts are running sufficient tests.<br>
>>>>>Per the requirements wiki page here [1], everyone needs to be<br>
>>>>>running more than just Tempest API tests, which I still see most<br>
>>>>>neutron third-party CI setups doing. I'd like to ask everyone who<br>
>>>>>operates a third-party CI account for Neutron to please look at the<br>
>>>>>link below and make sure you are running appropriate tests. If you<br>
>>>>>have questions, the weekly third-party meeting [2] is a great place<br>
>>>>>to ask questions.<br>
>>>>><br>
>>>>>Thanks,<br>
>>>>>Kyle<br>
>>>>><br>
>>>>>[1] <a href="https://wiki.openstack.org/wiki/NeutronThirdPartyTesting" target="_blank">https://wiki.openstack.org/wiki/NeutronThirdPartyTesting</a><br>
>>>>>[2] <a href="https://wiki.openstack.org/wiki/Meetings/ThirdParty" target="_blank">https://wiki.openstack.org/wiki/Meetings/ThirdParty</a><br>
>>>>><br>
>>>>>_______________________________________________<br>
>>>>>OpenStack-dev mailing list<br>
>>>>><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>><br>
>>>><br>
>>>>_______________________________________________<br>
>>>>OpenStack-dev mailing list<br>
>>>><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>>><br>
>>>>_______________________________________________<br>
>>>>OpenStack-dev mailing list<br>
>>>><a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
>>>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
>>><br>
>><br>
><br>
<br>
<br>
_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br><br clear="all"><div><br></div>-- <br><div>Kevin Benton</div>
</div>