Open Stack

Thanks Edgar for updating the APIC status!!! 

Edgar and Kyle: *****PLEASE NOTE******!!!!  I need your understanding and advice on the following:

We are still stuck with a problem stemming from a design limitation of Jenkins that prevents us from being compliant with Neutron 3rd Party CI requirements for our DFA CI.

The issue is that Jenkins only allows our scripts to (programmatically) return either Success or Fail. There is no option to return "Aborted", "Not Tested", or "Skipped".

Why does this matter? The DFA plugin is just being introduced, and initial DFA-enabling change sets have not yet been merged. Therefore, all other change sets will fail our Tempest tests, since they are not DFA-enabled.

Similarly, we were recently blocked in our APIC CI with a critical bug, causing all change sets without this fix to fail on our APIC testbed.

In these cases, we would like to enter a "throttled" or "partially blocked" mode, where we would skip testing on change sets we know will fail, and (in an ideal world) signal this shortcoming to Gerrit e.g. by returning a "Skipped" status. Unfortunately, this option is not available in Jenkins scripts, as Jenkins is currently designed. The only options we have available is "Success" or all "Fail", which are both misleading. We would also incorrectly report success or fail on one of the following test commits:
https://review.openstack.org/#/c/114393/
https://review.openstack.org/#/c/40296/

I've brought this issue up on the openstack-infra IRC, and jeblair confirmed the Jenkins limitation, but asked me to get consensus from the Neutron community as to this being a problem/requirement. I've also sent out an e-mail on the Neutron ML trying to start a discussion on this problem (no traction). I plan on bringing this up in the 3rd Party CI IRC on Monday, assuming there is time permitted in the open discussion.

I'm also investigating 

For the short term, I would like to propose the following:
* We bring this up on the 3rd Party CI IRC on Monday to get a solution or workaround, if available. If a solution is available, let's consider including that as a hint when we come up with CI requirements for handling CIs bocked by some critical fix.
* I'm also looking into using a REST API to cancel a Jenkins job programmatically.
* If no solution or workaround is available, we work with infra team or with Jenkins team to create a solution.
* Until a solution is available, for plugins which are blocked by a critical bug, we post a status/notes indicating the plugin's situation on our 3rd party CI status wiki, e.g.:

Vendor 			Plugin/Driver Name 	Contact Name 		Status 	Notes
My Vendor Name 	My Plugin CI 		My Contact Person 	T 	Throttled / Partially blocked / Awaiting Intial Commits

The status/notes should be clear and understood by the Neutron team.  The console logs for change sets where the tests were skipped should also contain a message that all testing is being skipped for that commit.

Note that when the DFA initial commits are merged, then this issue would go away for the DFA CI. However, this problem will reappear every time a blocking critical bug shows up for a 3rd party CI setup, or a new plugin is introduced and the hardware-enabling commits are not yet merged.  (That is, until we have a solution for the Jenkins limitation).

Let me know what you think.

Thanks,
Dane

-----Original Message-----
From: Edgar Magana [mailto:edgar.magana at workday.com] 
Sent: Friday, August 22, 2014 1:57 PM
To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run

Sorry my bad but I just changed.

Edgar

On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <leblancd at cisco.com> wrote:

>Edgar:
>
>I'm still seeing the comment "Results are not accurate. Needs 
>clarification..."
>
>Dane
>
>-----Original Message-----
>From: Edgar Magana [mailto:edgar.magana at workday.com]
>Sent: Thursday, August 21, 2014 2:58 PM
>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not 
>for usage questions)
>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are 
>required to be run
>
>Dane,
>
>Wiki has been updated.
>
>Thanks,
>
>Edgar
>
>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <leblancd at cisco.com> wrote:
>
>>Edgar:
>>
>>The status on the wiki page says "Results are not accurate. Needs 
>>clarification from Cisco".
>>Can you please tell me what we are missing?
>>
>>-Dane
>>
>>-----Original Message-----
>>From: Dane Leblanc (leblancd)
>>Sent: Tuesday, August 19, 2014 3:05 PM
>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for usage
>>questions)
>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are 
>>required to be run
>>
>>The APIC CI did run tests against that commit (after some queue latency):
>>
>>http://128.107.233.28:8080/job/apic/1860/
>>http://cisco-neutron-ci.cisco.com/logs/apic/1860/
>>
>>But the review comments never showed up on Gerrit. This seems to be an 
>>intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from 
>>this Jenkins/Gerrit server. Whenever we disable another one of our 
>>other Jenkins jobs (in this case, we disabled DFA for some rework), 
>>the review comments sometimes stop showing up on Gerrit.
>>
>>-----Original Message-----
>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>>Sent: Tuesday, August 19, 2014 1:33 PM
>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not 
>>for usage questions)
>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are 
>>required to be run
>>
>>I was looking to one of the most recent Neutron commits:
>>https://review.openstack.org/#/c/115175/
>>
>>
>>I could not find the APIC report.
>>
>>Edgar
>>
>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
>>wrote:
>>
>>>From which commit is it missing?
>>>https://review.openstack.org/#/c/114629/
>>>https://review.openstack.org/#/c/114393/
>>>
>>>-----Original Message-----
>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>>>Sent: Tuesday, August 19, 2014 12:28 PM
>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not 
>>>for usage questions)
>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are 
>>>required to be run
>>>
>>>Dane,
>>>
>>>Are you sure about it?
>>>I just went to this commit and I could not find the APIC tests.
>>>
>>>Thanks,
>>>
>>>Edgar
>>>
>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
>>>wrote:
>>>
>>>>Edgar:
>>>>
>>>>The Cisco APIC should be reporting results for both APIC-related and 
>>>>non-APIC related changes now.
>>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/).
>>>>
>>>>Will you be updating the wiki page?
>>>>
>>>>-Dane
>>>>
>>>>-----Original Message-----
>>>>From: Dane Leblanc (leblancd)
>>>>Sent: Friday, August 15, 2014 8:18 PM
>>>>To: OpenStack Development Mailing List (not for usage questions)
>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are 
>>>>required to be run
>>>>
>>>>Also, you can add me as a contact person for the Cisco VPNaaS driver.
>>>>
>>>>-----Original Message-----
>>>>From: Dane Leblanc (leblancd)
>>>>Sent: Friday, August 15, 2014 8:14 PM
>>>>To: OpenStack Development Mailing List (not for usage questions)
>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are 
>>>>required to be run
>>>>
>>>>Edgar:
>>>>
>>>>For the Notes for the Cisco APIC, can you change the comment 
>>>>"results are fake" to something like "results are only valid for 
>>>>APIC-related commits"? I think this more accurately represents our 
>>>>current results (for reasons we chatted about on another thread).
>>>>
>>>>Thanks,
>>>>Dane
>>>>
>>>>-----Original Message-----
>>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>>>>Sent: Friday, August 15, 2014 6:36 PM
>>>>To: OpenStack Development Mailing List (not for usage questions)
>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are 
>>>>required to be run
>>>>Importance: High
>>>>
>>>>Team,
>>>>
>>>>I did a quick audit on the Neutron CI. Very sad results. Only few 
>>>>plugins and drivers are running properly and testing all Neutron 
>>>>commits.
>>>>I created a report here:
>>>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing
>>>>_
>>>>P
>>>>l
>>>>ugi
>>>>n
>>>>_and_Drivers
>>>>
>>>>
>>>>We will discuss the actions to take on the next Neutron IRC meeting.
>>>>So please, reach me out to clarify what is the status of your CI.
>>>>I had two commits to quickly verify the CI reliability:
>>>>
>>>>https://review.openstack.org/#/c/114393/
>>>>
>>>>https://review.openstack.org/#/c/40296/
>>>>
>>>>
>>>>I would expect all plugins and drivers passing on the first one and 
>>>>failing for the second but I got so many surprises.
>>>>
>>>>Neutron code quality and reliability is a top priority, if you 
>>>>ignore this report that plugin/driver will be candidate to be remove 
>>>>from Neutron tree.
>>>>
>>>>Cheers,
>>>>
>>>>Edgar
>>>>
>>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the 
>>>>dirty job!
>>>>
>>>>
>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <mestery at mestery.com> wrote:
>>>>
>>>>>Folks, I'm not sure if all CI accounts are running sufficient tests.
>>>>>Per the requirements wiki page here [1], everyone needs to be 
>>>>>running more than just Tempest API tests, which I still see most 
>>>>>neutron third-party CI setups doing. I'd like to ask everyone who 
>>>>>operates a third-party CI account for Neutron to please look at the 
>>>>>link below and make sure you are running appropriate tests. If you 
>>>>>have questions, the weekly third-party meeting [2] is a great place 
>>>>>to ask questions.
>>>>>
>>>>>Thanks,
>>>>>Kyle
>>>>>
>>>>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
>>>>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty
>>>>>
>>>>>_______________________________________________
>>>>>OpenStack-dev mailing list
>>>>>OpenStack-dev at lists.openstack.org
>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>
>>>>_______________________________________________
>>>>OpenStack-dev mailing list
>>>>OpenStack-dev at lists.openstack.org
>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>>_______________________________________________
>>>>OpenStack-dev mailing list
>>>>OpenStack-dev at lists.openstack.org
>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>