[openstack-dev] [neutron] [third-party] What tests are required to be run
Dane Leblanc (leblancd)
leblancd at cisco.com
Mon Aug 25 17:01:48 UTC 2014
Edgar, Kyle:
Kevin's suggestion should work for me (still hashing out the implementation). I've added an item to the 3rd Party IRC agenda anyway to discuss this corner case.
Thanks!
Dane
-----Original Message-----
From: Edgar Magana [mailto:edgar.magana at workday.com]
Sent: Monday, August 25, 2014 12:44 PM
To: OpenStack Development Mailing List (not for usage questions)
Subject: Re: [openstack-dev] [neutron] [third-party] What tests are required to be run
Dane,
I will second Kyle's idea. Let's discuss this during today IRC meeting if Kevin's suggestion does not work for you.
Thanks,
Edgar
On 8/25/14, 10:08 AM, "Kyle Mestery" <mestery at mestery.com> wrote:
>Dane, thanks for all the great work you're doing in the third-party CI
>area. It's great to see you working to share this knowledge with others
>as well!
>
>Did Kevin's idea work for you to move past this issue? If not, I
>suggest you put an item on the neutron meeting agenda today and we
>cover this there. You could put the item on the third-party meeting
>agenda as well.
>
>Thanks!
>Kyle
>
>On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd)
><leblancd at cisco.com> wrote:
>> Hi Kevin:
>>
>>
>>
>> Thanks, this is a great idea! I may try just a slight variation of
>>this concept. Maybe your idea could be the recommended way to create
>>a 3rd party CI for plugins that are just being introduced and need to
>>limit the scope of testing to a small set of plugin-related commits
>>(or plugins blocked on a certain fix).
>>
>>
>>
>> Thanks,
>> Dane
>>
>>
>>
>> From: Kevin Benton [mailto:blak111 at gmail.com]
>> Sent: Saturday, August 23, 2014 5:47 AM
>>
>>
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>>required to be run
>>
>>
>>
>> Can you disable posting of results directly from your Jenkins/Zuul
>>setup and have a script that just checks the log file for special
>>markers to determine if the vote should be FAILED/PASSED/SKIPPED?
>>Another advantage of this approach is that it gives you an
>>opportunity to detect when a job just failed to setup due to
>>infrastructure reasons and trigger a recheck without ever first
>>posting a failure to gerrit.
>>
>>
>>
>> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd)
>> <leblancd at cisco.com> wrote:
>>
>> Thanks Edgar for updating the APIC status!!!
>>
>> Edgar and Kyle: *****PLEASE NOTE******!!!! I need your understanding
>>and advice on the following:
>>
>> We are still stuck with a problem stemming from a design limitation
>> of Jenkins that prevents us from being compliant with Neutron 3rd
>> Party CI requirements for our DFA CI.
>>
>> The issue is that Jenkins only allows our scripts to
>>(programmatically) return either Success or Fail. There is no option
>>to return "Aborted", "Not Tested", or "Skipped".
>>
>> Why does this matter? The DFA plugin is just being introduced, and
>>initial DFA-enabling change sets have not yet been merged. Therefore,
>>all other change sets will fail our Tempest tests, since they are not
>>DFA-enabled.
>>
>> Similarly, we were recently blocked in our APIC CI with a critical
>> bug, causing all change sets without this fix to fail on our APIC testbed.
>>
>> In these cases, we would like to enter a "throttled" or "partially
>>blocked"
>> mode, where we would skip testing on change sets we know will fail,
>>and (in an ideal world) signal this shortcoming to Gerrit e.g. by
>>returning a "Skipped" status. Unfortunately, this option is not
>>available in Jenkins scripts, as Jenkins is currently designed. The
>>only options we have available is "Success" or all "Fail", which are
>>both misleading. We would also incorrectly report success or fail on
>>one of the following test
>> commits:
>> https://review.openstack.org/#/c/114393/
>> https://review.openstack.org/#/c/40296/
>>
>> I've brought this issue up on the openstack-infra IRC, and jeblair
>>confirmed the Jenkins limitation, but asked me to get consensus from
>>the Neutron community as to this being a problem/requirement. I've
>>also sent out an e-mail on the Neutron ML trying to start a
>>discussion on this problem (no traction). I plan on bringing this up
>>in the 3rd Party CI IRC on Monday, assuming there is time permitted
>>in the open discussion.
>>
>> I'm also investigating
>>
>> For the short term, I would like to propose the following:
>> * We bring this up on the 3rd Party CI IRC on Monday to get a
>>solution or workaround, if available. If a solution is available,
>>let's consider including that as a hint when we come up with CI
>>requirements for handling CIs bocked by some critical fix.
>> * I'm also looking into using a REST API to cancel a Jenkins job
>>programmatically.
>> * If no solution or workaround is available, we work with infra team
>>or with Jenkins team to create a solution.
>> * Until a solution is available, for plugins which are blocked by a
>>critical bug, we post a status/notes indicating the plugin's
>>situation on our 3rd party CI status wiki, e.g.:
>>
>> Vendor Plugin/Driver Name Contact Name
>> Status Notes
>> My Vendor Name My Plugin CI My Contact Person T
>> Throttled / Partially blocked / Awaiting Intial Commits
>>
>> The status/notes should be clear and understood by the Neutron team.
>>The
>> console logs for change sets where the tests were skipped should also
>>contain a message that all testing is being skipped for that commit.
>>
>> Note that when the DFA initial commits are merged, then this issue
>>would go away for the DFA CI. However, this problem will reappear
>>every time a blocking critical bug shows up for a 3rd party CI setup,
>>or a new plugin is introduced and the hardware-enabling commits are
>>not yet merged. (That is, until we have a solution for the Jenkins
>>limitation).
>>
>> Let me know what you think.
>>
>>
>> Thanks,
>> Dane
>>
>> -----Original Message-----
>> From: Edgar Magana [mailto:edgar.magana at workday.com]
>>
>> Sent: Friday, August 22, 2014 1:57 PM
>> To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
>>for usage questions)
>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>>required to be run
>>
>> Sorry my bad but I just changed.
>>
>> Edgar
>>
>> On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
>>wrote:
>>
>>>Edgar:
>>>
>>>I'm still seeing the comment "Results are not accurate. Needs
>>>clarification..."
>>>
>>>Dane
>>>
>>>-----Original Message-----
>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>>>Sent: Thursday, August 21, 2014 2:58 PM
>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
>>>for usage questions)
>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>>>required to be run
>>>
>>>Dane,
>>>
>>>Wiki has been updated.
>>>
>>>Thanks,
>>>
>>>Edgar
>>>
>>>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
>>>wrote:
>>>
>>>>Edgar:
>>>>
>>>>The status on the wiki page says "Results are not accurate. Needs
>>>>clarification from Cisco".
>>>>Can you please tell me what we are missing?
>>>>
>>>>-Dane
>>>>
>>>>-----Original Message-----
>>>>From: Dane Leblanc (leblancd)
>>>>Sent: Tuesday, August 19, 2014 3:05 PM
>>>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for
>>>>usage
>>>>questions)
>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are
>>>>required to be run
>>>>
>>>>The APIC CI did run tests against that commit (after some queue
>>>>latency):
>>>>
>>>>http://128.107.233.28:8080/job/apic/1860/
>>>>http://cisco-neutron-ci.cisco.com/logs/apic/1860/
>>>>
>>>>But the review comments never showed up on Gerrit. This seems to be
>>>>an intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered
>>>>from this Jenkins/Gerrit server. Whenever we disable another one of
>>>>our other Jenkins jobs (in this case, we disabled DFA for some
>>>>rework), the review comments sometimes stop showing up on Gerrit.
>>>>
>>>>-----Original Message-----
>>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>>>>Sent: Tuesday, August 19, 2014 1:33 PM
>>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
>>>>for usage questions)
>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>>>>required to be run
>>>>
>>>>I was looking to one of the most recent Neutron commits:
>>>>https://review.openstack.org/#/c/115175/
>>>>
>>>>
>>>>I could not find the APIC report.
>>>>
>>>>Edgar
>>>>
>>>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
>>>>wrote:
>>>>
>>>>>From which commit is it missing?
>>>>>https://review.openstack.org/#/c/114629/
>>>>>https://review.openstack.org/#/c/114393/
>>>>>
>>>>>-----Original Message-----
>>>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>>>>>Sent: Tuesday, August 19, 2014 12:28 PM
>>>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List
>>>>>(not for usage questions)
>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>>>>>required to be run
>>>>>
>>>>>Dane,
>>>>>
>>>>>Are you sure about it?
>>>>>I just went to this commit and I could not find the APIC tests.
>>>>>
>>>>>Thanks,
>>>>>
>>>>>Edgar
>>>>>
>>>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
>>>>>wrote:
>>>>>
>>>>>>Edgar:
>>>>>>
>>>>>>The Cisco APIC should be reporting results for both APIC-related
>>>>>>and non-APIC related changes now.
>>>>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/).
>>>>>>
>>>>>>Will you be updating the wiki page?
>>>>>>
>>>>>>-Dane
>>>>>>
>>>>>>-----Original Message-----
>>>>>>From: Dane Leblanc (leblancd)
>>>>>>Sent: Friday, August 15, 2014 8:18 PM
>>>>>>To: OpenStack Development Mailing List (not for usage questions)
>>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests
>>>>>>are required to be run
>>>>>>
>>>>>>Also, you can add me as a contact person for the Cisco VPNaaS driver.
>>>>>>
>>>>>>-----Original Message-----
>>>>>>From: Dane Leblanc (leblancd)
>>>>>>Sent: Friday, August 15, 2014 8:14 PM
>>>>>>To: OpenStack Development Mailing List (not for usage questions)
>>>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests
>>>>>>are required to be run
>>>>>>
>>>>>>Edgar:
>>>>>>
>>>>>>For the Notes for the Cisco APIC, can you change the comment
>>>>>>"results are fake" to something like "results are only valid for
>>>>>>APIC-related commits"? I think this more accurately represents our
>>>>>>current results (for reasons we chatted about on another thread).
>>>>>>
>>>>>>Thanks,
>>>>>>Dane
>>>>>>
>>>>>>-----Original Message-----
>>>>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>>>>>>Sent: Friday, August 15, 2014 6:36 PM
>>>>>>To: OpenStack Development Mailing List (not for usage questions)
>>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests
>>>>>>are required to be run
>>>>>>Importance: High
>>>>>>
>>>>>>Team,
>>>>>>
>>>>>>I did a quick audit on the Neutron CI. Very sad results. Only few
>>>>>>plugins and drivers are running properly and testing all Neutron
>>>>>>commits.
>>>>>>I created a report here:
>>>>>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existi
>>>>>>ng
>>>>>>_
>>>>>>P
>>>>>>l
>>>>>>ugi
>>>>>>n
>>>>>>_and_Drivers
>>>>>>
>>>>>>
>>>>>>We will discuss the actions to take on the next Neutron IRC meeting.
>>>>>>So please, reach me out to clarify what is the status of your CI.
>>>>>>I had two commits to quickly verify the CI reliability:
>>>>>>
>>>>>>https://review.openstack.org/#/c/114393/
>>>>>>
>>>>>>https://review.openstack.org/#/c/40296/
>>>>>>
>>>>>>
>>>>>>I would expect all plugins and drivers passing on the first one
>>>>>>and failing for the second but I got so many surprises.
>>>>>>
>>>>>>Neutron code quality and reliability is a top priority, if you
>>>>>>ignore this report that plugin/driver will be candidate to be
>>>>>>remove from Neutron tree.
>>>>>>
>>>>>>Cheers,
>>>>>>
>>>>>>Edgar
>>>>>>
>>>>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the
>>>>>>dirty job!
>>>>>>
>>>>>>
>>>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <mestery at mestery.com> wrote:
>>>>>>
>>>>>>>Folks, I'm not sure if all CI accounts are running sufficient tests.
>>>>>>>Per the requirements wiki page here [1], everyone needs to be
>>>>>>>running more than just Tempest API tests, which I still see most
>>>>>>>neutron third-party CI setups doing. I'd like to ask everyone who
>>>>>>>operates a third-party CI account for Neutron to please look at
>>>>>>>the link below and make sure you are running appropriate tests.
>>>>>>>If you have questions, the weekly third-party meeting [2] is a
>>>>>>>great place to ask questions.
>>>>>>>
>>>>>>>Thanks,
>>>>>>>Kyle
>>>>>>>
>>>>>>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
>>>>>>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty
>>>>>>>
>>>>>>>_______________________________________________
>>>>>>>OpenStack-dev mailing list
>>>>>>>OpenStack-dev at lists.openstack.org
>>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>
>>>>>>_______________________________________________
>>>>>>OpenStack-dev mailing list
>>>>>>OpenStack-dev at lists.openstack.org
>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>>_______________________________________________
>>>>>>OpenStack-dev mailing list
>>>>>>OpenStack-dev at lists.openstack.org
>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>>
>> --
>>
>> Kevin Benton
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>_______________________________________________
>OpenStack-dev mailing list
>OpenStack-dev at lists.openstack.org
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
_______________________________________________
OpenStack-dev mailing list
OpenStack-dev at lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list