Open Stack

Can you disable posting of results directly from your Jenkins/Zuul setup
and have a script that just checks the log file for special markers to
determine if the vote should be FAILED/PASSED/SKIPPED? Another advantage of
this approach is that it gives you an opportunity to detect when a job just
failed to setup due to infrastructure reasons and trigger a recheck without
ever first posting a failure to gerrit.


On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd) <leblancd at cisco.com
> wrote:

> Thanks Edgar for updating the APIC status!!!
>
> Edgar and Kyle: *****PLEASE NOTE******!!!!  I need your understanding and
> advice on the following:
>
> We are still stuck with a problem stemming from a design limitation of
> Jenkins that prevents us from being compliant with Neutron 3rd Party CI
> requirements for our DFA CI.
>
> The issue is that Jenkins only allows our scripts to (programmatically)
> return either Success or Fail. There is no option to return "Aborted", "Not
> Tested", or "Skipped".
>
> Why does this matter? The DFA plugin is just being introduced, and initial
> DFA-enabling change sets have not yet been merged. Therefore, all other
> change sets will fail our Tempest tests, since they are not DFA-enabled.
>
> Similarly, we were recently blocked in our APIC CI with a critical bug,
> causing all change sets without this fix to fail on our APIC testbed.
>
> In these cases, we would like to enter a "throttled" or "partially
> blocked" mode, where we would skip testing on change sets we know will
> fail, and (in an ideal world) signal this shortcoming to Gerrit e.g. by
> returning a "Skipped" status. Unfortunately, this option is not available
> in Jenkins scripts, as Jenkins is currently designed. The only options we
> have available is "Success" or all "Fail", which are both misleading. We
> would also incorrectly report success or fail on one of the following test
> commits:
> https://review.openstack.org/#/c/114393/
> https://review.openstack.org/#/c/40296/
>
> I've brought this issue up on the openstack-infra IRC, and jeblair
> confirmed the Jenkins limitation, but asked me to get consensus from the
> Neutron community as to this being a problem/requirement. I've also sent
> out an e-mail on the Neutron ML trying to start a discussion on this
> problem (no traction). I plan on bringing this up in the 3rd Party CI IRC
> on Monday, assuming there is time permitted in the open discussion.
>
> I'm also investigating
>
> For the short term, I would like to propose the following:
> * We bring this up on the 3rd Party CI IRC on Monday to get a solution or
> workaround, if available. If a solution is available, let's consider
> including that as a hint when we come up with CI requirements for handling
> CIs bocked by some critical fix.
> * I'm also looking into using a REST API to cancel a Jenkins job
> programmatically.
> * If no solution or workaround is available, we work with infra team or
> with Jenkins team to create a solution.
> * Until a solution is available, for plugins which are blocked by a
> critical bug, we post a status/notes indicating the plugin's situation on
> our 3rd party CI status wiki, e.g.:
>
> Vendor                  Plugin/Driver Name      Contact Name
> Status  Notes
> My Vendor Name  My Plugin CI            My Contact Person       T
>  Throttled / Partially blocked / Awaiting Intial Commits
>
> The status/notes should be clear and understood by the Neutron team.  The
> console logs for change sets where the tests were skipped should also
> contain a message that all testing is being skipped for that commit.
>
> Note that when the DFA initial commits are merged, then this issue would
> go away for the DFA CI. However, this problem will reappear every time a
> blocking critical bug shows up for a 3rd party CI setup, or a new plugin is
> introduced and the hardware-enabling commits are not yet merged.  (That is,
> until we have a solution for the Jenkins limitation).
>
> Let me know what you think.
>
> Thanks,
> Dane
>
> -----Original Message-----
> From: Edgar Magana [mailto:edgar.magana at workday.com]
> Sent: Friday, August 22, 2014 1:57 PM
> To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not for
> usage questions)
> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> required to be run
>
> Sorry my bad but I just changed.
>
> Edgar
>
> On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <leblancd at cisco.com> wrote:
>
> >Edgar:
> >
> >I'm still seeing the comment "Results are not accurate. Needs
> >clarification..."
> >
> >Dane
> >
> >-----Original Message-----
> >From: Edgar Magana [mailto:edgar.magana at workday.com]
> >Sent: Thursday, August 21, 2014 2:58 PM
> >To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
> >for usage questions)
> >Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> >required to be run
> >
> >Dane,
> >
> >Wiki has been updated.
> >
> >Thanks,
> >
> >Edgar
> >
> >On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
> wrote:
> >
> >>Edgar:
> >>
> >>The status on the wiki page says "Results are not accurate. Needs
> >>clarification from Cisco".
> >>Can you please tell me what we are missing?
> >>
> >>-Dane
> >>
> >>-----Original Message-----
> >>From: Dane Leblanc (leblancd)
> >>Sent: Tuesday, August 19, 2014 3:05 PM
> >>To: 'Edgar Magana'; OpenStack Development Mailing List (not for usage
> >>questions)
> >>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are
> >>required to be run
> >>
> >>The APIC CI did run tests against that commit (after some queue latency):
> >>
> >>http://128.107.233.28:8080/job/apic/1860/
> >>http://cisco-neutron-ci.cisco.com/logs/apic/1860/
> >>
> >>But the review comments never showed up on Gerrit. This seems to be an
> >>intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered from
> >>this Jenkins/Gerrit server. Whenever we disable another one of our
> >>other Jenkins jobs (in this case, we disabled DFA for some rework),
> >>the review comments sometimes stop showing up on Gerrit.
> >>
> >>-----Original Message-----
> >>From: Edgar Magana [mailto:edgar.magana at workday.com]
> >>Sent: Tuesday, August 19, 2014 1:33 PM
> >>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
> >>for usage questions)
> >>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> >>required to be run
> >>
> >>I was looking to one of the most recent Neutron commits:
> >>https://review.openstack.org/#/c/115175/
> >>
> >>
> >>I could not find the APIC report.
> >>
> >>Edgar
> >>
> >>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
> >>wrote:
> >>
> >>>From which commit is it missing?
> >>>https://review.openstack.org/#/c/114629/
> >>>https://review.openstack.org/#/c/114393/
> >>>
> >>>-----Original Message-----
> >>>From: Edgar Magana [mailto:edgar.magana at workday.com]
> >>>Sent: Tuesday, August 19, 2014 12:28 PM
> >>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
> >>>for usage questions)
> >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> >>>required to be run
> >>>
> >>>Dane,
> >>>
> >>>Are you sure about it?
> >>>I just went to this commit and I could not find the APIC tests.
> >>>
> >>>Thanks,
> >>>
> >>>Edgar
> >>>
> >>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
> >>>wrote:
> >>>
> >>>>Edgar:
> >>>>
> >>>>The Cisco APIC should be reporting results for both APIC-related and
> >>>>non-APIC related changes now.
> >>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/).
> >>>>
> >>>>Will you be updating the wiki page?
> >>>>
> >>>>-Dane
> >>>>
> >>>>-----Original Message-----
> >>>>From: Dane Leblanc (leblancd)
> >>>>Sent: Friday, August 15, 2014 8:18 PM
> >>>>To: OpenStack Development Mailing List (not for usage questions)
> >>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> >>>>required to be run
> >>>>
> >>>>Also, you can add me as a contact person for the Cisco VPNaaS driver.
> >>>>
> >>>>-----Original Message-----
> >>>>From: Dane Leblanc (leblancd)
> >>>>Sent: Friday, August 15, 2014 8:14 PM
> >>>>To: OpenStack Development Mailing List (not for usage questions)
> >>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are
> >>>>required to be run
> >>>>
> >>>>Edgar:
> >>>>
> >>>>For the Notes for the Cisco APIC, can you change the comment
> >>>>"results are fake" to something like "results are only valid for
> >>>>APIC-related commits"? I think this more accurately represents our
> >>>>current results (for reasons we chatted about on another thread).
> >>>>
> >>>>Thanks,
> >>>>Dane
> >>>>
> >>>>-----Original Message-----
> >>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
> >>>>Sent: Friday, August 15, 2014 6:36 PM
> >>>>To: OpenStack Development Mailing List (not for usage questions)
> >>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> >>>>required to be run
> >>>>Importance: High
> >>>>
> >>>>Team,
> >>>>
> >>>>I did a quick audit on the Neutron CI. Very sad results. Only few
> >>>>plugins and drivers are running properly and testing all Neutron
> >>>>commits.
> >>>>I created a report here:
> >>>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existing
> >>>>_
> >>>>P
> >>>>l
> >>>>ugi
> >>>>n
> >>>>_and_Drivers
> >>>>
> >>>>
> >>>>We will discuss the actions to take on the next Neutron IRC meeting.
> >>>>So please, reach me out to clarify what is the status of your CI.
> >>>>I had two commits to quickly verify the CI reliability:
> >>>>
> >>>>https://review.openstack.org/#/c/114393/
> >>>>
> >>>>https://review.openstack.org/#/c/40296/
> >>>>
> >>>>
> >>>>I would expect all plugins and drivers passing on the first one and
> >>>>failing for the second but I got so many surprises.
> >>>>
> >>>>Neutron code quality and reliability is a top priority, if you
> >>>>ignore this report that plugin/driver will be candidate to be remove
> >>>>from Neutron tree.
> >>>>
> >>>>Cheers,
> >>>>
> >>>>Edgar
> >>>>
> >>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the
> >>>>dirty job!
> >>>>
> >>>>
> >>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <mestery at mestery.com> wrote:
> >>>>
> >>>>>Folks, I'm not sure if all CI accounts are running sufficient tests.
> >>>>>Per the requirements wiki page here [1], everyone needs to be
> >>>>>running more than just Tempest API tests, which I still see most
> >>>>>neutron third-party CI setups doing. I'd like to ask everyone who
> >>>>>operates a third-party CI account for Neutron to please look at the
> >>>>>link below and make sure you are running appropriate tests. If you
> >>>>>have questions, the weekly third-party meeting [2] is a great place
> >>>>>to ask questions.
> >>>>>
> >>>>>Thanks,
> >>>>>Kyle
> >>>>>
> >>>>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
> >>>>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty
> >>>>>
> >>>>>_______________________________________________
> >>>>>OpenStack-dev mailing list
> >>>>>OpenStack-dev at lists.openstack.org
> >>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>
> >>>>
> >>>>_______________________________________________
> >>>>OpenStack-dev mailing list
> >>>>OpenStack-dev at lists.openstack.org
> >>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>
> >>>>_______________________________________________
> >>>>OpenStack-dev mailing list
> >>>>OpenStack-dev at lists.openstack.org
> >>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>
> >>
> >
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140823/a8983ec7/attachment.html>