[openstack-dev] [neutron] [third-party] What tests are required to be run
Kevin Benton
blak111 at gmail.com
Tue Aug 26 23:46:29 UTC 2014
Depending on the environment, the rebasing/cherry-picking step adds a
decent amount of complexity to the setup. Without rebasing, you can just
place the gerrit refs directly into a devstack local.conf file and be on
your way. If you want to rebase then you need to have a separate process
that builds the new branch before it's passed to devstack.
This had come up in another email thread[1] when I was trying to get the
refs that upstream OpenStack creates for a patch, but that information
isn't currently available publicly without scraping the logs of the
OpenStack CI Jenkins build.
1. http://lists.openstack.org/pipermail/openstack-dev/2014-July/040795.html
On Tue, Aug 26, 2014 at 3:52 PM, Salvatore Orlando <sorlando at nicira.com>
wrote:
> Hi Karthik,
>
> what do you mean that the plugin is incompatible with
> https://review.openstack.org/#/c/114393/?
> you're mentioning a rebase issue - but the patch in question appears to
> cleanly apply to master.
>
> Is your probably because patch #114393 does not have in its log some
> changes you need to accommodate test_lib changes?
> Are these changes you need already merged? In this case you might try to
> rebase the patch you're going to test on master before running devstack,
> which I think it's also what happens in the upstream gate.
>
> Salvatore
>
>
>
> On 26 August 2014 21:57, Karthik Natarajan <natarajk at brocade.com> wrote:
>
>> Hi Edgar,
>>
>> We are also facing CI issues when the neutron patch set is not rebased
>> with latest changes.
>> For e.g. CI audit that you posted today (
>> https://review.openstack.org/#/c/114393/) is not rebased with neutron
>> test_lib related changes.
>> We had refactored the Brocade Vyatta plugin unit tests to accommodate the
>> test_lib related changes.
>> But our plugin is not compatible with the patch you have posted. So CI is
>> failing.
>>
>> I had a discussion with Dane Leblanc on this. We also need to post the
>> SKIPPED status for such patch sets.
>> We will also experiment with Kevin's suggestion.
>>
>> Thanks,
>> Karthik
>>
>> -----Original Message-----
>> From: Dane Leblanc (leblancd) [mailto:leblancd at cisco.com]
>> Sent: Monday, August 25, 2014 10:02 AM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>> required to be run
>>
>> Edgar, Kyle:
>>
>> Kevin's suggestion should work for me (still hashing out the
>> implementation). I've added an item to the 3rd Party IRC agenda anyway to
>> discuss this corner case.
>>
>> Thanks!
>> Dane
>>
>> -----Original Message-----
>> From: Edgar Magana [mailto:edgar.magana at workday.com]
>> Sent: Monday, August 25, 2014 12:44 PM
>> To: OpenStack Development Mailing List (not for usage questions)
>> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>> required to be run
>>
>> Dane,
>>
>> I will second Kyle's idea. Let's discuss this during today IRC meeting if
>> Kevin's suggestion does not work for you.
>>
>> Thanks,
>>
>> Edgar
>>
>> On 8/25/14, 10:08 AM, "Kyle Mestery" <mestery at mestery.com> wrote:
>>
>> >Dane, thanks for all the great work you're doing in the third-party CI
>> >area. It's great to see you working to share this knowledge with others
>> >as well!
>> >
>> >Did Kevin's idea work for you to move past this issue? If not, I
>> >suggest you put an item on the neutron meeting agenda today and we
>> >cover this there. You could put the item on the third-party meeting
>> >agenda as well.
>> >
>> >Thanks!
>> >Kyle
>> >
>> >On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd)
>> ><leblancd at cisco.com> wrote:
>> >> Hi Kevin:
>> >>
>> >>
>> >>
>> >> Thanks, this is a great idea! I may try just a slight variation of
>> >>this concept. Maybe your idea could be the recommended way to create
>> >>a 3rd party CI for plugins that are just being introduced and need to
>> >>limit the scope of testing to a small set of plugin-related commits
>> >>(or plugins blocked on a certain fix).
>> >>
>> >>
>> >>
>> >> Thanks,
>> >> Dane
>> >>
>> >>
>> >>
>> >> From: Kevin Benton [mailto:blak111 at gmail.com]
>> >> Sent: Saturday, August 23, 2014 5:47 AM
>> >>
>> >>
>> >> To: OpenStack Development Mailing List (not for usage questions)
>> >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>> >>required to be run
>> >>
>> >>
>> >>
>> >> Can you disable posting of results directly from your Jenkins/Zuul
>> >>setup and have a script that just checks the log file for special
>> >>markers to determine if the vote should be FAILED/PASSED/SKIPPED?
>> >>Another advantage of this approach is that it gives you an
>> >>opportunity to detect when a job just failed to setup due to
>> >>infrastructure reasons and trigger a recheck without ever first
>> >>posting a failure to gerrit.
>> >>
>> >>
>> >>
>> >> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd)
>> >> <leblancd at cisco.com> wrote:
>> >>
>> >> Thanks Edgar for updating the APIC status!!!
>> >>
>> >> Edgar and Kyle: *****PLEASE NOTE******!!!! I need your understanding
>> >>and advice on the following:
>> >>
>> >> We are still stuck with a problem stemming from a design limitation
>> >> of Jenkins that prevents us from being compliant with Neutron 3rd
>> >> Party CI requirements for our DFA CI.
>> >>
>> >> The issue is that Jenkins only allows our scripts to
>> >>(programmatically) return either Success or Fail. There is no option
>> >>to return "Aborted", "Not Tested", or "Skipped".
>> >>
>> >> Why does this matter? The DFA plugin is just being introduced, and
>> >>initial DFA-enabling change sets have not yet been merged. Therefore,
>> >>all other change sets will fail our Tempest tests, since they are not
>> >>DFA-enabled.
>> >>
>> >> Similarly, we were recently blocked in our APIC CI with a critical
>> >> bug, causing all change sets without this fix to fail on our APIC
>> testbed.
>> >>
>> >> In these cases, we would like to enter a "throttled" or "partially
>> >>blocked"
>> >> mode, where we would skip testing on change sets we know will fail,
>> >>and (in an ideal world) signal this shortcoming to Gerrit e.g. by
>> >>returning a "Skipped" status. Unfortunately, this option is not
>> >>available in Jenkins scripts, as Jenkins is currently designed. The
>> >>only options we have available is "Success" or all "Fail", which are
>> >>both misleading. We would also incorrectly report success or fail on
>> >>one of the following test
>> >> commits:
>> >> https://review.openstack.org/#/c/114393/
>> >> https://review.openstack.org/#/c/40296/
>> >>
>> >> I've brought this issue up on the openstack-infra IRC, and jeblair
>> >>confirmed the Jenkins limitation, but asked me to get consensus from
>> >>the Neutron community as to this being a problem/requirement. I've
>> >>also sent out an e-mail on the Neutron ML trying to start a
>> >>discussion on this problem (no traction). I plan on bringing this up
>> >>in the 3rd Party CI IRC on Monday, assuming there is time permitted
>> >>in the open discussion.
>> >>
>> >> I'm also investigating
>> >>
>> >> For the short term, I would like to propose the following:
>> >> * We bring this up on the 3rd Party CI IRC on Monday to get a
>> >>solution or workaround, if available. If a solution is available,
>> >>let's consider including that as a hint when we come up with CI
>> >>requirements for handling CIs bocked by some critical fix.
>> >> * I'm also looking into using a REST API to cancel a Jenkins job
>> >>programmatically.
>> >> * If no solution or workaround is available, we work with infra team
>> >>or with Jenkins team to create a solution.
>> >> * Until a solution is available, for plugins which are blocked by a
>> >>critical bug, we post a status/notes indicating the plugin's
>> >>situation on our 3rd party CI status wiki, e.g.:
>> >>
>> >> Vendor Plugin/Driver Name Contact Name
>> >> Status Notes
>> >> My Vendor Name My Plugin CI My Contact Person T
>> >> Throttled / Partially blocked / Awaiting Intial Commits
>> >>
>> >> The status/notes should be clear and understood by the Neutron team.
>> >>The
>> >> console logs for change sets where the tests were skipped should also
>> >>contain a message that all testing is being skipped for that commit.
>> >>
>> >> Note that when the DFA initial commits are merged, then this issue
>> >>would go away for the DFA CI. However, this problem will reappear
>> >>every time a blocking critical bug shows up for a 3rd party CI setup,
>> >>or a new plugin is introduced and the hardware-enabling commits are
>> >>not yet merged. (That is, until we have a solution for the Jenkins
>> >>limitation).
>> >>
>> >> Let me know what you think.
>> >>
>> >>
>> >> Thanks,
>> >> Dane
>> >>
>> >> -----Original Message-----
>> >> From: Edgar Magana [mailto:edgar.magana at workday.com]
>> >>
>> >> Sent: Friday, August 22, 2014 1:57 PM
>> >> To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
>> >>for usage questions)
>> >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>> >>required to be run
>> >>
>> >> Sorry my bad but I just changed.
>> >>
>> >> Edgar
>> >>
>> >> On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
>> >>wrote:
>> >>
>> >>>Edgar:
>> >>>
>> >>>I'm still seeing the comment "Results are not accurate. Needs
>> >>>clarification..."
>> >>>
>> >>>Dane
>> >>>
>> >>>-----Original Message-----
>> >>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>> >>>Sent: Thursday, August 21, 2014 2:58 PM
>> >>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
>> >>>for usage questions)
>> >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>> >>>required to be run
>> >>>
>> >>>Dane,
>> >>>
>> >>>Wiki has been updated.
>> >>>
>> >>>Thanks,
>> >>>
>> >>>Edgar
>> >>>
>> >>>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
>> >>>wrote:
>> >>>
>> >>>>Edgar:
>> >>>>
>> >>>>The status on the wiki page says "Results are not accurate. Needs
>> >>>>clarification from Cisco".
>> >>>>Can you please tell me what we are missing?
>> >>>>
>> >>>>-Dane
>> >>>>
>> >>>>-----Original Message-----
>> >>>>From: Dane Leblanc (leblancd)
>> >>>>Sent: Tuesday, August 19, 2014 3:05 PM
>> >>>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for
>> >>>>usage
>> >>>>questions)
>> >>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are
>> >>>>required to be run
>> >>>>
>> >>>>The APIC CI did run tests against that commit (after some queue
>> >>>>latency):
>> >>>>
>> >>>>http://128.107.233.28:8080/job/apic/1860/
>> >>>>http://cisco-neutron-ci.cisco.com/logs/apic/1860/
>> >>>>
>> >>>>But the review comments never showed up on Gerrit. This seems to be
>> >>>>an intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered
>> >>>>from this Jenkins/Gerrit server. Whenever we disable another one of
>> >>>>our other Jenkins jobs (in this case, we disabled DFA for some
>> >>>>rework), the review comments sometimes stop showing up on Gerrit.
>> >>>>
>> >>>>-----Original Message-----
>> >>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>> >>>>Sent: Tuesday, August 19, 2014 1:33 PM
>> >>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
>> >>>>for usage questions)
>> >>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>> >>>>required to be run
>> >>>>
>> >>>>I was looking to one of the most recent Neutron commits:
>> >>>>https://review.openstack.org/#/c/115175/
>> >>>>
>> >>>>
>> >>>>I could not find the APIC report.
>> >>>>
>> >>>>Edgar
>> >>>>
>> >>>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
>> >>>>wrote:
>> >>>>
>> >>>>>From which commit is it missing?
>> >>>>>https://review.openstack.org/#/c/114629/
>> >>>>>https://review.openstack.org/#/c/114393/
>> >>>>>
>> >>>>>-----Original Message-----
>> >>>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>> >>>>>Sent: Tuesday, August 19, 2014 12:28 PM
>> >>>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List
>> >>>>>(not for usage questions)
>> >>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
>> >>>>>required to be run
>> >>>>>
>> >>>>>Dane,
>> >>>>>
>> >>>>>Are you sure about it?
>> >>>>>I just went to this commit and I could not find the APIC tests.
>> >>>>>
>> >>>>>Thanks,
>> >>>>>
>> >>>>>Edgar
>> >>>>>
>> >>>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
>> >>>>>wrote:
>> >>>>>
>> >>>>>>Edgar:
>> >>>>>>
>> >>>>>>The Cisco APIC should be reporting results for both APIC-related
>> >>>>>>and non-APIC related changes now.
>> >>>>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/).
>> >>>>>>
>> >>>>>>Will you be updating the wiki page?
>> >>>>>>
>> >>>>>>-Dane
>> >>>>>>
>> >>>>>>-----Original Message-----
>> >>>>>>From: Dane Leblanc (leblancd)
>> >>>>>>Sent: Friday, August 15, 2014 8:18 PM
>> >>>>>>To: OpenStack Development Mailing List (not for usage questions)
>> >>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests
>> >>>>>>are required to be run
>> >>>>>>
>> >>>>>>Also, you can add me as a contact person for the Cisco VPNaaS
>> driver.
>> >>>>>>
>> >>>>>>-----Original Message-----
>> >>>>>>From: Dane Leblanc (leblancd)
>> >>>>>>Sent: Friday, August 15, 2014 8:14 PM
>> >>>>>>To: OpenStack Development Mailing List (not for usage questions)
>> >>>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests
>> >>>>>>are required to be run
>> >>>>>>
>> >>>>>>Edgar:
>> >>>>>>
>> >>>>>>For the Notes for the Cisco APIC, can you change the comment
>> >>>>>>"results are fake" to something like "results are only valid for
>> >>>>>>APIC-related commits"? I think this more accurately represents our
>> >>>>>>current results (for reasons we chatted about on another thread).
>> >>>>>>
>> >>>>>>Thanks,
>> >>>>>>Dane
>> >>>>>>
>> >>>>>>-----Original Message-----
>> >>>>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
>> >>>>>>Sent: Friday, August 15, 2014 6:36 PM
>> >>>>>>To: OpenStack Development Mailing List (not for usage questions)
>> >>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests
>> >>>>>>are required to be run
>> >>>>>>Importance: High
>> >>>>>>
>> >>>>>>Team,
>> >>>>>>
>> >>>>>>I did a quick audit on the Neutron CI. Very sad results. Only few
>> >>>>>>plugins and drivers are running properly and testing all Neutron
>> >>>>>>commits.
>> >>>>>>I created a report here:
>> >>>>>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existi
>> >>>>>>ng
>> >>>>>>_
>> >>>>>>P
>> >>>>>>l
>> >>>>>>ugi
>> >>>>>>n
>> >>>>>>_and_Drivers
>> >>>>>>
>> >>>>>>
>> >>>>>>We will discuss the actions to take on the next Neutron IRC meeting.
>> >>>>>>So please, reach me out to clarify what is the status of your CI.
>> >>>>>>I had two commits to quickly verify the CI reliability:
>> >>>>>>
>> >>>>>>https://review.openstack.org/#/c/114393/
>> >>>>>>
>> >>>>>>https://review.openstack.org/#/c/40296/
>> >>>>>>
>> >>>>>>
>> >>>>>>I would expect all plugins and drivers passing on the first one
>> >>>>>>and failing for the second but I got so many surprises.
>> >>>>>>
>> >>>>>>Neutron code quality and reliability is a top priority, if you
>> >>>>>>ignore this report that plugin/driver will be candidate to be
>> >>>>>>remove from Neutron tree.
>> >>>>>>
>> >>>>>>Cheers,
>> >>>>>>
>> >>>>>>Edgar
>> >>>>>>
>> >>>>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the
>> >>>>>>dirty job!
>> >>>>>>
>> >>>>>>
>> >>>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <mestery at mestery.com> wrote:
>> >>>>>>
>> >>>>>>>Folks, I'm not sure if all CI accounts are running sufficient
>> tests.
>> >>>>>>>Per the requirements wiki page here [1], everyone needs to be
>> >>>>>>>running more than just Tempest API tests, which I still see most
>> >>>>>>>neutron third-party CI setups doing. I'd like to ask everyone who
>> >>>>>>>operates a third-party CI account for Neutron to please look at
>> >>>>>>>the link below and make sure you are running appropriate tests.
>> >>>>>>>If you have questions, the weekly third-party meeting [2] is a
>> >>>>>>>great place to ask questions.
>> >>>>>>>
>> >>>>>>>Thanks,
>> >>>>>>>Kyle
>> >>>>>>>
>> >>>>>>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
>> >>>>>>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty
>> >>>>>>>
>> >>>>>>>_______________________________________________
>> >>>>>>>OpenStack-dev mailing list
>> >>>>>>>OpenStack-dev at lists.openstack.org
>> >>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>>
>> >>>>>>
>> >>>>>>_______________________________________________
>> >>>>>>OpenStack-dev mailing list
>> >>>>>>OpenStack-dev at lists.openstack.org
>> >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>>
>> >>>>>>_______________________________________________
>> >>>>>>OpenStack-dev mailing list
>> >>>>>>OpenStack-dev at lists.openstack.org
>> >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>>>>
>> >>>>
>> >>>
>> >>
>> >>
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> --
>> >>
>> >> Kevin Benton
>> >>
>> >>
>> >> _______________________________________________
>> >> OpenStack-dev mailing list
>> >> OpenStack-dev at lists.openstack.org
>> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> >>
>> >
>> >_______________________________________________
>> >OpenStack-dev mailing list
>> >OpenStack-dev at lists.openstack.org
>> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
--
Kevin Benton
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140826/bbc5267c/attachment.html>
More information about the OpenStack-dev
mailing list