Open Stack

Hi Karthik,

what do you mean that the plugin is incompatible with
https://review.openstack.org/#/c/114393/?
you're mentioning a rebase issue - but the patch in question appears to
cleanly apply to master.

Is your probably because patch #114393 does not have in its log some
changes you need to accommodate test_lib changes?
Are these changes you need already merged? In this case you might try to
rebase the patch you're going to test on master before running devstack,
which I think it's also what happens in the upstream gate.

Salvatore



On 26 August 2014 21:57, Karthik Natarajan <natarajk at brocade.com> wrote:

> Hi Edgar,
>
> We are also facing CI issues when the neutron patch set is not rebased
> with latest changes.
> For e.g. CI audit that you posted today (
> https://review.openstack.org/#/c/114393/) is not rebased with neutron
> test_lib related changes.
> We had refactored the Brocade Vyatta plugin unit tests to accommodate the
> test_lib related changes.
> But our plugin is not compatible with the patch you have posted. So CI is
> failing.
>
> I had a discussion with Dane Leblanc on this. We also need to post the
> SKIPPED status for such patch sets.
> We will also experiment with Kevin's suggestion.
>
> Thanks,
> Karthik
>
> -----Original Message-----
> From: Dane Leblanc (leblancd) [mailto:leblancd at cisco.com]
> Sent: Monday, August 25, 2014 10:02 AM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> required to be run
>
> Edgar, Kyle:
>
> Kevin's suggestion should work for me (still hashing out the
> implementation).  I've added an item to the 3rd Party IRC agenda anyway to
> discuss this corner case.
>
> Thanks!
> Dane
>
> -----Original Message-----
> From: Edgar Magana [mailto:edgar.magana at workday.com]
> Sent: Monday, August 25, 2014 12:44 PM
> To: OpenStack Development Mailing List (not for usage questions)
> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> required to be run
>
> Dane,
>
> I will second Kyle's idea. Let's discuss this during today IRC meeting if
> Kevin's suggestion does not work for you.
>
> Thanks,
>
> Edgar
>
> On 8/25/14, 10:08 AM, "Kyle Mestery" <mestery at mestery.com> wrote:
>
> >Dane, thanks for all the great work you're doing in the third-party CI
> >area. It's great to see you working to share this knowledge with others
> >as well!
> >
> >Did Kevin's idea work for you to move past this issue? If not, I
> >suggest you put an item on the neutron meeting agenda today and we
> >cover this there. You could put the item on the third-party meeting
> >agenda as well.
> >
> >Thanks!
> >Kyle
> >
> >On Sun, Aug 24, 2014 at 9:43 AM, Dane Leblanc (leblancd)
> ><leblancd at cisco.com> wrote:
> >> Hi Kevin:
> >>
> >>
> >>
> >> Thanks, this is a great idea! I may try just a slight variation of
> >>this  concept. Maybe your idea could be the recommended way to create
> >>a 3rd party  CI for plugins that are just being introduced and need to
> >>limit the scope of  testing to a small set of plugin-related commits
> >>(or plugins blocked on a  certain fix).
> >>
> >>
> >>
> >> Thanks,
> >> Dane
> >>
> >>
> >>
> >> From: Kevin Benton [mailto:blak111 at gmail.com]
> >> Sent: Saturday, August 23, 2014 5:47 AM
> >>
> >>
> >> To: OpenStack Development Mailing List (not for usage questions)
> >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> >>required  to be run
> >>
> >>
> >>
> >> Can you disable posting of results directly from your Jenkins/Zuul
> >>setup and  have a script that just checks the log file for special
> >>markers to determine  if the vote should be FAILED/PASSED/SKIPPED?
> >>Another advantage of this  approach is that it gives you an
> >>opportunity to detect when a job just  failed to setup due to
> >>infrastructure reasons and trigger a recheck without  ever first
> >>posting a failure to gerrit.
> >>
> >>
> >>
> >> On Fri, Aug 22, 2014 at 3:06 PM, Dane Leblanc (leblancd)
> >> <leblancd at cisco.com> wrote:
> >>
> >> Thanks Edgar for updating the APIC status!!!
> >>
> >> Edgar and Kyle: *****PLEASE NOTE******!!!!  I need your understanding
> >>and  advice on the following:
> >>
> >> We are still stuck with a problem stemming from a design limitation
> >> of Jenkins that prevents us from being compliant with Neutron 3rd
> >> Party CI requirements for our DFA CI.
> >>
> >> The issue is that Jenkins only allows our scripts to
> >>(programmatically)  return either Success or Fail. There is no option
> >>to return "Aborted", "Not  Tested", or "Skipped".
> >>
> >> Why does this matter? The DFA plugin is just being introduced, and
> >>initial  DFA-enabling change sets have not yet been merged. Therefore,
> >>all other  change sets will fail our Tempest tests, since they are not
> >>DFA-enabled.
> >>
> >> Similarly, we were recently blocked in our APIC CI with a critical
> >> bug, causing all change sets without this fix to fail on our APIC
> testbed.
> >>
> >> In these cases, we would like to enter a "throttled" or "partially
> >>blocked"
> >> mode, where we would skip testing on change sets we know will fail,
> >>and (in  an ideal world) signal this shortcoming to Gerrit e.g. by
> >>returning a  "Skipped" status. Unfortunately, this option is not
> >>available in Jenkins  scripts, as Jenkins is currently designed. The
> >>only options we have  available is "Success" or all "Fail", which are
> >>both misleading. We would  also incorrectly report success or fail on
> >>one of the following test
> >> commits:
> >> https://review.openstack.org/#/c/114393/
> >> https://review.openstack.org/#/c/40296/
> >>
> >> I've brought this issue up on the openstack-infra IRC, and jeblair
> >>confirmed  the Jenkins limitation, but asked me to get consensus from
> >>the Neutron  community as to this being a problem/requirement. I've
> >>also sent out an  e-mail on the Neutron ML trying to start a
> >>discussion on this problem (no  traction). I plan on bringing this up
> >>in the 3rd Party CI IRC on Monday,  assuming there is time permitted
> >>in the open discussion.
> >>
> >> I'm also investigating
> >>
> >> For the short term, I would like to propose the following:
> >> * We bring this up on the 3rd Party CI IRC on Monday to get a
> >>solution or  workaround, if available. If a solution is available,
> >>let's consider  including that as a hint when we come up with CI
> >>requirements for handling  CIs bocked by some critical fix.
> >> * I'm also looking into using a REST API to cancel a Jenkins job
> >>programmatically.
> >> * If no solution or workaround is available, we work with infra team
> >>or with  Jenkins team to create a solution.
> >> * Until a solution is available, for plugins which are blocked by a
> >>critical  bug, we post a status/notes indicating the plugin's
> >>situation on our 3rd  party CI status wiki, e.g.:
> >>
> >> Vendor                  Plugin/Driver Name      Contact Name
> >> Status  Notes
> >> My Vendor Name  My Plugin CI            My Contact Person       T
> >> Throttled / Partially blocked / Awaiting Intial Commits
> >>
> >> The status/notes should be clear and understood by the Neutron team.
> >>The
> >> console logs for change sets where the tests were skipped should also
> >>contain a message that all testing is being skipped for that commit.
> >>
> >> Note that when the DFA initial commits are merged, then this issue
> >>would go  away for the DFA CI. However, this problem will reappear
> >>every time a  blocking critical bug shows up for a 3rd party CI setup,
> >>or a new plugin is  introduced and the hardware-enabling commits are
> >>not yet merged.  (That is,  until we have a solution for the Jenkins
> >>limitation).
> >>
> >> Let me know what you think.
> >>
> >>
> >> Thanks,
> >> Dane
> >>
> >> -----Original Message-----
> >> From: Edgar Magana [mailto:edgar.magana at workday.com]
> >>
> >> Sent: Friday, August 22, 2014 1:57 PM
> >> To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
> >>for  usage questions)
> >> Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> >>required  to be run
> >>
> >> Sorry my bad but I just changed.
> >>
> >> Edgar
> >>
> >> On 8/21/14, 2:13 PM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
> >>wrote:
> >>
> >>>Edgar:
> >>>
> >>>I'm still seeing the comment "Results are not accurate. Needs
> >>>clarification..."
> >>>
> >>>Dane
> >>>
> >>>-----Original Message-----
> >>>From: Edgar Magana [mailto:edgar.magana at workday.com]
> >>>Sent: Thursday, August 21, 2014 2:58 PM
> >>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
> >>>for usage questions)
> >>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> >>>required to be run
> >>>
> >>>Dane,
> >>>
> >>>Wiki has been updated.
> >>>
> >>>Thanks,
> >>>
> >>>Edgar
> >>>
> >>>On 8/21/14, 7:57 AM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
> >>>wrote:
> >>>
> >>>>Edgar:
> >>>>
> >>>>The status on the wiki page says "Results are not accurate. Needs
> >>>>clarification from Cisco".
> >>>>Can you please tell me what we are missing?
> >>>>
> >>>>-Dane
> >>>>
> >>>>-----Original Message-----
> >>>>From: Dane Leblanc (leblancd)
> >>>>Sent: Tuesday, August 19, 2014 3:05 PM
> >>>>To: 'Edgar Magana'; OpenStack Development Mailing List (not for
> >>>>usage
> >>>>questions)
> >>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests are
> >>>>required to be run
> >>>>
> >>>>The APIC CI did run tests against that commit (after some queue
> >>>>latency):
> >>>>
> >>>>http://128.107.233.28:8080/job/apic/1860/
> >>>>http://cisco-neutron-ci.cisco.com/logs/apic/1860/
> >>>>
> >>>>But the review comments never showed up on Gerrit. This seems to be
> >>>>an intermittent quirk of Jenkins/Gerrit: We have 3 CIs triggered
> >>>>from this Jenkins/Gerrit server. Whenever we disable another one of
> >>>>our other Jenkins jobs (in this case, we disabled DFA for some
> >>>>rework), the review comments sometimes stop showing up on Gerrit.
> >>>>
> >>>>-----Original Message-----
> >>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
> >>>>Sent: Tuesday, August 19, 2014 1:33 PM
> >>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List (not
> >>>>for usage questions)
> >>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> >>>>required to be run
> >>>>
> >>>>I was looking to one of the most recent Neutron commits:
> >>>>https://review.openstack.org/#/c/115175/
> >>>>
> >>>>
> >>>>I could not find the APIC report.
> >>>>
> >>>>Edgar
> >>>>
> >>>>On 8/19/14, 9:48 AM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
> >>>>wrote:
> >>>>
> >>>>>From which commit is it missing?
> >>>>>https://review.openstack.org/#/c/114629/
> >>>>>https://review.openstack.org/#/c/114393/
> >>>>>
> >>>>>-----Original Message-----
> >>>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
> >>>>>Sent: Tuesday, August 19, 2014 12:28 PM
> >>>>>To: Dane Leblanc (leblancd); OpenStack Development Mailing List
> >>>>>(not for usage questions)
> >>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests are
> >>>>>required to be run
> >>>>>
> >>>>>Dane,
> >>>>>
> >>>>>Are you sure about it?
> >>>>>I just went to this commit and I could not find the APIC tests.
> >>>>>
> >>>>>Thanks,
> >>>>>
> >>>>>Edgar
> >>>>>
> >>>>>On 8/17/14, 8:47 PM, "Dane Leblanc (leblancd)" <leblancd at cisco.com>
> >>>>>wrote:
> >>>>>
> >>>>>>Edgar:
> >>>>>>
> >>>>>>The Cisco APIC should be reporting results for both APIC-related
> >>>>>>and non-APIC related changes now.
> >>>>>>(See http://cisco-neutron-ci.cisco.com/logs/apic/1738/).
> >>>>>>
> >>>>>>Will you be updating the wiki page?
> >>>>>>
> >>>>>>-Dane
> >>>>>>
> >>>>>>-----Original Message-----
> >>>>>>From: Dane Leblanc (leblancd)
> >>>>>>Sent: Friday, August 15, 2014 8:18 PM
> >>>>>>To: OpenStack Development Mailing List (not for usage questions)
> >>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests
> >>>>>>are required to be run
> >>>>>>
> >>>>>>Also, you can add me as a contact person for the Cisco VPNaaS driver.
> >>>>>>
> >>>>>>-----Original Message-----
> >>>>>>From: Dane Leblanc (leblancd)
> >>>>>>Sent: Friday, August 15, 2014 8:14 PM
> >>>>>>To: OpenStack Development Mailing List (not for usage questions)
> >>>>>>Subject: RE: [openstack-dev] [neutron] [third-party] What tests
> >>>>>>are required to be run
> >>>>>>
> >>>>>>Edgar:
> >>>>>>
> >>>>>>For the Notes for the Cisco APIC, can you change the comment
> >>>>>>"results are fake" to something like "results are only valid for
> >>>>>>APIC-related commits"? I think this more accurately represents our
> >>>>>>current results (for reasons we chatted about on another thread).
> >>>>>>
> >>>>>>Thanks,
> >>>>>>Dane
> >>>>>>
> >>>>>>-----Original Message-----
> >>>>>>From: Edgar Magana [mailto:edgar.magana at workday.com]
> >>>>>>Sent: Friday, August 15, 2014 6:36 PM
> >>>>>>To: OpenStack Development Mailing List (not for usage questions)
> >>>>>>Subject: Re: [openstack-dev] [neutron] [third-party] What tests
> >>>>>>are required to be run
> >>>>>>Importance: High
> >>>>>>
> >>>>>>Team,
> >>>>>>
> >>>>>>I did a quick audit on the Neutron CI. Very sad results. Only few
> >>>>>>plugins and drivers are running properly and testing all Neutron
> >>>>>>commits.
> >>>>>>I created a report here:
> >>>>>>https://wiki.openstack.org/wiki/Neutron_Plugins_and_Drivers#Existi
> >>>>>>ng
> >>>>>>_
> >>>>>>P
> >>>>>>l
> >>>>>>ugi
> >>>>>>n
> >>>>>>_and_Drivers
> >>>>>>
> >>>>>>
> >>>>>>We will discuss the actions to take on the next Neutron IRC meeting.
> >>>>>>So please, reach me out to clarify what is the status of your CI.
> >>>>>>I had two commits to quickly verify the CI reliability:
> >>>>>>
> >>>>>>https://review.openstack.org/#/c/114393/
> >>>>>>
> >>>>>>https://review.openstack.org/#/c/40296/
> >>>>>>
> >>>>>>
> >>>>>>I would expect all plugins and drivers passing on the first one
> >>>>>>and failing for the second but I got so many surprises.
> >>>>>>
> >>>>>>Neutron code quality and reliability is a top priority, if you
> >>>>>>ignore this report that plugin/driver will be candidate to be
> >>>>>>remove from Neutron tree.
> >>>>>>
> >>>>>>Cheers,
> >>>>>>
> >>>>>>Edgar
> >>>>>>
> >>>>>>P.s. I hate to be the inquisitor hereŠ but someone has to do the
> >>>>>>dirty job!
> >>>>>>
> >>>>>>
> >>>>>>On 8/14/14, 8:30 AM, "Kyle Mestery" <mestery at mestery.com> wrote:
> >>>>>>
> >>>>>>>Folks, I'm not sure if all CI accounts are running sufficient tests.
> >>>>>>>Per the requirements wiki page here [1], everyone needs to be
> >>>>>>>running more than just Tempest API tests, which I still see most
> >>>>>>>neutron third-party CI setups doing. I'd like to ask everyone who
> >>>>>>>operates a third-party CI account for Neutron to please look at
> >>>>>>>the link below and make sure you are running appropriate tests.
> >>>>>>>If you have questions, the weekly third-party meeting [2] is a
> >>>>>>>great place to ask questions.
> >>>>>>>
> >>>>>>>Thanks,
> >>>>>>>Kyle
> >>>>>>>
> >>>>>>>[1] https://wiki.openstack.org/wiki/NeutronThirdPartyTesting
> >>>>>>>[2] https://wiki.openstack.org/wiki/Meetings/ThirdParty
> >>>>>>>
> >>>>>>>_______________________________________________
> >>>>>>>OpenStack-dev mailing list
> >>>>>>>OpenStack-dev at lists.openstack.org
> >>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>
> >>>>>>
> >>>>>>_______________________________________________
> >>>>>>OpenStack-dev mailing list
> >>>>>>OpenStack-dev at lists.openstack.org
> >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>>
> >>>>>>_______________________________________________
> >>>>>>OpenStack-dev mailing list
> >>>>>>OpenStack-dev at lists.openstack.org
> >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>
> >>>
> >>
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >>
> >>
> >>
> >>
> >> --
> >>
> >> Kevin Benton
> >>
> >>
> >> _______________________________________________
> >> OpenStack-dev mailing list
> >> OpenStack-dev at lists.openstack.org
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>
> >
> >_______________________________________________
> >OpenStack-dev mailing list
> >OpenStack-dev at lists.openstack.org
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140827/6a3dcfda/attachment.html>