[openstack-dev] [nova][infra] The same SRIOV / NFV CI failures missed a regression, why?
Anita Kuno
anteaya at anteaya.info
Tue Mar 29 14:33:53 UTC 2016
On 03/29/2016 08:56 AM, Znoinski, Waldemar wrote:
> Hi Jay, Matt, Levi et al,
>
> The Intel NFV CI problems were fixed last week.
> Also, I think if the CIs would have voting right it would attract more attention.
>
> I'd be happy to work with OpenStack Infra team on getting our NFV CI the voting level/rights.
Individual projects decide for themselves who, if anyone, may vote +1
verified or -1 verified on their repos. infra has set up the structure
for teams to do this and is not involved in the decision of who may vote.
> To do it right, though, it'd be great to have some form of SLA.
Any agreement you create is between you and an individual project, in
this case Nova.
> This would help maintaining only the 3rdparty CIs that meet the SLA voting. Is there one or discussed? I'd imagine there would be a need for one anyways if we'd go with Jay's proposal of collocated hardware/engineers.
>
> Waldek
>
> >-----Original Message-----
> >From: Moshe Levi [mailto:moshele at mellanox.com]
> >Sent: Tuesday, March 29, 2016 8:41 AM
> >To: OpenStack Development Mailing List (not for usage questions)
> ><openstack-dev at lists.openstack.org>
> >Subject: Re: [openstack-dev] [nova] The same SRIOV / NFV CI failures missed
> >a regression, why?
> >
> >
> >
> >> -----Original Message-----
> >> From: Jay Pipes [mailto:jaypipes at gmail.com]
> >> Sent: Friday, March 25, 2016 10:20 PM
> >> To: openstack-dev at lists.openstack.org
> >> Subject: Re: [openstack-dev] [nova] The same SRIOV / NFV CI failures
> >> missed a regression, why?
> >>
> >> On 03/24/2016 09:35 AM, Matt Riedemann wrote:
> >> > We have another mitaka-rc-potential bug [1] due to a regression when
> >> > detaching SR-IOV interfaces in the libvirt driver.
> >> >
> >> > There were two NFV CIs that ran on the original change [2].
> >> >
> >> > Both failed with the same devstack setup error [3][4].
> >> >
> >> > So it sucks that we have a regression, it sucks that no one watched
> >> > for those CI results before approving the change, and it really
> >> > sucks in this case since it was specifically reported from mellanox
> >> > for sriov which failed in [4]. But it happens.
> >> >
> >> > What I'd like to know is, have the CI problems been fixed? There is
> >> > a change up to fix the regression [5] and this time the Mellanox CI
> >> > check is passing [6]. The Intel NFV CI hasn't reported, but with the
> >> > mellanox one also testing the suspend scenario, it's probably good
> >enough.
> >>
> >> From the commit message of the original patch that introduced the
> >> regression:
> >>
> >> "This fix was tested on a real environment containing the above type of
> >VMs.
> >> test_driver.test_detach_sriov_ports was slightly modified so that the
> >> VIF from which data is sent to _detach_pci_devices will contain the
> >> correct SRIOV values (pci_slot, vlan and hw_veb VIF type)"
> >>
> >> I'm not sure if the above statement could ever have been true
> >> considering the AttributeError that occurred in the bug...
> >>
> >> In any case, I think that it's pretty clear that the CI systems for
> >> NFV and PCI have been less than reliable at functionally testing the
> >> PCI and NFV-specific functionality in Nova.
> >>
> >> This isn't trying to put down the people that work on those systems --
> >> I know first hand that it can be difficult to build and maintain CI
> >> systems that report in to upstream, and I appreciate the effort that goes
> >into this.
> >>
> >> But, going forward, I think we need to do something as a concerned
> >> community.
> >>
> >> How about this for a proposal?
> >>
> >> 1) We establish a joint lab environment that contains heterogeneous
> >> hardware to which all interested hardware vendors must provide
> >hardware.
> >>
> >> 2) The OpenStack Foundation and the hardware vendors each foot some
> >> portion of the bill to hire 2 or more systems administrators to
> >> maintain this lab environment.
> >>
> >> 3) The upstream Infrastructure team works with the hired system
> >> administrators to create a single CI system that can spawn functional
> >> test jobs on the lab hardware and report results back to upstream
> >> Gerrit
> >>
> >> Given the will to do this, I think the benefits of more trusted
> >> testing results for the PCI and SR-IOV/NFV areas would more than make up
> >for the cost.
> >+1 I like this proposal. We can help by providing Mellanox hardware and
> >share our CI knowledge.
> >
> >>
> >> Best,
> >> -jay
> >>
> >>
> >__________________________________________________________
> >______
> >> __________
> >> OpenStack Development Mailing List (not for usage questions)
> >> Unsubscribe:
> >> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >__________________________________________________________
> >________________
> >OpenStack Development Mailing List (not for usage questions)
> >Unsubscribe: OpenStack-dev-
> >request at lists.openstack.org?subject:unsubscribe
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> --------------------------------------------------------------
> Intel Research and Development Ireland Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
>
>
> This e-mail and any attachments may contain confidential material for the sole
> use of the intended recipient(s). Any review or distribution by others is
> strictly prohibited. If you are not the intended recipient, please contact the
> sender and delete all copies.
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
More information about the OpenStack-dev
mailing list