[openstack-dev] [tc] [trove][stable] proboscis tests randomly failing in stable branches; bug 1538506

Matt Riedemann mriedem at linux.vnet.ibm.com
Wed Mar 9 21:14:39 UTC 2016



On 3/9/2016 2:15 PM, Amrith Kumar wrote:
> Thanks Matt.
>
> Agreed on 1, 2, 3, and 4. The reason for #5 is that there are other changes (for which we have FFE's) which will be merged and a new version of the client requested.
>
> Specifically, this change
>
> https://review.openstack.org/290177
>
> Thanks,
>
> -amrith
>
>> -----Original Message-----
>> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
>> Sent: Wednesday, March 09, 2016 2:53 PM
>> To: openstack-dev at lists.openstack.org
>> Subject: Re: [openstack-dev] [tc] [trove][stable] proboscis tests randomly
>> failing in stable branches; bug 1538506
>>
>>
>>
>> On 3/9/2016 1:34 PM, Amrith Kumar wrote:
>>> Matt,
>>>
>>> We discussed this at the Trove meeting. Here's my current understanding
>> of the situation:
>>>
>>> 1. Your change https://review.openstack.org/#/c/290048/ which reverts
>> the trove client side of the change should be merged.
>>>
>>> 2. This change (the Trove API side)
>> https://review.openstack.org/#/c/245845/ should also be reverted. I'm
>> assuming that you'll submit a change for this as well for completeness.
>>
>> I can submit a change for this.
>>
>>>
>>> 3. We need to blacklist python-troveclient 2.1.0 on Liberty, this is
>>> your change https://review.openstack.org/#/c/290021/
>>>
>>> 4. We need to blacklist python-troveclient 2.1.0 on master, this is
>> currently *NOT* what your change https://review.openstack.org/290168 does.
>> I disagree with the approach of just increasing the minimum requirement
>> from 1.2.0 to >=2.1.0.
>>
>> I'm OK with blacklisting 2.1.0 on master and can make that change in the
>> same patch.
>>
>>>
>>> 5. We need to request a new python-troveclient. Whether it should be
>> called 2.1.1 or 2.2.0, I'm not positive. My thought is 2.1.1. But out of
>> an abundance of caution, I'm going to look to #openstack-
>> release/#openstack-infra for guidance on this.
>>
>> There are no other changes to python-troveclient since 2.1.0 was released:
>>
>> user at xubuntu:~/git/python-troveclient$ git log --oneline --no-merges
>> 2.1.0..
>> user at xubuntu:~/git/python-troveclient$
>>
>>
>> So 2.1.1 is fine.
>>
>>>
>>> Thanks,
>>>
>>> -amrith
>>>
>>>> -----Original Message-----
>>>> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
>>>> Sent: Wednesday, March 09, 2016 12:42 PM
>>>> To: openstack-dev at lists.openstack.org
>>>> Subject: Re: [openstack-dev] [trove][stable] proboscis tests randomly
>>>> failing in stable branches; bug 1538506
>>>>
>>>>
>>>>
>>>> On 3/9/2016 8:54 AM, Amrith Kumar wrote:
>>>>> Matt,
>>>>>
>>>>> As I understand it, you have 4 changes up for review.
>>>>>
>>>>>        Change [1] seeks to revert the change [6] to python-troveclient
>> on
>>>>>        master and reinstate the slave_of parameter.
>>>>>
>>>>>        Change [2] is doing for stable/liberty, the same things that
>> were
>>>>>        done for master in [9] on master, and [7] on Kilo earlier.
>>>>>
>>>>>        Change [3] is looking to blacklist python-troveclient 2.1.0 on
>>>>>        stable/liberty.
>>>>>
>>>>>        Change [4] is looking to bump the minimum python-troveclient
>>>> version
>>>>>        on master to 2.1.0.
>>>>
>>>> Right, and there was actually another for that beat me to this:
>>>>
>>>> https://review.openstack.org/#/c/290115/
>>>>
>>>>>
>>>>> Here are three questions I have:
>>>>>
>>>>> (1) Wouldn't backward compatibility also require that you revert the
>>>>> other change that removed slave_of on the server [5] ? After all,
>>>>> just like a python client that calls create() with slave_of, a REST
>>>>> client could call the endpoint with slave_of. What is the policy for
>>>>> REST API backward compatibility?
>>>>
>>>> I guess it depends on the Trove team. In Nova, backward compatibility
>>>> in the API is serious business and that's why we have left all sorts
>>>> of warts in the v2.0 API and couldn't remove them. But with the v2.1
>>>> API and microversions, we're able to move the API forward (Ironic
>>>> also has microversions, and I think cinder/neutron/keystone are
>>>> working on adding that support).
>>>>
>>>> So if maintaining backward compatibility in the trove API is
>>>> important to the trove team and it's users, then yes the API change
>>>> should probably be reverted. For anyone doing CD with Trove, they are
>>>> already broken, but people upgrading from liberty to mitaka could save
>> themselves the break.
>>>>
>>>>>
>>>>> (2) Wouldn't [4] just block 2.1.0 in global-requirements on master
>>>>> (mitaka)?
>>>>
>>>> I'm not sure I understand, [4] here bumps the minimum required
>>>> version of python-troveclient to be 2.1.0, not block it. As noted in
>>>> the review, I don't know that it's really necessary to bump to 2.1.0
>>>> iff we land and release [1].
>>>>
>>>>>
>>>>> (3) Something, potentially your patch set [2], should also fix the
>>>>> test that is invoking create() with slave_of, no?
>>>>
>>>> [2] is stable/liberty which still has slave_of in the create API, so
>>>> I don't think that needs to be fixed in stable/liberty.
>>>>
>>>>>
>>>>> -amrith
>>>>>
>>>>> [1] https://review.openstack.org/290048/
>>>>> [2] https://review.openstack.org/290023/
>>>>> [3] https://review.openstack.org/290021/
>>>>> [4] https://review.openstack.org/290168/
>>>>> [5] https://review.openstack.org/#/c/245845/
>>>>> [6] https://review.openstack.org/#/c/245738/
>>>>> [7] https://review.openstack.org/#/c/246735/
>>>>>
>>>>> On 03/08/2016 08:24 PM, Matt Riedemann wrote:
>>>>>>
>>>>>>
>>>>>> On 3/8/2016 4:44 PM, Matt Riedemann wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 3/8/2016 3:17 PM, Matt Riedemann wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3/8/2016 1:18 PM, Amrith Kumar wrote:
>>>>>>>>> Matt,
>>>>>>>>>
>>>>>>>>> See inline below.
>>>>>>>>>
>>>>>>>>> -amrith
>>>>>>>>>
>>>>>>>>>> -----Original Message-----
>>>>>>>>>> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
>>>>>>>>>> Sent: Tuesday, March 08, 2016 2:00 PM
>>>>>>>>>> To: openstack-dev at lists.openstack.org
>>>>>>>>>> Subject: Re: [openstack-dev] [trove][stable] proboscis tests
>>>>>>>>>> randomly failing in stable branches; bug 1538506
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On 3/8/2016 12:52 PM, Matt Riedemann wrote:
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 3/8/2016 12:35 PM, Amrith Kumar wrote:
>>>>>>>>>>>> Matt,
>>>>>>>>>>>>
>>>>>>>>>>>> The correct solution for liberty is that we should fix the
>> tests.
>>>>>>>>>>>> Here's why I believe that this is the case.
>>>>>>>>>>>>
>>>>>>>>>>>> In pertinent part, the backtrace from your bug includes:
>>>>>>>>>>>>
>>>>>>>>>>>> 2016-01-27 07:02:06.777 | File
>>>>>>>>>>>> "/home/jenkins/workspace/periodic-trove-python27-liberty/trov
>>>>>>>>>>>> e/
>>>>>>>>>>>> tests/
>>>>>>>>>>>>
>>>>>>>>>>>> api/replication.py",
>>>>>>>>>>>> line 83, in create_slave
>>>>>>>>>>>> 2016-01-27 07:02:06.777 | slave_of=instance_info.id)
>>>>>>>>>>>>
>>>>>>>>>>>> This is in the tests, not the client.
>>>>>>>>>>>>
>>>>>>>>>>>> The test is now generating a parameter that the client no
>>>>>>>>>>>> longer understands.
>>>>>>>>>>>>
>>>>>>>>>>>> For a user, here are the various situations that could occur.
>>>>>>>>>>>>
>>>>>>>>>>>> 1. User running python-troveclient from the latest 2.1.0 and
>>>>>>>>>>>> a server from liberty. All is good.
>>>>>>>>>>>> 2. User running an old python-troveclient and a server from
>>>>>>>>>>>> liberty.
>>>>>>>>>>>> All is good.
>>>>>>>>>>>
>>>>>>>>>>> Will this work with python-troveclient 1.2.0 which is the
>>>>>>>>>>> minimum required troveclient for stable/liberty?
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/openstack/requirements/blob/stable/liberty/
>>>>>>>>>>> gl
>>>>>>>>>>> obal-r
>>>>>>>>>>>
>>>>>>>>>>> equirements.txt#L174
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> 3. User running an old python-troveclient and a new server
>>>>>>>>>>>> from mitaka, this is not supported.
>>>>>>>>>>>
>>>>>>>>>>> How is this not supported?  If it's not supported, the minimum
>>>>>>>>>>> required version of troveclient in master g-r is wrong, since
>>>>>>>>>>> it hasn't changed since liberty:
>>>>>>>>>>>
>>>>>>>>>>> https://github.com/openstack/requirements/blob/master/global-r
>>>>>>>>>>> eq
>>>>>>>>>>> uireme
>>>>>>>>>>>
>>>>>>>>>>> nts.txt#L202
>>>>>>>>>>
>>>>>>>>>> I've confirmed that running master (mitaka) unit tests for
>>>>>>>>>> trove against python-troveclient 1.2.0 don't work:
>>>>>>>>>>
>>>>>>>>>> http://paste.openstack.org/show/489719/
>>>>>>>>>
>>>>>>>>> [amrith] Just like the bug you listed, this appears to be a case
>>>>>>>>> of a broken test. Would you please LP it.
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> 4. User running a current python-troveclient from the latest
>>>>>>>>>>>> 2.1.0 and a server from mitaka, All is good.
>>>>>>>>>>>>
>>>>>>>>>>>> What you have here is a case where a test is generating an
>>>>>>>>>>>> argument
>>>>>>>>>>>> (slave_of) that the client (master, 2.1.0) no longer
>> understands.
>>>>>>>>>>>> There's nothing amiss there, except that the test needs to be
>>>>>>>>>>>> fixed.
>>>>>>>>>>>>
>>>>>>>>>>>> When you get back, let's continue the conversation either in
>>>>>>>>>>>> email or IRC #openstack-dev. Looking forward to hearing your
>>>>>>>>>>>> feedback on this.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>
>>>>>>>>>>>> -amrith
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
>>>>>>>>>>>>> Sent: Tuesday, March 08, 2016 12:11 PM
>>>>>>>>>>>>> To: openstack-dev at lists.openstack.org
>>>>>>>>>>>>> Subject: Re: [openstack-dev] [trove][stable] proboscis tests
>>>>>>>>>>>>> randomly failing in stable branches; bug 1538506
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 3/8/2016 10:17 AM, Matt Riedemann wrote:
>>>>>>>>>>>>>> This is a call for help on resolving bug 1538506 [1] where
>>>>>>>>>>>>>> the proboscis tests randomly fail on the stable branches
>>>>>>>>>>>>>> with something
>>>>>>>>>>>>> like:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> TypeError: create() got an unexpected keyword argument
>>>> 'slave_of'
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Craig Vyvial has a proposed stable/kilo change [2] but it
>>>>>>>>>>>>>> has some issues, at least from me as a reviewer of that
>> change.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> The tests are failing the periodic-stable job daily [3].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Can we get someone to help out with this? Otherwise I'm
>>>>>>>>>>>>>> inclined to say the tests should be disabled on the stable
>>>>>>>>>>>>>> branches, but that's pretty drastic.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Note that this is the kind of thing that breaks stable
>>>>>>>>>>>>>> branch policy, which is going to be part of a review when
>>>>>>>>>>>>>> applying for the stable:follows-policy tag. [4]  And the
>>>>>>>>>>>>>> stable:follows-policy tag may be used to determine when a
>>>>>>>>>>>>>> stable branch for a project goes end of life if it's not
>>>>>>>>>>>>>> being maintained.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] https://bugs.launchpad.net/trove/+bug/1538506
>>>>>>>>>>>>>> [2] https://review.openstack.org/#/c/276934/
>>>>>>>>>>>>>> [3] http://goo.gl/fqf11U
>>>>>>>>>>>>>> [4]
>>>>>>>>>>>>>> http://governance.openstack.org/reference/tags/stable_follo
>>>>>>>>>>>>>> ws
>>>>>>>>>>>>>> -polic
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> y.h
>>>>>>>>>>>>>> tml
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is my solution for liberty, cap
>>>>>>>>>>>>> python-troveclient<2.1.0 in
>>>>>>>>>>>>> global- requirements on stable/liberty [1].
>>>>>>>>>>>>>
>>>>>>>>>>>>> Then there is a change to trove on stable/liberty to use the
>>>>>>>>>>>>> g-r version range for python-troveclient for unit tests [2].
>>>>>>>>>>>>>
>>>>>>>>>>>>> Alternatively, we could avoid the cap in stable/liberty by:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. Reverting https://review.openstack.org/#/c/245738/
>>>>>>>>>>>>> 2. Blacklisting python-troveclient 2.1.0 in g-r 3. Release
>>>>>>>>>>>>> python- troveclient 2.2.0.
>>>>>>>>>>>>>
>>>>>>>>>>>>> It's getting late in the mitaka release to be messing with
>>>>>>>>>>>>> this though since we're already past client freeze.
>>>>>>>>>>>>>
>>>>>>>>>>>>> [1] https://review.openstack.org/#/c/290021/
>>>>>>>>>>>>> [2] https://review.openstack.org/#/c/290023/
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Matt Riedemann
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ____________________________________________________________
>>>>>>>>>>>>> __
>>>>>>>>>>>>> ______
>>>>>>>>>>>>>
>>>>>>>>>>>>> ______
>>>>>>>>>>>>>
>>>>>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>>>>> Unsubscribe:
>>>>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscrib
>>>>>>>>>>>>> e
>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
>>>>>>>>>>>>> k-
>>>>>>>>>>>>> dev
>>>>>>>>>>>>
>>>>>>>>>>>> _____________________________________________________________
>>>>>>>>>>>> __
>>>>>>>>>>>> ______
>>>>>>>>>>>>
>>>>>>>>>>>> _____
>>>>>>>>>>>>
>>>>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>>>> Unsubscribe:
>>>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>>>> -d
>>>>>>>>>>>> ev
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Matt Riedemann
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> _______________________________________________________________
>>>>>>>>>> __
>>>>>>>>>> _________
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>> Unsubscribe:
>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d
>>>>>>>>>> ev
>>>>>>>>>
>>>>>>>>> ________________________________________________________________
>>>>>>>>> __
>>>>>>>>> ________
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>> Unsubscribe:
>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-de
>>>>>>>>> v
>>>>>>>>>
>>>>>>>>
>>>>>>>> Well, the point I'm trying to get to is, I don't think
>>>>>>>> troveclient <
>>>>>>>> 2.1.0 will work with the trove-api in mitaka.
>>>>>>>>
>>>>>>>> For example, in my change to revert the slave_of removal in the
>>>> client:
>>>>>>>>
>>>>>>>> https://review.openstack.org/#/c/290048/
>>>>>>>>
>>>>>>>> That fails in the API:
>>>>>>>>
>>>>>>>> http://logs.openstack.org/48/290048/2/check/gate-trove-functional
>>>>>>>> -d
>>>>>>>> svm-mysql/ddd3089/logs/screen-tr-api.txt.gz#_2016-03-08_19_30_32_
>>>>>>>> 79
>>>>>>>> 0
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> "Validation error: instance Additional properties are not allowed
>>>>>>>> (u'slave_of' was unexpected)"
>>>>>>>>
>>>>>>>> So if my client application that I wrote in liberty was using
>>>>>>>> slave_of, it's happily working until the cloud that my
>>>>>>>> application is running on upgrades to mitaka, then things break.
>>>>>>>>
>>>>>>>> Part of the issue here is the backward incompatible change in the
>>>>>>>> trove API. The other part is, the trove API in mitaka requires
>>>>>>>> troveclient>=2.1.0 because that's what removed slave_of.
>>>>>>>>
>>>>>>>> Right?
>>>>>>>>
>>>>>>>> I'm trying to sort out what a valid troveclient is for master,
>>>>>>>> because I don't really trust what's in global-requirements since
>>>>>>>> trove hasn't been testing with those versions, at least not at
>>>>>>>> the lower bound of 1.2.0.
>>>>>>>>
>>>>>>>
>>>>>>> Here is a change that bumps the minimum required version of
>>>>>>> python-troveclient for mitaka to 2.1.0:
>>>>>>>
>>>>>>> https://review.openstack.org/#/c/290168/
>>>>>>>
>>>>>>
>>>>>> Here is my preferred option now.
>>>>>>
>>>>>> 1. Revert the slave_of removal from python-troveclient and provide
>>>>>> it as a compat shim until liberty-eol:
>>>>>>
>>>>>> https://review.openstack.org/#/c/290048/
>>>>>>
>>>>>> If slave_of is used, a deprecation warning is emitted (which we
>>>>>> should have been doing when this was deprecated in the first place).
>>>>>> It also never sends slave_of to the API, it's just a proxy to
>>>>>> replica_of. So this should work with both the liberty and mitaka
>>>>>> versions of the trove API.
>>>>>>
>>>>>> 2. Block python-troveclient 2.1.0 from stable/liberty g-r:
>>>>>>
>>>>>> https://review.openstack.org/#/c/290021/
>>>>>>
>>>>>> That blocks the bad 2.1.0 version from being used in stable/liberty
>>>>>> which is needed for...
>>>>>>
>>>>>> 3. Using the g-r versions of python-troveclient when testing on
>>>>>> stable/liberty:
>>>>>>
>>>>>> https://review.openstack.org/#/c/290023/
>>>>>>
>>>>>> If trove is following global-requirements, it should be using the
>>>>>> versions of python-troveclient from global-requirements rather than
>>>>>> installing it from the latest tarball on trunk. master and
>>>>>> stable/kilo are already doing this.
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> ____________________________________________________________________
>>>>> __ ____ OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>>
>>>> Matt Riedemann
>>>>
>>>>
>>>> _____________________________________________________________________
>>>> _____ OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>> ______________________________________________________________________
>>> ____ OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>> --
>>
>> Thanks,
>>
>> Matt Riedemann
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

I'd recommend that you land the revert in the client and release that as 
2.1.1, then worry about an FFE for the client since that's going to be a 
2.2.0 minor version bump.

-- 

Thanks,

Matt Riedemann




More information about the OpenStack-dev mailing list