Open Stack

Matt,

We discussed this at the Trove meeting. Here's my current understanding of the situation:

1. Your change https://review.openstack.org/#/c/290048/ which reverts the trove client side of the change should be merged.

2. This change (the Trove API side) https://review.openstack.org/#/c/245845/ should also be reverted. I'm assuming that you'll submit a change for this as well for completeness.

3. We need to blacklist python-troveclient 2.1.0 on Liberty, this is your change https://review.openstack.org/#/c/290021/

4. We need to blacklist python-troveclient 2.1.0 on master, this is currently *NOT* what your change https://review.openstack.org/290168 does. I disagree with the approach of just increasing the minimum requirement from 1.2.0 to >=2.1.0. 

5. We need to request a new python-troveclient. Whether it should be called 2.1.1 or 2.2.0, I'm not positive. My thought is 2.1.1. But out of an abundance of caution, I'm going to look to #openstack-release/#openstack-infra for guidance on this.

Thanks,

-amrith

> -----Original Message-----
> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
> Sent: Wednesday, March 09, 2016 12:42 PM
> To: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [trove][stable] proboscis tests randomly
> failing in stable branches; bug 1538506
> 
> 
> 
> On 3/9/2016 8:54 AM, Amrith Kumar wrote:
> > Matt,
> >
> > As I understand it, you have 4 changes up for review.
> >
> >      Change [1] seeks to revert the change [6] to python-troveclient on
> >      master and reinstate the slave_of parameter.
> >
> >      Change [2] is doing for stable/liberty, the same things that were
> >      done for master in [9] on master, and [7] on Kilo earlier.
> >
> >      Change [3] is looking to blacklist python-troveclient 2.1.0 on
> >      stable/liberty.
> >
> >      Change [4] is looking to bump the minimum python-troveclient
> version
> >      on master to 2.1.0.
> 
> Right, and there was actually another for that beat me to this:
> 
> https://review.openstack.org/#/c/290115/
> 
> >
> > Here are three questions I have:
> >
> > (1) Wouldn't backward compatibility also require that you revert the
> > other change that removed slave_of on the server [5] ? After all, just
> > like a python client that calls create() with slave_of, a REST client
> > could call the endpoint with slave_of. What is the policy for REST API
> > backward compatibility?
> 
> I guess it depends on the Trove team. In Nova, backward compatibility in
> the API is serious business and that's why we have left all sorts of warts
> in the v2.0 API and couldn't remove them. But with the v2.1 API and
> microversions, we're able to move the API forward (Ironic also has
> microversions, and I think cinder/neutron/keystone are working on adding
> that support).
> 
> So if maintaining backward compatibility in the trove API is important to
> the trove team and it's users, then yes the API change should probably be
> reverted. For anyone doing CD with Trove, they are already broken, but
> people upgrading from liberty to mitaka could save themselves the break.
> 
> >
> > (2) Wouldn't [4] just block 2.1.0 in global-requirements on master
> > (mitaka)?
> 
> I'm not sure I understand, [4] here bumps the minimum required version of
> python-troveclient to be 2.1.0, not block it. As noted in the review, I
> don't know that it's really necessary to bump to 2.1.0 iff we land and
> release [1].
> 
> >
> > (3) Something, potentially your patch set [2], should also fix the
> > test that is invoking create() with slave_of, no?
> 
> [2] is stable/liberty which still has slave_of in the create API, so I
> don't think that needs to be fixed in stable/liberty.
> 
> >
> > -amrith
> >
> > [1] https://review.openstack.org/290048/
> > [2] https://review.openstack.org/290023/
> > [3] https://review.openstack.org/290021/
> > [4] https://review.openstack.org/290168/
> > [5] https://review.openstack.org/#/c/245845/
> > [6] https://review.openstack.org/#/c/245738/
> > [7] https://review.openstack.org/#/c/246735/
> >
> > On 03/08/2016 08:24 PM, Matt Riedemann wrote:
> >>
> >>
> >> On 3/8/2016 4:44 PM, Matt Riedemann wrote:
> >>>
> >>>
> >>> On 3/8/2016 3:17 PM, Matt Riedemann wrote:
> >>>>
> >>>>
> >>>> On 3/8/2016 1:18 PM, Amrith Kumar wrote:
> >>>>> Matt,
> >>>>>
> >>>>> See inline below.
> >>>>>
> >>>>> -amrith
> >>>>>
> >>>>>> -----Original Message-----
> >>>>>> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
> >>>>>> Sent: Tuesday, March 08, 2016 2:00 PM
> >>>>>> To: openstack-dev at lists.openstack.org
> >>>>>> Subject: Re: [openstack-dev] [trove][stable] proboscis tests
> >>>>>> randomly failing in stable branches; bug 1538506
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On 3/8/2016 12:52 PM, Matt Riedemann wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>> On 3/8/2016 12:35 PM, Amrith Kumar wrote:
> >>>>>>>> Matt,
> >>>>>>>>
> >>>>>>>> The correct solution for liberty is that we should fix the tests.
> >>>>>>>> Here's why I believe that this is the case.
> >>>>>>>>
> >>>>>>>> In pertinent part, the backtrace from your bug includes:
> >>>>>>>>
> >>>>>>>> 2016-01-27 07:02:06.777 | File
> >>>>>>>> "/home/jenkins/workspace/periodic-trove-python27-liberty/trove/
> >>>>>>>> tests/
> >>>>>>>>
> >>>>>>>> api/replication.py",
> >>>>>>>> line 83, in create_slave
> >>>>>>>> 2016-01-27 07:02:06.777 | slave_of=instance_info.id)
> >>>>>>>>
> >>>>>>>> This is in the tests, not the client.
> >>>>>>>>
> >>>>>>>> The test is now generating a parameter that the client no
> >>>>>>>> longer understands.
> >>>>>>>>
> >>>>>>>> For a user, here are the various situations that could occur.
> >>>>>>>>
> >>>>>>>> 1. User running python-troveclient from the latest 2.1.0 and a
> >>>>>>>> server from liberty. All is good.
> >>>>>>>> 2. User running an old python-troveclient and a server from
> >>>>>>>> liberty.
> >>>>>>>> All is good.
> >>>>>>>
> >>>>>>> Will this work with python-troveclient 1.2.0 which is the
> >>>>>>> minimum required troveclient for stable/liberty?
> >>>>>>>
> >>>>>>> https://github.com/openstack/requirements/blob/stable/liberty/gl
> >>>>>>> obal-r
> >>>>>>>
> >>>>>>> equirements.txt#L174
> >>>>>>>
> >>>>>>>
> >>>>>>>> 3. User running an old python-troveclient and a new server from
> >>>>>>>> mitaka, this is not supported.
> >>>>>>>
> >>>>>>> How is this not supported?  If it's not supported, the minimum
> >>>>>>> required version of troveclient in master g-r is wrong, since it
> >>>>>>> hasn't changed since liberty:
> >>>>>>>
> >>>>>>> https://github.com/openstack/requirements/blob/master/global-req
> >>>>>>> uireme
> >>>>>>>
> >>>>>>> nts.txt#L202
> >>>>>>
> >>>>>> I've confirmed that running master (mitaka) unit tests for trove
> >>>>>> against python-troveclient 1.2.0 don't work:
> >>>>>>
> >>>>>> http://paste.openstack.org/show/489719/
> >>>>>
> >>>>> [amrith] Just like the bug you listed, this appears to be a case
> >>>>> of a broken test. Would you please LP it.
> >>>>>
> >>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>>> 4. User running a current python-troveclient from the latest
> >>>>>>>> 2.1.0 and a server from mitaka, All is good.
> >>>>>>>>
> >>>>>>>> What you have here is a case where a test is generating an
> >>>>>>>> argument
> >>>>>>>> (slave_of) that the client (master, 2.1.0) no longer understands.
> >>>>>>>> There's nothing amiss there, except that the test needs to be
> >>>>>>>> fixed.
> >>>>>>>>
> >>>>>>>> When you get back, let's continue the conversation either in
> >>>>>>>> email or IRC #openstack-dev. Looking forward to hearing your
> >>>>>>>> feedback on this.
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>>
> >>>>>>>> -amrith
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>> -----Original Message-----
> >>>>>>>>> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
> >>>>>>>>> Sent: Tuesday, March 08, 2016 12:11 PM
> >>>>>>>>> To: openstack-dev at lists.openstack.org
> >>>>>>>>> Subject: Re: [openstack-dev] [trove][stable] proboscis tests
> >>>>>>>>> randomly failing in stable branches; bug 1538506
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On 3/8/2016 10:17 AM, Matt Riedemann wrote:
> >>>>>>>>>> This is a call for help on resolving bug 1538506 [1] where
> >>>>>>>>>> the proboscis tests randomly fail on the stable branches with
> >>>>>>>>>> something
> >>>>>>>>> like:
> >>>>>>>>>>
> >>>>>>>>>> TypeError: create() got an unexpected keyword argument
> 'slave_of'
> >>>>>>>>>>
> >>>>>>>>>> Craig Vyvial has a proposed stable/kilo change [2] but it has
> >>>>>>>>>> some issues, at least from me as a reviewer of that change.
> >>>>>>>>>>
> >>>>>>>>>> The tests are failing the periodic-stable job daily [3].
> >>>>>>>>>>
> >>>>>>>>>> Can we get someone to help out with this? Otherwise I'm
> >>>>>>>>>> inclined to say the tests should be disabled on the stable
> >>>>>>>>>> branches, but that's pretty drastic.
> >>>>>>>>>>
> >>>>>>>>>> Note that this is the kind of thing that breaks stable branch
> >>>>>>>>>> policy, which is going to be part of a review when applying
> >>>>>>>>>> for the stable:follows-policy tag. [4]  And the
> >>>>>>>>>> stable:follows-policy tag may be used to determine when a
> >>>>>>>>>> stable branch for a project goes end of life if it's not
> >>>>>>>>>> being maintained.
> >>>>>>>>>>
> >>>>>>>>>> [1] https://bugs.launchpad.net/trove/+bug/1538506
> >>>>>>>>>> [2] https://review.openstack.org/#/c/276934/
> >>>>>>>>>> [3] http://goo.gl/fqf11U
> >>>>>>>>>> [4]
> >>>>>>>>>> http://governance.openstack.org/reference/tags/stable_follows
> >>>>>>>>>> -polic
> >>>>>>>>>>
> >>>>>>>>>> y.h
> >>>>>>>>>> tml
> >>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> This is my solution for liberty, cap python-troveclient<2.1.0
> >>>>>>>>> in
> >>>>>>>>> global- requirements on stable/liberty [1].
> >>>>>>>>>
> >>>>>>>>> Then there is a change to trove on stable/liberty to use the
> >>>>>>>>> g-r version range for python-troveclient for unit tests [2].
> >>>>>>>>>
> >>>>>>>>> Alternatively, we could avoid the cap in stable/liberty by:
> >>>>>>>>>
> >>>>>>>>> 1. Reverting https://review.openstack.org/#/c/245738/
> >>>>>>>>> 2. Blacklisting python-troveclient 2.1.0 in g-r 3. Release
> >>>>>>>>> python- troveclient 2.2.0.
> >>>>>>>>>
> >>>>>>>>> It's getting late in the mitaka release to be messing with
> >>>>>>>>> this though since we're already past client freeze.
> >>>>>>>>>
> >>>>>>>>> [1] https://review.openstack.org/#/c/290021/
> >>>>>>>>> [2] https://review.openstack.org/#/c/290023/
> >>>>>>>>>
> >>>>>>>>> --
> >>>>>>>>>
> >>>>>>>>> Thanks,
> >>>>>>>>>
> >>>>>>>>> Matt Riedemann
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> ______________________________________________________________
> >>>>>>>>> ______
> >>>>>>>>>
> >>>>>>>>> ______
> >>>>>>>>>
> >>>>>>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>>>>>> Unsubscribe:
> >>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-
> >>>>>>>>> dev
> >>>>>>>>
> >>>>>>>> _______________________________________________________________
> >>>>>>>> ______
> >>>>>>>>
> >>>>>>>> _____
> >>>>>>>>
> >>>>>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>>>>> Unsubscribe:
> >>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d
> >>>>>>>> ev
> >>>>>>>>
> >>>>>>>
> >>>>>>
> >>>>>> --
> >>>>>>
> >>>>>> Thanks,
> >>>>>>
> >>>>>> Matt Riedemann
> >>>>>>
> >>>>>>
> >>>>>> _________________________________________________________________
> >>>>>> _________
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>>> Unsubscribe:
> >>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>> __________________________________________________________________
> >>>>> ________
> >>>>>
> >>>>>
> >>>>>
> >>>>> OpenStack Development Mailing List (not for usage questions)
> >>>>> Unsubscribe:
> >>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >>>>>
> >>>>
> >>>> Well, the point I'm trying to get to is, I don't think troveclient
> >>>> <
> >>>> 2.1.0 will work with the trove-api in mitaka.
> >>>>
> >>>> For example, in my change to revert the slave_of removal in the
> client:
> >>>>
> >>>> https://review.openstack.org/#/c/290048/
> >>>>
> >>>> That fails in the API:
> >>>>
> >>>> http://logs.openstack.org/48/290048/2/check/gate-trove-functional-d
> >>>> svm-mysql/ddd3089/logs/screen-tr-api.txt.gz#_2016-03-08_19_30_32_79
> >>>> 0
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> "Validation error: instance Additional properties are not allowed
> >>>> (u'slave_of' was unexpected)"
> >>>>
> >>>> So if my client application that I wrote in liberty was using
> >>>> slave_of, it's happily working until the cloud that my application
> >>>> is running on upgrades to mitaka, then things break.
> >>>>
> >>>> Part of the issue here is the backward incompatible change in the
> >>>> trove API. The other part is, the trove API in mitaka requires
> >>>> troveclient>=2.1.0 because that's what removed slave_of.
> >>>>
> >>>> Right?
> >>>>
> >>>> I'm trying to sort out what a valid troveclient is for master,
> >>>> because I don't really trust what's in global-requirements since
> >>>> trove hasn't been testing with those versions, at least not at the
> >>>> lower bound of 1.2.0.
> >>>>
> >>>
> >>> Here is a change that bumps the minimum required version of
> >>> python-troveclient for mitaka to 2.1.0:
> >>>
> >>> https://review.openstack.org/#/c/290168/
> >>>
> >>
> >> Here is my preferred option now.
> >>
> >> 1. Revert the slave_of removal from python-troveclient and provide it
> >> as a compat shim until liberty-eol:
> >>
> >> https://review.openstack.org/#/c/290048/
> >>
> >> If slave_of is used, a deprecation warning is emitted (which we
> >> should have been doing when this was deprecated in the first place).
> >> It also never sends slave_of to the API, it's just a proxy to
> >> replica_of. So this should work with both the liberty and mitaka
> >> versions of the trove API.
> >>
> >> 2. Block python-troveclient 2.1.0 from stable/liberty g-r:
> >>
> >> https://review.openstack.org/#/c/290021/
> >>
> >> That blocks the bad 2.1.0 version from being used in stable/liberty
> >> which is needed for...
> >>
> >> 3. Using the g-r versions of python-troveclient when testing on
> >> stable/liberty:
> >>
> >> https://review.openstack.org/#/c/290023/
> >>
> >> If trove is following global-requirements, it should be using the
> >> versions of python-troveclient from global-requirements rather than
> >> installing it from the latest tarball on trunk. master and
> >> stable/kilo are already doing this.
> >>
> >
> >
> >
> >
> > ______________________________________________________________________
> > ____ OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> 
> --
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev