Open Stack

On 3/9/2016 3:14 PM, Matt Riedemann wrote:
>
>
> On 3/9/2016 2:15 PM, Amrith Kumar wrote:
>> Thanks Matt.
>>
>> Agreed on 1, 2, 3, and 4. The reason for #5 is that there are other
>> changes (for which we have FFE's) which will be merged and a new
>> version of the client requested.
>>
>> Specifically, this change
>>
>> https://review.openstack.org/290177
>>
>> Thanks,
>>
>> -amrith
>>
>>> -----Original Message-----
>>> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
>>> Sent: Wednesday, March 09, 2016 2:53 PM
>>> To: openstack-dev at lists.openstack.org
>>> Subject: Re: [openstack-dev] [tc] [trove][stable] proboscis tests
>>> randomly
>>> failing in stable branches; bug 1538506
>>>
>>>
>>>
>>> On 3/9/2016 1:34 PM, Amrith Kumar wrote:
>>>> Matt,
>>>>
>>>> We discussed this at the Trove meeting. Here's my current understanding
>>> of the situation:
>>>>
>>>> 1. Your change https://review.openstack.org/#/c/290048/ which reverts
>>> the trove client side of the change should be merged.
>>>>
>>>> 2. This change (the Trove API side)
>>> https://review.openstack.org/#/c/245845/ should also be reverted. I'm
>>> assuming that you'll submit a change for this as well for completeness.
>>>
>>> I can submit a change for this.
>>>
>>>>
>>>> 3. We need to blacklist python-troveclient 2.1.0 on Liberty, this is
>>>> your change https://review.openstack.org/#/c/290021/
>>>>
>>>> 4. We need to blacklist python-troveclient 2.1.0 on master, this is
>>> currently *NOT* what your change https://review.openstack.org/290168
>>> does.
>>> I disagree with the approach of just increasing the minimum requirement
>>> from 1.2.0 to >=2.1.0.
>>>
>>> I'm OK with blacklisting 2.1.0 on master and can make that change in the
>>> same patch.
>>>
>>>>
>>>> 5. We need to request a new python-troveclient. Whether it should be
>>> called 2.1.1 or 2.2.0, I'm not positive. My thought is 2.1.1. But out of
>>> an abundance of caution, I'm going to look to #openstack-
>>> release/#openstack-infra for guidance on this.
>>>
>>> There are no other changes to python-troveclient since 2.1.0 was
>>> released:
>>>
>>> user at xubuntu:~/git/python-troveclient$ git log --oneline --no-merges
>>> 2.1.0..
>>> user at xubuntu:~/git/python-troveclient$
>>>
>>>
>>> So 2.1.1 is fine.
>>>
>>>>
>>>> Thanks,
>>>>
>>>> -amrith
>>>>
>>>>> -----Original Message-----
>>>>> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
>>>>> Sent: Wednesday, March 09, 2016 12:42 PM
>>>>> To: openstack-dev at lists.openstack.org
>>>>> Subject: Re: [openstack-dev] [trove][stable] proboscis tests randomly
>>>>> failing in stable branches; bug 1538506
>>>>>
>>>>>
>>>>>
>>>>> On 3/9/2016 8:54 AM, Amrith Kumar wrote:
>>>>>> Matt,
>>>>>>
>>>>>> As I understand it, you have 4 changes up for review.
>>>>>>
>>>>>>        Change [1] seeks to revert the change [6] to
>>>>>> python-troveclient
>>> on
>>>>>>        master and reinstate the slave_of parameter.
>>>>>>
>>>>>>        Change [2] is doing for stable/liberty, the same things that
>>> were
>>>>>>        done for master in [9] on master, and [7] on Kilo earlier.
>>>>>>
>>>>>>        Change [3] is looking to blacklist python-troveclient 2.1.0 on
>>>>>>        stable/liberty.
>>>>>>
>>>>>>        Change [4] is looking to bump the minimum python-troveclient
>>>>> version
>>>>>>        on master to 2.1.0.
>>>>>
>>>>> Right, and there was actually another for that beat me to this:
>>>>>
>>>>> https://review.openstack.org/#/c/290115/
>>>>>
>>>>>>
>>>>>> Here are three questions I have:
>>>>>>
>>>>>> (1) Wouldn't backward compatibility also require that you revert the
>>>>>> other change that removed slave_of on the server [5] ? After all,
>>>>>> just like a python client that calls create() with slave_of, a REST
>>>>>> client could call the endpoint with slave_of. What is the policy for
>>>>>> REST API backward compatibility?
>>>>>
>>>>> I guess it depends on the Trove team. In Nova, backward compatibility
>>>>> in the API is serious business and that's why we have left all sorts
>>>>> of warts in the v2.0 API and couldn't remove them. But with the v2.1
>>>>> API and microversions, we're able to move the API forward (Ironic
>>>>> also has microversions, and I think cinder/neutron/keystone are
>>>>> working on adding that support).
>>>>>
>>>>> So if maintaining backward compatibility in the trove API is
>>>>> important to the trove team and it's users, then yes the API change
>>>>> should probably be reverted. For anyone doing CD with Trove, they are
>>>>> already broken, but people upgrading from liberty to mitaka could save
>>> themselves the break.
>>>>>
>>>>>>
>>>>>> (2) Wouldn't [4] just block 2.1.0 in global-requirements on master
>>>>>> (mitaka)?
>>>>>
>>>>> I'm not sure I understand, [4] here bumps the minimum required
>>>>> version of python-troveclient to be 2.1.0, not block it. As noted in
>>>>> the review, I don't know that it's really necessary to bump to 2.1.0
>>>>> iff we land and release [1].
>>>>>
>>>>>>
>>>>>> (3) Something, potentially your patch set [2], should also fix the
>>>>>> test that is invoking create() with slave_of, no?
>>>>>
>>>>> [2] is stable/liberty which still has slave_of in the create API, so
>>>>> I don't think that needs to be fixed in stable/liberty.
>>>>>
>>>>>>
>>>>>> -amrith
>>>>>>
>>>>>> [1] https://review.openstack.org/290048/
>>>>>> [2] https://review.openstack.org/290023/
>>>>>> [3] https://review.openstack.org/290021/
>>>>>> [4] https://review.openstack.org/290168/
>>>>>> [5] https://review.openstack.org/#/c/245845/
>>>>>> [6] https://review.openstack.org/#/c/245738/
>>>>>> [7] https://review.openstack.org/#/c/246735/
>>>>>>
>>>>>> On 03/08/2016 08:24 PM, Matt Riedemann wrote:
>>>>>>>
>>>>>>>
>>>>>>> On 3/8/2016 4:44 PM, Matt Riedemann wrote:
>>>>>>>>
>>>>>>>>
>>>>>>>> On 3/8/2016 3:17 PM, Matt Riedemann wrote:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On 3/8/2016 1:18 PM, Amrith Kumar wrote:
>>>>>>>>>> Matt,
>>>>>>>>>>
>>>>>>>>>> See inline below.
>>>>>>>>>>
>>>>>>>>>> -amrith
>>>>>>>>>>
>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
>>>>>>>>>>> Sent: Tuesday, March 08, 2016 2:00 PM
>>>>>>>>>>> To: openstack-dev at lists.openstack.org
>>>>>>>>>>> Subject: Re: [openstack-dev] [trove][stable] proboscis tests
>>>>>>>>>>> randomly failing in stable branches; bug 1538506
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On 3/8/2016 12:52 PM, Matt Riedemann wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/8/2016 12:35 PM, Amrith Kumar wrote:
>>>>>>>>>>>>> Matt,
>>>>>>>>>>>>>
>>>>>>>>>>>>> The correct solution for liberty is that we should fix the
>>> tests.
>>>>>>>>>>>>> Here's why I believe that this is the case.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In pertinent part, the backtrace from your bug includes:
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2016-01-27 07:02:06.777 | File
>>>>>>>>>>>>> "/home/jenkins/workspace/periodic-trove-python27-liberty/trov
>>>>>>>>>>>>> e/
>>>>>>>>>>>>> tests/
>>>>>>>>>>>>>
>>>>>>>>>>>>> api/replication.py",
>>>>>>>>>>>>> line 83, in create_slave
>>>>>>>>>>>>> 2016-01-27 07:02:06.777 | slave_of=instance_info.id)
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is in the tests, not the client.
>>>>>>>>>>>>>
>>>>>>>>>>>>> The test is now generating a parameter that the client no
>>>>>>>>>>>>> longer understands.
>>>>>>>>>>>>>
>>>>>>>>>>>>> For a user, here are the various situations that could occur.
>>>>>>>>>>>>>
>>>>>>>>>>>>> 1. User running python-troveclient from the latest 2.1.0 and
>>>>>>>>>>>>> a server from liberty. All is good.
>>>>>>>>>>>>> 2. User running an old python-troveclient and a server from
>>>>>>>>>>>>> liberty.
>>>>>>>>>>>>> All is good.
>>>>>>>>>>>>
>>>>>>>>>>>> Will this work with python-troveclient 1.2.0 which is the
>>>>>>>>>>>> minimum required troveclient for stable/liberty?
>>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/openstack/requirements/blob/stable/liberty/
>>>>>>>>>>>> gl
>>>>>>>>>>>> obal-r
>>>>>>>>>>>>
>>>>>>>>>>>> equirements.txt#L174
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> 3. User running an old python-troveclient and a new server
>>>>>>>>>>>>> from mitaka, this is not supported.
>>>>>>>>>>>>
>>>>>>>>>>>> How is this not supported?  If it's not supported, the minimum
>>>>>>>>>>>> required version of troveclient in master g-r is wrong, since
>>>>>>>>>>>> it hasn't changed since liberty:
>>>>>>>>>>>>
>>>>>>>>>>>> https://github.com/openstack/requirements/blob/master/global-r
>>>>>>>>>>>> eq
>>>>>>>>>>>> uireme
>>>>>>>>>>>>
>>>>>>>>>>>> nts.txt#L202
>>>>>>>>>>>
>>>>>>>>>>> I've confirmed that running master (mitaka) unit tests for
>>>>>>>>>>> trove against python-troveclient 1.2.0 don't work:
>>>>>>>>>>>
>>>>>>>>>>> http://paste.openstack.org/show/489719/
>>>>>>>>>>
>>>>>>>>>> [amrith] Just like the bug you listed, this appears to be a case
>>>>>>>>>> of a broken test. Would you please LP it.
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> 4. User running a current python-troveclient from the latest
>>>>>>>>>>>>> 2.1.0 and a server from mitaka, All is good.
>>>>>>>>>>>>>
>>>>>>>>>>>>> What you have here is a case where a test is generating an
>>>>>>>>>>>>> argument
>>>>>>>>>>>>> (slave_of) that the client (master, 2.1.0) no longer
>>> understands.
>>>>>>>>>>>>> There's nothing amiss there, except that the test needs to be
>>>>>>>>>>>>> fixed.
>>>>>>>>>>>>>
>>>>>>>>>>>>> When you get back, let's continue the conversation either in
>>>>>>>>>>>>> email or IRC #openstack-dev. Looking forward to hearing your
>>>>>>>>>>>>> feedback on this.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> -amrith
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>> From: Matt Riedemann [mailto:mriedem at linux.vnet.ibm.com]
>>>>>>>>>>>>>> Sent: Tuesday, March 08, 2016 12:11 PM
>>>>>>>>>>>>>> To: openstack-dev at lists.openstack.org
>>>>>>>>>>>>>> Subject: Re: [openstack-dev] [trove][stable] proboscis tests
>>>>>>>>>>>>>> randomly failing in stable branches; bug 1538506
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 3/8/2016 10:17 AM, Matt Riedemann wrote:
>>>>>>>>>>>>>>> This is a call for help on resolving bug 1538506 [1] where
>>>>>>>>>>>>>>> the proboscis tests randomly fail on the stable branches
>>>>>>>>>>>>>>> with something
>>>>>>>>>>>>>> like:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> TypeError: create() got an unexpected keyword argument
>>>>> 'slave_of'
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Craig Vyvial has a proposed stable/kilo change [2] but it
>>>>>>>>>>>>>>> has some issues, at least from me as a reviewer of that
>>> change.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> The tests are failing the periodic-stable job daily [3].
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Can we get someone to help out with this? Otherwise I'm
>>>>>>>>>>>>>>> inclined to say the tests should be disabled on the stable
>>>>>>>>>>>>>>> branches, but that's pretty drastic.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Note that this is the kind of thing that breaks stable
>>>>>>>>>>>>>>> branch policy, which is going to be part of a review when
>>>>>>>>>>>>>>> applying for the stable:follows-policy tag. [4]  And the
>>>>>>>>>>>>>>> stable:follows-policy tag may be used to determine when a
>>>>>>>>>>>>>>> stable branch for a project goes end of life if it's not
>>>>>>>>>>>>>>> being maintained.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> [1] https://bugs.launchpad.net/trove/+bug/1538506
>>>>>>>>>>>>>>> [2] https://review.openstack.org/#/c/276934/
>>>>>>>>>>>>>>> [3] http://goo.gl/fqf11U
>>>>>>>>>>>>>>> [4]
>>>>>>>>>>>>>>> http://governance.openstack.org/reference/tags/stable_follo
>>>>>>>>>>>>>>> ws
>>>>>>>>>>>>>>> -polic
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> y.h
>>>>>>>>>>>>>>> tml
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> This is my solution for liberty, cap
>>>>>>>>>>>>>> python-troveclient<2.1.0 in
>>>>>>>>>>>>>> global- requirements on stable/liberty [1].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Then there is a change to trove on stable/liberty to use the
>>>>>>>>>>>>>> g-r version range for python-troveclient for unit tests [2].
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Alternatively, we could avoid the cap in stable/liberty by:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> 1. Reverting https://review.openstack.org/#/c/245738/
>>>>>>>>>>>>>> 2. Blacklisting python-troveclient 2.1.0 in g-r 3. Release
>>>>>>>>>>>>>> python- troveclient 2.2.0.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> It's getting late in the mitaka release to be messing with
>>>>>>>>>>>>>> this though since we're already past client freeze.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> [1] https://review.openstack.org/#/c/290021/
>>>>>>>>>>>>>> [2] https://review.openstack.org/#/c/290023/
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Matt Riedemann
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ____________________________________________________________
>>>>>>>>>>>>>> __
>>>>>>>>>>>>>> ______
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> ______
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>>>>>> Unsubscribe:
>>>>>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscrib
>>>>>>>>>>>>>> e
>>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstac
>>>>>>>>>>>>>> k-
>>>>>>>>>>>>>> dev
>>>>>>>>>>>>>
>>>>>>>>>>>>> _____________________________________________________________
>>>>>>>>>>>>> __
>>>>>>>>>>>>> ______
>>>>>>>>>>>>>
>>>>>>>>>>>>> _____
>>>>>>>>>>>>>
>>>>>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>>>>> Unsubscribe:
>>>>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>>>>>>>>>>> -d
>>>>>>>>>>>>> ev
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> Matt Riedemann
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________________________
>>>>>>>>>>> __
>>>>>>>>>>> _________
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>>> Unsubscribe:
>>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-d
>>>>>>>>>>> ev
>>>>>>>>>>
>>>>>>>>>> ________________________________________________________________
>>>>>>>>>> __
>>>>>>>>>> ________
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>>>>> Unsubscribe:
>>>>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-de
>>>>>>>>>> v
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Well, the point I'm trying to get to is, I don't think
>>>>>>>>> troveclient <
>>>>>>>>> 2.1.0 will work with the trove-api in mitaka.
>>>>>>>>>
>>>>>>>>> For example, in my change to revert the slave_of removal in the
>>>>> client:
>>>>>>>>>
>>>>>>>>> https://review.openstack.org/#/c/290048/
>>>>>>>>>
>>>>>>>>> That fails in the API:
>>>>>>>>>
>>>>>>>>> http://logs.openstack.org/48/290048/2/check/gate-trove-functional
>>>>>>>>> -d
>>>>>>>>> svm-mysql/ddd3089/logs/screen-tr-api.txt.gz#_2016-03-08_19_30_32_
>>>>>>>>> 79
>>>>>>>>> 0
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> "Validation error: instance Additional properties are not allowed
>>>>>>>>> (u'slave_of' was unexpected)"
>>>>>>>>>
>>>>>>>>> So if my client application that I wrote in liberty was using
>>>>>>>>> slave_of, it's happily working until the cloud that my
>>>>>>>>> application is running on upgrades to mitaka, then things break.
>>>>>>>>>
>>>>>>>>> Part of the issue here is the backward incompatible change in the
>>>>>>>>> trove API. The other part is, the trove API in mitaka requires
>>>>>>>>> troveclient>=2.1.0 because that's what removed slave_of.
>>>>>>>>>
>>>>>>>>> Right?
>>>>>>>>>
>>>>>>>>> I'm trying to sort out what a valid troveclient is for master,
>>>>>>>>> because I don't really trust what's in global-requirements since
>>>>>>>>> trove hasn't been testing with those versions, at least not at
>>>>>>>>> the lower bound of 1.2.0.
>>>>>>>>>
>>>>>>>>
>>>>>>>> Here is a change that bumps the minimum required version of
>>>>>>>> python-troveclient for mitaka to 2.1.0:
>>>>>>>>
>>>>>>>> https://review.openstack.org/#/c/290168/
>>>>>>>>
>>>>>>>
>>>>>>> Here is my preferred option now.
>>>>>>>
>>>>>>> 1. Revert the slave_of removal from python-troveclient and provide
>>>>>>> it as a compat shim until liberty-eol:
>>>>>>>
>>>>>>> https://review.openstack.org/#/c/290048/
>>>>>>>
>>>>>>> If slave_of is used, a deprecation warning is emitted (which we
>>>>>>> should have been doing when this was deprecated in the first place).
>>>>>>> It also never sends slave_of to the API, it's just a proxy to
>>>>>>> replica_of. So this should work with both the liberty and mitaka
>>>>>>> versions of the trove API.
>>>>>>>
>>>>>>> 2. Block python-troveclient 2.1.0 from stable/liberty g-r:
>>>>>>>
>>>>>>> https://review.openstack.org/#/c/290021/
>>>>>>>
>>>>>>> That blocks the bad 2.1.0 version from being used in stable/liberty
>>>>>>> which is needed for...
>>>>>>>
>>>>>>> 3. Using the g-r versions of python-troveclient when testing on
>>>>>>> stable/liberty:
>>>>>>>
>>>>>>> https://review.openstack.org/#/c/290023/
>>>>>>>
>>>>>>> If trove is following global-requirements, it should be using the
>>>>>>> versions of python-troveclient from global-requirements rather than
>>>>>>> installing it from the latest tarball on trunk. master and
>>>>>>> stable/kilo are already doing this.
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> ____________________________________________________________________
>>>>>> __ ____ OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>
>>>>> --
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Matt Riedemann
>>>>>
>>>>>
>>>>> _____________________________________________________________________
>>>>> _____ OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>> ______________________________________________________________________
>>>> ____ OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>> --
>>>
>>> Thanks,
>>>
>>> Matt Riedemann
>>>
>>>
>>> __________________________________________________________________________
>>>
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>>
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
> I'd recommend that you land the revert in the client and release that as
> 2.1.1, then worry about an FFE for the client since that's going to be a
> 2.2.0 minor version bump.
>

I think the last problem now is, the reno change for trove on 
stable/liberty is failing because the mysql job always fails:

http://logs.openstack.org/84/252584/8/check/gate-trove-functional-dsvm-mysql/072142e/console.html#_2016-03-10_03_02_17_051

Is that a known issue? Was something fixed on master that can be backported?

-- 

Thanks,

Matt Riedemann