[openstack-dev] [all] new testtools breaking gate
doug at doughellmann.com
Mon Aug 25 18:20:01 UTC 2014
On Aug 23, 2014, at 6:52 PM, Matt Riedemann <mriedem at linux.vnet.ibm.com> wrote:
> On 8/22/2014 12:22 PM, Clark Boylan wrote:
>> On Fri, Aug 22, 2014, at 05:55 AM, Ihar Hrachyshka wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA512
>>> Hi all,
>>> this week is quite bumpy for unit testing in gate. First, it was
>>> upgrade to new 'tox' version that broke quite some branches.
>> I did a ton of work to make the tox upgrade go smoothly because we knew
>> it would be somewhat painful. About a month ago I sent mail to this list
>>  describing the problem. This thread included a pointer to the bug
>> filed to track this  and example work around changes  which I
>> wrote and proposed for as many projects and branches as I had time to
>> test at that point.
>> Updating tox to 1.7.2 is important for a couple reasons. We get a lot of
>> confused developers wondering why using tox doesn't work to run their
>> tests when all of our documentation says just run tox. Well you needed a
>> special version (1.6.1). Communicating that to everyone that tries to
>> run tox is hard.
>> It is also important because tox adds new features like the hashseed
>> randomization. This is the cause of our problems but it is exposing real
>> bugs in openstack . We should be fixing these issues and hopefully my
>> proposed workarounds are only temporary.
>> I decided to push ahead  and upgrade tox a couple days ago for a
>> couple reasons. This is an important change as illustrated above and
>> feature freeze and stabilization are rapidly approaching and this
>> probably needed to get in soon to have a chance at getting in at all. I
>> felt this was appropriate because I had done a ton of work prior to make
>> things go as smoothly as possible.
>> Where things did not go smoothly was on the reviews for my workaround.
>> Some changes were basically ignored  others ran into procedural
>> paperwork associated with stable branches that are not quite appropriate
>> for changes of this type . I get that generally we only want to
>> backport things from master and that we have some specific way to cherry
>> pick things, but this type of change is to address issues with
>> stable/foo directly and has nothing to do with master. I did eventually
>> go through the "backport" dance for most of these changes despite this
>> not actually being a true backport.
>>  https://bugs.launchpad.net/cinder/+bug/1348818
>>  https://review.openstack.org/#/c/109700/
>>  https://review.openstack.org/#/c/109749/
>>  https://review.openstack.org/#/c/109759/
>>  https://review.openstack.org/#/c/109750/
>> With all of that out of the way are there suggestions for how we can do
>> this better next time? Do we need more time (I gave us about 4 weeks
>> which seemed like plenty to me)? Perhaps I should send more reminder
>> emails? Feedback is very welcome.
>>> And today new testtools 0.9.36 were released and were caught by gate,
>>> which resulted in the following unit test failures in multiple projects:
>>> TestCase.setUp was already called. Do not explicitly call setUp from
>>> your tests. In your own setUp, use super to call the base setUp.
>>> All branches are affected: havana, icehouse, and master.
>>> This is because the following check was released with the new version
>>> of the library:
>>> And the temporary fix is to merge the version pin patch in global
>>> requirements, backport it to stable branches, and merge the updates
>>> from Openstack Proposal Bot to all affected projects. The patch for
>>> master requirements is: https://review.openstack.org/#/c/116267/
>>> In the meantime, projects will need to fix their tests not to call
>>> setUp() and tearDown() twice. This will be the requirement to unpin
>>> the version of the library.
>>> So, please review, backport, and make sure it lands in project
>>> requirements files.
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>>> -----END PGP SIGNATURE-----
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
> Clark, given all the cat-herding involved I think you did a nice job. I actually thought the tox/hashseed thing was wrapped up until the other day when tempest/elastic-recheck were blowing up, so I guess now I know there was an infra flip switched. Whatever, that's fine, if someone asks "wtf is going on with these since 8/21?" someone from infra is pretty quick to point out the change, then it's just getting people that care enough about fixing the bugs to fix them. I don't think you or the infra team should be responsible for that in all projects affected, it doesn't scale.
+1 - You did a good job keeping us informed in the project meetings and I really appreciate the amount of testing you did yourself before making the changes.
> Maybe next time something like this comes up we get the PTLs to be the ones assigning a person (Clark's Infra Czar?!?!) responsible for coordinating these types of changes so they are ready.
> Matt Riedemann
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
More information about the OpenStack-dev