Open Stack

On Aug 23, 2014, at 6:52 PM, Matt Riedemann <mriedem at linux.vnet.ibm.com> wrote:

> 
> 
> On 8/22/2014 12:22 PM, Clark Boylan wrote:
>> On Fri, Aug 22, 2014, at 05:55 AM, Ihar Hrachyshka wrote:
>>> -----BEGIN PGP SIGNED MESSAGE-----
>>> Hash: SHA512
>>> 
>>> Hi all,
>>> 
>>> this week is quite bumpy for unit testing in gate. First, it was
>>> upgrade to new 'tox' version that broke quite some branches.
>>> 
>> I did a ton of work to make the tox upgrade go smoothly because we knew
>> it would be somewhat painful. About a month ago I sent mail to this list
>> [0] describing the problem. This thread included a pointer to the bug
>> filed to track this [1] and example work around changes [2] which I
>> wrote and proposed for as many projects and branches as I had time to
>> test at that point.
>> 
>> Updating tox to 1.7.2 is important for a couple reasons. We get a lot of
>> confused developers wondering why using tox doesn't work to run their
>> tests when all of our documentation says just run tox. Well you needed a
>> special version (1.6.1). Communicating that to everyone that tries to
>> run tox is hard.
>> 
>> It is also important because tox adds new features like the hashseed
>> randomization. This is the cause of our problems but it is exposing real
>> bugs in openstack [3]. We should be fixing these issues and hopefully my
>> proposed workarounds are only temporary.
>> 
>> I decided to push ahead [4] and upgrade tox a couple days ago for a
>> couple reasons. This is an important change as illustrated above and
>> feature freeze and stabilization are rapidly approaching and this
>> probably needed to get in soon to have a chance at getting in at all. I
>> felt this was appropriate because I had done a ton of work prior to make
>> things go as smoothly as possible.
>> 
>> Where things did not go smoothly was on the reviews for my workaround.
>> Some changes were basically ignored [5] others ran into procedural
>> paperwork associated with stable branches that are not quite appropriate
>> for changes of this type [6][7]. I get that generally we only want to
>> backport things from master and that we have some specific way to cherry
>> pick things, but this type of change is to address issues with
>> stable/foo directly and has nothing to do with master. I did eventually
>> go through the "backport" dance for most of these changes despite this
>> not actually being a true backport.
>> 
>> [0]
>> http://lists.openstack.org/pipermail/openstack-dev/2014-July/041283.html
>> [1] https://bugs.launchpad.net/cinder/+bug/1348818
>> [2] https://review.openstack.org/#/c/109700/
>> [3]
>> http://lists.openstack.org/pipermail/openstack-dev/2014-July/041496.html
>> [4]
>> http://lists.openstack.org/pipermail/openstack-dev/2014-August/042010.html
>> [5] https://review.openstack.org/#/c/109749/
>> [6] https://review.openstack.org/#/c/109759/
>> [7] https://review.openstack.org/#/c/109750/
>> 
>> With all of that out of the way are there suggestions for how we can do
>> this better next time? Do we need more time (I gave us about 4 weeks
>> which seemed like plenty to me)? Perhaps I should send more reminder
>> emails? Feedback is very welcome.
>> 
>> Thanks,
>> Clark
>>> 
>>> And today new testtools 0.9.36 were released and were caught by gate,
>>> which resulted in the following unit test failures in multiple projects:
>>> 
>>> TestCase.setUp was already called. Do not explicitly call setUp from
>>> your tests. In your own setUp, use super to call the base setUp.
>>> 
>>> All branches are affected: havana, icehouse, and master.
>>> 
>>> This is because the following check was released with the new version
>>> of the library:
>>> https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517
>>> 
>>> And the temporary fix is to merge the version pin patch in global
>>> requirements, backport it to stable branches, and merge the updates
>>> from Openstack Proposal Bot to all affected projects. The patch for
>>> master requirements is: https://review.openstack.org/#/c/116267/
>>> 
>>> In the meantime, projects will need to fix their tests not to call
>>> setUp() and tearDown() twice. This will be the requirement to unpin
>>> the version of the library.
>>> 
>>> So, please review, backport, and make sure it lands in project
>>> requirements files.
>>> 
>>> /Ihar
>>> -----BEGIN PGP SIGNATURE-----
>>> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>>> 
>>> iQEcBAEBCgAGBQJT9z3cAAoJEC5aWaUY1u57DtsIAOFtK2i4zkMcC79nOrc5w9DW
>>> oO2b064eyLwwbQEaWeeIL2JBSLBxqNV5zeN0eZB3Sq7LQLv0oPaUNTMFG2+gvask
>>> JHCTAGKz776Rt7ptcfmpHURwcT9L//+1HXvd+ADtO0sYKwgmvaBF7aA4WFa4TseG
>>> JCnAsi5OiOZZgTo/6U1B55srHkZr0DWxqTkKKysZJbR2Pr/ZT9io8yu9uucaz9VH
>>> uNLfggtCcjGgccl7IqSUtVRf3lsSGuvBAxVqMszSFJQmFCjy2E26GfsTApp9KXtQ
>>> gbCpEns8QCnt6KF9rygjHLMbYikjbITuUfSL2okZelX9VpKNx0CS29K/tRg5/BA=
>>> =YavB
>>> -----END PGP SIGNATURE-----
>>> 
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
> 
> Clark, given all the cat-herding involved I think you did a nice job.  I actually thought the tox/hashseed thing was wrapped up until the other day when tempest/elastic-recheck were blowing up, so I guess now I know there was an infra flip switched. Whatever, that's fine, if someone asks "wtf is going on with these since 8/21?" someone from infra is pretty quick to point out the change, then it's just getting people that care enough about fixing the bugs to fix them.  I don't think you or the infra team should be responsible for that in all projects affected, it doesn't scale.

+1 - You did a good job keeping us informed in the project meetings and I really appreciate the amount of testing you did yourself before making the changes.

> 
> Maybe next time something like this comes up we get the PTLs to be the ones assigning a person (Clark's Infra Czar?!?!) responsible for coordinating these types of changes so they are ready.

+1

Doug

> 
> -- 
> 
> Thanks,
> 
> Matt Riedemann
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev