Open Stack

On 8/22/2014 12:22 PM, Clark Boylan wrote:
> On Fri, Aug 22, 2014, at 05:55 AM, Ihar Hrachyshka wrote:
>> -----BEGIN PGP SIGNED MESSAGE-----
>> Hash: SHA512
>>
>> Hi all,
>>
>> this week is quite bumpy for unit testing in gate. First, it was
>> upgrade to new 'tox' version that broke quite some branches.
>>
> I did a ton of work to make the tox upgrade go smoothly because we knew
> it would be somewhat painful. About a month ago I sent mail to this list
> [0] describing the problem. This thread included a pointer to the bug
> filed to track this [1] and example work around changes [2] which I
> wrote and proposed for as many projects and branches as I had time to
> test at that point.
>
> Updating tox to 1.7.2 is important for a couple reasons. We get a lot of
> confused developers wondering why using tox doesn't work to run their
> tests when all of our documentation says just run tox. Well you needed a
> special version (1.6.1). Communicating that to everyone that tries to
> run tox is hard.
>
> It is also important because tox adds new features like the hashseed
> randomization. This is the cause of our problems but it is exposing real
> bugs in openstack [3]. We should be fixing these issues and hopefully my
> proposed workarounds are only temporary.
>
> I decided to push ahead [4] and upgrade tox a couple days ago for a
> couple reasons. This is an important change as illustrated above and
> feature freeze and stabilization are rapidly approaching and this
> probably needed to get in soon to have a chance at getting in at all. I
> felt this was appropriate because I had done a ton of work prior to make
> things go as smoothly as possible.
>
> Where things did not go smoothly was on the reviews for my workaround.
> Some changes were basically ignored [5] others ran into procedural
> paperwork associated with stable branches that are not quite appropriate
> for changes of this type [6][7]. I get that generally we only want to
> backport things from master and that we have some specific way to cherry
> pick things, but this type of change is to address issues with
> stable/foo directly and has nothing to do with master. I did eventually
> go through the "backport" dance for most of these changes despite this
> not actually being a true backport.
>
> [0]
> http://lists.openstack.org/pipermail/openstack-dev/2014-July/041283.html
> [1] https://bugs.launchpad.net/cinder/+bug/1348818
> [2] https://review.openstack.org/#/c/109700/
> [3]
> http://lists.openstack.org/pipermail/openstack-dev/2014-July/041496.html
> [4]
> http://lists.openstack.org/pipermail/openstack-dev/2014-August/042010.html
> [5] https://review.openstack.org/#/c/109749/
> [6] https://review.openstack.org/#/c/109759/
> [7] https://review.openstack.org/#/c/109750/
>
> With all of that out of the way are there suggestions for how we can do
> this better next time? Do we need more time (I gave us about 4 weeks
> which seemed like plenty to me)? Perhaps I should send more reminder
> emails? Feedback is very welcome.
>
> Thanks,
> Clark
>>
>> And today new testtools 0.9.36 were released and were caught by gate,
>> which resulted in the following unit test failures in multiple projects:
>>
>> TestCase.setUp was already called. Do not explicitly call setUp from
>> your tests. In your own setUp, use super to call the base setUp.
>>
>> All branches are affected: havana, icehouse, and master.
>>
>> This is because the following check was released with the new version
>> of the library:
>> https://github.com/testing-cabal/testtools/commit/5c3b92d90a64efaecdc4010a98002bfe8b888517
>>
>> And the temporary fix is to merge the version pin patch in global
>> requirements, backport it to stable branches, and merge the updates
>> from Openstack Proposal Bot to all affected projects. The patch for
>> master requirements is: https://review.openstack.org/#/c/116267/
>>
>> In the meantime, projects will need to fix their tests not to call
>> setUp() and tearDown() twice. This will be the requirement to unpin
>> the version of the library.
>>
>> So, please review, backport, and make sure it lands in project
>> requirements files.
>>
>> /Ihar
>> -----BEGIN PGP SIGNATURE-----
>> Version: GnuPG/MacGPG2 v2.0.22 (Darwin)
>>
>> iQEcBAEBCgAGBQJT9z3cAAoJEC5aWaUY1u57DtsIAOFtK2i4zkMcC79nOrc5w9DW
>> oO2b064eyLwwbQEaWeeIL2JBSLBxqNV5zeN0eZB3Sq7LQLv0oPaUNTMFG2+gvask
>> JHCTAGKz776Rt7ptcfmpHURwcT9L//+1HXvd+ADtO0sYKwgmvaBF7aA4WFa4TseG
>> JCnAsi5OiOZZgTo/6U1B55srHkZr0DWxqTkKKysZJbR2Pr/ZT9io8yu9uucaz9VH
>> uNLfggtCcjGgccl7IqSUtVRf3lsSGuvBAxVqMszSFJQmFCjy2E26GfsTApp9KXtQ
>> gbCpEns8QCnt6KF9rygjHLMbYikjbITuUfSL2okZelX9VpKNx0CS29K/tRg5/BA=
>> =YavB
>> -----END PGP SIGNATURE-----
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

Clark, given all the cat-herding involved I think you did a nice job.  I 
actually thought the tox/hashseed thing was wrapped up until the other 
day when tempest/elastic-recheck were blowing up, so I guess now I know 
there was an infra flip switched. Whatever, that's fine, if someone asks 
"wtf is going on with these since 8/21?" someone from infra is pretty 
quick to point out the change, then it's just getting people that care 
enough about fixing the bugs to fix them.  I don't think you or the 
infra team should be responsible for that in all projects affected, it 
doesn't scale.

Maybe next time something like this comes up we get the PTLs to be the 
ones assigning a person (Clark's Infra Czar?!?!) responsible for 
coordinating these types of changes so they are ready.

-- 

Thanks,

Matt Riedemann