[openstack-dev] [nova] Bogus -1 scores from turbo hipster

Matt Riedemann mriedem at linux.vnet.ibm.com
Wed Jan 8 14:48:23 UTC 2014

On Tuesday, January 07, 2014 4:53:01 PM, Michael Still wrote:
> Hi. Thanks for reaching out about this.
> It seems this patch has now passed turbo hipster, so I am going to
> treat this as a more theoretical question than perhaps you intended. I
> should note though that Joshua Hesketh and I have been trying to read
> / triage every turbo hipster failure, but that has been hard this week
> because we're both at a conference.
> The problem this patch faced is that we are having trouble defining
> what is a reasonable amount of time for a database migration to run
> for. Specifically:
> 2014-01-07 14:59:32,012 [output] 205 -> 206...
> 2014-01-07 14:59:32,848 [heartbeat]
> 2014-01-07 15:00:02,848 [heartbeat]
> 2014-01-07 15:00:32,849 [heartbeat]
> 2014-01-07 15:00:39,197 [output] done
> So applying migration 206 took slightly over a minute (67 seconds).
> Our historical data (mean + 2 standard deviations) says that this
> migration should take no more than 63 seconds. So this only just
> failed the test.
> However, we know there are issues with our methodology -- we've tried
> normalizing for disk IO bandwidth and it hasn't worked out as well as
> we'd hoped. This week's plan is to try to use mysql performance schema
> instead, but we have to learn more about how it works first.
> I apologise for this mis-vote.
> Michael
> On Wed, Jan 8, 2014 at 1:44 AM, Matt Riedemann
> <mriedem at linux.vnet.ibm.com> wrote:
>> On 12/30/2013 6:21 AM, Michael Still wrote:
>>> Hi.
>>> The purpose of this email to is apologise for some incorrect -1 review
>>> scores which turbo hipster sent out today. I think its important when
>>> a third party testing tool is new to not have flakey results as people
>>> learn to trust the tool, so I want to explain what happened here.
>>> Turbo hipster is a system which takes nova code reviews, and runs
>>> database upgrades against them to ensure that we can still upgrade for
>>> users in the wild. It uses real user datasets, and also times
>>> migrations and warns when they are too slow for large deployments. It
>>> started voting on gerrit in the last week.
>>> Turbo hipster uses zuul to learn about reviews in gerrit that it
>>> should test. We run our own zuul instance, which talks to the
>>> openstack.org zuul instance. This then hands out work to our pool of
>>> testing workers. Another thing zuul does is it handles maintaining a
>>> git repository for the workers to clone from.
>>> This is where things went wrong today. For reasons I can't currently
>>> explain, the git repo on our zuul instance ended up in a bad state (it
>>> had a patch merged to master which wasn't in fact merged upstream
>>> yet). As this code is stock zuul from openstack-infra, I have a
>>> concern this might be a bug that other zuul users will see as well.
>>> I've corrected the problem for now, and kicked off a recheck of any
>>> patch with a -1 review score from turbo hipster in the last 24 hours.
>>> I'll talk to the zuul maintainers tomorrow about the git problem and
>>> see what we can learn.
>>> Thanks heaps for your patience.
>>> Michael
>> How do I interpret the warning and -1 from turbo-hipster on my patch here
>> [1] with the logs here [2]?
>> I'm inclined to just do 'recheck migrations' on this since this patch
>> doesn't have anything to do with this -1 as far as I can tell.
>> [1] https://review.openstack.org/#/c/64725/4/
>> [2]
>> https://ssl.rcbops.com/turbo_hipster/logviewer/?q=/turbo_hipster/results/64/64725/4/check/gate-real-db-upgrade_nova_mysql_user_001/5186e53/user_001.log
>> --
>> Thanks,
>> Matt Riedemann
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev

Another question.  This patch [1] failed turbo-hipster after it was 
approved but I don't know if that's a gating or just voting job, i.e. 
should someone do 'reverify migrations' on that patch or just let it 
sit and ignore turbo-hipster?

[1] https://review.openstack.org/#/c/59824/



Matt Riedemann

More information about the OpenStack-dev mailing list