Open Stack

Hi. Thanks for reaching out about this.

It seems this patch has now passed turbo hipster, so I am going to
treat this as a more theoretical question than perhaps you intended. I
should note though that Joshua Hesketh and I have been trying to read
/ triage every turbo hipster failure, but that has been hard this week
because we're both at a conference.

The problem this patch faced is that we are having trouble defining
what is a reasonable amount of time for a database migration to run
for. Specifically:

2014-01-07 14:59:32,012 [output] 205 -> 206...
2014-01-07 14:59:32,848 [heartbeat]
2014-01-07 15:00:02,848 [heartbeat]
2014-01-07 15:00:32,849 [heartbeat]
2014-01-07 15:00:39,197 [output] done

So applying migration 206 took slightly over a minute (67 seconds).
Our historical data (mean + 2 standard deviations) says that this
migration should take no more than 63 seconds. So this only just
failed the test.

However, we know there are issues with our methodology -- we've tried
normalizing for disk IO bandwidth and it hasn't worked out as well as
we'd hoped. This week's plan is to try to use mysql performance schema
instead, but we have to learn more about how it works first.

I apologise for this mis-vote.

Michael

On Wed, Jan 8, 2014 at 1:44 AM, Matt Riedemann
<mriedem at linux.vnet.ibm.com> wrote:
>
>
> On 12/30/2013 6:21 AM, Michael Still wrote:
>>
>> Hi.
>>
>> The purpose of this email to is apologise for some incorrect -1 review
>> scores which turbo hipster sent out today. I think its important when
>> a third party testing tool is new to not have flakey results as people
>> learn to trust the tool, so I want to explain what happened here.
>>
>> Turbo hipster is a system which takes nova code reviews, and runs
>> database upgrades against them to ensure that we can still upgrade for
>> users in the wild. It uses real user datasets, and also times
>> migrations and warns when they are too slow for large deployments. It
>> started voting on gerrit in the last week.
>>
>> Turbo hipster uses zuul to learn about reviews in gerrit that it
>> should test. We run our own zuul instance, which talks to the
>> openstack.org zuul instance. This then hands out work to our pool of
>> testing workers. Another thing zuul does is it handles maintaining a
>> git repository for the workers to clone from.
>>
>> This is where things went wrong today. For reasons I can't currently
>> explain, the git repo on our zuul instance ended up in a bad state (it
>> had a patch merged to master which wasn't in fact merged upstream
>> yet). As this code is stock zuul from openstack-infra, I have a
>> concern this might be a bug that other zuul users will see as well.
>>
>> I've corrected the problem for now, and kicked off a recheck of any
>> patch with a -1 review score from turbo hipster in the last 24 hours.
>> I'll talk to the zuul maintainers tomorrow about the git problem and
>> see what we can learn.
>>
>> Thanks heaps for your patience.
>>
>> Michael
>>
>
> How do I interpret the warning and -1 from turbo-hipster on my patch here
> [1] with the logs here [2]?
>
> I'm inclined to just do 'recheck migrations' on this since this patch
> doesn't have anything to do with this -1 as far as I can tell.
>
> [1] https://review.openstack.org/#/c/64725/4/
> [2]
> https://ssl.rcbops.com/turbo_hipster/logviewer/?q=/turbo_hipster/results/64/64725/4/check/gate-real-db-upgrade_nova_mysql_user_001/5186e53/user_001.log
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Rackspace Australia