Open Stack

----- Original Message -----
> From: "Michael Still" <mikal at stillhq.com>
> To: "OpenStack Development Mailing List (not for usage questions)" <openstack-dev at lists.openstack.org>
> Sent: Tuesday, January 7, 2014 5:53:01 PM
> Subject: Re: [openstack-dev] [nova] Bogus -1 scores from turbo hipster
> 
> Hi. Thanks for reaching out about this.
> 
> It seems this patch has now passed turbo hipster, so I am going to
> treat this as a more theoretical question than perhaps you intended. I
> should note though that Joshua Hesketh and I have been trying to read
> / triage every turbo hipster failure, but that has been hard this week
> because we're both at a conference.
> 
> The problem this patch faced is that we are having trouble defining
> what is a reasonable amount of time for a database migration to run
> for. Specifically:
> 
> 2014-01-07 14:59:32,012 [output] 205 -> 206...
> 2014-01-07 14:59:32,848 [heartbeat]
> 2014-01-07 15:00:02,848 [heartbeat]
> 2014-01-07 15:00:32,849 [heartbeat]
> 2014-01-07 15:00:39,197 [output] done
> 
> So applying migration 206 took slightly over a minute (67 seconds).
> Our historical data (mean + 2 standard deviations) says that this
> migration should take no more than 63 seconds. So this only just
> failed the test.

FWIW migrations 206 is a dead man walking: https://review.openstack.org/#/c/64893/

Regarding Turbo hipster in general though...

> 
> However, we know there are issues with our methodology -- we've tried
> normalizing for disk IO bandwidth and it hasn't worked out as well as
> we'd hoped. This week's plan is to try to use mysql performance schema
> instead, but we have to learn more about how it works first.

What if you adjusted your algorithm so that any given run has to fail two or three times. Perhaps even taking it a step further and making sure it fails in the same spots. When I've done performance testing in the past I had a baseline set of results which I then compare to a set of results for any given branch/tag. How large your set is depends on how many resources you've got... but the more the better. It sounds like you've got a good set of baseline data (which presumably you regenerate periodically). But having a larger set of data for each merge proposal could help here...

I really appreciate this sort of DB migration performance testing BTW. I've not had any trouble with it on my branches thus far... but if I do the information you provided in this thread is encouraging.

Also. What happens if you go on vacation? Is there a turbo hipster kill switch somewhere? Like it will quit giving out -1's if you aren't around and a trunk baseline run fails or something?

> 
> I apologise for this mis-vote.
> 
> Michael
> 
> On Wed, Jan 8, 2014 at 1:44 AM, Matt Riedemann
> <mriedem at linux.vnet.ibm.com> wrote:
> >
> >
> > On 12/30/2013 6:21 AM, Michael Still wrote:
> >>
> >> Hi.
> >>
> >> The purpose of this email to is apologise for some incorrect -1 review
> >> scores which turbo hipster sent out today. I think its important when
> >> a third party testing tool is new to not have flakey results as people
> >> learn to trust the tool, so I want to explain what happened here.
> >>
> >> Turbo hipster is a system which takes nova code reviews, and runs
> >> database upgrades against them to ensure that we can still upgrade for
> >> users in the wild. It uses real user datasets, and also times
> >> migrations and warns when they are too slow for large deployments. It
> >> started voting on gerrit in the last week.
> >>
> >> Turbo hipster uses zuul to learn about reviews in gerrit that it
> >> should test. We run our own zuul instance, which talks to the
> >> openstack.org zuul instance. This then hands out work to our pool of
> >> testing workers. Another thing zuul does is it handles maintaining a
> >> git repository for the workers to clone from.
> >>
> >> This is where things went wrong today. For reasons I can't currently
> >> explain, the git repo on our zuul instance ended up in a bad state (it
> >> had a patch merged to master which wasn't in fact merged upstream
> >> yet). As this code is stock zuul from openstack-infra, I have a
> >> concern this might be a bug that other zuul users will see as well.
> >>
> >> I've corrected the problem for now, and kicked off a recheck of any
> >> patch with a -1 review score from turbo hipster in the last 24 hours.
> >> I'll talk to the zuul maintainers tomorrow about the git problem and
> >> see what we can learn.
> >>
> >> Thanks heaps for your patience.
> >>
> >> Michael
> >>
> >
> > How do I interpret the warning and -1 from turbo-hipster on my patch here
> > [1] with the logs here [2]?
> >
> > I'm inclined to just do 'recheck migrations' on this since this patch
> > doesn't have anything to do with this -1 as far as I can tell.
> >
> > [1] https://review.openstack.org/#/c/64725/4/
> > [2]
> > https://ssl.rcbops.com/turbo_hipster/logviewer/?q=/turbo_hipster/results/64/64725/4/check/gate-real-db-upgrade_nova_mysql_user_001/5186e53/user_001.log
> >
> > --
> >
> > Thanks,
> >
> > Matt Riedemann
> >
> >
> > _______________________________________________
> > OpenStack-dev mailing list
> > OpenStack-dev at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> --
> Rackspace Australia
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>