Open Stack

On 03/17/2017 09:39 AM, Andrea Frittoli wrote:
> 
> 
> On Fri, Mar 17, 2017 at 1:03 PM Sean Dague <sean at dague.net
> <mailto:sean at dague.net>> wrote:
> 
>     On 03/17/2017 08:27 AM, Jordan Pittier wrote:
>     > The patch that reduced the number of Tempest Scenarios we run in every
>     > job and also reduce the test run concurrency [0] was merged 13
>     days ago.
>     > Since, the situation (i.e the high number of false negative job
>     results)
>     > has not improved significantly. We need to keep looking
>     collectively at
>     > this.
> 
>     While the situation hasn't completely cleared out -
>     http://tinyurl.com/mdmdxlk - since we've merged this we've not seen that
>     job go over 25% failure rate in the gate, which it was regularly
>     crossing in the prior 2 week period. That does feel like progress. 
> 
>  
> I agree the situation improved a bit, but there are still too many failures.
> There is a peak of failures on Mar 12th in the graph, I remember looking
> at it briefly, as it was on a Sunday - and then by Monday it was back to
> normal. It's not clear yet to me what caused / fixed that peak. The mysql
> revert was merged on March 15th, which is too late to explain the change.

Well, also realize that graph doesn't have error bars on it, and is 12
hour rolling averaging. So if you end up with only 5 changes in 12 hours
going through, and 1 fails it's going to hit 20%. It's actually a
totally non ideal way to handle this data, but it's about as good as you
can get with graphite.

	-Sean

-- 
Sean Dague
http://dague.net