Open Stack

On 06/29/2014 03:25 PM, Ilya Shakhat wrote:
> Hi!
> 
> During last couple weeks there is an increasing demand on tracking
> 3rd-party CI statuses. We at Stackalytics decided to be in trend and (with
> some inspiration from Salvatore's proposal) implemented report that shows
> summary on external CI status. The initial version is available for Neutron
> - http://stackalytics.com/report/ci/neutron/7
> 
> The report shows summary of all CI jobs during specified period of time,
> including:
>  * stats of runs on merged patch sets:
>     - total number of runs
>     - success rate (success to total ratio)
>     - time of the latest run
>     - last test result
>   * stats for all patch sets (the same set as for merged)
>   * last test results for every merged patch set grouped by days (useful to
> see how different CIs correlate with each other and how often they run)
> 
> Under "merged patch set" report means "the last patch set in the merged
> change request", thus it is almost the same as the trunk code. CI
> configuration is taken from DriverLog's default data
> <https://git.openstack.org/cgit/stackforge/driverlog/tree/etc/default_data.json>.
> Standard Stackalytics screen is also available for CIs -
> http://stackalytics.com/?metric=ci, including votes breakdown and activity
> log.
> 
> Since this is the first version there are some open questions:
>  * Currently report shows results per CI id, but there are CIs that run
> tests against multiple drivers and this case is not supported. What would
> be more useful: to get stats for a driver or for CI?
>  * Most CIs run tests when patch set is posted. So even if change request
> is merged within selected time period corresponding CI results may be
> missing.
>  * Patterns for non-voting CIs need to be verified. For example Cisco CI
> now runs 5 jobs, but DriverLog data still contains old data.
> 
> Thanks,
> Ilya
> 
> 2014-06-16 17:52 GMT+04:00 Salvatore Orlando <sorlando at nicira.com>:
> 
>>
>> However, it would be great if we could start devising a solution for
>> having "health" reports from the various CI systems.
>> This report should report the following kind of information:
>> - timestamp of last run
>> - timestamp of last vote (a system might start job which then get aborted
>> for CI infra problems)
>> - % of success vs failures (not sure how important is that one but
>> provides a metric of comparison with upstream jenkins)
>> - % of disagreements with jenkins (this might allow us to quickly spot
>> those CI systems which are randomly -1'ing patches)
>>
>> The percentage metrics might be taken over a 48 hours or 7 days interval,
>> or both.
>> Does this idea sound reasonable?
>>
>>
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
Hi Ilya:

I look forward to hearing more about this dashboard and ensuring you or
someone else associated with this dashboard are available for questions
at the third party meeting tomorrow:
https://wiki.openstack.org/wiki/Meetings/ThirdParty

We missed you last week.

Thanks Ilya,
Anita.