Open Stack

> this is a reflection of the discussion I just had on #openstack-infra; it's 
> about (re-)using the central CI infrastructure for our Open-Source DRBD 
> driver too.
As this is done now, I'd like to discuss a few points.


First of all -- thanks to everybody involved; if you happen to find someone 
(Europe is a bad timezone for Openstack, I guess), people on IRC are *very*
helpful.


Secondly,
>   «Yup, so the two things I will start with is that multinode testing is 
>    still really rudimentary, we only just got tempest sort of working with 
>    it. ... »
I didn't follow all the conversations on the ML - is there an update for 
multi-node testing?
What's the policy? "Good idea, but not needed" or "A must-have"?

Especially for distributed/replicated storage drivers (like DRBD ;) I guess 
it would make sense to have some.


Third point, mostly of interest for other driver maintainers: we're 
following the various CI runs, and sporadically check failures to see 
whether we can improve our driver or the devstack installation script.
For that we use "Kibana" on http://logstash.openstack.org, with a filter 
string of

    project:"openstack/cinder" AND build_status:"FAILURE" AND
    build_name:"check-tempest-dsvm-full-drbd-devstack-nv" AND "Detailed logs"

This shows the initial line of the failed devstack runs (the one including 
the line with the log URL), which we can then paste into a script that 
fetches the (for us) relevant files for further analysis.
The newest failures are already at the top.

Another nice feature is using the existing Graphite installation to get 
a visual display of the success/failure ratio over time[1]; here we can see 
the impact of individual changes, eg. on June 19th we diagnosed (and fixed) 
an udev/blkid race with the kernel-attach, since then the number of 
failures has clearly gone down. I just pushed another patch that should us 
bring even more "into the green area" ;)

One more idea is to watch http://status.openstack.org/zuul/? with a filter 
string of "Cinder", and to open reviews that will finish soon, so that 
"current" behaviour can be easily checked.


I hope that this helps other people a bit.


Regards,

Phil



Ad 1:
http://graphite.openstack.org/render/?width=600&height=344&_salt=1434709688.361&from=-7days&title=DRBD%20Cinder%2FDevstack%20stats&colorList=red%2Cgreen%2Cblue&target=stats_counts.zuul.pipeline.check.job.check-tempest-dsvm-full-drbd-devstack-nv.FAILURE&target=stats_counts.zuul.pipeline.check.job.check-tempest-dsvm-full-drbd-devstack-nv.SUCCESS

-- 
: Ing. Philipp Marek
: LINBIT | Your Way to High Availability
: DRBD/HA support and consulting                 http://www.linbit.com :

DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.