Open Stack

Howdy!

I've been operating a shellci for a while now and overall it is very smooth.

The main new feature now is to automatically retrigger events that neither
definitely succeed (exit status 100) nor definitely fail (exit status 101).
In this case the CI will vote "0" with the logs and then automatically
schedule a new test, up to a max of 5 tests, until it can conclude +1 or -1.

This seems to work well. On my setup I often see tests fail due to network
problems (e.g. timeout during a Git checkout) and automatic retries filer
away this class of error very neatly, ultimately casting the +1/-1 votes
based purely on the relatively reliable contents of tempest.log. (Thanks
Jim Gray for pointing out that detectable intermittent bugs are really easy
to deal with http://www.hpl.hp.com/techreports/tandem/TR-85.7.pdf).

[The network problems themselves are a little suspicious but for now I see
them as a blessing that helps me make shellci robust and I don't worry too
much about the root cause.]

I'm running 5 parallel builds at the moment. I've tried ramping that up to
10 or 20 but then I start to see Vagrant/VirtualBox startup errors. Again,
I'm using this as a test case for the automatic retries, and not digging
too deep yet.

I'd like to be able to track the results via RSS feeds. Can I do this via
openstack.org or should I support that directly in shellci?

disk space will become an issue soon. Currently I'm running around 500
builds per day and each one generates around 8MB of logs. Compression and
redundancy-avoidance may go a long way towards reducing his problem,
however it seems a pity to have a "scarcity" mentality when it comes to
logging (I'd prefer to do more rather than less). Any bright ideas for
conveniently archiving some terabytes of logs?

Overall I'd say that shellci is quite reliable now. The main reason I don't
recommend it to others yet is that I'm a bit too busy to provide support
with setting it up this week.

Cheers,
-Luke
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140710/be26ab59/attachment.html>