Open Stack

On 04/12/14 13:37, Dan Prince wrote:
> On Thu, 2014-12-04 at 11:51 +0000, Derek Higgins wrote:
>> A month since my last update, sorry my bad
>>
>> since the last email we've had 5 incidents causing ci failures
>>
>> 26/11/2014 : Lots of ubuntu jobs failed over 24 hours (maybe half)
>> - We seem to suffer any time an ubuntu mirror isn't in sync causing hash
>> mismatch errors. For now I've pinned DNS on our proxy to a specific
>> server so we stop DNS round robining
> 
> This sound fine to me. I personally like the model where you pin to a
> specific mirror, perhaps one that is geographically closer to your
> datacenter. This also makes Squid caching (in the rack) happier in some
> cases.
> 
>>
>> 21/11/2014 : All tripleo jobs failed for about 16 hours
>> - Neutron started asserting that local_ip be set to a valid ip address,
>> on the seed we had been leaving it blank
>> - Cinder moved to using  oslo.concurreny which in turn requires that
>> lock_path be set, we are now setting it
> 
> 
> Thinking about how we might catch these ahead of time with our limited
> resources ATM. These sorts of failures all seem related to configuration
> and or requirements changes. I wonder if we were to selectively
> (automatically) run check experimental jobs on all reviews with
> associated tickets which have either doc changes or modify
> requirements.txt. Probably a bit of work to pull this off but if we had
> a report containing these results "coming down the pike" we might be
> able to catch them ahead of time.
Yup, this sounds like it could be beneficial, alternatively if we soon
have the capacity to run on more projects (capacity is increasing) we'll
be running on all reviews and we'll be able to generate the report your
talking about, either way we should do something like this soon.

> 
> 
>>
>> 8/11/2014 : All fedora tripleo jobs failed for about 60 hours (over a
>> weekend)
>> - A url being accessed on  https://bzr.linuxfoundation.org is no longer
>> available, we removed the dependency
>>
>> 7/11/2014 : All tripleo tests failed for about 24 hours
>> - Options were removed from nova.conf that had been deprecated (although
>> no deprecation warnings were being reported), we were still using these
>> in tripleo
>>
>> as always more details can be found here
>> https://etherpad.openstack.org/p/tripleo-ci-breakages
> 
> Thanks for sending this out! Very useful.
no problem
> 
> Dan
> 
>>
>> thanks,
>> Derek.
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>