Open Stack

On 06/13/2018 07:50 PM, Emilien Macchi wrote:
> TL;DR: gate queue was 25h+, we put all patches from gate on standby, do 
> not restore/recheck until further announcement.
> 
> We recently enabled the containerized undercloud for multinode jobs and 
> we believe this was a bit premature as the container download process 
> wasn't optimized so it's not pulling the mirrors for the same containers 
> multiple times yet.
> It caused the job runtime to increase and probably the load on docker.io 
> <http://docker.io> mirrors hosted by OpenStack Infra to be a bit slower 
> to provide the same containers multiple times. The time taken to prepare 
> containers on the undercloud and then for the overcloud caused the jobs 
> to randomly timeout therefore the gate to fail in a high amount of 
> times, so we decided to remove all jobs from the gate by abandoning the 
> patches temporarily (I have them in my browser and will restore when 
> things are stable again, please do not touch anything).
> 
> Steve Baker has been working on a series of patches that optimize the 
> way we prepare the containers but basically the workflow will be:
> - pull containers needed for the undercloud into a local registry, using 
> infra mirror if available
> - deploy the containerized undercloud
> - pull containers needed for the overcloud minus the ones already pulled 
> for the undercloud, using infra mirror if available
> - update containers on the overcloud
> - deploy the containerized undercloud

That sounds like a great improvement. Well done!

> With that process, we hope to reduce the runtime of the deployment and 
> therefore reduce the timeouts in the gate.
> To enable it, we need to land in that order: 
> https://review.openstack.org/#/c/571613/, 
> https://review.openstack.org/#/c/574485/, 
> https://review.openstack.org/#/c/571631/ and 
> https://review.openstack.org/#/c/568403.
> 
> In the meantime, we are disabling the containerized undercloud recently 
> enabled on all scenarios: https://review.openstack.org/#/c/575264/ for 
> mitigation with the hope to stabilize things until Steve's patches land.
> Hopefully, we can merge Steve's work tonight/tomorrow and re-enable the 
> containerized undercloud on scenarios after checking that we don't have 
> timeouts and reasonable deployment runtimes.
> 
> That's the plan we came with, if you have any question / feedback please 
> share it.
> -- 
> Emilien, Steve and Wes
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>