<div dir="ltr">Aleksandra,<div>Thanks for detailed status update on Infra/Tests side.</div><div><br></div><div>I'd like to clarify - are you talking only about build env? That we need to have certain docker + kernel versions?</div><div>Or you are talking about using certain docker+kernel versions in the ISO?</div><div><br></div><div>Thanks,</div></div><br><div class="gmail_quote"><div dir="ltr">On Mon, Dec 7, 2015 at 8:12 AM Sergii Golovatiuk <<a href="mailto:sgolovatiuk@mirantis.com">sgolovatiuk@mirantis.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div class="gmail_extra"><br>
<br><div class="gmail_quote"></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Mon, Dec 7, 2015 at 4:29 PM, Aleksandra Fedorova <span dir="ltr"><<a href="mailto:afedorova@mirantis.com" target="_blank">afedorova@mirantis.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi everyone,<br>
<br>
let me describe CI status and changes in CI caused by CentOS7<br>
migration. Please read carefully as your local and custom test<br>
environments might be affected as well.<br>
<br>
<br>
* CentOS7 in docker with aufs on Ubuntu Trusty host is not stable<br>
<br>
Containers of such type are required for Fuel ISO building process,<br>
but this configuration is extremely sensitive to the versions of<br>
kernel and docker packages.<br>
<br>
ISO builds fail on 3.16 kernel or docker 1.6.1, see [1].<br>
<br>
The current known-to-work configuration is:<br>
<br>
kernel 3.19<br>
docker 1.6.2<br>
<br>
We have updated main build slaves to this configuration, so now<br>
community build with CentOS7 is available at<br>
<br>
<a href="https://ci.fuel-infra.org/view/ISO/" rel="noreferrer" target="_blank">https://ci.fuel-infra.org/view/ISO/</a></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>There are some plans to remove docker from master node completely. I think we should force that action. </div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><br>
<br>
* fuel-web is not compatible with python2.6<br>
<br>
verify-fuel-web job, which was used to test fuel-web against python2.6<br>
is now disabled. Work on cleaning up other jobs is currently in<br>
progress in [2]<br>
<br>
* fuel-library deployment tests have been refactored and now use<br>
perestroika build scripts, see [3], to build fuel-library packages.<br>
<br>
This is a huge step forward for Fuel CI, as we simplified job<br>
configurations and configured deployment tests to use common template<br>
[4]. This template can also be used for deployment tests for fuel-ostf<br>
and other Fuel repositories.<br>
<br>
* regression in upstream Ubuntu kernel 3.13.0-72 [5]<br>
<br>
This issue is not related to CentOS7 merge but appeared in the same<br>
timeframe and affected the testing process.<br>
<br>
Deployment tests use external Ubuntu repository to install base OS on<br>
OpenStack nodes. In nightly tests we use daily snapshots of<br>
<a href="http://archive.ubuntu.com" rel="noreferrer" target="_blank">archive.ubuntu.com</a>. You can find them at<br>
<a href="http://mirror.fuel-infra.org/pkgs/" rel="noreferrer" target="_blank">http://mirror.fuel-infra.org/pkgs/</a><br>
<br>
It appears that on 3rd of December new kernel was uploaded to<br>
trusty-proposed repo which caused certain issues with LVM and leads to<br>
Smoke failures.<br>
<br>
For now to unblock CI we've pinned Ubuntu mirror used in tests to<br>
known "good" snapshot with 3.13.0-71. See [6].<br>
<br>
Please use this snapshot when running local or custom tests<br>
<br>
<a href="http://mirror.fuel-infra.org/pkgs/ubuntu-2015-12-02-170158/" rel="noreferrer" target="_blank">http://mirror.fuel-infra.org/pkgs/ubuntu-2015-12-02-170158/</a><br>
<br>
or disable trusty-proposed repository till issue is resolved.<br></blockquote><div><br></div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>It's very dangerous to disable proposed, as it will break client's deployments when package is moved from proposed to updates. There is a standard process to block the package flow from proposed to updates, so let's use it</div></div></div></div><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
[1] <a href="https://bugs.launchpad.net/mos/+bug/1522788" rel="noreferrer" target="_blank">https://bugs.launchpad.net/mos/+bug/1522788</a><br>
[2] <a href="https://bugs.launchpad.net/fuel/+bug/1523514" rel="noreferrer" target="_blank">https://bugs.launchpad.net/fuel/+bug/1523514</a><br>
[3] <a href="https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/builders/build-pkgs.sh#L59" rel="noreferrer" target="_blank">https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/builders/build-pkgs.sh#L59</a><br>
[4] <a href="https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/build-and-test-pkgs.yaml" rel="noreferrer" target="_blank">https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/build-and-test-pkgs.yaml</a><br>
[5] <a href="https://bugs.launchpad.net/fuel/+bug/1523092" rel="noreferrer" target="_blank">https://bugs.launchpad.net/fuel/+bug/1523092</a><br>
[6] <a href="https://github.com/fuel-infra/jenkins-jobs/commit/6745eef4964a8c365178953f5f5f45524bfeeb19" rel="noreferrer" target="_blank">https://github.com/fuel-infra/jenkins-jobs/commit/6745eef4964a8c365178953f5f5f45524bfeeb19</a><br>
<span><font color="#888888"><br>
--<br>
Aleksandra Fedorova<br>
Fuel CI Team Lead<br>
bookwar<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</font></span></blockquote></div></div></div>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><div dir="ltr">-- <br></div><div dir="ltr">Mike Scherbakov<br>#mihgen</div>