Hi everyone,
let me describe CI status and changes in CI caused by CentOS7
migration. Please read carefully as your local and custom test
environments might be affected as well.
* CentOS7 in docker with aufs on Ubuntu Trusty host is not stable
Containers of such type are required for Fuel ISO building process,
but this configuration is extremely sensitive to the versions of
kernel and docker packages.
ISO builds fail on 3.16 kernel or docker 1.6.1, see [1].
The current known-to-work configuration is:
kernel 3.19
docker 1.6.2
We have updated main build slaves to this configuration, so now
community build with CentOS7 is available at
https://ci.fuel-infra.org/view/ISO/
* fuel-web is not compatible with python2.6
verify-fuel-web job, which was used to test fuel-web against python2.6
is now disabled. Work on cleaning up other jobs is currently in
progress in [2]
* fuel-library deployment tests have been refactored and now use
perestroika build scripts, see [3], to build fuel-library packages.
This is a huge step forward for Fuel CI, as we simplified job
configurations and configured deployment tests to use common template
[4]. This template can also be used for deployment tests for fuel-ostf
and other Fuel repositories.
* regression in upstream Ubuntu kernel 3.13.0-72 [5]
This issue is not related to CentOS7 merge but appeared in the same
timeframe and affected the testing process.
Deployment tests use external Ubuntu repository to install base OS on
OpenStack nodes. In nightly tests we use daily snapshots of
archive.ubuntu.com. You can find them at
http://mirror.fuel-infra.org/pkgs/
It appears that on 3rd of December new kernel was uploaded to
trusty-proposed repo which caused certain issues with LVM and leads to
Smoke failures.
For now to unblock CI we've pinned Ubuntu mirror used in tests to
known "good" snapshot with 3.13.0-71. See [6].
Please use this snapshot when running local or custom tests
http://mirror.fuel-infra.org/pkgs/ubuntu-2015-12-02-170158/
or disable trusty-proposed repository till issue is resolved.
[1] https://bugs.launchpad.net/mos/+bug/1522788
[2] https://bugs.launchpad.net/fuel/+bug/1523514
[3] https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/builders/build-pkgs.sh#L59
[4] https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/build-and-test-pkgs.yaml
[5] https://bugs.launchpad.net/fuel/+bug/1523092
[6] https://github.com/fuel-infra/jenkins-jobs/commit/6745eef4964a8c365178953f5f5f45524bfeeb19
--
Aleksandra Fedorova
Fuel CI Team Lead
bookwar