[openstack-dev] [Fuel] Current Fuel CI status and issues related to CentOS7 migration

Aleksandra Fedorova afedorova at mirantis.com
Mon Dec 7 15:29:54 UTC 2015

Hi everyone,

let me describe CI status and changes in CI caused by CentOS7
migration. Please read carefully as your local and custom test
environments might be affected as well.

* CentOS7 in docker with aufs on Ubuntu Trusty host is not stable

Containers of such type are required for Fuel ISO building process,
but this configuration is extremely sensitive to the versions of
kernel and docker packages.

ISO builds fail on 3.16 kernel or docker 1.6.1, see [1].

The current known-to-work configuration is:

      kernel 3.19
      docker 1.6.2

We have updated main build slaves to this configuration, so now
community build with CentOS7 is available at


* fuel-web is not compatible with python2.6

verify-fuel-web job, which was used to test fuel-web against python2.6
is now disabled. Work on cleaning up other jobs is currently in
progress in [2]

* fuel-library deployment tests have been refactored and now use
perestroika build scripts, see [3], to build fuel-library packages.

This is a huge step forward for Fuel CI, as we simplified job
configurations and configured deployment tests to use common template
[4]. This template can also be used for deployment tests for fuel-ostf
and other Fuel repositories.

* regression in upstream Ubuntu kernel 3.13.0-72 [5]

This issue is not related to CentOS7 merge but appeared in the same
timeframe and affected the testing process.

Deployment tests use external Ubuntu repository to install base OS on
OpenStack nodes. In nightly tests we use daily snapshots of
archive.ubuntu.com. You can find them at

It appears that on 3rd of December new kernel was uploaded to
trusty-proposed repo which caused certain issues with LVM and leads to
Smoke failures.

For now to unblock CI we've pinned Ubuntu mirror used in tests to
known "good" snapshot with 3.13.0-71. See [6].

Please use this snapshot when running local or custom tests


or disable trusty-proposed repository till issue is resolved.

[1] https://bugs.launchpad.net/mos/+bug/1522788
[2] https://bugs.launchpad.net/fuel/+bug/1523514
[3] https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/builders/build-pkgs.sh#L59
[4] https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/build-and-test-pkgs.yaml
[5] https://bugs.launchpad.net/fuel/+bug/1523092
[6] https://github.com/fuel-infra/jenkins-jobs/commit/6745eef4964a8c365178953f5f5f45524bfeeb19

Aleksandra Fedorova
Fuel CI Team Lead

More information about the OpenStack-dev mailing list