[openstack-dev] [Fuel] Current Fuel CI status and issues related to CentOS7 migration

Mike Scherbakov mscherbakov at mirantis.com
Tue Dec 8 08:03:28 UTC 2015


Aleksandra,
Thanks for detailed status update on Infra/Tests side.

I'd like to clarify - are you talking only about build env? That we need to
have certain docker + kernel versions?
Or you are talking about using certain docker+kernel versions in the ISO?

Thanks,

On Mon, Dec 7, 2015 at 8:12 AM Sergii Golovatiuk <sgolovatiuk at mirantis.com>
wrote:

> Hi,
>
>
> On Mon, Dec 7, 2015 at 4:29 PM, Aleksandra Fedorova <
> afedorova at mirantis.com> wrote:
>
>> Hi everyone,
>>
>> let me describe CI status and changes in CI caused by CentOS7
>> migration. Please read carefully as your local and custom test
>> environments might be affected as well.
>>
>>
>> * CentOS7 in docker with aufs on Ubuntu Trusty host is not stable
>>
>> Containers of such type are required for Fuel ISO building process,
>> but this configuration is extremely sensitive to the versions of
>> kernel and docker packages.
>>
>> ISO builds fail on 3.16 kernel or docker 1.6.1, see [1].
>>
>> The current known-to-work configuration is:
>>
>>       kernel 3.19
>>       docker 1.6.2
>>
>> We have updated main build slaves to this configuration, so now
>> community build with CentOS7 is available at
>>
>>    https://ci.fuel-infra.org/view/ISO/
>
>
> There are some plans to remove docker from master node completely. I think
> we should force that action.
>
>>
>>
>> * fuel-web is not compatible with python2.6
>>
>> verify-fuel-web job, which was used to test fuel-web against python2.6
>> is now disabled. Work on cleaning up other jobs is currently in
>> progress in [2]
>>
>> * fuel-library deployment tests have been refactored and now use
>> perestroika build scripts, see [3], to build fuel-library packages.
>>
>> This is a huge step forward for Fuel CI, as we simplified job
>> configurations and configured deployment tests to use common template
>> [4]. This template can also be used for deployment tests for fuel-ostf
>> and other Fuel repositories.
>>
>> * regression in upstream Ubuntu kernel 3.13.0-72 [5]
>>
>> This issue is not related to CentOS7 merge but appeared in the same
>> timeframe and affected the testing process.
>>
>> Deployment tests use external Ubuntu repository to install base OS on
>> OpenStack nodes. In nightly tests we use daily snapshots of
>> archive.ubuntu.com. You can find them at
>> http://mirror.fuel-infra.org/pkgs/
>>
>> It appears that on 3rd of December new kernel was uploaded to
>> trusty-proposed repo which caused certain issues with LVM and leads to
>> Smoke failures.
>>
>> For now to unblock CI we've pinned Ubuntu mirror used in tests to
>> known "good" snapshot with 3.13.0-71. See [6].
>>
>> Please use this snapshot when running local or custom tests
>>
>>   http://mirror.fuel-infra.org/pkgs/ubuntu-2015-12-02-170158/
>>
>> or disable trusty-proposed repository till issue is resolved.
>>
>
> It's very dangerous to disable proposed, as it will break client's
> deployments when package is moved from proposed to updates. There is a
> standard process to block the package flow from proposed to updates, so
> let's use it
>
>
>>
>>
>> [1] https://bugs.launchpad.net/mos/+bug/1522788
>> [2] https://bugs.launchpad.net/fuel/+bug/1523514
>> [3]
>> https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/builders/build-pkgs.sh#L59
>> [4]
>> https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/build-and-test-pkgs.yaml
>> [5] https://bugs.launchpad.net/fuel/+bug/1523092
>> [6]
>> https://github.com/fuel-infra/jenkins-jobs/commit/6745eef4964a8c365178953f5f5f45524bfeeb19
>>
>> --
>> Aleksandra Fedorova
>> Fuel CI Team Lead
>> bookwar
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-- 
Mike Scherbakov
#mihgen
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151208/3c4a04b0/attachment.html>


More information about the OpenStack-dev mailing list