[openstack-dev] [Fuel] Current Fuel CI status and issues related to CentOS7 migration

Aleksandra Fedorova afedorova at mirantis.com
Tue Dec 8 10:42:38 UTC 2015


Hi, Mike,

yes, it is about build environment only.

On Fuel node we use standard kernel and docker packages from CentOS
repositories and there are no issues I am aware of.

On Tue, Dec 8, 2015 at 11:03 AM, Mike Scherbakov
<mscherbakov at mirantis.com> wrote:
> Aleksandra,
> Thanks for detailed status update on Infra/Tests side.
>
> I'd like to clarify - are you talking only about build env? That we need to
> have certain docker + kernel versions?
> Or you are talking about using certain docker+kernel versions in the ISO?
>
> Thanks,
>
> On Mon, Dec 7, 2015 at 8:12 AM Sergii Golovatiuk <sgolovatiuk at mirantis.com>
> wrote:
>>
>> Hi,
>>
>>
>> On Mon, Dec 7, 2015 at 4:29 PM, Aleksandra Fedorova
>> <afedorova at mirantis.com> wrote:
>>>
>>> Hi everyone,
>>>
>>> let me describe CI status and changes in CI caused by CentOS7
>>> migration. Please read carefully as your local and custom test
>>> environments might be affected as well.
>>>
>>>
>>> * CentOS7 in docker with aufs on Ubuntu Trusty host is not stable
>>>
>>> Containers of such type are required for Fuel ISO building process,
>>> but this configuration is extremely sensitive to the versions of
>>> kernel and docker packages.
>>>
>>> ISO builds fail on 3.16 kernel or docker 1.6.1, see [1].
>>>
>>> The current known-to-work configuration is:
>>>
>>>       kernel 3.19
>>>       docker 1.6.2
>>>
>>> We have updated main build slaves to this configuration, so now
>>> community build with CentOS7 is available at
>>>
>>>    https://ci.fuel-infra.org/view/ISO/
>>
>>
>> There are some plans to remove docker from master node completely. I think
>> we should force that action.
>>>
>>>
>>>
>>> * fuel-web is not compatible with python2.6
>>>
>>> verify-fuel-web job, which was used to test fuel-web against python2.6
>>> is now disabled. Work on cleaning up other jobs is currently in
>>> progress in [2]
>>>
>>> * fuel-library deployment tests have been refactored and now use
>>> perestroika build scripts, see [3], to build fuel-library packages.
>>>
>>> This is a huge step forward for Fuel CI, as we simplified job
>>> configurations and configured deployment tests to use common template
>>> [4]. This template can also be used for deployment tests for fuel-ostf
>>> and other Fuel repositories.
>>>
>>> * regression in upstream Ubuntu kernel 3.13.0-72 [5]
>>>
>>> This issue is not related to CentOS7 merge but appeared in the same
>>> timeframe and affected the testing process.
>>>
>>> Deployment tests use external Ubuntu repository to install base OS on
>>> OpenStack nodes. In nightly tests we use daily snapshots of
>>> archive.ubuntu.com. You can find them at
>>> http://mirror.fuel-infra.org/pkgs/
>>>
>>> It appears that on 3rd of December new kernel was uploaded to
>>> trusty-proposed repo which caused certain issues with LVM and leads to
>>> Smoke failures.
>>>
>>> For now to unblock CI we've pinned Ubuntu mirror used in tests to
>>> known "good" snapshot with 3.13.0-71. See [6].
>>>
>>> Please use this snapshot when running local or custom tests
>>>
>>>   http://mirror.fuel-infra.org/pkgs/ubuntu-2015-12-02-170158/
>>>
>>> or disable trusty-proposed repository till issue is resolved.
>>
>>
>> It's very dangerous to disable proposed, as it will break client's
>> deployments when package is moved from proposed to updates. There is a
>> standard process to block the package flow from proposed to updates, so
>> let's use it
>>
>>>
>>>
>>>
>>> [1] https://bugs.launchpad.net/mos/+bug/1522788
>>> [2] https://bugs.launchpad.net/fuel/+bug/1523514
>>> [3]
>>> https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/builders/build-pkgs.sh#L59
>>> [4]
>>> https://github.com/fuel-infra/jenkins-jobs/blob/a15c63461bddf857e0548d8b861bd4154e5f8f3a/servers/fuel-ci/build-and-test-pkgs.yaml
>>> [5] https://bugs.launchpad.net/fuel/+bug/1523092
>>> [6]
>>> https://github.com/fuel-infra/jenkins-jobs/commit/6745eef4964a8c365178953f5f5f45524bfeeb19
>>>
>>> --
>>> Aleksandra Fedorova
>>> Fuel CI Team Lead
>>> bookwar
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> --
> Mike Scherbakov
> #mihgen
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Aleksandra Fedorova
CI Team Lead
bookwar



More information about the OpenStack-dev mailing list