[openstack-dev] [tripleo] Help needed on debugging upgrade jobs on Pike

Emilien Macchi emilien at redhat.com
Tue Nov 7 00:34:15 UTC 2017


Thanks folks :-) you rock!

On Mon, Nov 6, 2017 at 5:05 AM, Jiří Stránský <jistr at redhat.com> wrote:
> On 6.11.2017 11:17, Jiří Stránský wrote:
>>
>> On 6.11.2017 10:52, Marios Andreou wrote:
>>>
>>> On Mon, Nov 6, 2017 at 11:09 AM, Marius Cornea <mariusc at redhat.com>
>>> wrote:
>>>
>>>> On Sat, Nov 4, 2017 at 2:27 AM, Emilien Macchi <emilien at redhat.com>
>>>> wrote:
>>>>>
>>>>> Since we've got promotion, we can now properly test upgrades from ocata
>>>>
>>>> to pike.
>>>>>
>>>>> It's now failing for various reasons, as you can see on:
>>>>> https://review.openstack.org/#/c/500625/
>>>>>
>>>>> I haven't filled bug yet but this is the kind of thing I see now:
>>>>>
>>>>> http://logs.openstack.org/25/500625/20/check/legacy-tripleo-ci-centos-7-
>>>>
>>>> scenario002-multinode-oooq-container-upgrades/62e7f14/
>>>> logs/undercloud/home/zuul/overcloud_upgrade_console.log.
>>>> txt.gz#_2017-11-04_00_14_17
>>>>
>>>> I think this is related to https://review.openstack.org/#/c/510577/
>>>> which introduced running os-net-config during the major upgrade
>>>> composable step. In case of environments without network isolation
>>>> /etc/os-net-config/config.json doesn't exist so the os-net-config
>>>> command fails. I filed https://bugs.launchpad.net/tripleo/+bug/1730328
>>>> to keep track of it.
>>>>
>>>>
>>> heh, beat me to it :) I was about to file that. Indeed from logs @ [0]
>>> you
>>> can see the step3 ansible-playbook failing for
>>>
>>> https://github.com/openstack/tripleo-heat-templates/blob/e463ca15fb2189fde7e7e2de136cfb2303d3171f/puppet/services/tripleo-packages.yaml#L56-L64
>>>
>>> I had a poke at one of the other jobs too since there are apparently
>>> multiple issues - I found a different one
>>> for legacy-tripleo-ci-centos-7-containers-multinode-upgrades and filed
>>> https://bugs.launchpad.net/tripleo/+bug/1730349 for that. It looks like
>>> all
>>> the upgrade_tasks pass there but then fails on docker-puppet
>>
>>
>> I'm not sure if it's related to that ^ error in particular
>
>
> Yea the backport [2] seems to have fixed that issue. The upgrade now
> completed successfully, but the job failed on validation. I've +A'd the
> backport as it gets us closer to green.
>
>
>> , but since we
>> landed deploy/upgrade scenario separation [1], the upgrade job on Pike
>> effectively started testing non-pacemaker to pacemaker upgrade, which
>> won't work. Due to a chicken-and-egg issue with landing related patches
>> we could not set the dependencies properly. There's a patch fixing this
>> issue and making the Pike upgrade pacemaker-to-pacemaker [2]. This may
>> not solve all the issues, but i think we need it merged to at least have
>> a chance at a green result.
>>
>>>
>>> [0]
>>>
>>> http://logs.openstack.org/25/500625/20/check/legacy-tripleo-ci-centos-7-scenario002-multinode-oooq-container-upgrades/62e7f14/logs/subnode-2/var/log/messages.txt.gz#_Nov__4_00_13_55
>>
>> [1] https://review.openstack.org/#/c/500552
>> [2] https://review.openstack.org/#/c/512305
>>>
>>>
>>> thanks,
>>>
>>> marios
>>>
>>>
>>>> I'm requesting some help from the upgrades squad, if they already saw
>>>>>
>>>>> the failures, etc. It would be great to have the jobs passing at some
>>>>> point, now the framework is in place and we had promotion.
>>>>>
>>>>> Thanks,
>>>>
>>>>
>>>> --
>>>>>
>>>>> Emilien Macchi
>>>>
>>>>
>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Emilien Macchi



More information about the OpenStack-dev mailing list