[bifrost][ironic][kayobe][kolla][requirements] Kayobe CI breakage following ironic-lib release
Hello, I have been looking into CI failures that started impacting Kayobe today on the master branch, just as we're planning to branch stable/train this week (Kayobe is cycle trailing). Our seed jobs are failing because ironic doesn't start correctly [1]. The traceback shows that it is using code from the latest ironic-lib (3.0.0) which was released just last week [2] and dropped Python 2 compatibility. We're using Kolla Train images [3], but the latest kolla/centos-source-bifrost-deploy:train includes Bifrost 7.0.0 which fetches all its dependencies using Git master branches [4]. This only got changed later on stable/train [5], but there is no newer 7.x release. One of these dependencies is the requirements repo, which has just bumped the upper constraint for ironic-lib to 3.0.0 [6]. I believe this is what caused our jobs to start failing today. So I think we need: - a 7.0.1 release of Bifrost - an update of Kolla to use this release instead of 7.0.0 Would the ironic and kolla project teams be willing to help with this to fix our gate? Thanks a lot. [1] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackcd... [2] https://review.opendev.org/#/c/698500/ [3] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackcd... [4] https://opendev.org/openstack/bifrost/src/tag/7.0.0/playbooks/roles/bifrost-... [5] https://review.opendev.org/#/q/I3005fc59d6c5e1f5000874342f956977d9929907 [6] https://review.opendev.org/#/c/698981/
Pierre, Bifrost's train branch can be patched to explicitly set stable/train instead of master and then we can release 7.0.1. That seems reasonable to me. Is that what your hoping for? If not, it is going to be a little more unplanned work at the moment because bifrost needs more work for python3. -Julia On Wed, Dec 18, 2019 at 2:22 PM Pierre Riteau <pierre@stackhpc.com> wrote:
Hello,
I have been looking into CI failures that started impacting Kayobe today on the master branch, just as we're planning to branch stable/train this week (Kayobe is cycle trailing).
Our seed jobs are failing because ironic doesn't start correctly [1]. The traceback shows that it is using code from the latest ironic-lib (3.0.0) which was released just last week [2] and dropped Python 2 compatibility.
We're using Kolla Train images [3], but the latest kolla/centos-source-bifrost-deploy:train includes Bifrost 7.0.0 which fetches all its dependencies using Git master branches [4]. This only got changed later on stable/train [5], but there is no newer 7.x release.
One of these dependencies is the requirements repo, which has just bumped the upper constraint for ironic-lib to 3.0.0 [6]. I believe this is what caused our jobs to start failing today.
So I think we need:
- a 7.0.1 release of Bifrost - an update of Kolla to use this release instead of 7.0.0
Would the ironic and kolla project teams be willing to help with this to fix our gate? Thanks a lot.
[1] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackcd... [2] https://review.opendev.org/#/c/698500/ [3] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackcd... [4] https://opendev.org/openstack/bifrost/src/tag/7.0.0/playbooks/roles/bifrost-... [5] https://review.opendev.org/#/q/I3005fc59d6c5e1f5000874342f956977d9929907 [6] https://review.opendev.org/#/c/698981/
Hi Julia, Thanks for the quick response. The stable/train branch was already patched by Mark a few weeks ago [1] and I have just noticed he proposed a release of 7.1.0 last week [2], so we just need someone from the release team to approve it. I can then submit a Kolla patch to use the new tarball. [1] https://review.opendev.org/#/c/687820/ [2] https://review.opendev.org/#/c/698940/ On Wed, 18 Dec 2019 at 23:54, Julia Kreger <juliaashleykreger@gmail.com> wrote:
Pierre,
Bifrost's train branch can be patched to explicitly set stable/train instead of master and then we can release 7.0.1. That seems reasonable to me.
Is that what your hoping for? If not, it is going to be a little more unplanned work at the moment because bifrost needs more work for python3.
-Julia
On Wed, Dec 18, 2019 at 2:22 PM Pierre Riteau <pierre@stackhpc.com> wrote:
Hello,
I have been looking into CI failures that started impacting Kayobe today on the master branch, just as we're planning to branch stable/train this week (Kayobe is cycle trailing).
Our seed jobs are failing because ironic doesn't start correctly [1]. The traceback shows that it is using code from the latest ironic-lib (3.0.0) which was released just last week [2] and dropped Python 2 compatibility.
We're using Kolla Train images [3], but the latest kolla/centos-source-bifrost-deploy:train includes Bifrost 7.0.0 which fetches all its dependencies using Git master branches [4]. This only got changed later on stable/train [5], but there is no newer 7.x release.
One of these dependencies is the requirements repo, which has just bumped the upper constraint for ironic-lib to 3.0.0 [6]. I believe this is what caused our jobs to start failing today.
So I think we need:
- a 7.0.1 release of Bifrost - an update of Kolla to use this release instead of 7.0.0
Would the ironic and kolla project teams be willing to help with this to fix our gate? Thanks a lot.
[1] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackcd... [2] https://review.opendev.org/#/c/698500/ [3] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackcd... [4] https://opendev.org/openstack/bifrost/src/tag/7.0.0/playbooks/roles/bifrost-... [5] https://review.opendev.org/#/q/I3005fc59d6c5e1f5000874342f956977d9929907 [6] https://review.opendev.org/#/c/698981/
Ack, that works. Thanks! On Wed, Dec 18, 2019 at 3:04 PM Pierre Riteau <pierre@stackhpc.com> wrote:
Hi Julia,
Thanks for the quick response.
The stable/train branch was already patched by Mark a few weeks ago [1] and I have just noticed he proposed a release of 7.1.0 last week [2], so we just need someone from the release team to approve it. I can then submit a Kolla patch to use the new tarball.
[1] https://review.opendev.org/#/c/687820/ [2] https://review.opendev.org/#/c/698940/
On Wed, 18 Dec 2019 at 23:54, Julia Kreger <juliaashleykreger@gmail.com> wrote:
Pierre,
Bifrost's train branch can be patched to explicitly set stable/train instead of master and then we can release 7.0.1. That seems reasonable to me.
Is that what your hoping for? If not, it is going to be a little more unplanned work at the moment because bifrost needs more work for python3.
-Julia
On Wed, Dec 18, 2019 at 2:22 PM Pierre Riteau <pierre@stackhpc.com> wrote:
Hello,
I have been looking into CI failures that started impacting Kayobe today on the master branch, just as we're planning to branch stable/train this week (Kayobe is cycle trailing).
Our seed jobs are failing because ironic doesn't start correctly [1]. The traceback shows that it is using code from the latest ironic-lib (3.0.0) which was released just last week [2] and dropped Python 2 compatibility.
We're using Kolla Train images [3], but the latest kolla/centos-source-bifrost-deploy:train includes Bifrost 7.0.0 which fetches all its dependencies using Git master branches [4]. This only got changed later on stable/train [5], but there is no newer 7.x release.
One of these dependencies is the requirements repo, which has just bumped the upper constraint for ironic-lib to 3.0.0 [6]. I believe this is what caused our jobs to start failing today.
So I think we need:
- a 7.0.1 release of Bifrost - an update of Kolla to use this release instead of 7.0.0
Would the ironic and kolla project teams be willing to help with this to fix our gate? Thanks a lot.
[1] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackcd... [2] https://review.opendev.org/#/c/698500/ [3] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackcd... [4] https://opendev.org/openstack/bifrost/src/tag/7.0.0/playbooks/roles/bifrost-... [5] https://review.opendev.org/#/q/I3005fc59d6c5e1f5000874342f956977d9929907 [6] https://review.opendev.org/#/c/698981/
And I clearly didn't actually look at the branch status presently. Yeah, I'll submit a 7.0.1 release right now On Wed, Dec 18, 2019 at 2:54 PM Julia Kreger <juliaashleykreger@gmail.com> wrote:
Pierre,
Bifrost's train branch can be patched to explicitly set stable/train instead of master and then we can release 7.0.1. That seems reasonable to me.
Is that what your hoping for? If not, it is going to be a little more unplanned work at the moment because bifrost needs more work for python3.
-Julia
On Wed, Dec 18, 2019 at 2:22 PM Pierre Riteau <pierre@stackhpc.com> wrote:
Hello,
I have been looking into CI failures that started impacting Kayobe today on the master branch, just as we're planning to branch stable/train this week (Kayobe is cycle trailing).
Our seed jobs are failing because ironic doesn't start correctly [1]. The traceback shows that it is using code from the latest ironic-lib (3.0.0) which was released just last week [2] and dropped Python 2 compatibility.
We're using Kolla Train images [3], but the latest kolla/centos-source-bifrost-deploy:train includes Bifrost 7.0.0 which fetches all its dependencies using Git master branches [4]. This only got changed later on stable/train [5], but there is no newer 7.x release.
One of these dependencies is the requirements repo, which has just bumped the upper constraint for ironic-lib to 3.0.0 [6]. I believe this is what caused our jobs to start failing today.
So I think we need:
- a 7.0.1 release of Bifrost - an update of Kolla to use this release instead of 7.0.0
Would the ironic and kolla project teams be willing to help with this to fix our gate? Thanks a lot.
[1] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackcd... [2] https://review.opendev.org/#/c/698500/ [3] https://dafcf7323a7c828b67e0-814491f7a8e968b50eeaf14b0c7f115f.ssl.cf1.rackcd... [4] https://opendev.org/openstack/bifrost/src/tag/7.0.0/playbooks/roles/bifrost-... [5] https://review.opendev.org/#/q/I3005fc59d6c5e1f5000874342f956977d9929907 [6] https://review.opendev.org/#/c/698981/
participants (2)
-
Julia Kreger
-
Pierre Riteau