Potentially all the following repos should be fixed to avoid the same issue with their stable branches: - https://opendev.org/openstack/monasca-agent/ - https://opendev.org/openstack/release-test/src/branch/master/bindep.txt - https://opendev.org/openstack/training-labs Also notice that the following projects that doesn't have stable branches so I think they could be ignored: - https://opendev.org/openstack/reviewday/ - https://opendev.org/openstack/js-openstack-lib/ FYI the previous list is a filtered version of the results obtained by using: ``` beagle search --ignore-comments -f link --file '(^bindep.txt)' --repo-pattern "openstack/*" 'python-libvirt' https://opendev.org/openstack/js-openstack-lib/src/branch/master/bindep.txt#... : python-libvirt [platform:dpkg] https://opendev.org/openstack/monasca-agent/src/branch/master/bindep.txt#n46 : python-libvirt [platform:dpkg !platform:ubuntu-focal] https://opendev.org/openstack/release-test/src/branch/master/bindep.txt#n17 : python-libvirt [platform:dpkg] https://opendev.org/openstack/reviewday/src/branch/master/bindep.txt#n10 : python-libvirt [platform:dpkg] https://opendev.org/openstack/training-labs/src/branch/master/bindep.txt#n48 : python-libvirt [platform:dpkg] ``` Le ven. 13 nov. 2020 à 12:09, Herve Beraud <hberaud@redhat.com> a écrit :

Thanks for the heads up.

Notice that I just submitted a series of patches [1] to fix all masakari's stable branches, feel free to abandon these patches if they aren't needed anymore.

[1] https://review.opendev.org/#/q/topic:fix-libvirt-focal+(status:open+OR+statu...)

Le ven. 13 nov. 2020 à 11:57, Radosław Piliszek < radoslaw.piliszek@gmail.com> a écrit :

...

On Fri, Nov 13, 2020 at 11:35 AM Herve Beraud <hberaud@redhat.com> wrote:

...

During the release job for: - masakari-monitors 9.0.1 (ussuri) - masakari-monitors 8.0.2 (train) - masakari-monitors 7.0.1 (stein)

uploads to PyPI failed [1][2][3] with:

"No package matching 'python-libvirt' is available" [4][5][6]

It's due to our recent move to focal [7] where the package

'python-libvirt' is named ''python3-libvirt'.This package is pulled by bindep during the job execution.

...

Current status: Tag (9.0.1, 8.0.2, 7.0.1) was pushed OK No tarball upload No PyPI upload

Once the issue is fixed, a new version should be released for the three

stables branches.

...

Version 9.0.1, 8.0.2, 7.0.1 these versions will never be uploaded on

PyPi as the fix requires new changes in the project repo [8][9][10].

...

Also notice that on Victoria and Wallaby this bindep requirement have

been dropped during the migration testing on ubuntu focal [11].

...

[1]

https://zuul.opendev.org/t/openstack/build/63539177d44b4ea48d3fbfe39a638275/...

...

[2] https://zuul.opendev.org/t/openstack/build/778ace86ec8f4ef894d740d81e8f700f/... [3] https://zuul.opendev.org/t/openstack/build/919cfc7df1e0408e9d6af4a9e8ef28ee/... [4] http://lists.openstack.org/pipermail/release-job-failures/2020-November/0014... [5] http://lists.openstack.org/pipermail/release-job-failures/2020-November/0014... [6] http://lists.openstack.org/pipermail/release-job-failures/2020-November/0014... [7] http://lists.openstack.org/pipermail/openstack-discuss/2020-November/018585.... [8] https://opendev.org/openstack/masakari-monitors/src/branch/stable/stein/bind... [9] https://opendev.org/openstack/masakari-monitors/src/branch/stable/train/bind... [10] https://opendev.org/openstack/masakari-monitors/src/branch/stable/ussuri/bin... [11] https://opendev.org/openstack/masakari-monitors/commit/03ef3555888fdd0ea38b4...

Thanks, Hervé.

I will just drop this from bindep as it should not be there in the first place (bindep really meaning system-level packages not available from PyPI, usually because they are not Python packages).

-yoctozepto

-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

Le ven. 13 nov. 2020 à 13:17, Radosław Piliszek <radoslaw.piliszek@gmail.com> a écrit :

On Fri, Nov 13, 2020 at 12:09 PM Herve Beraud <hberaud@redhat.com> wrote:

...

Thanks for the heads up.

Notice that I just submitted a series of patches [1] to fix all

masakari's stable branches, feel free to abandon these patches if they aren't needed anymore.

...

[1]

https://review.opendev.org/#/q/topic:fix-libvirt-focal+(status:open+OR+statu...)

...

Ah, sorry, did not notice that. I just went with [1] removing the pkg to avoid the naming issue altogether.

No problem

My question is whether we could detect such issues more proactively?

Good question... I don't think that this kind of a check should be hosted on the release management side. Here our issue is more an environment issue, It's due to the fact that we decided to run the job `publish-openstack-artifacts` on ubuntu-focal, so to detect this kind of issue early I think that jobs on the projects side should rely on ubuntu-focal too. I'm not a Zuul expert and I don't know if it is possible to add a condition based on the branches (per example) to choose the right env (focal or not). By following this way I guess it could be possible to fail early on the project side, and it could allow you to fix this kind of issue well before proposing a new release. In other words, if projects's jobs env are aligned with release management our job we could catch them early. We have other scenarios to consider. These are when no patches haven't been merged since a while on projects side, and where a release is proposed, in this case even if environments are aligned, in this case we aren't able to detect this kind of issue, and they will continue to occur during publication, as in this case. It could be worth it to involve the infra team to see how to fail early. Thoughts?

[1] https://review.opendev.org/762637

-yoctozepto

-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

---- On Fri, 13 Nov 2020 06:45:54 -0600 Herve Beraud <hberaud@redhat.com> wrote ----

Le ven. 13 nov. 2020 à 13:17, Radosław Piliszek <radoslaw.piliszek@gmail.com> a écrit : On Fri, Nov 13, 2020 at 12:09 PM Herve Beraud <hberaud@redhat.com> wrote:

...

Thanks for the heads up.

Notice that I just submitted a series of patches [1] to fix all masakari's stable branches, feel free to abandon these patches if they aren't needed anymore.

[1] https://review.opendev.org/#/q/topic:fix-libvirt-focal+(status:open+OR+statu...)

Ah, sorry, did not notice that. I just went with [1] removing the pkg to avoid the naming issue altogether.

No problem

My question is whether we could detect such issues more proactively?

Good question... I don't think that this kind of a check should be hosted on the release management side.Here our issue is more an environment issue, It's due to the fact that we decided to run the job `publish-openstack-artifacts` on ubuntu-focal, so to detect this kind of issue early I think that jobs on the projects side should rely on ubuntu-focal too. I'm not a Zuul expert and I don't know if it is possible to add a condition based on the branches (per example) to choose the right env (focal or not). By following this way I guess it could be possible to fail early on the project side, and it could allow you to fix this kind of issue well before proposing a new release. In other words, if projects's jobs env are aligned with release management our job we could catch them early.

`publish-openstack-artifacts` job should not run for Focal for the stable gate and running it on Focal from Victoria onwards (like all other common jobs). -gmann

We have other scenarios to consider. These are when no patches haven't been merged since a while on projects side, and where a release is proposed, in this case even if environments are aligned, in this case we aren't able to detect this kind of issue, and they will continue to occur during publication, as in this case. It could be worth it to involve the infra team to see how to fail early. Thoughts?

[1] https://review.opendev.org/762637

-yoctozepto

-- Hervé BeraudSenior Software Engineer at Red Hatirc: hberaudhttps://github.com/4383/https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

To close this topic, masakari-monitors's refs have been reenqueued and successfully released on PyPi: - https://pypi.org/project/masakari-monitors/9.0.1/ - https://pypi.org/project/masakari-monitors/8.0.2/ - https://pypi.org/project/masakari-monitors/7.0.1/ Also notice that monasca-agent's transition to stein-em was successfully done https://review.opendev.org/#/c/762402/ The clean up of pti-python-tarball and python-branch-tarball helped us to fix the issue ( https://review.opendev.org/#/c/762699/ ). Thanks to everyone who helped us on this topic. Le ven. 13 nov. 2020 à 16:55, Herve Beraud <hberaud@redhat.com> a écrit :

Notice that this patch will face the same error too https://review.opendev.org/#/c/762402/2

I'm going to hold this one until a fix is merged on the project side. Don't forget to update the sha when the issue is fixed.

Le ven. 13 nov. 2020 à 16:04, Ghanshyam Mann <gmann@ghanshyammann.com> a écrit :

...

---- On Fri, 13 Nov 2020 06:45:54 -0600 Herve Beraud <hberaud@redhat.com> wrote ----

...

Le ven. 13 nov. 2020 à 13:17, Radosław Piliszek <

radoslaw.piliszek@gmail.com> a écrit :

...

On Fri, Nov 13, 2020 at 12:09 PM Herve Beraud <hberaud@redhat.com> wrote:

...

Thanks for the heads up.

Notice that I just submitted a series of patches [1] to fix all

masakari's stable branches, feel free to abandon these patches if they aren't needed anymore.

...

[1]

https://review.opendev.org/#/q/topic:fix-libvirt-focal+(status:open+OR+statu...)

...

Ah, sorry, did not notice that. I just went with [1] removing the pkg to avoid the naming issue altogether.

No problem

My question is whether we could detect such issues more proactively?

Good question... I don't think that this kind of a check should be hosted on the release management side.Here our issue is more an environment issue, It's due to the fact that we decided to run the job `publish-openstack-artifacts` on ubuntu-focal, so to detect this kind of issue early I think that jobs on the projects side should rely on ubuntu-focal too. I'm not a Zuul expert and I don't know if it is possible to add a condition based on the branches (per example) to choose the right env (focal or not). By following this way I guess it could be possible to fail early on the project side, and it could allow you to fix this kind of issue well before proposing a new release. In other words, if projects's jobs env are aligned with release management our job we could catch them early.

`publish-openstack-artifacts` job should not run for Focal for the stable gate and running it on Focal from Victoria onwards (like all other common jobs).

-gmann

...

We have other scenarios to consider. These are when no patches haven't been merged since a while on projects side, and where a release is proposed, in this case even if environments are aligned, in this case we aren't able to detect this kind of issue, and they will continue to occur during publication, as in this case. It could be worth it to involve the infra team to see how to fail early. Thoughts?

[1] https://review.opendev.org/762637

-yoctozepto

-- Hervé BeraudSenior Software Engineer at Red Hatirc: hberaudhttps:// github.com/4383/https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE-----

wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

-- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/ https://twitter.com/4383hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----