March 2023 - openstack-discuss - lists.openstack.org

[nova][heat] The next steps to "fix" libvirt problems in Ubuntu Jammy
by Takashi Kajinami 03 Apr '23

03 Apr '23

Hello, Since we migrated our jobs from Ubuntu Focal to Ubuntu Jammy, heat gate jobs have become very flaky. Further investigation revealed that the issue is related to something in libvirt from Ubuntu Jammy and that prevents detaching devices from instances[1]. The same problem appears in different jobs[2] and we workaround the problem by disabling some affected jobs. In heat we also disabled some flaky tests but because of this we no longer run basic scenario tests which deploys instance/volume/network in a single stack, which means we lost the quite basic test coverage. My question is, is there anyone in the Nova team working on "fixing" this problem ? We might be able to implement some workaround (like checking status of the instances before attempting to delete it) but this should be fixed in libvirt side IMO, as this looks like a "regression" in Ubuntu Jammy. Probably we should report a bug against the libvirt package in Ubuntu but I'd like to hear some thoughts from the nova team because they are more directly affected by this problem. I'm now trying to set up a centos stream 9 job in Heat repo to see whether this can be reproduced if we use centos stream 9. I've been running that specific scenario test in centos stream 9 jobs in puppet repos but I've never seen this issue, so I suspect the issue is really specific to libvirt in Jammy. [1] https://bugs.launchpad.net/nova/+bug/1998274 [2] https://bugs.launchpad.net/nova/+bug/1998148 Thank you, Takashi

3 5

[ptls] PyPI maintainer cleanup - Action needed: Contact extra maintainers
by Jay Faulkner 03 Apr '23

03 Apr '23

Hi PTLs, The TC recently voted[1] to require humans be removed from PyPI access for OpenStack-managed projects. This helps ensure all releases are created via releases team tooling and makes it less likely for a user account compromise to impact OpenStack packages. Many projects have already updated https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L33 with a list of packages that contain extra maintainers. We'd like to request that PTLs, or their designate, reach out to any extra maintainers listed for projects you are responsible for and request they remove their access in accordance with policy. An example email, and detailed steps to follow have been provided at https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup-email-tem… . Thank you for your cooperation as we work to improve our security posture and harden against supply chain attacks. Thank you, Jay Faulkner TC Vice-Chair 1: https://opendev.org/openstack/governance/commit/979e339f899ef62d2a6871a99c9…

3 6

[barbican] database is growing and can not be purged
by Pavlo Shchelokovskyy 03 Apr '23

03 Apr '23

Hi all, we are observing the following behavior in Barbican: - OpenStack environment is using both encrypted Cinder volumes and encrypted local storage (lvm) for Nova instances - over the time, the secrets and orders tables are growing - many soft-deleted entries in secrets DB can not be purged by the db cleanup script As I understand what is happening - both Cinder and Nova create secrets in Barbican on behalf of the user when creating an encrypted volume or booting an instance with encrypted local storage. They both do it via castellan library, that under the hood creates orders in Barbican, waits for them to become active and returns to the caller only the ID of the generated secret. When time comes to delete the thing (volume or instance) Cinder/Nova again use castellan, but only delete the secret, not the order (they are not aware that there was any 'order' created anyway). As a result, the orders are left in place, and DB cleanup procedure does not delete soft-deleted secrets when there's an ACTIVE order referencing such secret. This is troublesomes on many levels - users who use Cinder or Nova may not even be aware that they are creating something in Barbican. Orders accumulating like that may eventually result in cryptic errors when e.g. when you run out of quota for orders. And what's more, default Barbican policies do allow 'normal' (creator) users to create an order, but not delete it (only project admin can do it), so even if the users are aware of Barbican involvement, they can not delete those orders manually anyway. Plus there's no good way in API to determine outright which orders are referencing deleted secrets. I see several ways of dealing with that and would like to ask for your opinion on what would be the best one: 1. Amend Barbican API to allow filtering orders by the secrets, when castellan deletes a secret - search for corresponding order and delete it as well, change default policy to actually allow order deletion by the same users who can create them. 2. Cascade-delete orders when deleting secrets - this is easy but probably violates that very policy that disallowed normal users to delete orders. 3. improve the database cleanup so it first marks any order that references a deleted secret also as deleted, so later when time comes both could be purged (or something like that). This also has a similar downside to the previous option by not being explicit enough. I've filed a bug for that https://storyboard.openstack.org/#!/story/2010625 and proposed a patch for option 2 (cascade delete), but would like to ask what would you see as the most appropriate way or may be there's something else that I've missed. Btw, the problem is probably even more pronounced with keypairs - when castellan is used to create those, under the hood both order and container are created besides the actual secrets, and again only the secret ids are returned to the caller. When time comes to delete things, the caller only knows about secret IDs, and can only delete them, leaving both container and order behind. Luckily, I did not find any place across OpenStack that actually creates keypairs using castellan... but the problem is definitely there. Best regards, -- Dr. Pavlo Shchelokovskyy Principal Software Engineer Mirantis Inc www.mirantis.com

2 2

[kolla] Image building question
by Satish Patel 03 Apr '23

03 Apr '23

Folks, I am playing with kolla image building to understand how it works. I am using the following command to build images and wanted to check with you folks if that is the correct way to do it. $ kolla-build -b ubuntu -t source keystone nova neutron glance Does the above command compile code from source or just download images from remote repositories and re-compile them? because in command output I've not noticed anything related to the compiling process going on. Here is the output of all images produced by kolla-build command. Do I need anything else or is this enough to deploy kolla? root@docker-reg:~# docker images REPOSITORY TAG IMAGE ID CREATED SIZE kolla/mariadb-server 15.1.0 2a497eee8269 26 minutes ago 595MB kolla/cron 15.1.0 342877f26a8a 30 minutes ago 250MB kolla/memcached 15.1.0 0d19a4902644 31 minutes ago 250MB kolla/mariadb-clustercheck 15.1.0 d84427d3c639 31 minutes ago 314MB kolla/mariadb-base 15.1.0 34447e3e59b6 31 minutes ago 314MB kolla/keepalived 15.1.0 82133b09fbf0 31 minutes ago 260MB kolla/prometheus-memcached-exporter 15.1.0 6c2d605f70ee 31 minutes ago 262MB <none> <none> e66b228c2a07 31 minutes ago 248MB kolla/rabbitmq 15.1.0 8de5c39379d3 32 minutes ago 309MB kolla/fluentd 15.1.0 adfd19027862 33 minutes ago 519MB kolla/haproxy-ssh 15.1.0 514357ac4d36 36 minutes ago 255MB kolla/haproxy 15.1.0 e5b9cfdf6dfc 37 minutes ago 257MB kolla/prometheus-haproxy-exporter 15.1.0 a679f65fd735 37 minutes ago 263MB kolla/prometheus-base 15.1.0 afeff3ed5dce 37 minutes ago 248MB kolla/glance-api 15.1.0 a2241f68f23a 38 minutes ago 1.04GB kolla/glance-base 15.1.0 7286772a03a4 About an hour ago 1.03GB kolla/neutron-infoblox-ipam-agent 15.1.0 f90ffc1a3326 About an hour ago 1.05GB kolla/neutron-server 15.1.0 69c844a2e3a9 About an hour ago 1.05GB kolla/neutron-l3-agent 15.1.0 4d87e6963c96 About an hour ago 1.05GB <none> <none> 486da9a6562e About an hour ago 1.05GB kolla/neutron-linuxbridge-agent 15.1.0 e5b3ca7e099c About an hour ago 1.04GB kolla/neutron-bgp-dragent 15.1.0 ac37377820c6 About an hour ago 1.04GB kolla/ironic-neutron-agent 15.1.0 90993adcd74b About an hour ago 1.04GB kolla/neutron-metadata-agent 15.1.0 8522f147f88d About an hour ago 1.04GB kolla/neutron-sriov-agent 15.1.0 8a92ce7d13c0 About an hour ago 1.04GB kolla/neutron-dhcp-agent 15.1.0 5c214b0171f5 About an hour ago 1.04GB kolla/neutron-metering-agent 15.1.0 7b3b91ecd77b About an hour ago 1.04GB kolla/neutron-openvswitch-agent 15.1.0 1f8807308814 About an hour ago 1.04GB kolla/neutron-base 15.1.0 f85b6a2e2725 About an hour ago 1.04GB kolla/nova-libvirt 15.1.0 0f3ecefe4752 About an hour ago 987MB kolla/nova-compute 15.1.0 241b7e7fafbe About an hour ago 1.47GB kolla/nova-spicehtml5proxy 15.1.0 b740820a7ad1 About an hour ago 1.15GB kolla/nova-novncproxy 15.1.0 1ba2f443d5c3 About an hour ago 1.22GB kolla/nova-compute-ironic 15.1.0 716612107532 About an hour ago 1.12GB kolla/nova-ssh 15.1.0 ae2397f4e1c1 About an hour ago 1.11GB kolla/nova-api 15.1.0 2aef02667ff8 About an hour ago 1.11GB kolla/nova-conductor 15.1.0 6f1da3400901 About an hour ago 1.11GB kolla/nova-scheduler 15.1.0 628326776b1d About an hour ago 1.11GB kolla/nova-serialproxy 15.1.0 28eb7a4a13f8 About an hour ago 1.11GB kolla/nova-base 15.1.0 e47420013283 About an hour ago 1.11GB kolla/keystone 15.1.0 e5530d829d5f 2 hours ago 947MB kolla/keystone-ssh 15.1.0 eaa7e3f3985a 2 hours ago 953MB kolla/keystone-fernet 15.1.0 8a4fa24853a8 2 hours ago 951MB kolla/keystone-base 15.1.0 b6f9562364a9 2 hours ago 945MB kolla/barbican-base 15.1.0 b2fdef1afb44 2 hours ago 915MB kolla/barbican-keystone-listener 15.1.0 58bd59de2c63 2 hours ago 915MB kolla/openstack-base 15.1.0 c805b4b3b1c1 2 hours ago 893MB kolla/base 15.1.0 f68e9ef3dd30 2 hours ago 248MB registry 2 8db46f9d7550 19 hours ago 24.2MB ubuntu 22.04 08d22c0ceb15 3 weeks ago 77.8MB

4 8

[tripleo] Removal of TripleO Master Integration and Component Lines
by Ronelle Landy 03 Apr '23

03 Apr '23

Hello All, Removal of TripleO Master Integration and Component Lines Per the decision to not maintain TripleO after the Zed release [1], the master/main integration and component lines are being removed in the following patches: https://review.rdoproject.org/r/c/config/+/48073 https://review.rdoproject.org/r/c/config/+/48074 https://review.rdoproject.org/r/c/rdo-jobs/+/48075 The last promoted release of master through TripleO is: https://trunk.rdoproject.org/centos9-master/current-tripleo/delorean.repo (hash: ddce25bad764dde7e0515094b4d40471), which was promoted on 03/28/2023. Check/gate testing for the master branch is in process of being removed as well. [1] https://review.opendev.org/c/openstack/governance/+/878799

3 2

[kolla] horizon image build failed
by Satish Patel 02 Apr '23

02 Apr '23

Folks, All other images build successfully but when i am trying to build horizon which failed with following error: $ kolla-build --registry docker-reg:4000 -b ubuntu -t source --tag zed horizon INFO:kolla.common.utils.horizon:Collecting XStatic-Dagre>=0.6.4.0 INFO:kolla.common.utils.horizon: Downloading XStatic_Dagre-0.6.4.1-py2.py3-none-any.whl (140 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 140.0/140.0 kB 14.4 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon:Collecting XStatic-Dagre-D3>=0.4.17.0 INFO:kolla.common.utils.horizon: Downloading XStatic_Dagre_D3-0.4.17.0-py2.py3-none-any.whl (357 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 357.4/357.4 kB 13.5 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon:Requirement already satisfied: XStatic-Font-Awesome>=4.7.0.0 in /var/lib/kolla/venv/lib/python3.10/site-packages (from vitrage-dashboard==3.6.1.dev2) (4.7.0.0) INFO:kolla.common.utils.horizon:Collecting XStatic-Graphlib>=2.1.7.0 INFO:kolla.common.utils.horizon: Downloading XStatic_Graphlib-2.1.7.0-py2.py3-none-any.whl (51 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 51.5/51.5 kB 114.3 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon:Requirement already satisfied: XStatic-jQuery>=1.8.2.1 in /var/lib/kolla/venv/lib/python3.10/site-packages (from vitrage-dashboard==3.6.1.dev2) (1.12.4.1) INFO:kolla.common.utils.horizon:Collecting XStatic-lodash>=4.16.4.1 INFO:kolla.common.utils.horizon: Downloading XStatic_lodash-4.16.4.2-py3-none-any.whl (167 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 167.9/167.9 kB 12.4 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon:Collecting XStatic-moment>=2.8.4.1 INFO:kolla.common.utils.horizon: Downloading XStatic_moment-2.8.4.3-py3-none-any.whl (58 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 58.0/58.0 kB 66.7 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon:Collecting XStatic-Moment-Timezone>=0.5.22.0 INFO:kolla.common.utils.horizon: Downloading XStatic-Moment-Timezone-0.5.22.0.tar.gz (99 kB) INFO:kolla.common.utils.horizon: ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 99.7/99.7 kB 45.1 MB/s eta 0:00:00 INFO:kolla.common.utils.horizon: Preparing metadata (setup.py): started INFO:kolla.common.utils.horizon: Preparing metadata (setup.py): finished with status 'error' INFO:kolla.common.utils.horizon: error: subprocess-exited-with-error INFO:kolla.common.utils.horizon: INFO:kolla.common.utils.horizon: × python setup.py egg_info did not run successfully. INFO:kolla.common.utils.horizon: │ exit code: 1 INFO:kolla.common.utils.horizon: ╰─> [6 lines of output] INFO:kolla.common.utils.horizon: Traceback (most recent call last): INFO:kolla.common.utils.horizon: File "<string>", line 2, in <module> INFO:kolla.common.utils.horizon: File "<pip-setuptools-caller>", line 34, in <module> INFO:kolla.common.utils.horizon: File "/tmp/pip-install-dqag1zef/xstatic-moment-timezone_60eeadc1dfb9492781fe3ca90e3b95c2/setup.py", line 2, in <module> INFO:kolla.common.utils.horizon: from xstatic.pkg import moment_timezone as xs INFO:kolla.common.utils.horizon: ImportError: cannot import name 'moment_timezone' from 'xstatic.pkg' (unknown location) INFO:kolla.common.utils.horizon: [end of output] INFO:kolla.common.utils.horizon: INFO:kolla.common.utils.horizon: note: This error originates from a subprocess, and is likely not a problem with pip. INFO:kolla.common.utils.horizon: INFO:kolla.common.utils.horizon:error: metadata-generation-failed INFO:kolla.common.utils.horizon:× Encountered error while generating package metadata. INFO:kolla.common.utils.horizon:╰─> See above for output. INFO:kolla.common.utils.horizon:note: This is an issue with the package mentioned above, not pip. INFO:kolla.common.utils.horizon:hint: See above for details. INFO:kolla.common.utils.horizon: INFO:kolla.common.utils.horizon:Removing intermediate container e6cd437ba529 ERROR:kolla.common.utils.horizon:Error'd with the following message ERROR:kolla.common.utils.horizon:The command '/bin/sh -c ln -s horizon-source/* horizon && sed -i /^horizon=/d /requirements/upper-constraints.txt && SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt /horizon && mkdir -p /etc/openstack-dashboard && cp -r /horizon/openstack_dashboard/conf/* /etc/openstack-dashboard/ && cp /horizon/openstack_dashboard/local/local_settings.py.example /etc/openstack-dashboard/local_settings && cp /horizon/manage.py /var/lib/kolla/venv/bin/manage.py && if [ "$(ls /plugins)" ]; then SETUPTOOLS_USE_DISTUTILS=stdlib python3 -m pip --no-cache-dir install --upgrade -c /requirements/upper-constraints.txt /plugins/*; fi && for locale in /var/lib/kolla/venv/lib/python3.10/site-packages/*/locale; do (cd ${locale%/*} && /var/lib/kolla/venv/bin/django-admin compilemessages) done && chmod 644 /usr/local/bin/kolla_extend_start' returned a non-zero code: 1 INFO:kolla.common.utils:========================= INFO:kolla.common.utils:Successfully built images INFO:kolla.common.utils:========================= INFO:kolla.common.utils:base INFO:kolla.common.utils:openstack-base INFO:kolla.common.utils:=========================== INFO:kolla.common.utils:Images that failed to build INFO:kolla.common.utils:===========================

4 8

[tc] No TC weekly meeting next week
by Nikolla, Kristi 31 Mar '23

31 Mar '23

Hi all, There will be no TC meeting on Tuesday, April 4th. The next TC meeting will be held on Tuesday, April 11 at 18.00 UTC. More information and an ICS file can be found here https://meetings.opendev.org/#Technical_Committee_Meeting Thank you, Kristi Nikolla

1 0

[release] Release countdown for week R-26, Apr 03 - 07
by Elõd Illés 31 Mar '23

31 Mar '23

Hi, Welcome back to the release countdown emails! These will be sent at major points in the 2023.2 Bobcat development cycle, which should conclude with a final release on October 4th, 2023. Development Focus ----------------- At this stage in the release cycle, focus should be on planning the 2023.2 Bobcat development cycle and approving 2023.2 Bobcat specs. General Information ------------------- 2023.2 Bobcat is a 28 weeks long development cycle. In case you haven't seen it yet, please take a look over the schedule for this release: https://releases.openstack.org/bobcat/schedule.html By default, the team PTL is responsible for handling the release cycle and approving release requests. This task can (and probably should) be delegated to release liaisons. Now is a good time to review release liaison information for your team and make sure it is up to date: https://opendev.org/openstack/releases/src/branch/master/data/release_liais… By default, all your team deliverables from the 2023.1 Antelope release are continued in 2023.2 Bobcat with a similar release model. Upcoming Deadlines & Dates -------------------------- Bobcat-1 milestone: May 11th, 2023 OpenInfra Summit + PTG Vancouver 2023 - June 13-15, 2023 Előd Illés irc: elodilles @ #openstack-release

1 0

[heat][qa][requirements] Pinning tempest in stable/xena constraints
by Takashi Kajinami 31 Mar '23

31 Mar '23

Hello, I have had some local discussions with gmann, but I'd really like to move this discussion forward to fix the broken stable/xena gate in heat so I will start this thread, hoping the thread can provide more context behind my proposal. Historically stable branches of heat have been frequently affected by any change in requirements of tempest. This is mainly because in our CI we install our own in-tree integration tests[1] into tempest venv where tempest and heat-tempest-plugin are installed. Because in-tree integration tests are tied to that specific stable branch, this has been often causing conflicts in requirements (master constraint vs stable/X constraint). [1] https://github.com/openstack/heat/tree/master/heat_integrationtests In the past we changed our test installation[2] to use stable constraint to avoid this conflicts, but this approach does no longer work since stable/xena because 1. stable/xena u-c no longer includes tempest 2. latest tempest CAN'T be installed with stable/xena u-c because current tempest requires fasteners>=0.16.0 which conflicts with 0.14.1 in stable/xena u-c. [2] https://review.opendev.org/c/openstack/heat/+/803890 https://review.opendev.org/c/openstack/heat/+/848215 I've proposed the change to pin tempest[3] in stable/xena u-c so that people can install tempest with stable/xena u-c. [3] https://review.opendev.org/c/openstack/requirements/+/878228 I understand the reason tempest was removed from u-c was that we should use the latest tempest to test recent stable releases.I agree we can keep tempest excluded for stable/yoga and onwards because tempest is installable with their u-c, but stable/xena u-c is no longer compatible with master. Adding pin to xena u-c does not mainly affect the policy to test stable branches with latest tempest because for that we anyway need to use more recent u-c. I'm still trying to find out the workaround within heat but IMO adding tempest pin to stable/xena u-c is harmless but beneficial in case anyone is trying to use tempest with stable/xena u-c. Thank you, Takashi Kajinami

4 10

Reminder! In Person PTG 2023 Team Signup Deadline
by Kendall Nelson 31 Mar '23

31 Mar '23

Hello Everyone, This is the last call to sign your team up for the *in-person* Project Teams Gathering (PTG) happening in Vancouver at the OpenInfra Summit! If you haven't already done so and your team is interested in participating, please complete the survey[1] by April 2nd, 2023 at 7:00 UTC. Registration for the PTG is included as a part of registration for the OpenInfra Summit in Vancouver. Prices increase May 5th so register soon! Thanks! -Kendall (diablo_rojo) [1] Team Survey: https://openinfrafoundation.formstack.com/forms/june2023_ptg_survey [2] Summit Registration: https://vancouver2023.openinfra.dev/ <https://vancouver2023.openinfra.dev/>

1 0