May 2023 - openstack-discuss - lists.openstack.org

Did StarlingX Break?
by John van Ommen 12 May '23

12 May '23

I've installed StarlingX 10+ times using the iso "bootimage.iso" from their website. I opted to do a reinstall yesterday, and it keeps hitting the same error at the same spot in the install. I've been able to replicate the error three times using two different HP servers. The installer basically reaches the point where it's installing package number 866 out of 1,217 packages, and fails with this error: process [/usr/libexec/anaconda/anaconda-yum --config /tmp/anaconda-yum.conf --tsfile /mnt/sysimage/anaconda-yum.yumtx --rpmlog /tmp/rpm-script.log --installroot /mnt/sysimage --release 7 --arch x86_64 --macro __dbi_htconfig hash nofsync /etc/selinux/targeted/contexts/files/file_contexts exited with status 1 I did a Google search on this error, and there's a slightly similar error reported on a RedHat forum from someone who had a bad ISO. But I know my ISO is working; it's the exact same ISO I've already installed from 10+ times. In the event that I have a bad DIMM or a bad NVME, I tried an identical server, and was able to reproduce the same error. In an earlier message on the list today, someone reported that missing repos may be impacting StarlingX, which definitely makes me wonder if I'm being impacted by that. I'm on the CentOS based version of StarlingX, version 7.0 I also tried to install from the Debian based version (8.0) and that one wouldn't even boot properly. It basically asks you what type of install you want to do, then crashes.

2 1

[kolla-ansible] TLS and internal VIP
by Nicolas Froger 12 May '23

12 May '23

Hello, I was debugging the monitoring stack of our deployment and I noticed that our Prometheus could not reach the OpenStack Exporter. The error is about a certificate name mismatch because Prometheus is scraping the exporter with the internal IP address instead of the internal FQDN while the certificate we have is only valid for the internal FQDN. Indeed, the Prometheus config specifies kolla_internal_vip_address as a target and uses HTTPS when kolla_enable_tls_internal is true. Replacing the target with kolla_internal_fqdn which is a DNS name for which the certificate is valid fixed my issue. My question is the following: should the internal certificate also be valid for the internal VIP when kolla_enable_tls_internal is set to true or is it okay if it's only valid for the FQDN? In the later case, does it make sense if I open an issue to use the FQDN instead of the IP address in the Prometheus config? Regards, -- Nicolas Froger

1 0

[neutron][ironic] Postgres support?
by Julia Kreger 11 May '23

11 May '23

Greetings folks, I have spent a substantial amount of time investigating CI issues as of recent, and I noticed that at some point the legacy "ironic-tempest-pxe_ipmitool-postgres" CI job which has been kept around as a non-voting postgres support canary, is now failing on the master branch. Specifically, looking through the logs[0], it appears that structurally the queries are no longer compatible. np0033933847 neutron-server[90677]: WARNING oslo_db.sqlalchemy.exc_filters [None req-155d0299-07ef-433a-8576-467127c82be4 None None] DBAPIError exception wrapped.: psycopg2.errors.GroupingError: column "subnetpools.address_scope_id" must appear in the GROUP BY clause or be used in an aggregate function May 03 18:41:57.893335 np0033933847 neutron-server[90677]: LINE 2: ...standard_attr_id AS floatingips_standard_attr_id, subnetpool... May 03 18:41:57.893335 np0033933847 neutron-server[90677]: ^ May 03 18:41:57.893335 np0033933847 neutron-server[90677]: ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last): May 03 18:41:57.893335 np0033933847 neutron-server[90677]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/base.py", line 1900, in _execute_context May 03 18:41:57.893335 np0033933847 neutron-server[90677]: ERROR oslo_db.sqlalchemy.exc_filters self.dialect.do_execute( May 03 18:41:57.893335 np0033933847 neutron-server[90677]: ERROR oslo_db.sqlalchemy.exc_filters File "/usr/local/lib/python3.8/dist-packages/sqlalchemy/engine/default.py", line 736, in do_execute May 03 18:41:57.893335 np0033933847 neutron-server[90677]: ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters) May 03 18:41:57.893335 np0033933847 neutron-server[90677]: ERROR oslo_db.sqlalchemy.exc_filters psycopg2.errors.GroupingError: column "subnetpools.address_scope_id" must appear in the GROUP BY clause or be used in an aggregate function May 03 18:41:57.893335 np0033933847 neutron-server[90677]: ERROR oslo_db.sqlalchemy.exc_filters LINE 2: ...standard_attr_id AS floatingips_standard_attr_id, subnetpool... May 03 18:41:57.893335 np0033933847 neutron-server[90677]: ERROR oslo_db.sqlalchemy.exc_filters ^ >From Ironic's point of view, we're wondering if this is expected? Is it likely to be fixed? If there are no plans to fix the DB queries, Ironic will have no choice but to drop the CI job. Thanks in advance! -Julia [0]: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/…

5 6

[ironic] Meeting cancelled; Metal3 PTG next Monday
by Jay Faulkner 11 May '23

11 May '23

Hi all, The Metal3 team informed us they are having a one day PTG, which will take place on May 15th, 2023 from 14:00 to 17:00 UTC. I strongly encourage any interested Ironic cores to review the invitation[1] and attend if interested. Because this overlaps our regularly scheduled Ironic meeting, and there is significant overlap between the Metal3 and Ironic communities, the Ironic meeting for Monday, May 15h is cancelled to allow our communities to collaborate without distraction. Thanks, Jay Faulkner Ironic PTL 1: https://groups.google.com/g/metal3-dev/c/62PCloXk-zQ

1 0

[kolla] [train] [neutron]
by Albert Braden 11 May '23

11 May '23

1 0

Extending the Forum
by Kendall Nelson 11 May '23

11 May '23

Hello Everyone, We received feedback that critical Forum conversations were not able to make the schedule due to the intentional work on our part to avoid PTG / Forum overlap which ended up restricting the number of Forum discussion proposals we were able to accept. To mitigate this, we have opened up the Forum room for a third day. We have created a form to collect critical Forum proposals. We will leave the form open for one week to give all timezones fair representation. The deadline to submit a proposal for a forum discussion is Thursday May, 18th at 7:00 UTC. Shortly after, the OpenInfra Foundation will make final decisions and program the Forum on Thursday. -Kendall Nelson [1] https://openinfrafoundation.formstack.com/forms/forum_expansion

1 0

[publiccloud-sig] Bi-weekly meeting times adjusted to 0700 UTC
by Felix Kronlage-Dammers 11 May '23

11 May '23

Hi, since some of us (to be honest, probably mostly me ;) have a conflicting meeting during summer time with the slot of the publiccloud-sig we discussed moving it to 0700 UTC during summer time. Since nobody in the past two meetings spoke against that, this was concluded yesterday. During summer time the publiccloud SIG will meet at 0700 UTC, not at 0800 UTC. The change is also reflected in the wiki page: https://wiki.openstack.org/wiki/PublicCloudSIG See you next time (and I'll set myself reminder to send out the reminder ;) felix -- GPG: 824CE0F0 / 2082 651E 5104 F989 4D18 BB2E 0B26 6738 824C E0F0 https://hazardous.org/ - fkr(a)hazardous.org - fkr@irc - @felixkronlage

1 0

[OSSA-2023-003] cinder, glance_store, nova, os-brick: Unauthorized volume access through deleted volume attachments (CVE-2023-2088)
by Jeremy Stanley 10 May '23

10 May '23

============================================================================ OSSA-2023-003: Unauthorized volume access through deleted volume attachments ============================================================================ :Date: May 10, 2023 :CVE: CVE-2023-2088 Affects ~~~~~~~ - Cinder: <20.2.1, >=21.0.0 <21.2.1, ==22.0.0 - Glance_store: <3.0.1, >=4.0.0 <4.1.1, >=4.2.0 <4.3.1 - Nova: <25.1.2, >=26.0.0 <26.1.2, ==27.0.0 - Os-brick: <5.2.3, >=6.0.0 <6.1.1, >=6.2.0 <6.2.2 Description ~~~~~~~~~~~ An unauthorized access to a volume could occur when an iSCSI or FC connection from a host is severed due to a volume being unmapped on the storage system and the device is later reused for another volume on the same host. **Scope:** Only deployments with iSCSI or FC volumes are affected. However, the fix for this issue includes a configuration change in Nova and Cinder that may impact you on your next upgrade regardless of what backend storage technology you use. See the *Configuration change* section below, and item 4(B) in the *Patches and Associated Deployment Changes* for details. This data leak can be triggered by two different situations. **Accidental case:** If there is a problem with network connectivity during a normal detach operation, OpenStack may fail to clean the situation up properly. Instead of force-detaching the compute node device, Nova ignores the error, assuming the instance has already been deleted. Due to this incomplete operation OpenStack may end up selecting the wrong multipath device when connecting another volume to an instance. **Intentional case:** A regular user can create an instance with a volume, and then delete the volume attachment directly in Cinder, which neglects to notify Nova. The compute node SCSI plumbing (over iSCSI/FC) will continue trying to connect to the original host/port/LUN, not knowing the attachment has been deleted. If a subsequent volume attachment re-uses the host/port/LUN for a different instance and volume, the original instance will gain access to it once the SCSI plumbing reconnects. Configuration Change -------------------- To prevent the intentional case, the Block Storage API provided by Cinder must only accept attachment delete requests from Nova for instance-attached volumes. A complicating factor is that Nova deletes an attachment by making a call to the Block Storage API on behalf of the user (that is, by passing the user's token), which makes the request indistinguishable from the user making this request directly. The solution is to have Nova include a service token along with the user's token so that Cinder can determine that the detach request is coming from Nova. The ability for Nova to pass a service token has been supported since Ocata, but has not been required until now. Thus, deployments that are not currently sending service user credentials from Nova will need to apply the relevant code changes and also make configuration changes to solve the problem. Patches and Associated Deployment Changes ----------------------------------------- Given the above analysis, a thorough fix must include the following elements: 1. The os-brick library must implement the ``force`` option for fibre channel, which which has only been available for iSCSI until now (covered by the linked patches). 2. Nova must call os-brick with the ``force`` option when disconnecting volumes from deleted instances (covered by the linked patches). 3. In deployments where Glance uses the cinder glance_store driver, glance must call os-brick with the ``force`` option when disconnecting volumes (covered by the linked patches). 4. Cinder must distinguish between safe and unsafe attachment delete requests and reject the unsafe ones. This part of the fix has two components: a. The Block Storage API will return a 409 (Conflict) for a request to delete an attachment if there is an instance currently using the attachment, **unless** the request is being made by a service (for example, Nova) on behalf of a user (covered by the linked patches). b. In order to recognize that a request is being made by a service on behalf of a user, Nova must be configured to send a service token along with the user token. If this configuration change is not made, the cinder change will reject **any** request to delete an attachment associated with a volume that is attached to an instance. Nova must be configured to send a service token to Cinder, and Cinder must be configured to accept service tokens. This is described in the following document and **IS NOT AUTOMATICALLY APPLIED BY THE LINKED PATCHES:** (Using service tokens to prevent long-running job failures) https://docs.openstack.org/cinder/latest/configuration/block-storage/servic… The Nova patch mentioned in step 2 includes a similar document more focused on Nova: doc/source/admin/configuration/service-user-token.rst 5. The cinder glance_store driver does not attach volumes to instances; instead, it attaches volumes directly to the Glance node. Thus, the Cinder change in step 4 will recognize an attachment-delete request coming from Glance as safe and allow it. (Of course, we expect that you will have applied the patches in steps 1 and 3 to your Glance nodes.) Errata ~~~~~~ An additional nova patch is required to fix a minor regression in periodic tasks and some nova-manage actions (errata 1). Also a patch to tempest is needed to account for behavior changes with fixes in place (errata 2). Patches ~~~~~~~ - https://review.opendev.org/882836 (2023.1/antelope cinder) - https://review.opendev.org/882851 (2023.1/antelope glance_store) - https://review.opendev.org/882858 (2023.1/antelope nova) - https://review.opendev.org/882859 (2023.1/antelope nova errata 1) - https://review.opendev.org/882843 (2023.1/antelope os-brick) - https://review.opendev.org/882835 (2023.2/bobcat cinder) - https://review.opendev.org/882834 (2023.2/bobcat glance_store) - https://review.opendev.org/882847 (2023.2/bobcat nova) - https://review.opendev.org/882852 (2023.2/bobcat nova errata 1) - https://review.opendev.org/882840 (2023.2/bobcat os-brick) - https://review.opendev.org/882876 (2023.2/bobcat tempest errata 2) - https://review.opendev.org/882869 (Wallaby nova) - https://review.opendev.org/882870 (Wallaby nova errata 1) - https://review.opendev.org/882839 (Xena cinder) - https://review.opendev.org/882855 (Xena glance_store) - https://review.opendev.org/882867 (Xena nova) - https://review.opendev.org/882868 (Xena nova errata 1) - https://review.opendev.org/882848 (Xena os-brick) - https://review.opendev.org/882838 (Yoga cinder) - https://review.opendev.org/882854 (Yoga glance_store) - https://review.opendev.org/882863 (Yoga nova) - https://review.opendev.org/882864 (Yoga nova errata 1) - https://review.opendev.org/882846 (Yoga os-brick) - https://review.opendev.org/882837 (Zed cinder) - https://review.opendev.org/882853 (Zed glance_store) - https://review.opendev.org/882860 (Zed nova) - https://review.opendev.org/882861 (Zed nova errata 1) - https://review.opendev.org/882844 (Zed os-brick) Credits ~~~~~~~ - Jan Wasilewski from Atman (CVE-2023-2088) - Gorka Eguileor from Red Hat (CVE-2023-2088) References ~~~~~~~~~~ - https://launchpad.net/bugs/2004555 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2088 Notes ~~~~~ - Limited Protection Against Accidents... If you are only concerned with protecting against the accidental case described earlier in this document, steps 1-3 above should be sufficient. Note, however, that only applying steps 1-3 leaves your cloud wide open to the intentional exploitation of this vulnerability. Therefore, we recommend that the full fix be applied to all deployments. - Using Configuration as a Short-Term Mitigation... An alternative approach to mitigation can be found in OSSN-0092 https://wiki.openstack.org/wiki/OSSN/OSSN-0092 - The stable/xena and stable/wallaby branches are under extended maintenance and will receive no new point releases, but patches for them are provided as a courtesy where available. OSSA History ~~~~~~~~~~~~ - 2023-05-10 - Errata 2 - 2023-05-10 - Errata 1 - 2023-05-10 - Original Version -- Jeremy Stanley OpenStack Vulnerability Management Team

1 0

Cinder Bug Report 2023-05-10
by Sofia Enriquez 10 May '23

10 May '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> *Medium* - [rbac] Volume type's extra_specs dict is empty. <https://bugs.launchpad.net/cinder/+bug/2018467> - *Status:* Unassigned. - cinder services report_state hangs for a longer period than report_interval when losing database connection. <https://bugs.launchpad.net/cinder/+bug/2018595> - *Status:* Unassigned. *Low* - HPE 3par - Unable to use small QoS Latency value (less than 1) <https://bugs.launchpad.net/cinder/+bug/2018994> - *Status:* Fix proposed to master <https://review.opendev.org/c/openstack/cinder/+/882266>. Cheers -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[cinder] create a multiattach volume in two ways?
by Tony Liu 10 May '23

10 May '23

Hi, [1] says "Currently you can create a multiattach volume in two ways." I see one way is with volume type. What's the another way? [1] https://docs.openstack.org/cinder/xena/admin/blockstorage-volume-multiattac… Thanks! Tony

2 3