Open Stack

Bug LP:#2004555 (CVE-2023-2088) identified that “an unauthorised access to
a volume could occur when an iSCSI or FC connection from a host is severed
due to a volume being unmapped on the storage system and the device is
later reused for another volume on the same host”.

While the bug affects iSCSI and FC, the fix introduces a breaking change
for configurations upgrading to the fixed packages if they do not have
service token support enabled. The new code introduces mandatory use of
service tokens to identify requests coming from a trusted control plane
service.

Service tokens are issued by Keystone for service users. For example, a
token can be issued to the Nova service user. They have a particular trust
level, defined by the service, that allows them to perform trusted actions.
In the case of Nova, the action is detaching volumes from instances.

Versions of Ubuntu OpenStack for Yoga and later will have new packages
versions released with CVE patches applied that require the use of service
tokens to enable the detaching and deletion of volumes on instances. These
packages will be released on July 24, 2023.

Versions of Ubuntu OpenStack for Xena and earlier will not have package
updates released and will still be vulnerable to the “unauthorised detach
of volumes” issue until a mitigation policy is applied. The mitigation
policy provided requires the use of service tokens.

For Charmed OpenStack releases, this means that some charms (identified
below) must be upgraded to enable service tokens prior to the packages
being upgraded and before the mitigation policy is applied. If not, users
will lose the ability to detach volumes from their instances.

= Patched packages =

The following package versions include patches for CVE-2023-2088. These
package versions deliver services that require the use of service tokens.
Package versions prior to those listed will not be patched for
CVE-2023-2088 and will therefore require policy updates for mitigation.

Ubuntu 23.04 (Lunar)
* nova >= 3:27.0.0-0ubuntu1.3
* cinder >= 2:22.0.0-0ubuntu1.3
* python-os-brick >= 6.2.0-0ubuntu2.3
* ironic >= 1:21.4.0-0ubuntu1.1
* python-glance-store >= 4.3.0-0ubuntu1.3

Ubuntu Cloud Archive Antelope
* nova >= 3:27.0.0-0ubuntu1.3~cloud0
* cinder >= 2:22.0.0-0ubuntu1.3~cloud0
* python-os-brick >= 6.2.0-0ubuntu2.3~cloud0
* ironic >= 1:21.4.0-0ubuntu1.1~cloud0
* python-glance-store >= 4.3.0-0ubuntu1.3~cloud0

Ubuntu 22.10 (Kinetic)
* nova >= 3:26.1.1-0ubuntu1.1
* cinder >= 2:21.2.0-0ubuntu1.1
* python-os-brick >= os-brick - 6.1.0-0ubuntu1.3
* ironic >= 1:21.1.0-0ubuntu1.1
* python-glance-store >= 4.1.0-0ubuntu1.3

Ubuntu Cloud Archive Zed
* nova >= 3:26.1.1-0ubuntu1.1~cloud0
* cinder >= 2:21.2.0-0ubuntu1.1~cloud0
* python-os-brick >= os-brick - 6.1.0-0ubuntu1.3~cloud0
* ironic >= 1:21.1.0-0ubuntu1.1~cloud0
* python-glance-store >= 4.1.0-0ubuntu1.3~cloud0

Ubuntu 22.04 (Jammy)
* nova >= 3:25.1.1-0ubuntu1.1
* cinder >= 2:20.2.0-0ubuntu1.1
* python-os-brick >= 5.2.2-0ubuntu1.2
* ironic >= 1:20.1.0-0ubuntu1.1
* python-glance-store >= 3.0.0-0ubuntu1.3

Ubuntu Cloud Archive Yoga
* nova >= 3:25.1.1-0ubuntu1.1~cloud0
* cinder >= 2:20.2.0-0ubuntu1.1~cloud0
* python-os-brick >= 5.2.2-0ubuntu1.2~cloud0
* ironic >= 1:20.1.0-0ubuntu1.1~cloud0
* python-glance-store >= 3.0.0-0ubuntu1.3~cloud0

= Charms with Service Tokens Enabled by Default =

Following is the list of charms and earliest revision that supports service
tokens:

Channel: 2023.1/stable
* Keystone revision >= 645
* Cinder revision >= 639
* Nova-compute revision >= 678
* Nova-cloud-controller revision >= 674
* Ironic-api revision >= 54
* Ironic-conductor
- s390x revision >= 151
- arm64 revision >= 152
- ppc64el revision >= 153
- amd64 revision >= 149

Channel: zed/stable
* Keystone revision >= 646
* Cinder revision >= 647
* Nova-compute revision >= 683
* Nova-cloud-controller revision >= 676
* Ironic-api revision >= 53
* Ironic-conductor
- s390x revision >= 148
- arm64 revision >= 154
- ppc64el revision >= 150
- amd64 revision >= 147

Channel: yoga/stable
* Keystone revision >= 647
* Cinder revision >= 638
* Nova-compute revision >= 681
* Nova-cloud-controller revision >= 679
* Ironic-api revision >= 48
* Ironic-conductor revision >= 145

Channel: xena/stable
* Keystone revision >= 651
* Cinder revision >= 640
* Nova-compute revision >= 680
* Nova-cloud-controller revision >= 677
* Ironic-api revision >= 50
* Ironic-conductor revision >= 143

Channel: wallaby/stable
* Keystone revision >= 653
* Cinder revision >= 648
* Nova-compute revision >= 684
* Nova-cloud-controller revision >= 675
* Ironic-api revision >= 52
* Ironic-conductor revision >= 144

Channel: victoria/stable
* Keystone revision >= 650
* Cinder revision >= 646
* Nova-compute revision >= 679
* Nova-cloud-controller revision >= 678
* Ironic-api revision >= 49
* Ironic-conductor revision >= 142

Channel: ussuri/stable
* Keystone revision >= 652
* Cinder revision >= 645
* Nova-compute revision >= 682
* Nova-cloud-controller revision >= 680
* Ironic-api revision >= 51
* Ironic-conductor revision >= 155

= Using a custom Cinder policy as a mitigation =

It’s possible to restrict the access to the operations affected by this CVE
by providing a custom Cinder policy file that will prevent users from
deleting attachments, detach and force detach. In order to use this
mitigation, service token support must be enabled. The exact method for
customising this depends on the deployment tooling being used. For Charmed
OpenStack users it is the following:

1) Identify the nova service user id running the following command

openstack user show --domain service_domain -f value -c id nova

The command output looks like this

ae034831d6bf41099327bf9d34988fbc

2) Create a policy file named cve-2023-2088-cinder.yaml with the following
content:

"is_service": "role:service or service_user_id:<nova_service_uuid>"
"legacy_system_admin_or_project_member": "(role:admin) or (role:member and
project_id:%(project_id)s)"
"volume:attachment_delete": "rule:legacy_system_admin_or_project_member and
rule:is_service"
"volume_extension:volume_actions:terminate_connection":
"rule:legacy_system_admin_or_project_member and rule:is_service"
"volume_extension:volume_actions:detach":
"rule:legacy_system_admin_or_project_member and rule:is_service"
"volume_extension:volume_admin_actions:force_detach": "!"

Replacing “<nova_service_uuid>” would result in a snippet that looks like
this:

"is_service": "role:service or
service_user_id:ae034831d6bf41099327bf9d34988fbc"
"legacy_system_admin_or_project_member": "(role:admin) or (role:member and
project_id:%(project_id)s)"
"volume:attachment_delete": "rule:legacy_system_admin_or_project_member and
rule:is_service"
"volume_extension:volume_actions:terminate_connection":
"rule:legacy_system_admin_or_project_member and rule:is_service"
"volume_extension:volume_actions:detach":
"rule:legacy_system_admin_or_project_member and rule:is_service"
"volume_extension:volume_admin_actions:force_detach": "!"

3) Create zip files from the yaml file:

zip cve-2023-2088-cinder.zip cve-2023-2088-cinder.yaml

4) Attach the zip file as a resource to the cinder applications:

juju attach-resource cinder policyd-override=cve-2023-2088-cinder.zip

5) Enable the overrides via the use-policyd-override charm option:

juju config cinder use-policyd-override=true

For more details on using policy overrides see the OpenStack Charm Guide.

= Frequently Asked Questions =

Q: What can I do if my deployed cloud doesn’t have service tokens enabled
and my deployment tooling doesn’t have the option to enable them?
A: Pin the version of the affected packages to block the upgrades to the
versions listed in “Affected packages” section or newer, until service
tokens are enabled.

Q: For Charmed OpenStack, what’s the upgrade order for charms and deb
packages?
A: Upgrade the charms to the latest versions available in the charm channel
tracked by your environment or at the least to the revisions listed in
section “Charms with Service Token Enabled By Default”. After that, upgrade
the deb packages with your preferred method (e.g. Landscape, apt-get, etc).

= References =

* CVE-2023-2088: https://ubuntu.com/security/CVE-2023-2088
* OSSA-2023-003 Unauthorized volume access through deleted volume
attachments: https://security.openstack.org/ossa/OSSA-2023-003.html
* OSSN-0092 Using Configuration as a Short-Term Mitigation for
OSSA-2023-003: https://wiki.openstack.org/wiki/OSSN/OSSN-0092
* Bug 2004555: https://bugs.launchpad.net/nova/+bug/2004555
* OpenStack Charm Guide Policy overrides:
https://docs.openstack.org/charm-guide/latest/admin/policy-overrides.html
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.openstack.org/pipermail/openstack-discuss/attachments/20230717/93e2c8e1/attachment.htm>