From fungi at yuggoth.org Wed May 10 17:15:45 2023 From: fungi at yuggoth.org (Jeremy Stanley) Date: Wed, 10 May 2023 17:15:45 +0000 Subject: [openstack-announce] [OSSA-2023-003] cinder, glance_store, nova, os-brick: Unauthorized volume access through deleted volume attachments (CVE-2023-2088) Message-ID: <20230510171544.jj6tbd2owpq5pog3@yuggoth.org> ============================================================================ OSSA-2023-003: Unauthorized volume access through deleted volume attachments ============================================================================ :Date: May 10, 2023 :CVE: CVE-2023-2088 Affects ~~~~~~~ - Cinder: <20.2.1, >=21.0.0 <21.2.1, ==22.0.0 - Glance_store: <3.0.1, >=4.0.0 <4.1.1, >=4.2.0 <4.3.1 - Nova: <25.1.2, >=26.0.0 <26.1.2, ==27.0.0 - Os-brick: <5.2.3, >=6.0.0 <6.1.1, >=6.2.0 <6.2.2 Description ~~~~~~~~~~~ An unauthorized access to a volume could occur when an iSCSI or FC connection from a host is severed due to a volume being unmapped on the storage system and the device is later reused for another volume on the same host. **Scope:** Only deployments with iSCSI or FC volumes are affected. However, the fix for this issue includes a configuration change in Nova and Cinder that may impact you on your next upgrade regardless of what backend storage technology you use. See the *Configuration change* section below, and item 4(B) in the *Patches and Associated Deployment Changes* for details. This data leak can be triggered by two different situations. **Accidental case:** If there is a problem with network connectivity during a normal detach operation, OpenStack may fail to clean the situation up properly. Instead of force-detaching the compute node device, Nova ignores the error, assuming the instance has already been deleted. Due to this incomplete operation OpenStack may end up selecting the wrong multipath device when connecting another volume to an instance. **Intentional case:** A regular user can create an instance with a volume, and then delete the volume attachment directly in Cinder, which neglects to notify Nova. The compute node SCSI plumbing (over iSCSI/FC) will continue trying to connect to the original host/port/LUN, not knowing the attachment has been deleted. If a subsequent volume attachment re-uses the host/port/LUN for a different instance and volume, the original instance will gain access to it once the SCSI plumbing reconnects. Configuration Change -------------------- To prevent the intentional case, the Block Storage API provided by Cinder must only accept attachment delete requests from Nova for instance-attached volumes. A complicating factor is that Nova deletes an attachment by making a call to the Block Storage API on behalf of the user (that is, by passing the user's token), which makes the request indistinguishable from the user making this request directly. The solution is to have Nova include a service token along with the user's token so that Cinder can determine that the detach request is coming from Nova. The ability for Nova to pass a service token has been supported since Ocata, but has not been required until now. Thus, deployments that are not currently sending service user credentials from Nova will need to apply the relevant code changes and also make configuration changes to solve the problem. Patches and Associated Deployment Changes ----------------------------------------- Given the above analysis, a thorough fix must include the following elements: 1. The os-brick library must implement the ``force`` option for fibre channel, which which has only been available for iSCSI until now (covered by the linked patches). 2. Nova must call os-brick with the ``force`` option when disconnecting volumes from deleted instances (covered by the linked patches). 3. In deployments where Glance uses the cinder glance_store driver, glance must call os-brick with the ``force`` option when disconnecting volumes (covered by the linked patches). 4. Cinder must distinguish between safe and unsafe attachment delete requests and reject the unsafe ones. This part of the fix has two components: a. The Block Storage API will return a 409 (Conflict) for a request to delete an attachment if there is an instance currently using the attachment, **unless** the request is being made by a service (for example, Nova) on behalf of a user (covered by the linked patches). b. In order to recognize that a request is being made by a service on behalf of a user, Nova must be configured to send a service token along with the user token. If this configuration change is not made, the cinder change will reject **any** request to delete an attachment associated with a volume that is attached to an instance. Nova must be configured to send a service token to Cinder, and Cinder must be configured to accept service tokens. This is described in the following document and **IS NOT AUTOMATICALLY APPLIED BY THE LINKED PATCHES:** (Using service tokens to prevent long-running job failures) https://docs.openstack.org/cinder/latest/configuration/block-storage/service-token.html The Nova patch mentioned in step 2 includes a similar document more focused on Nova: doc/source/admin/configuration/service-user-token.rst 5. The cinder glance_store driver does not attach volumes to instances; instead, it attaches volumes directly to the Glance node. Thus, the Cinder change in step 4 will recognize an attachment-delete request coming from Glance as safe and allow it. (Of course, we expect that you will have applied the patches in steps 1 and 3 to your Glance nodes.) Errata ~~~~~~ An additional nova patch is required to fix a minor regression in periodic tasks and some nova-manage actions (errata 1). Also a patch to tempest is needed to account for behavior changes with fixes in place (errata 2). Patches ~~~~~~~ - https://review.opendev.org/882836 (2023.1/antelope cinder) - https://review.opendev.org/882851 (2023.1/antelope glance_store) - https://review.opendev.org/882858 (2023.1/antelope nova) - https://review.opendev.org/882859 (2023.1/antelope nova errata 1) - https://review.opendev.org/882843 (2023.1/antelope os-brick) - https://review.opendev.org/882835 (2023.2/bobcat cinder) - https://review.opendev.org/882834 (2023.2/bobcat glance_store) - https://review.opendev.org/882847 (2023.2/bobcat nova) - https://review.opendev.org/882852 (2023.2/bobcat nova errata 1) - https://review.opendev.org/882840 (2023.2/bobcat os-brick) - https://review.opendev.org/882876 (2023.2/bobcat tempest errata 2) - https://review.opendev.org/882869 (Wallaby nova) - https://review.opendev.org/882870 (Wallaby nova errata 1) - https://review.opendev.org/882839 (Xena cinder) - https://review.opendev.org/882855 (Xena glance_store) - https://review.opendev.org/882867 (Xena nova) - https://review.opendev.org/882868 (Xena nova errata 1) - https://review.opendev.org/882848 (Xena os-brick) - https://review.opendev.org/882838 (Yoga cinder) - https://review.opendev.org/882854 (Yoga glance_store) - https://review.opendev.org/882863 (Yoga nova) - https://review.opendev.org/882864 (Yoga nova errata 1) - https://review.opendev.org/882846 (Yoga os-brick) - https://review.opendev.org/882837 (Zed cinder) - https://review.opendev.org/882853 (Zed glance_store) - https://review.opendev.org/882860 (Zed nova) - https://review.opendev.org/882861 (Zed nova errata 1) - https://review.opendev.org/882844 (Zed os-brick) Credits ~~~~~~~ - Jan Wasilewski from Atman (CVE-2023-2088) - Gorka Eguileor from Red Hat (CVE-2023-2088) References ~~~~~~~~~~ - https://launchpad.net/bugs/2004555 - http://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2023-2088 Notes ~~~~~ - Limited Protection Against Accidents... If you are only concerned with protecting against the accidental case described earlier in this document, steps 1-3 above should be sufficient. Note, however, that only applying steps 1-3 leaves your cloud wide open to the intentional exploitation of this vulnerability. Therefore, we recommend that the full fix be applied to all deployments. - Using Configuration as a Short-Term Mitigation... An alternative approach to mitigation can be found in OSSN-0092 https://wiki.openstack.org/wiki/OSSN/OSSN-0092 - The stable/xena and stable/wallaby branches are under extended maintenance and will receive no new point releases, but patches for them are provided as a courtesy where available. OSSA History ~~~~~~~~~~~~ - 2023-05-10 - Errata 2 - 2023-05-10 - Errata 1 - 2023-05-10 - Original Version -- Jeremy Stanley OpenStack Vulnerability Management Team -------------- next part -------------- Using Configuration as a Short-Term Mitigation for OSSA-2023-003 --- ### Summary ### An unauthorized access to a volume could occur when an iSCSI or FC connection from a host is severed due to a volume being unmapped on the storage system and the device is later reused for another volume on the same host. ### Affected Services / Software ### - cinder: <20.2.1, >=21.0.0 <21.2.1, ==22.0.0 - glance_store: <3.0.1, >=4.0.0 <4.1.1, >=4.2.0 <4.3.1 - nova: <25.1.2, >=26.0.0 <26.1.2, ==27.0.0 - os-brick: <5.2.3, >=6.0.0 <6.1.1, >=6.2.0 <6.2.2 ### Discussion ### It is recommended to apply the fixes provided in OSSA-2023-003 https://security.openstack.org/ossa/OSSA-2023-003.html but updating an OpenStack deployment may take a long time requiring a proper maintenance window and may even require a validation process of the release prior to the deployment, so operators may prefer applying tactical configuration changes to their cloud to prevent harmful actions while they go through their standarized process. In this case the fastest way to prevent an unsafe attach deletion is twofold: 1. ensuring that Nova uses a user with a service role to send its token on all the requests made to Cinder on behalf of users, and 2. Cinder protects the vulnerable APIs via policy. ### Recommended Actions ### If the deployment has Glance using Cinder as a backend, in order to use this alternative short-term mitigation, Glance must be configured to use a single user having the service role for all its requests to Cinder. If your deployment is *not* using a single user (that is, instead of all the image-volumes being stored in a single project, they are stored in each user's project), then you cannot use this Short-Term Mitigation strategy, but must instead apply the full change described in the previous section. Steps for Mitigation: A. Ensure the users used by Nova (and Glance, if applicable) have the service role * In Nova, this is the user configured in the [service_user] section of nova.conf * In Glance, this user will only be defined if you are using Cinder as a Glance storage backend. It is defined in the [cinder] section of glance.conf ** if you do not use Cinder as a backend for Glance, you do not need to define a service user for Glance B. Configure Nova to send the service token https://docs.openstack.org/cinder/latest/configuration/block-storage/service-token.html * Glance does not have the ability to send a service token to Cinder; instead, the mitigation-by-policy strategy described below relies upon the user configured in glance in the [cinder]/cinder_store_user_name option in glance.conf having been granted the service role in Keystone C. Configure the cinder policies as per https://docs.openstack.org/cinder/latest/configuration/block-storage/policy-config-HOWTO.html to have the following: "is_service": "role:service or service_user_id:" "volume:attachment_delete": "rule:xena_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:terminate_connection": "rule:xena_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:detach": "rule:xena_system_admin_or_project_member and rule:is_service" "volume_extension:volume_admin_actions:force_detach": "!" Notes: - The operator should replace "" with the actual UUID of the user configured in the [service_user] section of nova.conf - If the role name in Keystone to identify a service is not "service"' then the operator should also replace "role:service" accordingly and also make appropriate adjustment to the [keystone_authtoken]/service_token_roles setting in your cinder configuration file. - It may not be obvious why there are four policy targets defined. It's because the Block Storage API v3 provides four different API calls by which an attachment-delete may be accomplished: ** DELETE /v3/attachments/{attachment_id} (introduced in microversion 3.27, the preferred way to do this) ** POST /v3/volumes/{volume_id} with 'os-detach' action in the request body ** POST /v3/volumes/{volume_id} with 'os-terminate_connection' action in the request body ** POST /v3/volumes/{volume_id} with 'os-force_detach' action in the request body Limitations: The drawback to this configuration-only approach is that while it protects against the intentional case described earlier, it does not protect against the accidental case. Additionally, it is not fine-grained enough to distinguish acceptable end-user Block Storage API attachment-delete requests from unsafe ones (the Cinder code change is required for that). For these reasons, we emphasize that this is only a short-term mitigation, and we recommend that the full fix be applied as soon as possible in your deployment. Warning: Note that if you deploy this short-term mitigation, you should roll back the policy changes after the full fix is applied, or end users will continue to be unable to make acceptable attachment-delete requests. ### Credits ### Jan Wasilewski, Atman Gorka Eguileor, Red Hat ### Contacts / References ### Authors: - Brian Rosmaita, Red Hat - Dan Smith, Red Hat - Gorka Eguileor, Red Hat - Jeremy Stanley, OpenInfra Foundation - Nick Tait, Red Hat This OSSN: https://wiki.openstack.org/wiki/OSSN/OSSN-0092#Discussion Original Launchpad bug: https://launchpad.net/bugs/2004555 Mailing List : [security-sig] tag on openstack-discuss at lists.openstack.org OpenStack Security Project : https://launchpad.net/~openstack-ossg CVE: CVE-2023-2088 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 963 bytes Desc: not available URL: