OSSN-0094: Ensuring Volume Safety with Nova and Watcher
== Summary ==
A vulnerability has been identified in OpenStack Nova and OpenStack Watcher
in conjunction with volume swap operations performed by the Watcher service.
Under specific circumstances, this can lead to a situation where two Nova
libvirt instances could reference the same block device, allowing accidental
information disclosure to the unauthorized instance.
== Affected Services / Software ==
Services: Nova,Watcher
Releases: all supported releases
== Discussion ==
The issue occurs when Watcher's zone migration strategy performs the
following
sequence of events:
1. Watcher initiates a volume swap using Nova's internal-only volume
swap API
2. Watcher initiates a live migration of the same instance
3. In some error cases connection details may have failed to update
storage references. These invalid details are used during the live
migration.
=== Required Access ===
The swap volume, live migration and all Watcher APIs are admin only so with
default policy is only possible to create the inconsistent state
described in
this OSSN if you have admin rights on the relevant OpenStack project.
=== Further Watcher Hardening ===
The Watcher service, when first created, often implemented its own means
to perform operations. Many of those operations can now be done natively
via other OpenStack services. In the specific context of OSSN-0094,
the ability to migrate Cinder volumes between storage backends is such an
example.
Additionally, the Cinder volume migration in Watcher created a new Keystone
user with the admin role assigned for the instance owners' project and then
used that user to perform API requests on behalf of the project. This code
has been removed.
Finally, due to limited error handling and no validation that the objects
involved were migrated properly, some error scenarios could have led to
a source volume being deleted despite not having been migrated properly.
=== Resolution ===
Nova will now reject any request to swap a volume that has an empty
migration
status, effectively restricting the usage of this API to Cinder. This brings
the API validation in line with the documentation.
Watchers internal implementation of swap volume has been deleted and updated
to use Cinder's native volume migration as a replacement. Watcher no longer
creates temporary Keystone users in normal operation.
=== Patches ===
Patches for Nova and Watcher have been backported to all supported stable
branches and committed to master branch.
stable/2025.1:
* Watcher: https://review.opendev.org/c/openstack/watcher/+/957770
* Nova: https://review.opendev.org/c/openstack/nova/+/957759
stable/2024.2:
* Watcher: https://review.opendev.org/c/openstack/watcher/+/957773
* Nova: https://review.opendev.org/c/openstack/nova/+/957762
stable/2024.1:
* Watcher: https://review.opendev.org/c/openstack/watcher/+/957774
* Nova: https://review.opendev.org/c/openstack/nova/+/957764
== Recommended Actions ==
* Operators using Watcher's zone migration strategy should apply the
provided Watcher and Nova patches as soon as possible.
* Operators should refrain from using the swap volume migration action
in Watcher. The compatibility code for swap volume that uses a
Cinder-based migration may be removed in a future API version.
* Operators should audit all users with the admin role and ensure no
temporary Watcher-created users remain.
* Operators using custom policy for volume attachment
(''/servers/{server_id}/os-volume_attachments/{volume_id}'') or live
migration API should review the state of existing instances which have
had volume migrations. Any instance in an inconsistent state can be
resolved by hard rebooting the instance using Nova's API.
== Contacts / References ==
* Author: Sean Mooney <smooney(a)redhat.com>, Jay Faulkner <jay(a)jvf.cc>
* This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0094
* Original LaunchPad Bug : https://bugs.launchpad.net/nova/+bug/2112187
* Mailing List : [Security] tag on openstack-discuss(a)lists.openstack.org
* OpenStack Security Project : https://launchpad.net/~openstack-ossg
* CVE: None