Open Stack

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Ceph user credential leakage to consumers of OpenStack Manila
- -------------------------------------------------------------

### Summary ###

OpenStack Manila users can request access on a share to any
arbitrary cephx user, including privileged pre-existing users
of a Ceph cluster. They can then retrieve access secret keys
for these pre-existing ceph users via Manila APIs. A cephx
client user name and access secret key are required to mount
a Native CephFS manila share. With a secret key, a manila user
can impersonate a pre-existing ceph user and gain capabilities
to manipulate resources that the manila user was never intended
to have access to. It is possible to even obtain the default
ceph "admin" user's key in this manner, and execute any commands
as the ceph administrator.


### Affected Services / Software ###

- - OpenStack Shared File Systems Service (Manila) versions Mitaka (2.0.0)
  through Victoria (11.0.0)
- - Ceph Luminous (<=v12.2.13), Mimic (<=v13.2.10),
  Nautilus (<=v14.2.15), Octopus (<=v15.2.7)

### Discussion ###

OpenStack Manila can provide users with Native CephFS shared
file systems. When a user creates a "share" (short for
"shared file system") via Manila, a CephFS "subvolume" is
created on the Ceph cluster and exported. After creating
their share, a user  can specify who can have access to the
share with the help of "cephx" client user names. A cephx
client corresponds to Ceph Client Users [2]. When access
is provided, a client user "access key" is returned via
manila.

A ceph client user account is required to access any ceph
resource. This includes interacting with Ceph cluster
infrastructure daemons (ceph-mgr, ceph-mds, ceph-mon, ceph-osd)
or consuming Ceph storage via RBD, RGW or CephFS. Deployment and
orchestration services like ceph-ansible, nfs-ganesha, kolla,
tripleo need ceph client users to work, as do OpenStack services
such as cinder, manila, glance and nova for their own interactions
with Ceph. For the purpose of illustrating this vulnerability,
we'll call them "pre-existing" users of the Ceph cluster. Another
example of a pre-existing user includes the "admin" user that
is created by default on the ceph cluster.

In theory, manila's cephx users are no different from a ceph
client user. When a manila user requests access to a share,
a corresponding ceph user account is created if one already
does not exist. If a ceph user account already exists, the
existing capabilities of that user are adjusted to provide
them permissions to access the manila share in question.
There is no reasonable way for this mechanism to know what
pre-existing ceph client users must be protected against
unauthorized abuse. Therefore there is a risk that a
manila user can claim to be a pre-existing ceph user to
steal their access secret key.

To resolve this issue, the ceph interface that manila uses
was patched to no longer allow manila to claim a pre-existing
user account that didn't create. By consequence this means
that manila users cannot use cephx usernames that correspond
to ceph client users that exist outside of manila.


### Recommended Actions ###

#. Upgrade your ceph software to the latest patched releases of
   ceph to take advantage of the fix to this vulnerability.

#. Audit cephx access keys provisioned via manila. You may use
   "ceph auth ls" and ensure that no clients have been compromised.
   If they have been, you may need to delete and recreate the
   client credentials to prevent unauthorized access.

#. The audit can also be performed on manila by enumerating all
   CephFS shares and their access rules as a system administrator. If a
   reserved ceph client username has been used, you may deny access
   and recreate the client credential on ceph to refresh the
   access secret.

No code changes were necessary in the OpenStack Shared File
System service (manila). With an upgraded ceph, when manila
users attempt to provide share access to a cephx username
that they cannot use, the access rule's "state" attribute is
set to "error" because this operation is no longer permitted.

### Patches ###

The Ceph community has provided the following patches:

Ceph Octopus: https://github.com/ceph/ceph/commit/1b8a634fdcd94dfb3ba650793fb1b6d09af65e05
Ceph Nautilus: https://github.com/ceph/ceph/commit/7e3e4e73783a98bb07ab399438eb3aab41a6fc8b
Ceph Luminous: https://github.com/ceph/ceph/commit/956ceb853a58f6b6847b31fac34f2f0228a70579

The fixes are in the latest releases of Ceph Nautilus (14.2.16) and Ceph
Octopus (15.2.8). The patch for Luminous was provided as a courtesy to possible
users of OpenStack Manila, however the Ceph community no longer produces
releases for Luminous or Mimic as they are end of life. See `here for
information about ceph releases.
<https://docs.ceph.com/en/latest/releases/general/>`_

### Contacts / References ###

Author:
- - Pacha Ravi, Goutham gouthamr at redhat.com (Red Hat)

Credits:
- - Garbutt, John john at johngarbutt.com (StackHPC)
- - Babel, Jahson jahson.babel at cc.in2p3.fr (Centre de Calcul de l'IN2P3)

This OSSN : https://wiki.openstack.org/wiki/OSSN/OSSN-0087
Original LaunchPad Bug : https://launchpad.net/bugs/1904015
Mailing List : [Security] tag on openstack-discuss at lists.openstack.org
OpenStack Security Project : https://launchpad.net/~openstack-ossg
CVE: CVE-2020-27781

-----BEGIN PGP SIGNATURE-----

wsFcBAEBCAAGBQJf2raCAAoJENJP32eYWZR3mREP/ij+So0KHK7dD3WAdcVK
0JdGzwOjX2Bc4/7g5RPzn4RaxZKicsBOWqESCTTBl94oG4XvTax3fW0E6VlL
L6XoV+At1cEvptONDoZ0faCSHfTng1J73rHMo9v+cmxmOuEwXReghArS86tS
KIeRWviW9hyNmZfhJxuAC9ICR0HglhT5VHqNtAjL5WzoFGMtC4VeJ7e8rf8r
PLjvYGOPzNDj8wAn5UvTnJgkT1tbbIZQai4o+QlDJK5eEuEQnwGTUQ/umx/a
z2DeuCnDDxJeOFcWEgkDzTQsE6e7dO4FvoIIsZ3u5pA0Rhw31QfpupUsLAYH
WhAjt7cImKRTfza/zVuS7PAko2fMmuNHyHEQQh2Y80S4nkdo/WAfUaBft8eO
vdNlvunBQA2E7mlK6oxNF22k/pX49N47vVqGttPFA1kNPFg2qLcc8mJIxpjf
V4sJVfMO/DuQaU1zTH/P9KsYm/hlyVzLELEvDR43jmWg4p4btZT8Z2OVD80r
9/dRMcbRQl3yq1C2L7yyKVOc6Pw9HJ90ixXpgv7ZT6TbwAPzFe/euITmA6H0
EGzinRg7JmfFFnf/9FBEZJRL46/idMLVNRo2XioA6+o2kngkwSsD+uJTlMFG
XQJINZwrbL1xpuh3VGaOI44yCimsQwNGURH2NmZ7uFBKy6CHYXX42ydnQk51
w9qp
=o0Lx
-----END PGP SIGNATURE-----