Open Stack

Since there were no objections to the revised plan, I've switched the
report to public, set the advisory task to won't fix and added a
confirmed task for a security note. Thanks!

** Also affects: ossn
   Importance: Undecided
       Status: New

** Changed in: ossa
       Status: Incomplete => Won't Fix

** Changed in: ossn
       Status: New => Confirmed

** Changed in: ossn
     Assignee: (unassigned) => Brian Rosmaita (brian-rosmaita)

** Description changed:

- This issue is being treated as a potential security risk under embargo.
- Please do not make any public mention of embargoed (private) security
- vulnerabilities before their coordinated publication by the OpenStack
- Vulnerability Management Team in the form of an official OpenStack
- Security Advisory. This includes discussion of the bug or associated
- fixes in public forums such as mailing lists, code review systems and
- bug trackers. Please also avoid private disclosure to other individuals
- not already approved for access to this information, and provide this
- same reminder to those who are made aware of the issue prior to
- publication. All discussion should remain confined to this private bug
- report, and any proposed fixes should be added to the bug as
- attachments.
- 
  Cinder + ceph backend, secret key leak
  
  Conditions: cinder + ceph backend + rbd_keyring_conf set in cinder
  config files
  
  As an authenticated simple user create a cinder volume that ends up on a ceph backend,
  Then reuse the os.initialize_connection api call
  (used by nova-compute/cinder-backup to attach volumes locally to the host running the services):
  
  curl -g -i -X POST https://<cinder_controller>/v3/c495530af57611e9bc14bbaa251e1e96/volumes/7e59b91e-d426-4294-bfc5-dfdebcb21879/action \
      -H "Accept: application/json" \
      -H "Content-Type: application/json" \
      -H "OpenStack-API-Version: volume 3.15" \
      -H "X-Auth-Token: $TOKEN" \
      -d '{"os-initialize_connection": {"connector":{}}}'
  
  If you do not want to forge the http request, openstack clients and
  extensions may prove helpful.
  
  As root:
  
  apt-get install python3-oslo.privsep virtualenv python3-dev python3-os-brick gcc ceph-common
  virtualenv -p python3 venv_openstack
  source venv_openstack/bin/activate
  pip install python-openstackclient
  pip install python-cinderclient
  pip install os-brick
  pip install python-brick-cinderclient-ext
  cinder create vol 1
  cinder --debug local-attach 7e59b91e-d426-4294-bfc5-dfdebcb21879
  
  This leaks the ceph credentials for the whole ceph cluster, leaving anyone able to go through ceph acls to get access
  to all the volumes within the cluster.
  
  {
     "connection_info" : {
        "data" : {
           "access_mode" : "rw",
           "secret_uuid" : "SECRET_UUID",
           "cluster_name" : "ceph",
           "encrypted" : false,
           "auth_enabled" : true,
           "discard" : true,
           "qos_specs" : {
              "write_iops_sec" : "3050",
              "read_iops_sec" : "3050"
           },
           "keyring" : "SECRETFILETOHIDE",
           "ports" : [
              "6789",
              "6789",
              "6789"
           ],
           "name" : "volumes/volume-7e59b91e-d426-4294-bfc5-dfdebcb21879",
           "secret_type" : "ceph",
           "hosts" : [
              "ceph_host1",
              "ceph_host2",
              ...
           ],
           "volume_id" : "7e59b91e-d426-4294-bfc5-dfdebcb21879",
           "auth_username" : "cinder"
        },
        "driver_volume_type" : "rbd"
     }
  }
  
  Quick workaround:
  1. Remove rbd_keyring_conf param from any cinder config file, this will mitigate the information disclosure.
  2. For cinder backups to still work, providers should instead deploy their ceph keyring secrets directly on cinder-backup hosts
  (/etc/cinder/<backend_name>.keyring.conf, to be confirmed).
  
  Note that nova-compute hosts should not be impacted by the change, because ceph secrets are expected to be stored in
  libvirt secrets already, thus making this keyring disclose useless to it.
  (to be confirmed, there may be other compute drivers that might be impacted by this)
  
  Quick code fix:
  Mandatory: revert this commit https://review.opendev.org/#/c/456672/
  Optional: revert this one https://review.opendev.org/#/c/465044/, harmless in itself, but pointless once the first one has been reverted
  
  Long term code fix proposals:
  What the os.initialize_connection api call is meant to: allow simple users to use cinder as block storage as a service
  in order to attach volumes outside the scope of any virtual machines/nova.
  Thus, information returned by this call should give enough information for a volume attach to be possible for the caller but they should not disclose
  anything that would allow him to do more than that.
  Since it is not possible at all with ceph to do so (no tenant isolation within ceph cluster),
  the related cinder backend for ceph should not implement this route at all
  There is indeed no reason why cinder should disclose anything here about ceph cluster, including hosts, cluster-ids,
  if the attach is doomed to fail for users missing secret informations anyway.
  Then, any 'admin' service using this call to locally attach the volumes (nova-compute, cinder-backup...) should be modified to:
  - check caller rw permissions on requested volumes
  - escalate the request
  - go through a new admin api route, not this 'user' one

** Information type changed from Private Security to Public

** Tags added: security

-- 
You received this bug notification because you are a member of OpenStack
Security SIG, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1849624

Title:
  ceph backend, secret key leak

Status in Cinder:
  New
Status in OpenStack Security Advisory:
  Won't Fix
Status in OpenStack Security Notes:
  Confirmed

Bug description:
  Cinder + ceph backend, secret key leak

  Conditions: cinder + ceph backend + rbd_keyring_conf set in cinder
  config files

  As an authenticated simple user create a cinder volume that ends up on a ceph backend,
  Then reuse the os.initialize_connection api call
  (used by nova-compute/cinder-backup to attach volumes locally to the host running the services):

  curl -g -i -X POST https://<cinder_controller>/v3/c495530af57611e9bc14bbaa251e1e96/volumes/7e59b91e-d426-4294-bfc5-dfdebcb21879/action \
      -H "Accept: application/json" \
      -H "Content-Type: application/json" \
      -H "OpenStack-API-Version: volume 3.15" \
      -H "X-Auth-Token: $TOKEN" \
      -d '{"os-initialize_connection": {"connector":{}}}'

  If you do not want to forge the http request, openstack clients and
  extensions may prove helpful.

  As root:

  apt-get install python3-oslo.privsep virtualenv python3-dev python3-os-brick gcc ceph-common
  virtualenv -p python3 venv_openstack
  source venv_openstack/bin/activate
  pip install python-openstackclient
  pip install python-cinderclient
  pip install os-brick
  pip install python-brick-cinderclient-ext
  cinder create vol 1
  cinder --debug local-attach 7e59b91e-d426-4294-bfc5-dfdebcb21879

  This leaks the ceph credentials for the whole ceph cluster, leaving anyone able to go through ceph acls to get access
  to all the volumes within the cluster.

  {
     "connection_info" : {
        "data" : {
           "access_mode" : "rw",
           "secret_uuid" : "SECRET_UUID",
           "cluster_name" : "ceph",
           "encrypted" : false,
           "auth_enabled" : true,
           "discard" : true,
           "qos_specs" : {
              "write_iops_sec" : "3050",
              "read_iops_sec" : "3050"
           },
           "keyring" : "SECRETFILETOHIDE",
           "ports" : [
              "6789",
              "6789",
              "6789"
           ],
           "name" : "volumes/volume-7e59b91e-d426-4294-bfc5-dfdebcb21879",
           "secret_type" : "ceph",
           "hosts" : [
              "ceph_host1",
              "ceph_host2",
              ...
           ],
           "volume_id" : "7e59b91e-d426-4294-bfc5-dfdebcb21879",
           "auth_username" : "cinder"
        },
        "driver_volume_type" : "rbd"
     }
  }

  Quick workaround:
  1. Remove rbd_keyring_conf param from any cinder config file, this will mitigate the information disclosure.
  2. For cinder backups to still work, providers should instead deploy their ceph keyring secrets directly on cinder-backup hosts
  (/etc/cinder/<backend_name>.keyring.conf, to be confirmed).

  Note that nova-compute hosts should not be impacted by the change, because ceph secrets are expected to be stored in
  libvirt secrets already, thus making this keyring disclose useless to it.
  (to be confirmed, there may be other compute drivers that might be impacted by this)

  Quick code fix:
  Mandatory: revert this commit https://review.opendev.org/#/c/456672/
  Optional: revert this one https://review.opendev.org/#/c/465044/, harmless in itself, but pointless once the first one has been reverted

  Long term code fix proposals:
  What the os.initialize_connection api call is meant to: allow simple users to use cinder as block storage as a service
  in order to attach volumes outside the scope of any virtual machines/nova.
  Thus, information returned by this call should give enough information for a volume attach to be possible for the caller but they should not disclose
  anything that would allow him to do more than that.
  Since it is not possible at all with ceph to do so (no tenant isolation within ceph cluster),
  the related cinder backend for ceph should not implement this route at all
  There is indeed no reason why cinder should disclose anything here about ceph cluster, including hosts, cluster-ids,
  if the attach is doomed to fail for users missing secret informations anyway.
  Then, any 'admin' service using this call to locally attach the volumes (nova-compute, cinder-backup...) should be modified to:
  - check caller rw permissions on requested volumes
  - escalate the request
  - go through a new admin api route, not this 'user' one

To manage notifications about this bug go to:
https://bugs.launchpad.net/cinder/+bug/1849624/+subscriptions