Open Stack

Hi OpenStack Community,

I am having some issues with key management in a multinode devstack (from master branch 27th July '18) environment where Barbican is the configured key_manager.  I have followed setup instructions from the following pages:

  *   https://docs.openstack.org/barbican/latest/contributor/devstack.html (manual configuration)
  *   https://docs.openstack.org/cinder/latest/configuration/block-storage/volume-encryption.html

So far:

  *   Unencrypted block volumes can be attached to instances on any compute node
  *   Instances with unencrypted volumes can also be live migrated to other compute node
  *   Encrypted bootable volumes created successfully
  *   Instances can be launched using these encrypted volumes when the instance is spawned on demo_machine1 (controller & compute node)
  *   Instances cannot be launched using encrypted volumes when the instance is spawned on demo_machine2 or demo_machine3 (compute only), the same failure can be seen in nova logs from both compute nodes:

Jul 30 14:35:18 demo_machine2 nova-compute[25686]: DEBUG cinderclient.v3.client [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] GET call to cinderv3 for http://10.0.0.63/volume/v3/3f22a0262a7b4832a08c24ac0295cbd9/volumes/296148bf-edb8-4c9f-88c2-44464907f7e7/encryption used request id req-71fa7f20-c0bc-46c3-9f07-5866344d31a1 {{(pid=25686) request /usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py:844}}

Jul 30 14:35:18 demo_machine2 nova-compute[25686]: DEBUG os_brick.encryptors [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] Using volume encryption metadata '{u'cipher': u'aes-xts-plain64', u'encryption_key_id': u'da7ee21c-67ff-4d74-95a0-18ee6c25d85a', u'provider': u'luks', u'key_size': 256, u'control_location': u'front-end'}' for connection: {'status': u'attaching', 'detached_at': u'', u'volume_id': u'296148bf-edb8-4c9f-88c2-44464907f7e7', 'attach_mode': u'null', 'driver_volume_type': u'iscsi', 'instance': u'e0dc6eac-09bb-4232-bea7-7b8b161cfa31', 'attached_at': u'2018-07-30T13:35:17.000000', 'serial': u'296148bf-edb8-4c9f-88c2-44464907f7e7', 'data': {'device_path': '/dev/disk/by-id/scsi-SEMC_SYMMETRIX_900049_wy000', u'target_discovered': True, u'encrypted': True, u'qos_specs': None, u'target_iqn': u'iqn.1992-04.com.emc:600009700bcbb7112504018f00000000', u'target_portal': u'192.168.0.60:3260', u'volume_id': u'296148bf-edb8-4c9f-88c2-44464907f7e7', u'target_lun': 1, u'access_mode': u'rw'}} {{(pid=25686) get_encryption_metadata /usr/local/lib/python2.7/dist-packages/os_brick/encryptors/__init__.py:125}}

Jul 30 14:35:18 demo_machine2 nova-compute[25686]: WARNING keystoneauth.identity.generic.base [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] Failed to discover available identity versions when contacting http://localhost/identity/v3. Attempting to parse version from URL.: NotFound: Not Found (HTTP 404)

Jul 30 14:35:18 demo_machine2 nova-compute[25686]: ERROR castellan.key_manager.barbican_key_manager [None req-3c977faa-a64c-4536-82c8-d1dbaf856b99 admin admin] Error creating Barbican client: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Not Found (HTTP 404): DiscoveryFailure: Could not find versioned identity endpoints when attempting to authenticate. Please check that your auth_url is correct. Not Found (HTTP 404)

All instance of Nova have [key_manager] configured as follows:
[key_manager]
backend = barbican
auth_url = http://10.0.0.63/identity/
### Tried with and without the below config options, same result
# auth_type = password
# password = devstack
# username = barbican

Any assistance here would be greatly appreciated, I have spent a lot of time looking for some additional information for the use of Barbican in multinode devstack environments or with live migration but there is nothing out there, everything is for all-in-one environments and I'm not having any issues when everything is on one node. I am wondering if at this point there is something I am missing in terms of services in a multinode devstack environment, qualification of barbican in a multinode environment is outside of the recommended test config but following the docs it looks very straight forward.

Some information on the three nodes in my environment are below, if there is any other information I can provide let me know, thanks for the help!

Node & Service Breakdown
Node 1 (Controller & Compute)
stack at demo_machine1:~$ openstack service list
+----------------------------------+-------------+----------------+
| ID                               | Name        | Type           |
+----------------------------------+-------------+----------------+
| 43a1334c755c4c81969565097cc9c30c | cinder      | volume         |
| 52a8927c09154e33900f24c7c95a9f8b | cinderv2    | volumev2       |
| 5427a9dff3b6477197062e1747843c4d | nova_legacy | compute_legacy |
| 5b319b6d50634661998fdd8dc70a85e3 | nova        | compute        |
| 5ffbb2e9f7c84c9e9601ab7aba0cf5e1 | placement   | placement      |
| 787fd29afe2f41b0bb44f9c301fd22c5 | cinderv3    | volumev3       |
| 96813e167b8842aba9d8b94fad67904f | neutron     | network        |
| 993e615a03cc49e3be94840c0b82636b | swift       | object-store   |
| b3834468ffc44f30b792459611f5f4e9 | cinder      | block-storage  |
| cab9ff9e175f4566a1865ea35a377d0d | barbican    | key-manager    |
| d12f710b815442fb970c22087b6e8f4f | glance      | image          |
| eb80de21e42b4e978985db979b175f79 | keystone    | identity       |
+----------------------------------+-------------+----------------+

stack at demo_machine1:~$ openstack endpoint list
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+
| ID                               | Region    | Service Name | Service Type   | Enabled | Interface | URL                                             |
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+
| 00b276609956454d8d80dd0dde0df231 | RegionOne | cinder       | volume         | True    | public    | http://10.0.0.63/volume/v1/$(project_id)s    |
| 18e5d431143d47ed980ee0ffbf0d03d7 | RegionOne | barbican     | key-manager    | True    | public    | http://10.0.0.63/key-manager                 |
| 20cfe0a80cc94b6eb8ea8e6784839198 | RegionOne | barbican     | key-manager    | True    | internal  | http://10.0.0.63/key-manager                 |
| 3a740b472e7349f19d0cf110c1792122 | RegionOne | cinderv3     | volumev3       | True    | public    | http://10.0.0.63/volume/v3/$(project_id)s    |
| 4d957921fe894abba296331869f82f7f | RegionOne | cinderv2     | volumev2       | True    | public    | http://10.0.0.63/volume/v2/$(project_id)s    |
| 4df258794fde476ab82502c682848e58 | RegionOne | swift        | object-store   | True    | admin     | http://10.0.0.63:8080                        |
| 719eabec7cb94580af9f928278589878 | RegionOne | keystone     | identity       | True    | public    | http://10.0.0.63/identity                    |
| 792f4c99085f4b008643b08aff463759 | RegionOne | keystone     | identity       | True    | admin     | http://10.0.0.63/identity                    |
| 9e8c27c6e22f4a70865bfcdd815ed3c0 | RegionOne | cinder       | block-storage  | True    | public    | http://10.0.0.63/volume/v3/$(project_id)s    |
| a271f19f29d443a0b5545626584389d7 | RegionOne | glance       | image          | True    | public    | http://10.0.0.63/image                       |
| a975403a2ff149bb88ce2d2227d17a80 | RegionOne | nova         | compute        | True    | public    | http://10.0.0.63/compute/v2.1                |
| b65b46e83b4547588eb694d63cb5cdd5 | RegionOne | swift        | object-store   | True    | public    | http://10.0.0.63:8080/v1/AUTH_$(project_id)s |
| bfd1f91ba18b4bc0bc83586ee358a73c | RegionOne | placement    | placement      | True    | public    | http://10.0.0.63/placement                   |
| d38a11dcfe824fe28f70b45422277d26 | RegionOne | nova_legacy  | compute_legacy | True    | public    | http://10.0.0.63/compute/v2/$(project_id)s   |
| ea9139e670e84ff39d1c052347a04695 | RegionOne | neutron      | network        | True    | public    | http://10.0.0.63:9696/                       |
+----------------------------------+-----------+--------------+----------------+---------+-----------+-------------------------------------------------+

      stack at demo_machine1:~$ openstack secret store
+---------------+---------------------------------------------------------------------------------+
| Field         | Value                                                                           |
+---------------+---------------------------------------------------------------------------------+
| Secret href   | http://10.0.0.63/key-manager/v1/secrets/72a3955b-a494-4352-b1f6-ae3f322e5656 |
| Name          | None                                                                            |
| Created       | 2018-07-30T12:58:33+00:00                                                       |
| Status        | ACTIVE                                                                          |
| Content types | None                                                                            |
| Algorithm     | aes                                                                             |
| Bit length    | 256                                                                             |
| Secret type   | opaque                                                                          |
| Mode          | cbc                                                                             |
| Expiration    | None                                                                            |
+---------------+---------------------------------------------------------------------------------+

Node 2 & 3 (Compute Only)
Services:
      stack at demo_machine2:~$ sudo systemctl list-unit-files | grep devstack@*
      devstack at n-api-meta.service<mailto:devstack at n-api-meta.service>                enabled
      devstack at n-cpu.service<mailto:devstack at n-cpu.service>                     enabled
      devstack at q-agt.service<mailto:devstack at q-agt.service>                     enabled

      stack at demo_machine3:~$ sudo systemctl list-unit-files | grep devstack@*
      devstack at n-api-meta.service<mailto:devstack at n-api-meta.service>                enabled
      devstack at n-cpu.service<mailto:devstack at n-cpu.service>                     enabled
      devstack at q-agt.service<mailto:devstack at q-agt.service>                     enabled


********************************************************************



Michael McAleer
Software Engineer 1, Core Technologies
Dell EMC | Enterprise Storage Division
Phone: +353 21 428 1729
Michael.Mcaleer at Dell.com<mailto:Michael.Mcaleer at Dell.com>
Ireland COE, Ovens, Co. Cork, Ireland


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20180730/1b78efa3/attachment.html>