<div dir="ltr">Hi Lee,<div><br></div><div>sorry for long delay in my response, but I wanted to fully test it and thanks to your draft I was able to decrypt my encrypted volume. Your procedure is well described, so I just collected all steps in one place and wanted to share in this conversation:</div><div><br></div><div><div class="gmail-gE gmail-iv gmail-gt" style="padding:20px 0px 0px;font-size:0.875rem;font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif"><br class="gmail-Apple-interchange-newline"><table cellpadding="0" class="gmail-cf gmail-gJ" style="border-collapse:collapse;margin-top:0px;width:auto;font-size:0.875rem;letter-spacing:0.2px;display:block"><tbody style="display:block"><tr class="gmail-acZ" style="height:auto;display:flex"><td class="gmail-gF gmail-gK" style="white-space:nowrap;padding:0px;vertical-align:top;width:1401.56px;line-height:20px;display:block;max-height:20px"><table cellpadding="0" class="gmail-cf gmail-ix" style="border-collapse:collapse;table-layout:fixed;width:1401px"><tbody><tr><td class="gmail-c2" style="display:flex"><h3 class="gmail-iw" style="overflow:hidden;font-size:0.75rem;font-weight:inherit;margin:inherit;text-overflow:ellipsis;letter-spacing:0.3px;color:rgb(95,99,104);line-height:20px"><span class="gmail-qu" tabindex="-1"><span name="Jan Wasilewski" class="gmail-gD" style="color:rgb(32,33,36);font-size:0.875rem;font-weight:bold;display:inline;vertical-align:top;letter-spacing:0.2px;line-height:20px">Jan Wasilewski</span></span></h3></td></tr></tbody></table></td><td class="gmail-gH gmail-bAk" style="text-align:right;white-space:nowrap;vertical-align:top;display:block;max-height:20px"><div class="gmail-gK" style="padding:0px;display:flex"><img class="gmail-f gmail-gW" src="https://mail.google.com/mail/u/0/images/cleardot.gif" title="image001.jpg, noname" alt="Załączniki" style="vertical-align: top; background: url("https://www.gstatic.com/images/icons/material/system/1x/attachment_black_20dp.png") 50% 50% / 20px no-repeat; height: 20px; width: 20px; opacity: 0.54;"><span id="gmail-:15l" class="gmail-g3" title="23 lut 2021, 11:04" alt="23 lut 2021, 11:04" tabindex="-1" style="vertical-align:top;margin:0px;font-size:0.75rem;letter-spacing:0.3px;color:rgb(95,99,104);display:block;line-height:20px">11:04 (1 minutę temu)</span><div class="gmail-zd gmail-bi4" tabindex="0" style="display:inline-block;height:20px;margin-left:20px;outline:0px"><span class="gmail-T-KT" style="display:inline-flex;height:20px;text-align:center;width:20px;padding:0px;margin:0px;border:none;outline:none"></span></div></div></td><td class="gmail-gH" style="text-align:right;white-space:nowrap;vertical-align:top;display:flex"></td><td class="gmail-gH gmail-acX gmail-bAm" rowspan="2" style="text-align:right;white-space:nowrap;vertical-align:top;display:block;max-height:20px"><div class="gmail-T-I gmail-J-J5-Ji gmail-T-I-Js-IF gmail-aaq gmail-T-I-ax7 gmail-L3" tabindex="0" style="display:inline-flex;border-radius:2px 0px 0px 2px;font-size:0.875rem;text-align:center;margin:0px 0px 0px 20px;height:20px;line-height:18px;min-width:0px;outline:none;padding:0px;background:transparent;color:rgb(68,68,68);border:none"><img class="gmail-hB gmail-T-I-J3" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt="" style="background: url("https://www.gstatic.com/images/icons/material/system/1x/reply_black_20dp.png") 50% 50% / 20px no-repeat; height: 20px; margin: 0px; vertical-align: middle; width: 20px; opacity: 0.7; display: inline-block; padding: 0px; transition: opacity 0.15s cubic-bezier(0.4, 0, 0.2, 1) 0s;"></div><div id="gmail-:14a" class="gmail-T-I gmail-J-J5-Ji gmail-T-I-Js-Gs gmail-aap gmail-T-I-awG gmail-T-I-ax7 gmail-L3" tabindex="0" style="display:inline-flex;border-radius:0px 2px 2px 0px;font-size:0.875rem;text-align:center;margin:0px 0px 0px 20px;height:20px;line-height:18px;min-width:0px;outline:none;padding:0px;background:transparent;color:rgb(68,68,68);border:none"><img class="gmail-hA gmail-T-I-J3" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt="" style="background: url("https://www.gstatic.com/images/icons/material/system/1x/more_vert_black_20dp.png") 50% 50% / 20px no-repeat; height: 20px; width: 20px; margin: 0px; vertical-align: middle; opacity: 0.7; display: inline-block; padding: 0px; transition: opacity 0.15s cubic-bezier(0.4, 0, 0.2, 1) 0s;"></div></td></tr><tr class="gmail-acZ gmail-xD" style="height:auto;display:flex"><td colspan="3"><table cellpadding="0" class="gmail-cf gmail-adz" style="border-collapse:collapse;table-layout:fixed;white-space:nowrap;width:1672px"><tbody><tr><td class="gmail-ady" style="overflow:visible;text-overflow:ellipsis;display:flex;line-height:20px"><div class="gmail-iw gmail-ajw" style="overflow:hidden;max-width:92%;display:inline-block"><span class="gmail-bfM" style="display:inline-block;height:18px;vertical-align:text-top;width:18px;background:url("https://ssl.gstatic.com/mail/sprites/smime-69772d50adad95aedb3d9639945f861d.png") 0px 0px no-repeat"></span> <span class="gmail-hb" style="vertical-align:top;color:rgb(95,99,104);font-size:0.75rem;letter-spacing:0.3px;line-height:20px">do <span dir="ltr" name="mnie" class="gmail-g2" style="vertical-align:top">mnie</span></span></div><div id="gmail-:158" class="gmail-ajy" tabindex="0" style="display:inline-flex;margin-left:4px;vertical-align:top;border:none;outline:none"><img class="gmail-ajz" src="https://mail.google.com/mail/u/0/images/cleardot.gif" alt="" style="background: url("https://www.gstatic.com/images/icons/material/system/1x/arrow_drop_down_black_20dp.png") 50% 50% / 20px no-repeat; cursor: pointer; padding: 0px; vertical-align: baseline; height: 20px; width: 20px; border: none; margin: 0px 0px 0px auto; right: 0px; top: 0px; display: flex; opacity: 0.54;"></div></td></tr></tbody></table></td></tr></tbody></table></div><div id="gmail-:15p" style="font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif;font-size:medium"><div class="gmail-qQVYZb"></div><div class="gmail-utdU2e"></div><div class="gmail-lQs8Hd"></div><div class="gmail-btm"></div></div><div class="gmail-" style="font-size:medium"><div class="gmail-aHl" style="font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif"></div><div id="gmail-:159" tabindex="-1" style="font-family:Roboto,RobotoDraft,Helvetica,Arial,sans-serif"></div><div id="gmail-:15n" class="gmail-ii gmail-gt" style="font-size:0.875rem;direction:ltr;margin:8px 0px 0px;padding:0px"><div id="gmail-:15o" class="gmail-a3s gmail-aiL" style="overflow:hidden;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:small;line-height:1.5">**Description:**<br><br>As an admin, you would like to decrypt the volume, which is attached to compute node and check, that your barbican secret key is correct(i.e. customer is saying, that the barbican secret key doesn't work). This procedure describes, how you can simply test it.<br><br>**Starting point:**<br><br>Volume is encrypted and attached to an instance(instance has to be shutdown to make qemu commands operational). Our volume id is: ca8da832-a88d-4f91-ab2d-2bd3efbca4a3<br><br>**Procedure:**<br><br>Log in to a compute node that is hosting your instance. List volumes attached to your instance:<br>```<br><font face="monospace">[TEST]root@comp-09:/home/jwasilewski# virsh domblklist ec9081e4-e1e4-40a2-bf8c-c87c14b79d5a<br>Target Source<br>------------------------------------------------<br>vda /dev/dm-29<br>vdb /dev/disk/by-id/wwn-0x6e00084100ee7e7e7fe79b5900003a89</font><br>```<br>In our case vdb volume is an encrypted one. We can check it by qemu-img command:<br>```<br><font face="monospace">[TEST]root@comp-09:/home/jwasilewski# qemu-img info /dev/disk/by-id/wwn-0x6e00084100ee7e7e7fe79b5900003a89<br>image: /dev/disk/by-id/wwn-0x6e00084100ee7e7e7fe79b5900003a89<br>file format: luks<br>virtual size: 20G (21472739328 bytes)<br>disk size: 0<br>encrypted: yes<br>Format specific information:<br> ivgen alg: plain64<br> hash alg: sha256<br> cipher alg: aes-256<br> uuid: 009f60f7-e871-4eac-88da-b274e80eb247<br> cipher mode: xts<br> slots:<br> [0]:<br> active: true<br> iters: 900838<br> key offset: 4096<br> stripes: 4000<br> [1]:<br> active: false<br> key offset: 262144<br> [2]:<br> active: false<br> key offset: 520192<br> [3]:<br> active: false<br> key offset: 778240<br> [4]:<br> active: false<br> key offset: 1036288<br> [5]:<br> active: false<br> key offset: 1294336<br> [6]:<br> active: false<br> key offset: 1552384<br> [7]:<br> active: false<br> key offset: 1810432<br> payload offset: 2097152<br> master key iters: 56302</font><br>```<br><br>We would like to decrypt the volume. We need to retrieve symmetric key which is allocated to this volume from barbican. We need to find a secret store associated with our volume, so we have to login to OpenStack database and execute:<br>```<br><font face="monospace">mysql> select * from volumes where id = 'ca8da832-a88d-4f91-ab2d-2bd3efbca4a3'\G<br>*************************** 1. row ***************************<br> created_at: 2021-02-12 13:41:40<br> updated_at: 2021-02-17 12:33:34<br> deleted_at: NULL<br> deleted: 0<br> id: ca8da832-a88d-4f91-ab2d-2bd3efbca4a3<br> ec2_id: NULL<br> user_id: wfoij24f0sdfs0934nkl<br> project_id: 234sfds90klfgd093n<br> host: cinder-01@huawei_backend#StoragePool001<br> size: 20<br> availability_zone: nova<br> status: in-use<br> attach_status: attached<br> scheduled_at: 2021-02-12 13:41:40<br> launched_at: 2021-02-12 13:41:42<br> terminated_at: NULL<br> display_name: encrypted-volume<br> display_description:<br> provider_location: {"huawei_sn": "2102352VVA10L2000001", "huawei_lun_id": "14985", "huawei_lun_wwn": "6e00084100ee7e7e7fe79b5900003a89"}<br> provider_auth: NULL<br> snapshot_id: NULL<br> volume_type_id: 3129bdc2-6162-4729-9eab-d0c97db2335a<br> source_volid: NULL<br> bootable: 0<br> provider_geometry: NULL<br> _name_id: NULL<br> encryption_key_id: b13d2017-e3e5-4f5f-a836-918ec130dc0a<br> migration_status: NULL<br> replication_status: disabled<br>replication_extended_status: NULL<br> replication_driver_data: NULL<br> consistencygroup_id: NULL<br> provider_id: NULL<br> multiattach: 0<br> previous_status: NULL<br> cluster_name: NULL<br> group_id: NULL<br> service_uuid: 674de52f-1c9a-402f-88c9-6b79c91a4249<br> shared_targets: 1<br>1 row in set (0.00 sec)</font><br>```<br>So encryption_key_id is the value that we were looking for.<br>Then we can simply get our secret store:<br>```<br><font face="monospace">[TEST]root@zabbix-1:~# openstack secret get <a href="http://controller.tc.tester-pl.pl:9311/v1/secrets/b13d2017-e3e5-4f5f-a836-918ec130dc0a">http://controller.tc.tester-pl.pl:9311/v1/secrets/b13d2017-e3e5-4f5f-a836-918ec130dc0a</a><br>+---------------+----------------------------------------------------------------------------------------+<br>| Field | Value |<br>+---------------+----------------------------------------------------------------------------------------+<br>| Secret href | <a href="http://controller.tc.tester-pl.pl:9311/v1/secrets/b13d2017-e3e5-4f5f-a836-918ec130dc0a">http://controller.tc.tester-pl.pl:9311/v1/secrets/b13d2017-e3e5-4f5f-a836-918ec130dc0a</a> |<br>| Name | None |<br>| Created | 2021-02-12T13:41:39+00:00 |<br>| Status | ACTIVE |<br>| Content types | {u'default': u'application/octet-stream'} |<br>| Algorithm | aes |<br>| Bit length | 512 |<br>| Secret type | symmetric |<br>| Mode | None |<br>| Expiration | None |<br>+---------------+----------------------------------------------------------------------------------------+</font><br>```<br>And of course encryption key, by command(we will save it to file **my_symmetric_key.key**):<br>```<br><font face="monospace">barbican secret get --payload_content_type application/octet-stream <a href="http://controller.tester-pl.pl:9311/v1/secrets/b13d2017-e3e5-4f5f-a836-918ec130dc0a">http://controller.tester-pl.pl:9311/v1/secrets/b13d2017-e3e5-4f5f-a836-918ec130dc0a</a> --file my_symmetric_key.key<br></font>```<br>We need to transfer symmetric key to passphrase then:<br>```<br><font face="monospace">[TEST]root@barbican-01:/var/log/barbican# hexdump -e '16/1 "%02x"' my_symmetric_key.key<br></font>```<br>The output is our LUKS Passphrase. We can go to our compute node and decrypt a volume:<br>```<br><font face="monospace">[TEST]root@comp-09:/home/jwasilewski# cryptsetup luksOpen /dev/disk/by-id/wwn-0x6e00084100ee7e7e7fe79b5900003a89 my-encrypted-volume-decrypted<br>Enter passphrase for /dev/disk/by-id/wwn-0x6e00084100ee7e7e7fe79b5900003a89:</font><br>```<br><br>Then we can confirm, that our volume is decrypted:<br>```<br><font face="monospace">[TEST]root@comp-09:/home/jwasilewski# qemu-img info /dev/mapper/my-encrypted-volume-decrypted<br>image: /dev/mapper/my-encrypted-volume-decrypted<br>file format: raw<br>virtual size: 20G (21472739328 bytes)<br>disk size: 0</font><br>```<br><br><br>Thanks again for sharing it, I believe it is a super-useful procedure.</div><div id="gmail-:15o" class="gmail-a3s gmail-aiL" style="overflow:hidden;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:small;line-height:1.5"><br></div><div id="gmail-:15o" class="gmail-a3s gmail-aiL" style="overflow:hidden;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:small;line-height:1.5">Best regards,</div><div id="gmail-:15o" class="gmail-a3s gmail-aiL" style="overflow:hidden;font-variant-numeric:normal;font-variant-east-asian:normal;font-stretch:normal;font-size:small;line-height:1.5">Jan</div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">czw., 11 lut 2021 o 13:23 Lee Yarwood <<a href="mailto:lyarwood@redhat.com">lyarwood@redhat.com</a>> napisał(a):<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On 10-02-21 17:43:03, Lee Yarwood wrote:<br>
> On 10-02-21 11:29:06, Jan Wasilewski wrote:<br>
>> Thank you for a nice description of how everything is organized. It is much<br>
>> easier to understand the full workflow.<br>
>> <br>
>>> I'll try to find some time to write these up again later in the week.<br>
>> That would be great, I will try to do this by myself, but I'm wondering if<br>
>> it's possible to do "all magic" directly from a payload that is visible<br>
>> from barbican CLI.<br>
<br>
My thanks to gcharot for writing the following up downstream a while ago<br>
and highlighting some easy ways of achieving this.<br>
<br>
The following assumes that the volume is already mapped and connected to<br>
the localhost, in this case I'm just using the LVs used by the default<br>
LVM/iSCSI c-vol backend in my devstack env.<br>
<br>
It also assumes you have access to the secrets associated with the<br>
encrypted volume, by default admins do not.<br>
<br>
- Starting with an encrypted volume<br>
<br>
$ sudo qemu-img info --output=json /dev/stack-volumes-lvmdriver-1/volume-d4cc53db-6add-4c29-9f96-42a5498f8bd0 | jq .format<br>
"luks"<br>
<br>
- Fetch and store the key locally<br>
<br>
$ openstack secret get --payload_content_type 'application/octet-stream' <a href="http://192.168.122.208/key-manager/v1/secrets/6fd4f879-005d-4b7d-9e5f-2505f010be7c" rel="noreferrer" target="_blank">http://192.168.122.208/key-manager/v1/secrets/6fd4f879-005d-4b7d-9e5f-2505f010be7c</a> --file mysecret.key<br>
<br>
- Use dmcrypt to decrypt the device using the key as a passphrase<br>
<br>
$ yes $(hexdump -e '16/1 "%02x"' mysecret.key) | sudo cryptsetup luksOpen /dev/stack-volumes-lvmdriver-1/volume-d4cc53db-6add-4c29-9f96-42a5498f8bd0 volume-d4cc53db-6add-4c29-9f96-42a5498f8bd0<br>
<br>
- This should leave you with the decrypted volume under /dev/mapper<br>
<br>
$ sudo qemu-img info /dev/mapper/volume-d4cc53db-6add-4c29-9f96-42a5498f8bd0<br>
image: /dev/mapper/volume-d4cc53db-6add-4c29-9f96-42a5498f8bd0<br>
file format: raw<br>
virtual size: 0.998 GiB (1071644672 bytes)<br>
disk size: 0 B<br>
<br>
Hope this helps!<br>
<br>
-- <br>
Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76<br>
</blockquote></div>