Creating multiple servers with a shared volume
Hi, I am planning to spin hundreds of servers with a shared storage. I have created one instance and attached a 1 TB volume (multiattach) and created a snapshot of the instance. But when I spin another server using the snapshot, I noticed that instead of expected shared volume, individual 1 TB size additional (dedicated) volume has been attached to each VM. Is there a mechanism where I can create multiple instances along with a shared volume attached to the instance instead of attaching later? Regards, Danishka
Just to be sure, the flow is. - Create VM with root disk on a volume/image. - Create another 1TB volume. - Attach the volume to the VM as a second drive using a multi attach approach ( https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiatta... )? - Create snapshot of the original VM (with it's root drive + 1 TB volume)? - Create new VM from snapshot. - Instead of a new VM + 1 new root volume + the same 1TB multiattach volume attached, you get a new multiattach volume (new volume ID and everything)? I've never had the pleasure to play around with this, but creating a new VM from a snapshot could mean a new volume too. I'm not sure how snapshot will work with multi volume VMs. On Fri, Sep 24, 2021 at 11:20 AM open infra <openinfradn@gmail.com> wrote:
Hi,
I am planning to spin hundreds of servers with a shared storage. I have created one instance and attached a 1 TB volume (multiattach) and created a snapshot of the instance. But when I spin another server using the snapshot, I noticed that instead of expected shared volume, individual 1 TB size additional (dedicated) volume has been attached to each VM.
Is there a mechanism where I can create multiple instances along with a shared volume attached to the instance instead of attaching later?
Regards, Danishka
On Sat, Sep 25, 2021 at 7:37 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
Just to be sure, the flow is.
- Create VM with root disk on a volume/image. - Create another 1TB volume. - Attach the volume to the VM as a second drive using a multi attach approach ( https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiatta... )? - Create snapshot of the original VM (with it's root drive + 1 TB volume)? - Create new VM from snapshot. - Instead of a new VM + 1 new root volume + the same 1TB multiattach volume attached, you get a new multiattach volume (new volume ID and everything)?
Yes, here is the block device mapping of the snapshot (from a VM with 100GB of root disk and 1TB of multiattach volume) block_device_mapping[{"image_id": null, "delete_on_termination": true, "device_name": "/dev/vda", "disk_bus": "virtio", "volume_id": null, "volume_type": null, "destination_type": "volume", "source_type": "snapshot", "guest_format": null, "volume_size": 100, "device_type": "disk", "snapshot_id": "7b2c7f3a-8420-4a33-820e-2d2231d930c7", "boot_index": 0, "no_device": null, "tag": null}, {"image_id": null, "delete_on_termination": false, "device_name": "/dev/vdb", "disk_bus": null, "volume_id": null, "volume_type": null, "destination_type": "volume", "source_type": "snapshot", "guest_format": null, "volume_size": 1000, "device_type": null, "snapshot_id": "c2a0695f-8c9e-46f9-80a8-560c47521eeb", "boot_index": null, "no_device": null, "tag": null}] openstack image list +--------------------------------------+----------------+--------+ | ID | Name | Status | +--------------------------------------+----------------+--------+ | e9e0e6dc-6389-49a3-bd52-467520cd2c9e | vm-04-snap-new | active | +--------------------------------------+----------------+--------+ openstack volume list +--------------------------------------+---------------+--------+------+--------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+---------------+--------+------+--------------------------------------------------------------+ | b239c3da-3276-4497-ad4c-1e900e907426 | | in-use | 100 | Attached to VM-04 on /dev/vda | | 28088321-f87a-4392-bd6c-bcd0174184ea | | in-use | 100 | Attached to VM-03 on /dev/vda | | c693b9ad-6d47-4a2c-93e4-36baaf25f062 | shared-volume | in-use | 1000 | Attached to VM-03 on /dev/vdb Attached to VM-04 on /dev/vdb | Under volume >. snapshots I can see two new volumes created openstack volume snapshot list +--------------------------------------+-----------------------------+-------------+-----------+------+ | ID | Name | Description | Status | Size | +--------------------------------------+-----------------------------+-------------+-----------+------+ | c2a0695f-8c9e-46f9-80a8-560c47521eeb | snapshot for vm-04-snap-new | None | available | 1000 | | 7b2c7f3a-8420-4a33-820e-2d2231d930c7 | snapshot for vm-04-snap-new | | available | 100 | +--------------------------------------+-----------------------------+-------------+-----------+------+ controller-1:~$ openstack server create --flavor WinGPU16 --image vm-04-snap-new --max 10 vm +-------------------------------------+-------------------------------------------------------+ | Field | Value | +-------------------------------------+-------------------------------------------------------+ | OS-DCF:diskConfig | MANUAL | | OS-EXT-AZ:availability_zone | | | OS-EXT-SRV-ATTR:host | None | | OS-EXT-SRV-ATTR:hypervisor_hostname | None | | OS-EXT-SRV-ATTR:instance_name | | | OS-EXT-STS:power_state | NOSTATE | | OS-EXT-STS:task_state | scheduling | | OS-EXT-STS:vm_state | building | | OS-SRV-USG:launched_at | None | | OS-SRV-USG:terminated_at | None | | accessIPv4 | | | accessIPv6 | | | addresses | | | adminPass | se8pWoZ8AQ5Y | | config_drive | | | created | 2021-09-25T07:25:51Z | | flavor | WinGPU16 (e3cd48c5-9675-42ad-a766-29b92c674224) | | hostId | | | id | facfc95d-bcb5-452c-a7fb-fcfa22f4257b | | image | vm-04-snap-new (e9e0e6dc-6389-49a3-bd52-467520cd2c9e) | | key_name | None | | name | vm-1 | | progress | 0 | | project_id | 21639a96c609474390c837accd2de337 | | properties | | | security_groups | name='default' | | status | BUILD | | updated | 2021-09-25T07:25:51Z | | user_id | 19faf07bb5da40b897e5dc73073cdec6 | | volumes_attached | | +-------------------------------------+-------------------------------------------------------+ controller-1:~$ controller-1:~$ openstack server list +--------------------------------------+----------+---------+----------------------+----------------+----------+ | ID | Name | Status | Networks | Image | Flavor | +--------------------------------------+----------+---------+----------------------+----------------+----------+ | 24352c37-0c2d-4760-bb6e-aeb3f0432833 | vm-2 | ACTIVE | g3-ext=192.168.6.168 | vm-04-snap-new | WinGPU16 | | 39f2a8d5-a895-47ef-8cdc-bd9f9fe6acd2 | vm-7 | ACTIVE | g3-ext=192.168.6.214 | vm-04-snap-new | WinGPU16 | | 59330c7c-4c57-4aaa-9c83-69d05d670b53 | vm-6 | ACTIVE | g3-ext=192.168.6.251 | vm-04-snap-new | WinGPU16 | | 60fc3dc6-c873-4faa-99e6-657a98fa2275 | vm-3 | ACTIVE | g3-ext=192.168.6.210 | vm-04-snap-new | WinGPU16 | | 88af8a71-93f9-438b-93f3-4169b35842ca | vm-4 | ACTIVE | g3-ext=192.168.6.241 | vm-04-snap-new | WinGPU16 | | 90fd144a-2126-45f0-85b7-59d68bff4ba1 | vm-9 | ACTIVE | g3-ext=192.168.6.201 | vm-04-snap-new | WinGPU16 | | 91a069c3-7b31-40d2-b030-5cd23a53619f | vm-10 | ACTIVE | g3-ext=192.168.6.183 | vm-04-snap-new | WinGPU16 | | 9f01af21-101c-412f-a6e3-89632170b932 | vm-8 | ACTIVE | g3-ext=192.168.6.249 | vm-04-snap-new | WinGPU16 | | d345c99b-dac4-4ba2-8d2a-4d2aab5ab73c | vm-5 | ACTIVE | g3-ext=192.168.6.229 | vm-04-snap-new | WinGPU16 | | facfc95d-bcb5-452c-a7fb-fcfa22f4257b | vm-1 | ACTIVE | g3-ext=192.168.6.222 | vm-04-snap-new | WinGPU16 | | cec33607-a668-4ce8-ac16-c9f5b143d676 | VM-04 | SHUTOFF | g3-ext=192.168.6.239 | | WinGPU16 | | 87efb650-d7b0-4459-a401-80ae95cd35c4 | VM-03 | SHUTOFF | g3-ext=192.168.6.179 | | WinGPU16 | +--------------------------------------+----------+---------+----------------------+----------------+----------+ controller-1:~$ openstack volume list +--------------------------------------+---------------+--------+------+--------------------------------------------------------------+ | ID | Name | Status | Size | Attached to | +--------------------------------------+---------------+--------+------+--------------------------------------------------------------+ | d2d51653-23d5-44a6-a713-b69c010c440d | | in-use | 1000 | Attached to vm-10 on /dev/vdb | | 86f5b026-cdd6-46e1-9726-d159d38d2d1a | | in-use | 1000 | Attached to vm-8 on /dev/vdb | | 775a21c3-9229-4487-a30d-267a844a9ed4 | | in-use | 1000 | Attached to vm-9 on /dev/vdb | | 021cbae9-6610-4bbd-ac5d-6663c419532f | | in-use | 1000 | Attached to vm-7 on /dev/vdb | | f46cefa0-3cfe-4a52-b6d2-8bccb6d1b2d9 | | in-use | 1000 | Attached to vm-6 on /dev/vdb | | 887131fe-c2b8-41ca-b0c8-72dd75b4cd5b | | in-use | 1000 | Attached to vm-5 on /dev/vdb | | f5aafed0-4b4b-4bce-ba5e-e96f717ecfa3 | | in-use | 1000 | Attached to vm-4 on /dev/vdb | | ce07fe51-3808-4947-acea-a6344d39ef2b | | in-use | 1000 | Attached to vm-2 on /dev/vdb | | c7088c07-e4cd-4baa-8c1a-1deeab5b486c | | in-use | 1000 | Attached to vm-1 on /dev/vdb | | b1816c08-53c3-4391-8396-e710acf24edf | | in-use | 1000 | Attached to vm-3 on /dev/vdb | | e3b8ccc9-0bdd-4c4f-8256-7e4905637d56 | | in-use | 100 | Attached to vm-8 on /dev/vda | | a4a933e2-b139-4ba0-8cbe-523ac0eb5036 | | in-use | 100 | Attached to vm-10 on /dev/vda | | 81997801-840a-4f23-98d6-af961abad590 | | in-use | 100 | Attached to vm-9 on /dev/vda | | 3d22538f-6f88-4529-a689-df0c37f2a00f | | in-use | 100 | Attached to vm-7 on /dev/vda | | de0551a6-e27e-4dd9-9a5b-ac2569c0dc76 | | in-use | 100 | Attached to vm-4 on /dev/vda | | b080eb16-c17e-40bd-b345-40933d6d1977 | | in-use | 100 | Attached to vm-3 on /dev/vda | | 8b514c8f-f190-4617-be12-be21f15cbd2e | | in-use | 100 | Attached to vm-5 on /dev/vda | | 0e0b0a37-060f-4bb0-b78b-288830d627e5 | | in-use | 100 | Attached to vm-6 on /dev/vda | | 86fba86b-9fdd-4d4c-8d41-f56f03152206 | | in-use | 100 | Attached to vm-2 on /dev/vda | | 69c1de5d-f4ca-4fe8-a617-bd22863610cb | | in-use | 100 | Attached to vm-1 on /dev/vda | | b239c3da-3276-4497-ad4c-1e900e907426 | | in-use | 100 | Attached to VM-04 on /dev/vda | | 28088321-f87a-4392-bd6c-bcd0174184ea | | in-use | 100 | Attached to VM-03 on /dev/vda | | c693b9ad-6d47-4a2c-93e4-36baaf25f062 | shared-volume | in-use | 1000 | Attached to VM-03 on /dev/vdb Attached to VM-04 on /dev/vdb | +--------------------------------------+---------------+--------+------+--------------------------------------------------------------+ As you can see the original multi attach volume "shared-volume" is attached to VM-03 but all the other 1TB volumes are not shared but directly attach only per one sevrer/vm. Seems we have to attach the shared volume separately into each VM. I've never had the pleasure to play around with this, but creating a new VM
from a snapshot could mean a new volume too. I'm not sure how snapshot will work with multi volume VMs.
On Fri, Sep 24, 2021 at 11:20 AM open infra <openinfradn@gmail.com> wrote:
Hi,
I am planning to spin hundreds of servers with a shared storage. I have created one instance and attached a 1 TB volume (multiattach) and created a snapshot of the instance. But when I spin another server using the snapshot, I noticed that instead of expected shared volume, individual 1 TB size additional (dedicated) volume has been attached to each VM.
Is there a mechanism where I can create multiple instances along with a shared volume attached to the instance instead of attaching later?
Regards, Danishka
On 25-09-21 13:11:19, open infra wrote:
On Sat, Sep 25, 2021 at 7:37 AM Laurent Dumont <laurentfdumont@gmail.com> wrote:
Just to be sure, the flow is.
- Create VM with root disk on a volume/image. - Create another 1TB volume. - Attach the volume to the VM as a second drive using a multi attach approach ( https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiatta... )? - Create snapshot of the original VM (with it's root drive + 1 TB volume)? - Create new VM from snapshot. - Instead of a new VM + 1 new root volume + the same 1TB multiattach volume attached, you get a new multiattach volume (new volume ID and everything)?
Because you're attaching a snapshot and not the original volume? See below for more.
Yes, here is the block device mapping of the snapshot (from a VM with 100GB of root disk and 1TB of multiattach volume)
block_device_mapping[{"image_id": null, "delete_on_termination": true, "device_name": "/dev/vda", "disk_bus": "virtio", "volume_id": null, "volume_type": null, "destination_type": "volume", "source_type": "snapshot", "guest_format": null, "volume_size": 100, "device_type": "disk", "snapshot_id": "7b2c7f3a-8420-4a33-820e-2d2231d930c7", "boot_index": 0, "no_device": null, "tag": null}, {"image_id": null, "delete_on_termination": false, "device_name": "/dev/vdb", "disk_bus": null, "volume_id": null, "volume_type": null, "destination_type": "volume", "source_type": "snapshot", "guest_format": null, "volume_size": 1000, "device_type": null, "snapshot_id": "c2a0695f-8c9e-46f9-80a8-560c47521eeb", "boot_index": null, "no_device": null, "tag": null}]
As you can see above you're using source_type = "snapshot" and destination_type of "volume". As set out in the following docs this creates a new volume from a given snapshot and attaches that to the instance: https://docs.openstack.org/nova/latest/user/block-device-mapping.html#valid-... snapshot -> volume - this works exactly as passing type=snap does. It would create a volume from a Cinder volume snapshot and attach that volume to the instance. Can be marked bootable. https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#cr... block_device_mapping_v2.source_type (Optional) The source type of the block device. Valid values are: [..] snapshot: This is only valid with destination_type=volume; creates a volume backed by the given volume snapshot referenced via the block_device_mapping_v2.uuid parameter and attaches it to the server When attaching multiattach enabled volumes to multiple instances you need to use source_type = 'volume' *and* destination_type = 'volume'. Hope this helps, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
On Mon, Sep 27, 2021 at 2:15 PM Lee Yarwood <lyarwood@redhat.com> wrote:
On 25-09-21 13:11:19, open infra wrote:
On Sat, Sep 25, 2021 at 7:37 AM Laurent Dumont <laurentfdumont@gmail.com
wrote:
Just to be sure, the flow is.
- Create VM with root disk on a volume/image. - Create another 1TB volume. - Attach the volume to the VM as a second drive using a multi attach approach (
https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiatta...
)? - Create snapshot of the original VM (with it's root drive + 1 TB volume)? - Create new VM from snapshot. - Instead of a new VM + 1 new root volume + the same 1TB multiattach volume attached, you get a new multiattach volume (new volume ID and everything)?
Because you're attaching a snapshot and not the original volume? See below for more.
Yes, here is the block device mapping of the snapshot (from a VM with 100GB of root disk and 1TB of multiattach volume)
block_device_mapping[{"image_id": null, "delete_on_termination": true, "device_name": "/dev/vda", "disk_bus": "virtio", "volume_id": null, "volume_type": null, "destination_type": "volume", "source_type": "snapshot", "guest_format": null, "volume_size": 100, "device_type": "disk", "snapshot_id": "7b2c7f3a-8420-4a33-820e-2d2231d930c7", "boot_index": 0, "no_device": null, "tag": null}, {"image_id": null, "delete_on_termination": false, "device_name": "/dev/vdb", "disk_bus": null, "volume_id": null, "volume_type": null, "destination_type": "volume", "source_type": "snapshot", "guest_format": null, "volume_size": 1000, "device_type": null, "snapshot_id": "c2a0695f-8c9e-46f9-80a8-560c47521eeb", "boot_index": null, "no_device": null, "tag": null}]
As you can see above you're using source_type = "snapshot" and destination_type of "volume". As set out in the following docs this creates a new volume from a given snapshot and attaches that to the instance:
https://docs.openstack.org/nova/latest/user/block-device-mapping.html#valid-...
snapshot -> volume - this works exactly as passing type=snap does. It would create a volume from a Cinder volume snapshot and attach that volume to the instance. Can be marked bootable.
https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#cr...
block_device_mapping_v2.source_type (Optional)
The source type of the block device. Valid values are: [..] snapshot: This is only valid with destination_type=volume; creates a volume backed by the given volume snapshot referenced via the block_device_mapping_v2.uuid parameter and attaches it to the server
When attaching multiattach enabled volumes to multiple instances you need to use source_type = 'volume' *and* destination_type = 'volume'.
Hope this helps,
It very clears what went wrong. The above block device mapping was found when I try to figure out what went wrong. But I am not sure where exactly I should define block device mapping when I use CLI. Appreciate if you can give an example. Thanks a lot Lee Yarwood!
-- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
On 28-09-21 18:17:14, open infra wrote:
On Mon, Sep 27, 2021 at 2:15 PM Lee Yarwood <lyarwood@redhat.com> wrote:
On 25-09-21 13:11:19, open infra wrote:
On Sat, Sep 25, 2021 at 7:37 AM Laurent Dumont <laurentfdumont@gmail.com
wrote:
Just to be sure, the flow is.
- Create VM with root disk on a volume/image. - Create another 1TB volume. - Attach the volume to the VM as a second drive using a multi attach approach (
https://docs.openstack.org/cinder/latest/admin/blockstorage-volume-multiatta...
)? - Create snapshot of the original VM (with it's root drive + 1 TB volume)? - Create new VM from snapshot. - Instead of a new VM + 1 new root volume + the same 1TB multiattach volume attached, you get a new multiattach volume (new volume ID and everything)?
Because you're attaching a snapshot and not the original volume? See below for more.
Yes, here is the block device mapping of the snapshot (from a VM with 100GB of root disk and 1TB of multiattach volume)
block_device_mapping[{"image_id": null, "delete_on_termination": true, "device_name": "/dev/vda", "disk_bus": "virtio", "volume_id": null, "volume_type": null, "destination_type": "volume", "source_type": "snapshot", "guest_format": null, "volume_size": 100, "device_type": "disk", "snapshot_id": "7b2c7f3a-8420-4a33-820e-2d2231d930c7", "boot_index": 0, "no_device": null, "tag": null}, {"image_id": null, "delete_on_termination": false, "device_name": "/dev/vdb", "disk_bus": null, "volume_id": null, "volume_type": null, "destination_type": "volume", "source_type": "snapshot", "guest_format": null, "volume_size": 1000, "device_type": null, "snapshot_id": "c2a0695f-8c9e-46f9-80a8-560c47521eeb", "boot_index": null, "no_device": null, "tag": null}]
As you can see above you're using source_type = "snapshot" and destination_type of "volume". As set out in the following docs this creates a new volume from a given snapshot and attaches that to the instance:
https://docs.openstack.org/nova/latest/user/block-device-mapping.html#valid-...
snapshot -> volume - this works exactly as passing type=snap does. It would create a volume from a Cinder volume snapshot and attach that volume to the instance. Can be marked bootable.
https://docs.openstack.org/api-ref/compute/?expanded=create-server-detail#cr...
block_device_mapping_v2.source_type (Optional)
The source type of the block device. Valid values are: [..] snapshot: This is only valid with destination_type=volume; creates a volume backed by the given volume snapshot referenced via the block_device_mapping_v2.uuid parameter and attaches it to the server
When attaching multiattach enabled volumes to multiple instances you need to use source_type = 'volume' *and* destination_type = 'volume'.
Hope this helps,
It very clears what went wrong. The above block device mapping was found when I try to figure out what went wrong. But I am not sure where exactly I should define block device mapping when I use CLI. Appreciate if you can give an example.
Thanks a lot Lee Yarwood!
Okay so given an existing multiattach volume of 02a27e61-b242-460e-8cf6-8525c5353698 I can create two instances with it attached as a secondary disks as follows: $ openstack volume show -c multiattach -f json 02a27e61-b242-460e-8cf6-8525c5353698 { "multiattach": true } $ openstack --os-compute-api-version 2.latest server create \ --image cirros-0.5.2-x86_64-disk \ --flavor 1 \ --block-device source_type=volume,destination_type=volume,uuid=02a27e61-b242-460e-8cf6-8525c5353698 \ --network private test-1 $ openstack --os-compute-api-version 2.latest server create \ --image cirros-0.5.2-x86_64-disk \ --flavor 1 \ --block-device source_type=volume,destination_type=volume,uuid=02a27e61-b242-460e-8cf6-8525c5353698 \ --network private test-2 You should be able to see both attachments listed against the volume: $ openstack volume show 02a27e61-b242-460e-8cf6-8525c5353698 -c attachments -f json { "attachments": [ { "id": "02a27e61-b242-460e-8cf6-8525c5353698", "attachment_id": "2f3a515a-3ef4-499e-a48a-04c6cd45a05a", "volume_id": "02a27e61-b242-460e-8cf6-8525c5353698", "server_id": "74adb548-f3ed-4d68-b282-b278dd1ec3f2", "host_name": "localhost.localdomain", "device": "/dev/vdb", "attached_at": "2021-09-28T13:31:31.000000" }, { "id": "02a27e61-b242-460e-8cf6-8525c5353698", "attachment_id": "5f0cdd6e-4586-4e9d-ad70-dfcf2a093c03", "volume_id": "02a27e61-b242-460e-8cf6-8525c5353698", "server_id": "2fdc931e-781a-4430-86af-4debe92c885a", "host_name": "localhost.localdomain", "device": "/dev/vdb", "attached_at": "2021-09-28T13:31:45.000000" } ] } Hope this helps, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
participants (3)
-
Laurent Dumont
-
Lee Yarwood
-
open infra