On Mon, Aug 26, 2024 at 06:42:40PM GMT, Jan Horstmann wrote:
Hello cinder team and other interested parties,
I have recently started thinking about ways to offer local storage as part of the OpenStack block storage service. Naturally any solution would require to transparently migrate volumes to and with the instances they are attached to, so that the user experience of being able to access their storage from anywhere would not be degraded.
Fortunately, with the device mapper clone target ([1]), a mechanism is available in the linux kernel which allows to do exactly that: Create copies of devices (even network attached ones) while allowing to immediately use the target device.
I have detailed the motivation and some implications of having local storage in the block storage service in a blog post [2], which I do not want to repeat here in order to not make this mail longer than it needs to be. The blog post also outlines a way to implement a cinder driver leveraging this mechanism, which is loosely based on the way cinder migrates volumes between backends, i.e.: When a volume is going to be attached, a new volume is created on the instance's host, which is backed by a device mapper clone target with the original volume as its source; Then the volume objects are switched and a local connector is returned, which may be used immediately.
Since then I went ahead and created a proof of concept cinder driver ([3]), which roughly follows what is outlined in the post (some lessons were already learned during implementation). The driver is based on the in-tree lvm driver for basic volume management and can do basic things like create/delete, attach/detach and also handles live migration and rebuilding of instances. There is nothing currently regarding snapshots and volume migrations will definitly not work as expected yet, but it should be sufficient to get a feeling for the general idea.
please note that the linked driver version contains a bug, which leads to a read-only attachment of the volume due to the `access_mode` metadata not being moved to the correct volume. There is a fixed version at the link below: https://github.com/janhorstmann/openstack-cinder/tree/56f4b5cc71548883208253...
I am currently looking for feedback on the idea and the implementation and would like to know if this is of interest to others. I am happpy to answer any questions or present this in more detail.
In case anyone wants to take the driver for a spin I am going to leave some instructions on how to set up the environment in [4].
Regards, Jan
[1] https://docs.kernel.org/admin-guide/device-mapper/dm-clone.html
[2] https://scs.community/2024/07/12/local-block-storage-via-dm-clone/
[3] https://github.com/janhorstmann/openstack-cinder/tree/59e1808a24d112d2e8d089...
[4] - Set up an OpenStack environment - Create an lvm2 volume group and run a cinder volume service on every hypervisor - Configure a new backend on every volume service with the same `volume_backend_name` Configure it like you would an lvm backend, but with `volume_driver = cinder.volume.drivers.dmclone.DMCloneVolumeDriver`, e.g.:
[local-volumes] volume_driver = cinder.volume.drivers.dmclone.DMCloneVolumeDriver volume_group = local volume_backend_name = local-volumes target_helper = tgtadm target_protocol = iscsi lvm_type = default
I have tested with the values above, but any target driver should work
-- Jan Horstmann Senior Cloud Engineer
Mail: horstmann@osism.tech Web: https://osism.tech
OSISM GmbH Talweg 8 / 75417 Mühlacker / Deutschland
Geschäftsführer: Christian Berendt Unternehmenssitz: Mühlacker Amtsgericht Mannheim, HRB 750852
-- Jan Horstmann Senior Cloud Engineer Mail: horstmann@osism.tech Web: https://osism.tech OSISM GmbH Talweg 8 / 75417 Mühlacker / Deutschland Geschäftsführer: Christian Berendt Unternehmenssitz: Mühlacker Amtsgericht Mannheim, HRB 750852