[cinder] Offering local storage using the device mapper clone target

26 Aug 2024

      Hello cinder team and other interested parties,

I have recently started thinking about ways to offer local storage as part of
the OpenStack block storage service. Naturally any solution would require to
transparently migrate volumes to and with the instances they are attached to,
so that the user experience of being able to access their storage from anywhere
would not be degraded.

Fortunately, with the device mapper clone target ([1]), a mechanism is
available in the linux kernel which allows to do exactly that: Create copies
of devices (even network attached ones) while allowing to immediately use the
target device.

I have detailed the motivation and some implications of having local
storage in the block storage service in a blog post [2], which I do not want to
repeat here in order to not make this mail longer than it needs to be.
The blog post also outlines a way to implement a cinder driver leveraging
this mechanism, which is loosely based on the way cinder migrates volumes
between backends, i.e.: When a volume is going to be attached, a new volume is
created on the instance's host, which is backed by a device mapper clone target
with the original volume as its source; Then the volume objects are switched and
a local connector is returned, which may be used immediately.

Since then I went ahead and created a proof of concept cinder driver ([3]),
which roughly follows what is outlined in the post (some lessons were already
learned during implementation). The driver is based on the in-tree lvm driver
for basic volume management and can do basic things like create/delete,
attach/detach and also handles live migration and rebuilding of instances. There
is nothing currently regarding snapshots and volume migrations will definitly
not work as expected yet, but it should be sufficient to get a feeling for the
general idea.

I am currently looking for feedback on the idea and the implementation and would
like to know if this is of interest to others. I am happpy to answer any
questions or present this in more detail.

In case anyone wants to take the driver for a spin I am going to leave some
instructions on how to set up the environment in [4].

Regards,
Jan

[1]
https://docs.kernel.org/admin-guide/device-mapper/dm-clone.html

[2]
https://scs.community/2024/07/12/local-block-storage-via-dm-clone/

[3]
https://github.com/janhorstmann/openstack-cinder/tree/59e1808a24d112d2e8d089...

[4]
- Set up an OpenStack environment
- Create an lvm2 volume group and run a cinder volume service on every hypervisor
- Configure a new backend on every volume service with the same
  `volume_backend_name`
  Configure it like you would an lvm backend, but with
  `volume_driver = cinder.volume.drivers.dmclone.DMCloneVolumeDriver`, e.g.:

  [local-volumes]
  volume_driver = cinder.volume.drivers.dmclone.DMCloneVolumeDriver
  volume_group = local
  volume_backend_name = local-volumes
  target_helper = tgtadm
  target_protocol = iscsi
  lvm_type = default

  I have tested with the values above, but any target driver should work

-- 
Jan Horstmann
Senior Cloud Engineer

Mail: horstmann@osism.tech
Web: https://osism.tech

OSISM GmbH
Talweg 8 / 75417 Mühlacker / Deutschland

Geschäftsführer: Christian Berendt
Unternehmenssitz: Mühlacker
Amtsgericht Mannheim, HRB 750852

Jan Horstmann

Jan Horstmann

tags

participants (1)