Open Stack

Hello,

this is hopefully the conclusive (positively) email of this thread.

at the end we had to patch also Nova, because the ephimeral volumes
are created from the rbd driver in Nova.

Summary,
if you want OpenStack Juno to honor the following ceph configuration:

rbd default format = 2
rbd default features = 13

the rbd driver of the various openstack components need to know about
these new ceph features.

Here is the juno backported patches for Nova glance_store and cinder

Nova:
https://github.com/zioproto/nova/tree/rbd_default_features

glance_store:
https://github.com/zioproto/glance_store/tree/rbd_features_0.1.8

Cinder:
https://github.com/zioproto/cinder/tree/2014.2.4.backport-ceph-object-map

Deb packages available here:
http://ubuntu.mirror.cloud.switch.ch/engines/packages/

All this is completely fixed in Kilo, but because a lot of people are
still running Juno this was worthed to share.

Saverio

2015-12-08 9:34 GMT+01:00 Saverio Proto <zioproto at gmail.com>:
> Hello there,
>
> finally yesterday I found fast way to backport the rbd driver in juno
> glance_store.
>
> I found this repository with the right patch I was looking for:
> https://github.com/vumrao/glance_store.git (branch rbd_default_features)
>
> I reworked the patch on top of stable juno:
> https://github.com/zioproto/glance_store/commit/564129f865e10e7fcd5378a0914847323139f901
>
> and I created my ubuntu packages.
>
> Now everything works. I am testing the deb packages in my staging
> cluster. I do have cinder and glance
> honoring the ceph.conf default features. All volumes and images are
> created in the ceph
> backend with the object-map.
>
> If anyone is running juno and wants to enable this feature we have
> packages published here:
> http://ubuntu.mirror.cloud.switch.ch/engines/packages/
>
> Saverio
>
>
>
> 2015-11-26 11:36 GMT+01:00 Saverio Proto <zioproto at gmail.com>:
>> Hello,
>>
>> I think it is worth to update the list on this issue, because a lot of
>> operators are running Juno, and might want to enable the object map
>> feature in their rbd backend.
>>
>> our cinder backport seems to work great.
>>
>> however, most of volumes are CoW from glance images. Glance uses as
>> well the rbd backend.
>>
>> This means that if glance images do not have the rbd object map
>> features, the cinder volumes will have flags "object map invalid".
>>
>> So, we are now trying to backport this feature of the rbd driver in
>> glance as well.
>>
>> Saverio
>>
>>
>>
>> 2015-11-24 13:12 GMT+01:00 Saverio Proto <zioproto at gmail.com>:
>>> Hello there,
>>>
>>> we were able finally to backport the patch to Juno:
>>> https://github.com/zioproto/cinder/tree/backport-ceph-object-map
>>>
>>> we are testing this version. Everything good so far.
>>>
>>> this will require in your ceph.conf
>>> rbd default format = 2
>>> rbd default features = 13
>>>
>>> if anyone is willing to test this on his Juno setup I can also share
>>> .deb packages for Ubuntu
>>>
>>> Saverio
>>>
>>>
>>>
>>> 2015-11-16 16:21 GMT+01:00 Saverio Proto <zioproto at gmail.com>:
>>>> Thanks,
>>>>
>>>> I tried to backport this patch to Juno but it is not that trivial for
>>>> me. I have 2 tests failing, about volume cloning and create a volume
>>>> without layering.
>>>>
>>>> https://github.com/zioproto/cinder/commit/0d26cae585f54c7bda5ba5b423d8d9ddc87e0b34
>>>> https://github.com/zioproto/cinder/commits/backport-ceph-object-map
>>>>
>>>> I guess I will stop trying to backport this patch and wait for the
>>>> upgrade to Kilo of our Openstack installation to have the feature.
>>>>
>>>> If anyone ever backported this feature to Juno it would be nice to
>>>> know, so I can use the patch to generate deb packages.
>>>>
>>>> thanks
>>>>
>>>> Saverio
>>>>
>>>> 2015-11-12 17:55 GMT+01:00 Josh Durgin <jdurgin at redhat.com>:
>>>>> On 11/12/2015 07:41 AM, Saverio Proto wrote:
>>>>>>
>>>>>> So here is my best guess.
>>>>>> Could be that I am missing this patch ?
>>>>>>
>>>>>>
>>>>>> https://github.com/openstack/cinder/commit/6211d8fa2033c2a607c20667110c5913cf60dd53
>>>>>
>>>>>
>>>>> Exactly, you need that patch for cinder to use rbd_default_features
>>>>> from ceph.conf instead of its own default of only layering.
>>>>>
>>>>> In infernalis and later version of ceph you can also add object map to
>>>>> existing rbd images via the 'rbd feature enable' and 'rbd object-map
>>>>> rebuild' commands.
>>>>>
>>>>> Josh
>>>>>
>>>>>> proto at controller:~$ apt-cache policy python-cinder
>>>>>> python-cinder:
>>>>>>    Installed: 1:2014.2.3-0ubuntu1.1~cloud0
>>>>>>    Candidate: 1:2014.2.3-0ubuntu1.1~cloud0
>>>>>>
>>>>>>
>>>>>> Thanks
>>>>>>
>>>>>> Saverio
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2015-11-12 16:25 GMT+01:00 Saverio Proto <zioproto at gmail.com>:
>>>>>>>
>>>>>>> Hello there,
>>>>>>>
>>>>>>> I am investigating why my cinder is slow deleting volumes.
>>>>>>>
>>>>>>> you might remember my email from few days ago with subject:
>>>>>>> "cinder volume_clear=zero makes sense with rbd ?"
>>>>>>>
>>>>>>> so it comes out that volume_clear has nothing to do with the rbd driver.
>>>>>>>
>>>>>>> cinder was not guilty, it was really ceph rbd slow itself to delete big
>>>>>>> volumes.
>>>>>>>
>>>>>>> I was able to reproduce the slowness just using the rbd client.
>>>>>>>
>>>>>>> I was also able to fix the slowness just using the rbd client :)
>>>>>>>
>>>>>>> This is fixed in ceph hammer release, introducing a new feature.
>>>>>>>
>>>>>>>
>>>>>>> http://www.sebastien-han.fr/blog/2015/07/06/ceph-enable-the-object-map-feature/
>>>>>>>
>>>>>>> Enabling the object map feature rbd is now super fast to delete large
>>>>>>> volumes.
>>>>>>>
>>>>>>> However how I am in trouble with cinder. Looks like my cinder-api
>>>>>>> (running juno here) ignores the changes in my ceph.conf file.
>>>>>>>
>>>>>>> cat cinder.conf | grep rbd
>>>>>>>
>>>>>>> volume_driver=cinder.volume.drivers.rbd.RBDDriver
>>>>>>> rbd_user=cinder
>>>>>>> rbd_max_clone_depth=5
>>>>>>> rbd_ceph_conf=/etc/ceph/ceph.conf
>>>>>>> rbd_flatten_volume_from_snapshot=False
>>>>>>> rbd_pool=volumes
>>>>>>> rbd_secret_uuid=secret
>>>>>>>
>>>>>>> But when I create a volume with cinder, The options in ceph.conf are
>>>>>>> ignored:
>>>>>>>
>>>>>>> cat /etc/ceph/ceph.conf | grep rbd
>>>>>>> rbd default format = 2
>>>>>>> rbd default features = 13
>>>>>>>
>>>>>>> But the volume:
>>>>>>>
>>>>>>> rbd image 'volume-78ca9968-77e8-4b68-9744-03b25b8068b1':
>>>>>>>      size 102400 MB in 25600 objects
>>>>>>>      order 22 (4096 kB objects)
>>>>>>>      block_name_prefix: rbd_data.533f4356fe034
>>>>>>>      format: 2
>>>>>>>      features: layering
>>>>>>>      flags:
>>>>>>>
>>>>>>>
>>>>>>> so my first question is:
>>>>>>>
>>>>>>> does anyone use cinder with rbd driver and object map feature enabled
>>>>>>> ? Does it work for anyone ?
>>>>>>>
>>>>>>> thank you
>>>>>>>
>>>>>>> Saverio
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-operators mailing list
>>>>>> OpenStack-operators at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>>>>>
>>>>>