Open Stack

Hi,

we're using OpenStack with Ceph in production and also have customers  
doing that.
 From my point of view fixing nova to be able to deal with shared  
storage of course would improve many things, but it doesn't liberate  
you from monitoring your systems. Filling up a ceph cluster should be  
avoided and therefore proper monitoring is required.

I assume you were able to resolve the frozen instances?

Regards,
Eugen


Zitat von Sean Mooney <smooney at redhat.com>:

> On Tue, 2021-01-05 at 14:17 -0300, Rodrigo Barbieri wrote:
>> Hi Nova folks and OpenStack operators!
>>
>> I have had some trouble recently where while using the "images_type = rbd"
>> libvirt option my ceph cluster got filled up without I noticing and froze
>> all my nova services and instances.
>>
>> I started digging and investigating why and how I could prevent or
>> workaround this issue, but I didn't find a very reliable clean way.
>>
>> I documented all my steps and investigation in bug 1908133 [0]. It has been
>> marked as a duplicate of 1522307 [1] which has been around for quite some
>> time, so I am wondering if any operators have been using nova + ceph in
>> production with "images_type = rbd" config set and how you have been
>> handling/working around the issue.
>
> this is indeed a know issue and the long term plan to fix it was to  
> track shared storae
> as a sharing resouce provide in plamcent. that never happend so  
> there si currenlty no mechanium
> available to prevent this explcitly in nova.
>
> the disk filter which is nolonger used could prevnet the boot of a   
> vm that would fill the ceph pool but
> it could not protect against two concurrent request form filling the pool.
>
> placement can protect against that due to the transational nature of  
> allocations which serialise
> all resouce useage however since each host reports the total size of  
> the ceph pool as its local storage that wont work out of the box.
>
> as a quick hack what you can do is set the  
> [DEFAULT]/disk_allocation_ratio=(1/number of compute nodes)
> https://docs.openstack.org/nova/latest/configuration/config.html#DEFAULT.disk_allocation_ratio
> on each of your compute agents configs.
>
>
> that will prevent over subscription however it has other negitve sidefects.
> mainly that you will fail to scudle instance that could boot if a  
> host exced its 1/n usage
> so unless you have perfectly blanced consumtion this is not a good approch.
>
> a better appoch but one that requires external scripting is to have  
> a chron job that will update the resrved
>  usaage of each of the disk_gb inventores to the actull amount of of  
> stoarge allocated form the pool.
>
> the real fix however is for nova to tack its shared usage in  
> placment correctly as a sharing resouce provide.
>
> its possible you might be able to do that via the porvider.yaml file
>
> by overriding the local disk_gb to 0 on all comupte nodes
> then creating a singel haring resouce provider of disk_gb that  
> models the ceph pool.
>
> https://specs.openstack.org/openstack/nova-specs/specs/ussuri/approved/provider-config-file.html
> currently that does not support the addtion of providers to placment  
> agggreate so while it could be used to 0 out the comptue node
> disk inventoies and to create a sharing provider it with the  
> MISC_SHARES_VIA_AGGREGATE trait it cant do the final step of mapping
> which compute nodes can consume form sharing provider via the  
> agggrate but you could do that form.
> that assume that "sharing resouce provdiers" actully work.
>
>
> bacialy what it comes down to today is you need to monitor the  
> avaiable resouce yourslef externally and ensure you never run out of  
> space.
> that sucks but untill we proably track things in plamcent there is  
> nothign we can really do.
> the two approch i suggested above might work for a subset of  
> usecasue but really this is a feature that need native suport in  
> nova to adress properly.
>
>>
>> Thanks in advance!
>>
>> [0] https://bugs.launchpad.net/nova/+bug/1908133
>> [1] https://bugs.launchpad.net/nova/+bug/1522307
>>