Open Stack

On 05/07, Arnaud wrote:
> Hey,
>
> Thanks for your answer!
> OK, I understand the why ;) also because we hit some issues on our deployment.
> So we increase the number of threads to 100 but we also enable the deferred deletion (keeping in mind the quota usage downsides that it brings).

Hi Arnaud,

Deferred deletion should reduce the number of required native threads
since delete calls will complete faster.


> We also disabled the periodic task to compute usage and use the less precise way from db.

Are you referring to the 'rbd_exclusive_cinder_pool' configuration
option?  Because that should already have the optimum default (True
value).


> First question here: do you think we are going the right path?
>
> One thing we are not yet sure is how to calculate correctly the number of threads to use.
> Should we do basic math with the number of deletion per minutes? Or should we take the number of volumes in the backend into account? Something in the middle?

Native threads on the RBD driver are not only used for deletion, they
are used for *all* RBD calls.

We haven't defined any particular method to calculate the optimum number
of threads on a system, but I can think of 2 possible avenues to
explore:

- Performance testing: Run a set of tests with a high number of
  concurrent requests and different operations and see how Cinder
  performs.  I wouldn't bother with individual attach and detach to VM
  operations because those are noops on the Cinder side, creating volume
  from image with either different images or cache disabled would be
  better.

- Reporting native thread usage: To really know if the number of native
  threads is sufficient or not you could modify the Cinder volume
  manager (and possibly also eventlet.tpool to gather statistics on the
  number of used/free native threads and number of queued requests that
  are waiting for a native thread to pick them up.

Cheers,
Gorka.

>
> Thanks!
>
> Arnaud
>
>
>
> Le 5 juillet 2022 18:06:14 GMT+02:00, Rajat Dhasmana <rdhasman at redhat.com> a écrit :
> >Hi Arnaud,
> >
> >We discussed this in last week's cinder meeting and unfortunately we
> >haven't tested it thoroughly so we don't have any performance numbers to
> >share.
> >What we can tell is the reason why RBD requires a higher number of native
> >threads. RBD calls C code which could potentially block green threads hence
> >blocking the main operation therefore all of the calls in RBD to execute
> >operations are wrapped to use native threads so depending on the operations
> >we want to
> >perform concurrently, we can set the value of
> >backend_native_threads_pool_size for RBD.
> >
> >Thanks and regards
> >Rajat Dhasmana
> >
> >On Mon, Jun 27, 2022 at 9:35 PM Arnaud Morin <arnaud.morin at gmail.com> wrote:
> >
> >> Hey all,
> >>
> >> Is there any recommendation on the number of threads to use when using
> >> RBD backend (option backend_native_threads_pool_size)?
> >> The doc is saying that 20 is the default but it should be increased,
> >> specially for the RBD driver, but up to which value?
> >>
> >> Is there anyone tuning this parameter in their openstack deployments?
> >>
> >> If yes, maybe we can add some recommendations on openstack large-scale
> >> doc about it?
> >>
> >> Cheers,
> >>
> >> Arnaud.
> >>
> >>