Open Stack

Great!
Thanks Jim.

I will later report our experience with conductor groups.

Belmiro
CERN

On Tue, Nov 12, 2019 at 5:58 PM Jim Rollenhagen <jim at jimrollenhagen.com>
wrote:

>
>
> On Tue, Nov 12, 2019 at 11:38 AM Belmiro Moreira <
> moreira.belmiro.email.lists at gmail.com> wrote:
>
>> Dan Smith just point me the conductor groups that were added in Stein.
>>
>> https://specs.openstack.org/openstack/nova-specs/specs/stein/implemented/ironic-conductor-groups.html
>> This is an interesting way to partition the deployment much better than
>> the multiple nova-computes setup.
>>
>
> Just a note, they aren't mutually exclusive. You can run multiple
> nova-computes to manage a single conductor group, whether for HA or because
> you're using groups for some other construct (cells, racks, halls, network
> zones, etc) which you want to shard further.
>
> // jim
>
>
>> Thanks,
>> Belmiro
>> CERN
>>
>> On Tue, Nov 12, 2019 at 5:06 PM Belmiro Moreira <
>> moreira.belmiro.email.lists at gmail.com> wrote:
>>
>>> Hi,
>>> using several cells for the Ironic deployment would be great however it
>>> doesn't work with the current architecture.
>>> The nova ironic driver gets all the nodes available in Ironic. This
>>> means that if we have several cells all of them will report the same nodes!
>>> The other possibility is to have a dedicated Ironic instance per cell,
>>> but in this case it will be very hard to manage a large deployment.
>>>
>>> What we are trying is to shard the ironic nodes between several
>>> nova-computes.
>>> nova/ironic deployment supports several nova-computes and it will be
>>> great if the RT nodes cycle is sharded between them.
>>>
>>> But anyway, this will also require speeding up the big lock.
>>> It would be great if a compute node can handle more than 500 nodes.
>>> Considering our use case: 15k/500 = 30 compute nodes.
>>>
>>> Belmiro
>>> CERN
>>>
>>>
>>>
>>> On Mon, Nov 11, 2019 at 9:13 PM Matt Riedemann <mriedemos at gmail.com>
>>> wrote:
>>>
>>>> On 11/11/2019 7:03 AM, Chris Dent wrote:
>>>> > Or using
>>>> > separate processes? For the ironic and vsphere contexts, increased
>>>> > CPU usage by the nova-compute process does not impact on the
>>>> > workload resources, so parallization is likely a good option.
>>>>
>>>> I don't know how much it would help - someone would have to actually
>>>> test it out and get metrics - but one easy win might just be using a
>>>> thread or process executor pool here [1] so that N compute nodes could
>>>> be processed through the update_available_resource periodic task
>>>> concurrently, maybe $ncpu or some factor thereof. By default make it
>>>> serialized for backward compatibility and non-ironic deployments.
>>>> Making
>>>> that too highly concurrent could have negative impacts on other things
>>>> running on that host, like the neutron agent, or potentially storming
>>>> conductor/rabbit with a ton of DB requests from that compute.
>>>>
>>>> That doesn't help with the scenario that the big
>>>> COMPUTE_RESOURCE_SEMAPHORE lock is held by the periodic task while
>>>> spawning, moving, or deleting an instance that also needs access to the
>>>> big lock to update the resource tracker, but baby steps if any steps in
>>>> this area of the code would be my recommendation.
>>>>
>>>> [1]
>>>>
>>>> https://github.com/openstack/nova/blob/20.0.0/nova/compute/manager.py#L8629
>>>>
>>>> --
>>>>
>>>> Thanks,
>>>>
>>>> Matt
>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20191112/edf7d166/attachment.html>