[nova][ptg] Allow compute nodes to use DISK_GB from shared storage RP by using aggregate relationship
spec: https://review.opendev.org/#/c/650188/ Agreements from the room[1]: * new config option 'using_shared_disk_provider' (just an example name) on the compute level to ignore the DISK_GB reporting from the driver * new config option 'sharing_disk_aggregate' (just an example name) on the compute level to tell nova compute what is the UUID of the placement aggregate that contains the sharing DISK_GB providers in placement. * the "using_shared_disk_provider" flag necessarly has to be explicit since if not, it would be a chicken-and-egg problem on a greenfields install as the shared RP wouldn't be created * deployer needs to create the sharing disk RP and report inventory / traits on it * deployer needs to define the placement aggregate and add the sharing disk RP into it * when compute restarts and sees that 'using_shared_disk_provider' = True in the config, it adds the its compute RP to the aggregate defined in 'sharing_disk_aggregate' Then if it sees that the root RP still has DISK_GB inventory then trigger a reshape * os-hypervisor API response (in a new microversion) will have a link to the sharing disk RP if the compute is so configured. TODO: * tpatil to update the spec [1] https://etherpad.openstack.org/p/nova-shanghai-ptg
Hi All,
TODO: * tpatil to update the spec
I will update the specs next week and upload it for review. Regards, tpatil ________________________________________ From: Balázs Gibizer <balazs.gibizer@est.tech> Sent: Friday, November 8, 2019 5:01 PM To: openstack-discuss Subject: [nova][ptg] Allow compute nodes to use DISK_GB from shared storage RP by using aggregate relationship spec: https://review.opendev.org/#/c/650188/ Agreements from the room[1]: * new config option 'using_shared_disk_provider' (just an example name) on the compute level to ignore the DISK_GB reporting from the driver * new config option 'sharing_disk_aggregate' (just an example name) on the compute level to tell nova compute what is the UUID of the placement aggregate that contains the sharing DISK_GB providers in placement. * the "using_shared_disk_provider" flag necessarly has to be explicit since if not, it would be a chicken-and-egg problem on a greenfields install as the shared RP wouldn't be created * deployer needs to create the sharing disk RP and report inventory / traits on it * deployer needs to define the placement aggregate and add the sharing disk RP into it * when compute restarts and sees that 'using_shared_disk_provider' = True in the config, it adds the its compute RP to the aggregate defined in 'sharing_disk_aggregate' Then if it sees that the root RP still has DISK_GB inventory then trigger a reshape * os-hypervisor API response (in a new microversion) will have a link to the sharing disk RP if the compute is so configured. TODO: * tpatil to update the spec [1] https://etherpad.openstack.org/p/nova-shanghai-ptg Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.
On 11/8/2019 2:01 AM, Balázs Gibizer wrote:
* deployer needs to create the sharing disk RP and report inventory / traits on it * deployer needs to define the placement aggregate and add the sharing disk RP into it * when compute restarts and sees that 'using_shared_disk_provider' = True in the config, it adds the its compute RP to the aggregate defined in 'sharing_disk_aggregate' Then if it sees that the root RP still has DISK_GB inventory then trigger a reshape
Does the compute host also get added to a nova host aggregate which mirrors the resource provider aggregate in placmeent or do we only need the placement resource provider sharing DISK_GB aggregate? -- Thanks, Matt
On Fri, Nov 8, 2019 at 08:03, Matt Riedemann <mriedemos@gmail.com> wrote:
On 11/8/2019 2:01 AM, Balázs Gibizer wrote:
* deployer needs to create the sharing disk RP and report inventory / traits on it * deployer needs to define the placement aggregate and add the sharing disk RP into it * when compute restarts and sees that 'using_shared_disk_provider' = True in the config, it adds the its compute RP to the aggregate defined in 'sharing_disk_aggregate' Then if it sees that the root RP still has DISK_GB inventory then trigger a reshape
Does the compute host also get added to a nova host aggregate which mirrors the resource provider aggregate in placmeent or do we only need the placement resource provider sharing DISK_GB aggregate?
As far as I see we only need the placement aggregate to make this work.
--
Thanks,
Matt
On 11/8/2019 2:01 AM, Balázs Gibizer wrote:
* when compute restarts and sees that 'using_shared_disk_provider' = True in the config, it adds the its compute RP to the aggregate defined in 'sharing_disk_aggregate' Then if it sees that the root RP still has DISK_GB inventory then trigger a reshape
Conversely, if the deployer decides to use local disk for the host again, what are the steps? 1. Change using_shared_disk_provider=False 2. Restart/SIGHUP compute service 3. Compute removes itself from the aggregate 4. Compute reshapes to add DISK_GB inventory on the root compute node resource provider and moves DISK_GB allocations from the sharing provider back to the root compute node provider. Correct? -- Thanks, Matt
On Fri, Nov 8, 2019 at 08:05, Matt Riedemann <mriedemos@gmail.com> wrote:
On 11/8/2019 2:01 AM, Balázs Gibizer wrote:
* when compute restarts and sees that 'using_shared_disk_provider' = True in the config, it adds the its compute RP to the aggregate defined in 'sharing_disk_aggregate' Then if it sees that the root RP still has DISK_GB inventory then trigger a reshape
Conversely, if the deployer decides to use local disk for the host again, what are the steps?
1. Change using_shared_disk_provider=False 2. Restart/SIGHUP compute service 3. Compute removes itself from the aggregate 4. Compute reshapes to add DISK_GB inventory on the root compute node resource provider and moves DISK_GB allocations from the sharing provider back to the root compute node provider.
Correct?
Seems correct to me. gibi
--
Thanks,
Matt
On Wed, Nov 13, 2019 at 3:32 PM Balázs Gibizer <balazs.gibizer@est.tech> wrote:
On Fri, Nov 8, 2019 at 08:05, Matt Riedemann <mriedemos@gmail.com> wrote:
On 11/8/2019 2:01 AM, Balázs Gibizer wrote:
* when compute restarts and sees that 'using_shared_disk_provider' = True in the config, it adds the its compute RP to the aggregate defined in 'sharing_disk_aggregate' Then if it sees that the root RP still has DISK_GB inventory then trigger a reshape
Conversely, if the deployer decides to use local disk for the host again, what are the steps?
1. Change using_shared_disk_provider=False 2. Restart/SIGHUP compute service 3. Compute removes itself from the aggregate 4. Compute reshapes to add DISK_GB inventory on the root compute node resource provider and moves DISK_GB allocations from the sharing provider back to the root compute node provider.
Correct?
Seems correct to me.
gibi
Me too. To be clear, I don't think operators would modify the above but if so, they would need reshapes.
--
Thanks,
Matt
On 11/13/2019 8:34 AM, Sylvain Bauza wrote:
Me too. To be clear, I don't think operators would modify the above but if so, they would need reshapes.
Maybe not, but this is the kind of detail that should be in the spec and functional tests to make sure it's solid since this is a big architectural change in nova. -- Thanks, Matt
On 11/13/2019 8:34 AM, Sylvain Bauza wrote:
Me too. To be clear, I don't think operators would modify the above but if so, they would need reshapes.
Maybe not, but this is the kind of detail that should be in the spec and functional tests to make sure it's solid since this is a big architectural change in nova.
It depends on how the aggregates are created on the nova and placement side. A) From placement point of view, operator can create a new aggregate and add shared storage RP to it (tag MISC_SHARES_VIA_AGGREGATE trait to this RP). The newly created valid UUID would then be set in the config option ``sharing_disk_aggregate`` on the compute node side. This aggregate UUID wouldn't be present in the nova aggregate. so it's not possible to add host to the nova aggregate unless a new aggregate is created on nova side. B) If nova aggregates are synced to the placement service and say below is the picture: Nova: Agg1 - metadata (pinned=True) - host1 - host2 Now, operator adds a new shared storage RP to Agg1 on placement side and then set Agg1 UUID in ``sharing_disk_aggregate`` on compute nodes along with ``using_shared_disk_provider`=True``, then it would add compute node RP to the Agg1 on the placement without any issues but when you want to reverse the configuration, using_shared_disk_provider=False, then it not that straight to remove the host from the placement/nova aggregate because there would be other traits set to compute RPs which could cause those functions stop working. We had same kind of discussion [1] when implementing forbidden aggregates where we want to sync traits set to the aggregates but later it was concluded that operator will do it manually. I will include the details Matt has pointed out in this email in my next patchset. [1] : http://lists.openstack.org/pipermail/openstack-discuss/2019-June/006950.html Regards, tpatil ________________________________________ From: Matt Riedemann <mriedemos@gmail.com> Sent: Wednesday, November 13, 2019 11:41 PM To: openstack-discuss@lists.openstack.org Subject: Re: [nova][ptg] Allow compute nodes to use DISK_GB from shared storage RP by using aggregate relationship On 11/13/2019 8:34 AM, Sylvain Bauza wrote:
Me too. To be clear, I don't think operators would modify the above but if so, they would need reshapes.
Maybe not, but this is the kind of detail that should be in the spec and functional tests to make sure it's solid since this is a big architectural change in nova. -- Thanks, Matt Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding.
participants (4)
-
Balázs Gibizer
-
Matt Riedemann
-
Patil, Tushar
-
Sylvain Bauza