Hi Eric,

Thank you for your response. Please see my response inline.

Regards, Madhuri

-----Original Message----- From: Eric Fried [mailto:openstack@fried.cc] Sent: Tuesday, June 4, 2019 12:07 AM To: openstack-discuss@lists.openstack.org Subject: Re: [Nova][Ironic] Reset Configurations in Baremetals Post Provisioning

Hi Madhuri-

...

For this purpose, we would need to change a trait of the server’s
flavor in Nova. This trait is mapped to a deploy step in Ironic
which does some operation(change BIOS config and reboot in this use
case).____

If your trait was something that wasn't tracked in the flavor (or elsewhere in the instance's db record), you could just update it directly in placement. Then you'd have to figure out how to make ironic notice that and effect the change. (Or perhaps the other way around: tell ironic you want to make the change, and it updates the trait in placement as part of the process.)

In this case, the trait is stored with flavor so it is known to Nova. The new trait should be added in the database and the old one removed. For an ex:

An instance with flavor bm_hyperthreading with trait:CUSTOM_HYPERTHREADING_ON=required is created in Nova. Now the user wants to turn off the hyperthreading, than they could update the flavor with trait:CUSTOM_HYPERTHREADING_OFF=required. This should remove the trait:CUSTOM_HYPERTHREADING_ON and add trait:CUSTOM_HYPERTHREADING_OFF associated with the new flavor.

...

In Nova, the only API to change trait in flavor is resize whereas
resize does migration and a reboot as well.____

In short, I am  looking for a Nova API that only changes the traits,
and trigger the ironic deploy steps but no reboot and migration.
Please suggest.____

It's inconvenient, but I'm afraid "resize" is the right way to get this done, because that's the only way to get the appropriate validation and changes effected in the general case.

Yes, resize seems to be the only valid one.

Now, there's a spec [1] we've been talking about for ~4.5 years that would let you do a resize without rebooting, when only a certain subset of properties are being changed. It is currently proposed for "upsizing" CPU, memory, and disk, and adding PCI devices, but clearly this ISS configuration would be a reasonable candidate to include.

Looking at the specs, it seems it's mostly talking about changing VMs resources without rebooting. However that's not the actual intent of the Ironic use case I explained in the email. Yes, it requires a reboot to reflect the BIOS changes. This reboot can be either be done by Nova IronicDriver or Ironic deploy step can also do it. So I am not sure if the spec actually satisfies the use case. I hope to get more response from the team to get more clarity.

In fact, it's possible that leading the charge with something this unobtrusive would reduce some of the points of contention that have stalled the blueprint up to this point.

Food for thought.

Thanks, efried

[1] https://review.opendev.org/#/c/141219/

Hi Jay,

-----Original Message----- From: Jay Pipes [mailto:jaypipes@gmail.com]

The absence of a trait on a provider should be represented by the provider not having a trait. Just have a single trait "CUSTOM_HYPERTHREADING" that you either place on the provider or do not place on a provider.

The flavor should then either request that the trait be present on a provider that the instance is scheduled to (trait:CUSTOM_HYPERTHREADING=required) or that the trait should *not* be present on a provider that the instance is scheduled to (trait:CUSTOM_HYPERTHREADING=forbidden).

I understand that these traits are used for scheduling while server create in Nova. Whereas these traits means more to Ironic. Ironic runs multiple deploy steps matching the name of traits in flavor[1].

The use case explained in the email is about changing some BIOS configuration post server create. By changing the trait in flavor from CUSTOM_HYPERTHREADING_ON to CUSTOM_HYPERTHREADING_OFF, Ironic should run the matching deploy step to disable hyperthreading in BIOS and do a reboot. But currently there isn't a way in Nova about telling Ironic about the trait has changed in flavor, so perform the corresponding deploy steps.

[1] https://docs.openstack.org/ironic/stein/admin/node-deployment.html#matching-...

Regards, Madhuri

Better still, add a standardized trait to os-traits for hyperthreading support, which is what I'd recommended in the original cpu-resource-tracking spec.

HW_CPU_HYPERTHREADING was added via [1] and has been in os-traits since 0.8.0.

efried

[1] https://review.opendev.org/#/c/576030/

On Fri, 7 Jun 2019, 18:02 Jay Pipes, jaypipes@gmail.com wrote:

On 6/7/19 11:23 AM, Eric Fried wrote:

...

...

Better still, add a standardized trait to os-traits for hyperthreading support, which is what I'd recommended in the original cpu-resource-tracking spec.

HW_CPU_HYPERTHREADING was added via [1] and has been in os-traits since 0.8.0.

I think we need a tri-state here. There are three options:

1. Give me a node with hyperthreading enabled 2. Give me a node with hyperthreading disabled 3. I don't care

For me, the lack of a trait is 3 - I wouldn't want existing flavours without this trait to cause hyperthreading to be disabled.

The ironic deploy templates feature wasn't designed to support forbidden traits - I don't think they were implemented at the time. The example use cases so far have involved encoding values into a trait name, e.g. CUSTOM_HYPERTHREADING_ON. Forbidden traits could be made to work in this case, but it doesn't really extend to non Boolean things such as RAID levels.

I'm not trying to shoot down new ideas, just explaining how we got here.

Excellent, I had a faint recollection of that...

-jay

Looking at the specs, it seems it's mostly talking about changing VMs resources without rebooting. However that's not the actual intent of the Ironic use case I explained in the email. Yes, it requires a reboot to reflect the BIOS changes. This reboot can be either be done by Nova IronicDriver or Ironic deploy step can also do it. So I am not sure if the spec actually satisfies the use case. I hope to get more response from the team to get more clarity.

Waitwait. The VM needs to be rebooted for the BIOS change to take effect? So (non-live) resize would actually satisfy your use case just fine. But the problem is that the ironic driver doesn't support resize at all?

Without digging too hard, that seems like it would be a fairly straightforward thing to add. It would be limited to only "same host" and initially you could only change this one attribute (anything else would have to fail).

Nova people, thoughts?

efried .

On Mon, 10 Jun 2019 at 06:18, Alex Xu soulxu@gmail.com wrote:

Eric Fried openstack@fried.cc 于2019年6月7日周五 上午1:59写道：

...

...

Looking at the specs, it seems it's mostly talking about changing VMs resources without rebooting. However that's not the actual intent of the Ironic use case I explained in the email. Yes, it requires a reboot to reflect the BIOS changes. This reboot can be either be done by Nova IronicDriver or Ironic deploy step can also do it. So I am not sure if the spec actually satisfies the use case. I hope to get more response from the team to get more clarity.

Waitwait. The VM needs to be rebooted for the BIOS change to take effect? So (non-live) resize would actually satisfy your use case just fine. But the problem is that the ironic driver doesn't support resize at all?

Without digging too hard, that seems like it would be a fairly straightforward thing to add. It would be limited to only "same host" and initially you could only change this one attribute (anything else would have to fail).

Nova people, thoughts?

Contribute another idea.

So just as Jay said in this thread. Those CUSTOM_HYPERTHREADING_ON and CUSTOM_HYPERTHREADING_OFF are configuration. Those configuration isn't used for scheduling. Actually, Traits is designed for scheduling.

So yes, there should be only one trait. CUSTOM_HYPERTHREADING, this trait is used for indicating the host support HT. About whether enable it in the instance is configuration info.

That is also pain for change the configuration in the flavor. The flavor is the spec of instance's virtual resource, not the configuration.

So another way is we should store the configuration into another place. Like the server's metadata.

So for the HT case. We only fill the CUSTOM_HYPERTHREADING trait in the flavor, and fill a server metadata 'hyperthreading_config=on' in server metadata. The nova will find out a BM node support HT. And ironic based on the server metadata 'hyperthreading_config=on' to enable the HT.

When change the configuration of HT to off, the user can update the server's metadata. Currently, the nova will send a rpc call to the compute node and calling a virt driver interface when the server metadata is updated. In the ironic virt driver, it can trigger a hyper-threading configuration deploy step to turn the HT off, and do a reboot of the instance. (The reboot is a step inside deploy-step, not part of ironic virt driver flow)

But yes, this changes some design to the original deploy-steps and deploy-templates. And we fill something into the server's metadata which I'm not sure nova people like it.

Anyway, just put my idea at here.

We did consider using metadata. The problem is that it is user-defined, so there is no way for an operator to restrict what can be done by a user. Flavors are operator-defined and so allow for selection from a 'menu' of types and configurations.

What might be nice is if we could use a flavor extra spec like this:

deploy-config:hyperthreading=enabled

The nova ironic virt driver could pass this to ironic, like it does with traits.

Then in the ironic deploy template, have fields like this:

name: Hyperthreading enabled config-type: hyperthreading config-value: enabled steps: <deploy steps>

Ironic would then match on the config-type and config-value to find a suitable deploy template.

As an extension, the deploy template could define a trait (or list of traits) that must be supported by a node in order for the template to be applied. Perhaps this would even be a standard relationship between config-type and traits?

Haven't thought this through completely, I'm sure it has holes.

...

efried .

On 6/11/2019 2:46 PM, Eric Fried wrote:

...

What might be nice is if we could use a flavor extra spec like this:

deploy-config:hyperthreading=enabled

The nova ironic virt driver could pass this to ironic, like it does with traits.

Then in the ironic deploy template, have fields like this:

name: Hyperthreading enabled config-type: hyperthreading config-value: enabled steps: <deploy steps>

Ironic would then match on the config-type and config-value to find a suitable deploy template.

As an extension, the deploy template could define a trait (or list of traits) that must be supported by a node in order for the template to be applied. Perhaps this would even be a standard relationship between config-type and traits?

This.

As rubber has hit road for traits-related-to-config, the pattern that has emerged as (IMO) most sensible has looked a lot like the above.

To get a bit more specific:

• HW_CPU_HYPERTHREADING is a trait indicating that a node is*capable*

of switching hyperthreading on. There is no trait, ever, anywhere, that indicates that is is on or off on a particular node.

• The ironic virt driver tags the node RP with the trait when it detects

that the node is capable.

• The flavor (or image) indicates a desire to enable hyperthreading as

Mark says: via a (non-Placement-ese) property that conveys information in a way that ironic can understand.

• A request filter [1] interprets the non-Placement-ese property and

adds HW_CPU_HYPERTHREADING as a required trait to the request if it's `enabled`, so the scheduler will ensure we land on a node that can handle it.

• During spawn, the ironic virt driver communicates whatever/however to

ironic based on the (non-Placement-ese) property in the flavor/image.

Getting back to the original issue of this thread, this still means we need to implement some limited subset of `resize` for ironic to allow us to turn this thing on or off on an established instance. That resize should still go through the scheduler so that, for example, the above process will punt if you try to switch on hyperthreading on a node that isn't capable (doesn't have the HW_CPU_HYPERTHREADING trait).

This sounds similar to the ARQ device profile stuff from the nova/cyborg spec [1] - is it?

Also, I'm reminded of the glare/artifactory discussion for baremetal node config we talked about at the PTG in Dublin [2] - how does this compare/contrast?

[1] https://review.opendev.org/#/c/603955/ [2] https://etherpad.openstack.org/p/nova-ptg-rocky (~L250)

Hi Mark,

Replied inline.

From: Mark Goddard [mailto:mark@stackhpc.com] Sent: Monday, June 3, 2019 2:16 PM To: Kumari, Madhuri madhuri.kumari@intel.com Cc: openstack-discuss@lists.openstack.org Subject: Re: [Nova][Ironic] Reset Configurations in Baremetals Post Provisioning

On Mon, 3 Jun 2019 at 06:57, Kumari, Madhuri <madhuri.kumari@intel.commailto:madhuri.kumari@intel.com> wrote: Hi Ironic, Nova Developers,

I am currently working on implementing Intel Speed Select(ISS) feature[1] in Ironic and I have a use case where I want to change ISS configuration in BIOS after a node is provisioned. Such use case of changing the configuration post deployment is common and not specific to ISS. A real-life example for such a required post-deploy configuration change is the change of BIOS settings to disable hyper-threading in order to address a security vulnerability. Currently there is no way of changing any BIOS configuration after a node is provisioned in Ironic. One solution for it is to allow manual deploy steps in Ironic[2](not implemented yet) which can be trigged by changing traits in Nova. For this purpose, we would need to change a trait of the server’s flavor in Nova. This trait is mapped to a deploy step in Ironic which does some operation(change BIOS config and reboot in this use case). In Nova, the only API to change trait in flavor is resize whereas resize does migration and a reboot as well. In short, I am looking for a Nova API that only changes the traits, and trigger the ironic deploy steps but no reboot and migration. Please suggest.

Hi, it is possible to modify a flavor (openstack flavor set <flavor> --property <key>=<value>). However, changes to a flavor are not reflected in instances that were previously created from that flavor. Internally, nova stores an 'embedded flavor' in the instance state. I'm not aware of any API that would allow modifying the embedded flavor, nor any process that would synchronise those changes to ironic.

The resize API in Nova allows changing the flavor of an instance. It does migration and reboots. But the API is not implemented for IronicDriver. Though this doesn’t match our use case but seems to be the only available one that allows changing a flavor and ultimately a trait.

Thanks in advance.

Regards, Madhuri [1] https://specs.openstack.org/openstack/ironic-specs/specs/not-implemented/sup... [2] https://storyboard.openstack.org/#!/story/2005129