[nova][ops] Migrating instances with old extra specs info.
Hi, We have an issue regarding live migrate old instances. On our infrastructure based on Newton, we had not well named our aggregation groups and we decided to completely change the naming conventions. We are using extra specs/metadata in flavors/aggregates and AggregateInstanceExtraSpecsFilter to select the set of hosts. After changing one of the key (lets call it 'kind') of extra spec/metadata to the new value, we are able to create new instances, but unable to live-migrate old one. This is because of stored flavor extra specs in instance_extra, which still holds the old value for the 'kind' key, and make the aggregate filter fail to match the hosts. In our opinion, it should be able to live-migrate such instance, since only the flavor metadata has changed, while other fields doesn't (vcpus, memory, disk). Is it a bug? Or maybe it is possible to change the instance flavor extra spec? -- Cheers, Roman Dobosz
On 12/03/2018 09:08 AM, Roman Dobosz wrote:
Hi,
We have an issue regarding live migrate old instances.
On our infrastructure based on Newton, we had not well named our aggregation groups and we decided to completely change the naming conventions. We are using extra specs/metadata in flavors/aggregates and AggregateInstanceExtraSpecsFilter to select the set of hosts.
After changing one of the key (lets call it 'kind') of extra spec/metadata to the new value, we are able to create new instances, but unable to live-migrate old one. This is because of stored flavor extra specs in instance_extra, which still holds the old value for the 'kind' key, and make the aggregate filter fail to match the hosts.
In our opinion, it should be able to live-migrate such instance, since only the flavor metadata has changed, while other fields doesn't (vcpus, memory, disk).
Is it a bug? Or maybe it is possible to change the instance flavor extra spec?
Not sure it's a bug, per-se... might just be something you need to manually change database records for, though. Sucks, since it's hacky, but not sure if it's something we should change in the API contract. Best, -jay
On 12/03/2018 09:08 AM, Roman Dobosz wrote:
Hi,
We have an issue regarding live migrate old instances.
On our infrastructure based on Newton, we had not well named our aggregation groups and we decided to completely change the naming conventions. We are using extra specs/metadata in flavors/aggregates and AggregateInstanceExtraSpecsFilter to select the set of hosts.
After changing one of the key (lets call it 'kind') of extra spec/metadata to the new value, we are able to create new instances, but unable to live-migrate old one. This is because of stored flavor extra specs in instance_extra, which still holds the old value for the 'kind' key, and make the aggregate filter fail to match the hosts.
In our opinion, it should be able to live-migrate such instance, since only the flavor metadata has changed, while other fields doesn't (vcpus, memory, disk).
Is it a bug? Or maybe it is possible to change the instance flavor extra spec?
Not sure it's a bug, per-se... might just be something you need to manually change database records for, though. Sucks, since it's hacky, but not sure if it's something we should change in the API contract. depending on what that extra spec was nova could not assume that changing it would not impact that vms runtime or how its scheduled. As an RFE however i might be worth considering if we could/show allow a resize to the same flavor so
On Mon, 2018-12-03 at 10:07 -0500, Jay Pipes wrote: that we can update the embeded flavor to pickup extra specs changes. if we allowed such a resize and the live resize spec makes progess then perhaps that woudl be a clean resolution to this usecase?
Best, -jay
On Mon, 03 Dec 2018 15:34:25 +0000 Sean Mooney <smooney@redhat.com> wrote:
if we allowed such a resize and the live resize spec makes progess then perhaps that woudl be a clean resolution to this usecase?
Unlike changing values during resize, overriding certain keys in extra specs could be tricky, and slippery a bit. I'd opt for nova-manage command for manipulating existing instances data instead. -- Cheers, Roman Dobosz
On Mon, 3 Dec 2018 10:07:18 -0500 Jay Pipes <jaypipes@gmail.com> wrote:
Is it a bug? Or maybe it is possible to change the instance flavor extra spec?
Not sure it's a bug, per-se... might just be something you need to
At least unexpected behavior :)
manually change database records for, though. Sucks, since it's hacky, but not sure if it's something we should change in the API contract.
I'd be happy with nova-manage command for update extra specs in instance info, it doesn't need to be an API change :) -- Cheers, Roman Dobosz
On 12/03/2018 11:32 AM, Roman Dobosz wrote:
On Mon, 3 Dec 2018 10:07:18 -0500 Jay Pipes <jaypipes@gmail.com> wrote:
Is it a bug? Or maybe it is possible to change the instance flavor extra spec?
Not sure it's a bug, per-se... might just be something you need to
At least unexpected behavior :)
It's not unexpected behaviour at all. The behaviour of Nova when changing a flavor's details has been (for many releases) that the flavor's details are changed but flavor data for already-launched instances does not change. Just like you can delete a flavor but it doesn't affect already-launched instances because we store the details of the original instance. The root of the problem you are experiencing is that flavor extra-specs is a pile of half-standardized random key-value information instead of structured, standardized quantitative and qualitative information (*gasp* that's what resource classes and traits are in placement...) Which is why I say this isn't actually unexpected behaviour or really a bug.
manually change database records for, though. Sucks, since it's hacky, but not sure if it's something we should change in the API contract.
I'd be happy with nova-manage command for update extra specs in instance info, it doesn't need to be an API change :)
mysql -u$USER -p$PASSWORD -Dnova -e"UPDATE instance_extra SET flavor = REPLACE(flavor, 'old string', 'new string')" There, no need for a nova-manage command. Best, -jay p.s. test the above on a copy of your production DB before you trust it... :)
On Mon, 3 Dec 2018 11:56:17 -0500 Jay Pipes <jaypipes@gmail.com> wrote:
Is it a bug? Or maybe it is possible to change the instance flavor extra spec? Not sure it's a bug, per-se... might just be something you need to At least unexpected behavior :) It's not unexpected behaviour at all. The behaviour of Nova when changing a flavor's details has been (for many releases) that the flavor's details are changed but flavor data for already-launched instances does not change. Just like you can delete a flavor but it doesn't affect already-launched instances because we store the details of the original instance.
For resource related things - sure. As for more ephemeral things like metadata - its arguable. Although, I tend to agree, that sometimes metadata could potentially hold sensitive information, which shouldn't be changed in that case.
The root of the problem you are experiencing is that flavor extra-specs is a pile of half-standardized random key-value information instead of structured, standardized quantitative and qualitative information (*gasp* that's what resource classes and traits are in placement...)
Yeah, I know. But we have that issue in Newton, where traits are nonexistent yet :)
Which is why I say this isn't actually unexpected behaviour or really a bug.
manually change database records for, though. Sucks, since it's hacky, but not sure if it's something we should change in the API contract. I'd be happy with nova-manage command for update extra specs in instance info, it doesn't need to be an API change :) mysql -u$USER -p$PASSWORD -Dnova -e"UPDATE instance_extra SET flavor = REPLACE(flavor, 'old string', 'new string')"
There, no need for a nova-manage command.
Well, thank you, kind sir!
p.s. test the above on a copy of your production DB before you trust it... :)
You bet :) -- Cheers, Roman Dobosz
participants (3)
-
Jay Pipes
-
Roman Dobosz
-
Sean Mooney