[nova][ops][stable] Any interest in backporting --dry-run and/or --instance options for heal_allocations?
I was helping someone recover from a stuck live migration today where the migration record was stuck in pre-migrating status and somehow the request never hit the compute or was lost. The guest was stopped on the guest and basically the live migration either never started or never completed properly (maybe rabbit dropped the request or the compute service was restarted, I don't know).
I instructed them to update the database to set the migration record status to 'error' and hard reboot the instance to get it running again.
Then they pointed out they were seeing this in the compute logs:
"There are allocations remaining against the source host that might need to be removed"
That's because the source node allocations are still tracked in placement by the migration record and the dest node allocations are tracked by the instance. Cleaning that up is non-trivial. I have a troubleshooting doc started for manually cleaning up that kind of stuff here [1] but ultimately just told them to delete the allocations in placement for both the migration and the instance and then run the heal_allocations command to recreate the allocations for the instance. Since this person's nova deployment was running Stein, they don't have the --dry-run [2] or --instance [3] options for the heal_allocations command. This isn't a huge problem but it does mean they could be healing allocations for instances they didn't expect.
They could work around this by installing nova from train or master in a VM/container/virtual environment and running it against the stein setup, but that's maybe more work than they want to do.
The question I'm posing is if people would like to see those options backported to stein and if so, would the stable team be OK with it? I'd say this falls into a gray area where these are things that are optional, not used by default, and are operational tooling so less risk to backport, but it's not zero risk. It's also worth noting that when I wrote those patches I did so with the intent that people could backport them at least internally.
[1] https://review.opendev.org/#/c/691427/ [2] https://review.opendev.org/#/c/651932/ [3] https://review.opendev.org/#/c/651945/
The question I'm posing is if people would like to see those options backported to stein and if so, would the stable team be OK with it? I'd say this falls into a gray area where these are things that are optional, not used by default, and are operational tooling so less risk to backport, but it's not zero risk. It's also worth noting that when I wrote those patches I did so with the intent that people could backport them at least internally.
Backporting features to operator tooling that helps them recover from bugs or other failures without doing database surgery seems like a good thing. Hard to argue that the risk outweighs the benefit, IMHO.
--Dan
On Tue, Nov 05, 2019 at 08:51:13AM -0800, Dan Smith wrote:
The question I'm posing is if people would like to see those options backported to stein and if so, would the stable team be OK with it? I'd say this falls into a gray area where these are things that are optional, not used by default, and are operational tooling so less risk to backport, but it's not zero risk. It's also worth noting that when I wrote those patches I did so with the intent that people could backport them at least internally.
Backporting features to operator tooling that helps them recover from bugs or other failures without doing database surgery seems like a good thing. Hard to argue that the risk outweighs the benefit, IMHO.
FWIW, I agree with this. #makeitso
Yours Tony.
On 11/5/19 08:45, Matt Riedemann wrote:
I was helping someone recover from a stuck live migration today where the migration record was stuck in pre-migrating status and somehow the request never hit the compute or was lost. The guest was stopped on the guest and basically the live migration either never started or never completed properly (maybe rabbit dropped the request or the compute service was restarted, I don't know).
I instructed them to update the database to set the migration record status to 'error' and hard reboot the instance to get it running again.
Then they pointed out they were seeing this in the compute logs:
"There are allocations remaining against the source host that might need to be removed"
That's because the source node allocations are still tracked in placement by the migration record and the dest node allocations are tracked by the instance. Cleaning that up is non-trivial. I have a troubleshooting doc started for manually cleaning up that kind of stuff here [1] but ultimately just told them to delete the allocations in placement for both the migration and the instance and then run the heal_allocations command to recreate the allocations for the instance. Since this person's nova deployment was running Stein, they don't have the --dry-run [2] or --instance [3] options for the heal_allocations command. This isn't a huge problem but it does mean they could be healing allocations for instances they didn't expect.
They could work around this by installing nova from train or master in a VM/container/virtual environment and running it against the stein setup, but that's maybe more work than they want to do.
The question I'm posing is if people would like to see those options backported to stein and if so, would the stable team be OK with it? I'd say this falls into a gray area where these are things that are optional, not used by default, and are operational tooling so less risk to backport, but it's not zero risk. It's also worth noting that when I wrote those patches I did so with the intent that people could backport them at least internally.
I think tools like this that provide significant operability benefit are worthwhile to backport and that the value is much greater than the risk.
Related but not nearly as simple, I've backported nova-manage db purge and nova-manage db archive_deleted_rows --purge, --before, and --all-cells downstream because of the amount of bugs support/operators have opened around database cleanup pain. These were all pretty difficult to backport with the number of differences and conflicts, but my point is that I understand the motivation well and support the idea.
The fact that the patches in question were written with backportability in mind is A Good Thing.
-melanie
[1] https://review.opendev.org/#/c/691427/ [2] https://review.opendev.org/#/c/651932/ [3] https://review.opendev.org/#/c/651945/
Yea - this is our number one pain point with Nova and Rocky, and having this backported would be invaluable.
Since we are on the topic some additional issues we are having.
- Sometimes heal_allocations just fails without a good error (e.g. Compute host could not be found.) - Errors are always sequential and always halt execution, so if you have a lot of errors, you'll end up fixing them all one-by-one. - Better logging when unexpected errors do happen (maybe something more verbose like --debug would be good?).
Best Regards, Erik Olof Gunnar Andersson
-----Original Message----- From: melanie witt melwittt@gmail.com Sent: Wednesday, November 6, 2019 9:02 AM To: openstack-discuss@lists.openstack.org Subject: Re: [nova][ops][stable] Any interest in backporting --dry-run and/or --instance options for heal_allocations?
On 11/5/19 08:45, Matt Riedemann wrote:
I was helping someone recover from a stuck live migration today where the migration record was stuck in pre-migrating status and somehow the request never hit the compute or was lost. The guest was stopped on the guest and basically the live migration either never started or never completed properly (maybe rabbit dropped the request or the compute service was restarted, I don't know).
I instructed them to update the database to set the migration record status to 'error' and hard reboot the instance to get it running again.
Then they pointed out they were seeing this in the compute logs:
"There are allocations remaining against the source host that might need to be removed"
That's because the source node allocations are still tracked in placement by the migration record and the dest node allocations are tracked by the instance. Cleaning that up is non-trivial. I have a troubleshooting doc started for manually cleaning up that kind of stuff here [1] but ultimately just told them to delete the allocations in placement for both the migration and the instance and then run the heal_allocations command to recreate the allocations for the instance. Since this person's nova deployment was running Stein, they don't have the --dry-run [2] or --instance [3] options for the heal_allocations command. This isn't a huge problem but it does mean they could be healing allocations for instances they didn't expect.
They could work around this by installing nova from train or master in a VM/container/virtual environment and running it against the stein setup, but that's maybe more work than they want to do.
The question I'm posing is if people would like to see those options backported to stein and if so, would the stable team be OK with it? I'd say this falls into a gray area where these are things that are optional, not used by default, and are operational tooling so less risk to backport, but it's not zero risk. It's also worth noting that when I wrote those patches I did so with the intent that people could backport them at least internally.
I think tools like this that provide significant operability benefit are worthwhile to backport and that the value is much greater than the risk.
Related but not nearly as simple, I've backported nova-manage db purge and nova-manage db archive_deleted_rows --purge, --before, and --all-cells downstream because of the amount of bugs support/operators have opened around database cleanup pain. These were all pretty difficult to backport with the number of differences and conflicts, but my point is that I understand the motivation well and support the idea.
The fact that the patches in question were written with backportability in mind is A Good Thing.
-melanie
[1] https://urldefense.com/v3/__https://review.opendev.org/*/c/691427/__;I w!2E0gRdhhnqPNNL0!37tRTxqquwil9Vw_imfj9qg3SczjE--jSBbK3qUS_UO_wOddekP_ GkxCspm5LX4aBQ$ [2] https://urldefense.com/v3/__https://review.opendev.org/*/c/651932/__;I w!2E0gRdhhnqPNNL0!37tRTxqquwil9Vw_imfj9qg3SczjE--jSBbK3qUS_UO_wOddekP_ GkxCspniVit4uQ$ [3] https://urldefense.com/v3/__https://review.opendev.org/*/c/651945/__;I w!2E0gRdhhnqPNNL0!37tRTxqquwil9Vw_imfj9qg3SczjE--jSBbK3qUS_UO_wOddekP_ GkxCsplE-E_TJw$
On 11/6/2019 5:07 PM, Erik Olof Gunnar Andersson wrote:
Yea - this is our number one pain point with Nova and Rocky, and having this backported would be invaluable.
I posted [1] today. If that's accepted I can work on Rocky afterward.
Since we are on the topic some additional issues we are having.
- Sometimes heal_allocations just fails without a good error (e.g. Compute host could not be found.)
- Errors are always sequential and always halt execution, so if you have a lot of errors, you'll end up fixing them all one-by-one.
- Better logging when unexpected errors do happen (maybe something more verbose like --debug would be good?).
Could you open a bug with more details about the issues you're hitting. Like in what case do you hit ComputeHostNotFound?
The sequential errors thing is pretty obvious but I'm not sure what to do about it off the top of my head besides some option to say "process as much as possible storing up all of the errors to dump at the end" kind of thing.
As for better logging about unexpected errors, it's hard to know what to log that's better when it's unexpected, you know? If you have examples can you throw those into the bug report?
[1] https://review.opendev.org/#/q/status:open+project:openstack/nova+branch:sta...
participants (5)
-
Dan Smith
-
Erik Olof Gunnar Andersson
-
Matt Riedemann
-
melanie witt
-
Tony Breeds