On 11/12/19 05:18, Sean Mooney wrote:
On Tue, 2019-11-12 at 05:46 +0000, Zhong, Luyao wrote:
Hi Nova experts,
"Not tracking error migrations and orphans in RT." is probably a bug. This may trigger some problems in update_available_resources in RT at the moment. That is some orphans or error migrations are using cpus/memory/disk etc, but we don't take these usage into consideration. And instance.resources is introduced from Train used to contain specific resources, we also track assigned specific resources in RT based on tracked migrations and instances. So this bug will also affect the specific resources tracking.
I draft an doc to clarify this bug and possible solutions: https://etherpad.openstack.org/p/track-err-migr-and-orphans-in-RT Looking forward to suggestions from you. Thanks in advance.
there are patche up to allow cleaning up orpahn instances https://review.opendev.org/#/c/627765/ https://review.opendev.org/#/c/648912/ if we can get those merged that woudl adress at least some of the proablem
I just wanted to mention: I have reviewed the cleanup patches ^ multiple times and I'm having a hard time getting past the fact that any way you slice it (AFAICT), the cleanup code will have a window where a valid guest could be destroyed erroneously (not an orphan). This is because the "get instance list by host" can miss instances that are mid-migration, because of how/where we update the instance.host field. Maybe this ^ could be acceptable (?) if we put a big fat warning on the config option help for 'reap_unknown'. But I was unsure of the answers about what recovery looks like in case a guest is erroneously destroyed for an instance that is in the middle of migrating. In the case of resize or cold migrate, a hard reboot would fix it AFAIK. What about for a live migration? If recovery is possible in every case, those would also need to be documented in the config option help for 'reap_unknown'. The patch has lots of complexities to think about and I'm left wondering if the pitfalls are better or worse than the current state. It would help if others joined in the review with their thoughts about it. -melanie