Open Stack

Chris and I were talking about the various bugs related to not cleaning 
up allocations during failures, of which I've reported two more today 
[1][2].

The discovery process here is really just me doing code inspection, then 
opening a bug, writing a recreate functional test and then fixing the 
bug. This is based on a few other bugs we already fixed (for evacuate, 
shelve offload, resize and live migration).

We got to talking about a report we could run at the end of a CI job run 
which dumps the allocations out of Placement and then we could use that 
to figure out what we are missing.

We actually already do something like this with 
nova/tools/hooks/post_test_hook.sh in the nova-next job, as seen here 
[3]. That runs the 'nova-manage db archive_deleted_rows' CLI a few times 
to make sure things are getting archived properly.

I was thinking we could leverage that post_test_hook script to do 
something similar for dumping allocations from Placement.

The devil, of course, is in the details, because we don't have a CLI for 
Placement yet. There are some patches for the OSC plugin though [4]. If 
we got those in, at least the ones to list resource providers, usages 
and allocations, we could get a start on this.

There would likely be other wrinkles, like if there are still instances 
lingering when the script runs, we'd have to take those into account. I 
think in a failed job, we wouldn't care as much, but in a successful job 
where Tempest passes, we'd expect allocations to be cleaned up because 
the instances shouldn't be leaking from Tempest.

Anyway, these are just some ideas if someone wanted to start hacking on 
this. It should be self-testing since like I said it's in the nova-next 
job so if you're just playing around with patches in nova you can see 
the results show up in that post_test_hook log.

[1] https://bugs.launchpad.net/nova/+bug/1713786
[2] https://bugs.launchpad.net/nova/+bug/1713796
[3] 
http://logs.openstack.org/61/496861/1/check/gate-tempest-dsvm-neutron-nova-next-full-ubuntu-xenial-nv/572ceed/logs/devstack-gate-post_test_hook.txt.gz
[4] https://review.openstack.org/#/q/project:openstack/osc-placement

-- 

Thanks,

Matt