[Openstack-operators] How to delete instances
gabe.westmaas at RACKSPACE.COM
Tue Aug 28 12:09:40 UTC 2012
> -----Original Message-----
> From: Narayan Desai [mailto:narayan.desai at gmail.com]
> Sent: Tuesday, August 28, 2012 7:54 AM
> To: Lorin Hochstein
> Cc: openstack-operators at lists.openstack.org
> Subject: Re: [Openstack-operators] How to delete instances
> On Mon, Aug 27, 2012 at 9:53 PM, Lorin Hochstein
> <lorin at nimbisservices.com> wrote:
> > You should be able to use "nova reset-state" and then "nova-delete" in
> > order to delete them if they're stuck. See
> > http://docs.openstack.org/essex/openstack-
> > state.html
> Hi Lorin. This doesn't work against essex systems, right? Is there an
> alternative method in that case?
> I've been using a simple script to bang on the db to set deleted=1 and
> deleted_at=NOW(), which works, but seems potentially wrong.
That will likely leave the instances still running on the hypervisor, just no longer visible. To perform the "reset-state" action, set vm_state to "error" and task_state to NULL, and then you should no longer be blocked from deleting because of task state or vm state.
However, in some of the output we saw from Vivek, it showed that he got a "204" back, which likely means the vm and task state are fine. Vivek, can you do anything else with these instances? Like change the password, or reboot them? My guess is no. I have seen this happen when either:
1) The nova-compute service is no longer running (and therefore not consuming from the rabbit queue)
2) The nova-compute service thinks it's still connected to rabbit, but some sort of network event has interrupted the connection from the rabbit side.
3) The nova-compute service is "stuck" in some other way, and no longer reading from the rabbit queue.
I'm sure there are some other causes as well, but typically this means the message isn't getting to the compute node successfully in some way! If you just want to get things working, I'd try restarting the nova-compute services and rabbitmq service. Although there was another thread that seemed to indicate that restarting the compute service also restarted all VMs - sorry, I work in the Xen world, so I'm not sure how that behaves outside of Xen!
Finally, as a good debugging step, you might try looking for those request IDs in all service logs to see how far you actually got. For example grep for: req-ed8fc900-02cd-4496-9b83-4891c7d3cb7d in nova-api.log, nova-network.log and all nova-compute.log files. You also might try looking at the number of messages in all your rabbit queues to see if there is one that is obviously high which could indicate a problem.
> OpenStack-operators mailing list
> OpenStack-operators at lists.openstack.org
More information about the OpenStack-operators