[Openstack] unrescue VM instance

Balazs Varhegyi varh1ibalazs at gmail.com
Wed Mar 16 08:58:42 UTC 2016


Found this in the cloud-init.log:

in cloud-init.log:
2016-02-10 14:02:09,604 - cc_resizefs.py[DEBUG]: Resizing / (ext4) using 
resize2fs /dev/disk/by-label/cloudimg-rootfs
2016-02-10 14:02:09,604 - util.py[DEBUG]: Running command ('resize2fs', 
'/dev/disk/by-label/cloudimg-rootfs') with allowed return 
codes[0](shell=False, capture=True)
2016-02-10 14:02:09,615 - util.py[WARNING]: Failed to resize filesystem 
(cmd=('resize2fs', '/dev/disk/by-label/cloudimg-rootfs'))
2016-02-10 14:02:09,616 - util.py[DEBUG]: Failed to resize filesystem 
(cmd=('resize2fs', '/dev/disk/by-label/cloudimg-rootfs'))
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cloudinit/config/cc_resizefs.py", 
line 172, in do_resize
util.subp(resize_cmd)
File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 1539, in 
subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ('resize2fs', '/dev/disk/by-label/cloudimg-rootfs')
Exit code: 1
Reason: -
Stdout: ''
Stderr: "resize2fs 1.42.9 (4-Feb-2014)\nPlease run 'e2fsck -f 
/dev/disk/by-label/cloudimg-rootfs' first.\n\n"
2016-02-10 14:02:09,627 - util.py[DEBUG]: Resizing took 0.023 seconds
2016-02-10 14:02:09,627 - util.py[WARNING]: Running resizefs (<module 
'cloudinit.config.cc_resizefs' from 
'/usr/lib/python2.7/dist-packages/cloudinit/config/cc_resizefs.pyc'>) failed
2016-02-10 14:02:09,627 - util.py[DEBUG]: Running resizefs (<module 
'cloudinit.config.cc_resizefs' from 
'/usr/lib/python2.7/dist-packages/cloudinit/config/cc_resizefs.pyc'>) failed
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/cloudinit/stages.py", line 658, 
in _run_modules
cc.run(run_name, mod.handle, func_args, freq=freq)
File "/usr/lib/python2.7/dist-packages/cloudinit/cloud.py", line 63, in run
return self._runners.run(name, functor, args, freq, clear_on_fail)
File "/usr/lib/python2.7/dist-packages/cloudinit/helpers.py", line 197, 
in run
results = functor(*args)
File "/usr/lib/python2.7/dist-packages/cloudinit/config/cc_resizefs.py", 
line 161, in handle
func=do_resize, args=(resize_cmd, log))
File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 1875, in 
log_time
ret = func(*args, **kwargs)
File "/usr/lib/python2.7/dist-packages/cloudinit/config/cc_resizefs.py", 
line 172, in do_resize
util.subp(resize_cmd)
File "/usr/lib/python2.7/dist-packages/cloudinit/util.py", line 1539, in 
subp
cmd=args)
ProcessExecutionError: Unexpected error while running command.
Command: ('resize2fs', '/dev/disk/by-label/cloudimg-rootfs')
Exit code: 1
Reason: -
Stdout: ''
Stderr: "resize2fs 1.42.9 (4-Feb-2014)\nPlease run 'e2fsck -f 
/dev/disk/by-label/cloudimg-rootfs' first.\n\n"

Is it possible that after we cleared up the disk openstack tried to 
resize the disk but it didn't manage and the image got corrupted?
Google search gave a case when someone had an image with this error that 
was not bootable: 
https://bugs.launchpad.net/diskimage-builder/+bug/1280709/ ("Daniele 
Venzano wroteon 2014-02-28:" )

Regards, Balazs


On 15/03/16 14:13, Balazs Varhegyi wrote:
>
> Hi Steve,
>
> Thanks for the suggestion,
> I already managed to clear up some space in rescue mode (VM instance has
> 2,1T disk with 1T free space) and core machine has an extra 270G free
> disk too.
>
> Balazs
>
> On 15/03/16 13:56, Stephen Davies wrote:
>> Hi Balazs,
>>
>> Have you tried connecting to the image via NBD (network block device).
>>
>> 1) Mount the image.
>> 2) Repair/free some space.
>> 3) Start the instance.
>>
>> http://blog.vmsplice.net/2011/02/how-to-access-virtual-machine-image.html 
>>
>>
>>
>> Steve
>>
>>
>>
>>
>> On 15/03/16 11:20, Tomas Vondra wrote:
>>> Balazs Varhegyi <varh1ibalazs at ...> writes:
>>>
>>>> Hi guys,
>>>> I have an openstack instance that run out of storage on core machine
>>>> and
>>>> the instance got paused.
>>>> I cleared up the core machine so it has more disk now but when I start
>>>> the machine I can't ssh in.
>>>> In rescue mode I'm able to ssh in but I want to unrescue the 
>>>> machine to
>>>> be able to use it again without creating a new instance and migrate
>>>> everything from this failed instance manually.
>>>> I enabled virtlib debug log but nothing interesting came up (or at
>>>> least
>>>> I didn't notice), the only thing that indicates an error is:
>>>> "Domain id=53 name='instance-00000281'
>>>> uuid=f2715c2c-6d4e-5cf1-7606-f5552a59cb56 is tainted: host-cpu"
>>>> "nova list" shows the machine has the correct IP set but I'm not
>>>> able to
>>>> ping it on private IP nor via floating IP.
>>>> I tracked the network configuration from libvirt.xml and it seems it's
>>>> configured correctly to connect to the internal bridge: "br-int" on
>>>> Open
>>>> VSwitch.
>>>>
>>>> do you have any idea what should I try to unrescue the machine to and
>>>> get it into active state?
>>>>
>>>> Regards,
>>>> Balazs Varhegyi
>>>>
>>>>
>>> Hi!
>>> What does the log and console show when the VM is in normal state?
>>> The rescue mode is not a magical solution to all problem. It just 
>>> runs a
>>> clean OS and attaches your broken one as a secondary disk. YOU have
>>> to do
>>> the magic to rescue it.
>>> Tomas
>>>
>>>
>>>
>>> _______________________________________________
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     : openstack at lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160316/5d3280d3/attachment.html>


More information about the Openstack mailing list