Open Stack

On Thu, Nov 13, 2014 at 01:55:06PM -0800, Surojit Pathak wrote:
> Hi all,
> 
> [Issue observed]
> If we issue 'nova reboot <server>', we get to have the console output of the
> latest bootup of the server only. The console output of the previous boot
> for the same server vanishes due to truncation[1]. If we do reboot from
> within the VM instance [ #sudo reboot ], or reboot the instance with 'virsh
> reboot <instance>' the behavior is not the same, where the console.log keeps
> increasing, with the new output being appended.
> This loss of history makes some debugging scenario difficult due to lack of
> information being available.
> 
> Please point me to any solution/blueprint for this issue, if already
> planned. Otherwise, please comment on my analysis and proposals as solution,
> below -
> 
> [Analysis]
> Nova's libvirt driver on compute node tries to do a graceful restart of the
> server instance, by attempting a soft_reboot first. If soft_reboot fails, it
> attempts a hard_reboot. As part of soft_reboot, it brings down the instance
> by calling shutdown(), and then calls createWithFlags() to bring this up.
> Because of this, qemu-kvm process for the instance gets terminated and new
> process is launched. In QEMU, the chardev file is opened with O_TRUNC, and
> thus we lose the previous content of the console.log file.
> On the other-hand, during 'virsh reboot <instance>', the same qemu-kvm
> process continues, and libvirt actually does a qemuDomainSetFakeReboot().
> Thus the same file continues capturing the new console output as a
> continuation into the same file.

Nova and libvirt have support for issuing a graceful reboot via the QEMU
guest agent. So if you make sure that is installed, and tell Nova to use
it, then Nova won't have to stop & recreate the QEMU process and thus
won't have the problem of overwriting the logs.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|