<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=utf-8">
</head>
<body bgcolor="#FFFFFF" text="#000000">
Hi all,<br>
<br>
[Issue observed]<br>
If we issue 'nova reboot <server>', we get to have the console
output of the latest bootup of the server only. The console output
of the previous boot for the same server vanishes due to
truncation[1]. If we do reboot from within the VM instance [ #sudo
reboot ], or reboot the instance with 'virsh reboot
<instance>' the behavior is not the same, where the
console.log keeps increasing, with the new output being appended.<br>
This loss of history makes some debugging scenario difficult due to
lack of information being available.<br>
<br>
Please point me to any solution/blueprint for this issue, if already
planned. Otherwise, please comment on my analysis and proposals as
solution, below -<br>
<br>
[Analysis]<br>
Nova's libvirt driver on compute node tries to do a graceful restart
of the server instance, by attempting a soft_reboot first. If
soft_reboot fails, it attempts a hard_reboot. As part of
soft_reboot, it brings down the instance by calling shutdown(), and
then calls createWithFlags() to bring this up. Because of this,
qemu-kvm process for the instance gets terminated and new process is
launched. In QEMU, the chardev file is opened with O_TRUNC, and thus
we lose the previous content of the console.log file.<br>
On the other-hand, during 'virsh reboot <instance>', the same
qemu-kvm process continues, and libvirt actually does a
qemuDomainSetFakeReboot(). Thus the same file continues capturing
the new console output as a continuation into the same file.<br>
<br>
[Proposals for solution]<br>
1. NOVA, driven by certain configuration, will backup the console
file, before creating the domain, during reboot scenario. viz. doing
a backup of console.log as console.log.0. How many such backups of
log-file to keep, what can be the maximum size of the file,
'logrotate` to be used or not - all these can come to NOVA as
configuration parameter.<br>
Pros - As NOVA libvirt driver is not using libvirt's reboot()
functionality knowingly, this problem can be better addressed from
the same layer.<br>
Cons - NOVA's libvirt layer building awareness of the console
files is not clean from modularity.<br>
<br>
2. virDomainCreateWithFlags() will have a new flag value to indicate
logs to be appended instead of truncated, if FILE option is used.
This config will be passed to QEMU, while spawning the process.<br>
- Changes will be not in OpenStack Code, but in libvirt and
QEMU.<br>
Cons - We may have to do the similar implementation for all the
drivers of libvirt.<br>
Pros - This feature's use-case is there in case of 'virsh
shutdown <instance>', followed by a 'virsh start
<instance>' too.<br>
<br>
Regards,<br>
<b>Suro<br>
</b><br>
Surojit Pathak<br>
<b><br>
<br>
</b>Refs -<br>
[1] Snippet <br>
# tail -f
/opt/stack/data/nova/instances/cea9a3d9-f833-4ded-90b8-c85b7da3f758/console.log<br>
...<br>
[ OK ] Started udev Coldplug all Devices.<br>
[ OK ] Started Create static device nodes in /dev.<br>
Starting udev Kernel Device Manager...<br>
[ 36.938075] EXT4-fs (vda1): re-mounted. Opts: (null)<br>
[ OK ] Started Remount Root and Kernel File Systems.<br>
Starting Load/Save Random Seed...<br>
[ OK ] Reached target Local File Systems (Pre).<br>
Starting Configure read-only root support...<br>
[ OK ] Started Load/Save Random Seed.<br>
<b>tail:
/opt/stack/data/nova/instances/cea9a3d9-f833-4ded-90b8-c85b7da3f758/console.log:
file truncated</b><br>
[ 0.000000] Initializing cgroup subsys cpuset<br>
[ 0.000000] Initializing cgroup subsys cpu<br>
[ 0.000000] Initializing cgroup subsys cpuacct<br>
[ 0.000000] Linux version 3.11.10-301.fc20.x86_64 (<a
class="moz-txt-link-abbreviated"
href="mailto:mockbuild@bkernel01.phx2.fedoraproject.org">mockbuild@bkernel01.phx2.fedoraproject.org</a>)
(gcc version 4.8.2 20131017 (Red Hat 4.8.2-1) (GCC) ) #1 SMP Thu Dec
5 14:01:17 UTC 2013<br>
[ 0.000000] Command line: ro
root=UUID=314b4a27-3885-49e8-9415-af098db4fd2a no_timer_check
console=tty1 console=ttyS0,115200n8
initrd=/boot/initramfs-3.11.10-301.fc20.x86_64.img
BOOT_IMAGE=/boot/vmlinuz-3.11.10-301.fc20.x86_64<br>
....<br>
</body>
</html>