Open Stack

On Wed, Jul 09, 2014 at 06:23:27PM -0400, Sean Dague wrote:
> The libvirt logs needed are huge, so we can't run them all the time. And
> realistically, I don't think they provided us the info we needed. There
> has been at least one fail on Dan's log hack patch for this scenario
> today, so maybe it will be in there.

I did finally get lucky and hit the failure, and the libvirtd.log has
provided the info to narrow down the problem in QEMU I believe. I'm
going to be talking with QEMU developers about it based on this info
now.

FYI, the logs are approximately 3 MB compressed for a full tempest
run. If turned on this would be either the 3rd or 4th largest log
file we'd be collecting, adding 8-10% to the total size of all.

Currently I had to do a crude hack to enable it

diff --git a/etc/nova/rootwrap.d/compute.filters b/etc/nova/rootwrap.d/compute.filters
index b79851b..7e4469a 100644
--- a/etc/nova/rootwrap.d/compute.filters
+++ b/etc/nova/rootwrap.d/compute.filters
@@ -226,3 +226,6 @@ cp: CommandFilter, cp, root
 # nova/virt/xenapi/vm_utils.py:
 sync: CommandFilter, sync, root
 
+apt-get: CommandFilter, apt-get, root
+service: CommandFilter, service, root
+augtool: CommandFilter, augtool, root
diff --git a/nova/virt/libvirt/driver.py b/nova/virt/libvirt/driver.py
index 99edf12..93e60af 100644
--- a/nova/virt/libvirt/driver.py
+++ b/nova/virt/libvirt/driver.py
@@ -28,6 +28,7 @@ Supports KVM, LXC, QEMU, UML, and XEN.
@@ -611,6 +619,16 @@ class LibvirtDriver(driver.ComputeDriver):
                         {'type': CONF.libvirt.virt_type, 'arch': arch})
 
     def init_host(self, host):
+        utils.execute("apt-get", "-y", "install", "augeas-tools", run_as_root=True)
+        utils.execute("augtool",
+                      process_input="""set /files/etc/libvirt/libvirtd.conf/log_filters "1:libvirt.c 1:qemu 1:conf 1:security 3:object 3:event 3:json 3:file 1:util"
+set /files/etc/libvirt/libvirtd.conf/log_outputs "1:file:/var/log/libvirt/libvirtd.log"
+save
+""", run_as_root=True)
+        utils.execute("service", "libvirt-bin", "restart",
+                      run_as_root=True)
+        time.sleep(10)
+



If we genuinely can't enable it all the time, then I think we really need
to figure out a way to let us turn it on selectively per review, in a bit
of an easier manner. devstack lets you set DEBUG_LIBVIRT environment
variable to turn this on, but there's no way for people to get that env
var set in the gate runs - AFAICT infra team would have to toggle that
globally each time it was needed which isn't really practical.

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|