[openstack-dev] [nova] top gate bug is libvirt snapshot

Sean Dague sean at dague.net
Thu Jul 10 12:32:37 UTC 2014


On 07/10/2014 05:03 AM, Daniel P. Berrange wrote:
> On Wed, Jul 09, 2014 at 06:23:27PM -0400, Sean Dague wrote:
>> The libvirt logs needed are huge, so we can't run them all the time. And
>> realistically, I don't think they provided us the info we needed. There
>> has been at least one fail on Dan's log hack patch for this scenario
>> today, so maybe it will be in there.
> 
> I did finally get lucky and hit the failure, and the libvirtd.log has
> provided the info to narrow down the problem in QEMU I believe. I'm
> going to be talking with QEMU developers about it based on this info
> now.
> 
> FYI, the logs are approximately 3 MB compressed for a full tempest
> run. If turned on this would be either the 3rd or 4th largest log
> file we'd be collecting, adding 8-10% to the total size of all.

It's larger than anything other that the ceilometer logs, which are
their own issue. Remember that we are doing 20 - 30k runs a week. So 3MB
+ 20k = 60 GB / week. We're currently trying to keep 6 months of logs.
So * 26 = 1.5 TB of libvirt logs. We're currently limited by having a
max of 14 x 1 TB volumes on our log server in Rax.

We're hoping to fix that with using swift for log storage, if we get
that in place, we could probably do that on every run.

Is it possible to make libvirt log to 2 log files? One that is the
normal light load, and an enhanced error log? Then we could maybe make a
decision on cleanup time about if we need the error log saved or not.
Like if things failed we'd keep it. This all starts to get more
complicated, but might be worth exploring.

> 
> Currently I had to do a crude hack to enable it
> 
> diff --git a/etc/nova/rootwrap.d/compute.filters b/etc/nova/rootwrap.d/compute.filters
> index b79851b..7e4469a 100644
> --- a/etc/nova/rootwrap.d/compute.filters
> +++ b/etc/nova/rootwrap.d/compute.filters
> @@ -226,3 +226,6 @@ cp: CommandFilter, cp, root
>  # nova/virt/xenapi/vm_utils.py:
>  sync: CommandFilter, sync, root
>  
> +apt-get: CommandFilter, apt-get, root
> +service: CommandFilter, service, root
> +augtool: CommandFilter, augtool, root
> diff --git a/nova/virt/libvirt/driver.py b/nova/virt/libvirt/driver.py
> index 99edf12..93e60af 100644
> --- a/nova/virt/libvirt/driver.py
> +++ b/nova/virt/libvirt/driver.py
> @@ -28,6 +28,7 @@ Supports KVM, LXC, QEMU, UML, and XEN.
> @@ -611,6 +619,16 @@ class LibvirtDriver(driver.ComputeDriver):
>                          {'type': CONF.libvirt.virt_type, 'arch': arch})
>  
>      def init_host(self, host):
> +        utils.execute("apt-get", "-y", "install", "augeas-tools", run_as_root=True)
> +        utils.execute("augtool",
> +                      process_input="""set /files/etc/libvirt/libvirtd.conf/log_filters "1:libvirt.c 1:qemu 1:conf 1:security 3:object 3:event 3:json 3:file 1:util"
> +set /files/etc/libvirt/libvirtd.conf/log_outputs "1:file:/var/log/libvirt/libvirtd.log"
> +save
> +""", run_as_root=True)
> +        utils.execute("service", "libvirt-bin", "restart",
> +                      run_as_root=True)
> +        time.sleep(10)
> +
> 
> 
> 
> If we genuinely can't enable it all the time, then I think we really need
> to figure out a way to let us turn it on selectively per review, in a bit
> of an easier manner. devstack lets you set DEBUG_LIBVIRT environment
> variable to turn this on, but there's no way for people to get that env
> var set in the gate runs - AFAICT infra team would have to toggle that
> globally each time it was needed which isn't really practical.
> 
> Regards,
> Daniel
> 


-- 
Sean Dague
http://dague.net

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 482 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140710/2d0d2397/attachment.pgp>


More information about the OpenStack-dev mailing list