Open Stack

On Thu, Jul 10, 2014 at 08:32:37AM -0400, Sean Dague wrote:
> On 07/10/2014 05:03 AM, Daniel P. Berrange wrote:
> > On Wed, Jul 09, 2014 at 06:23:27PM -0400, Sean Dague wrote:
> >> The libvirt logs needed are huge, so we can't run them all the time. And
> >> realistically, I don't think they provided us the info we needed. There
> >> has been at least one fail on Dan's log hack patch for this scenario
> >> today, so maybe it will be in there.
> > 
> > I did finally get lucky and hit the failure, and the libvirtd.log has
> > provided the info to narrow down the problem in QEMU I believe. I'm
> > going to be talking with QEMU developers about it based on this info
> > now.
> > 
> > FYI, the logs are approximately 3 MB compressed for a full tempest
> > run. If turned on this would be either the 3rd or 4th largest log
> > file we'd be collecting, adding 8-10% to the total size of all.
> 
> It's larger than anything other that the ceilometer logs, which are
> their own issue. Remember that we are doing 20 - 30k runs a week. So 3MB
> + 20k = 60 GB / week. We're currently trying to keep 6 months of logs.
> So * 26 = 1.5 TB of libvirt logs. We're currently limited by having a
> max of 14 x 1 TB volumes on our log server in Rax.

Could we simply expire the libvirtd.log file after 2 weeks while
leaving everything else at 6 months ?

> We're hoping to fix that with using swift for log storage, if we get
> that in place, we could probably do that on every run.
> 
> Is it possible to make libvirt log to 2 log files? One that is the
> normal light load, and an enhanced error log? Then we could maybe make a
> decision on cleanup time about if we need the error log saved or not.
> Like if things failed we'd keep it. This all starts to get more
> complicated, but might be worth exploring.

Yes, you can easily do multiple log files at different levels

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|