<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Mon, Mar 24, 2014 at 3:49 AM, Sean Dague <span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Here is some preliminary views (it currently ignores the ceilometer<br>
logs, I haven't had a chance to dive in there yet).<br>
<br>
It actually looks like a huge part of the issue is olso.messaging, the<br>
bulk of screen-n-cond is oslo.messaging debug errors. It seems that in<br>
debug mode oslo.messaging is basically a 100% trace mode, which include<br>
logging every time a UUID is created and every payload.<br>
<br>
I'm not convinced why that's a useful. We don't log every sql statement<br>
we run (with full payload).<br>
<br></blockquote><div><br></div><div>Agreed. I turned off oslo.messaging logs [1] and the file sizes in a check-tempest-dsvm-full dropped drastically to [2]. nova-conductor logs dropped way down from 7.3MB to 214K.</div>
<div><br></div><div>[1] <a href="https://review.openstack.org/#/c/82255/" target="_blank">https://review.openstack.org/#/c/82255/</a></div><div>[2] <a href="http://logs.openstack.org/55/82255/1/check/check-tempest-dsvm-full/88d1e36/logs/?C=S;O=D" target="_blank">http://logs.openstack.org/55/82255/1/check/check-tempest-dsvm-full/88d1e36/logs/?C=S;O=D</a></div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
The recent integration of oslo.messaging would also explain the new<br>
growth of logs.<br>
<br>
Other issues include other oslo utils that have really verbose debug<br>
modes. Like lockutils emitting 4 DEBUG messages for every lock acquired.<br>
<br>
Part of the challenge is turning off DEBUG is currently embedded in code<br>
in oslo log, which makes it kind of awkward to set sane log levels for<br>
included libraries because it requires an oslo round trip with code to<br>
all the projects to do it.<br></blockquote><div><br></div><div>++</div><div><br></div><div>One possible solution is to start using the log_config_append and load the config from a logging.conf file. But we don't even copy over the sample file in devstack. So for icehouse we may want to do a cherry-pick from oslo-incubator to disable oslo.messaging</div>
<div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br>
-Sean<br>
<div><div><br>
On 03/21/2014 07:23 PM, Clark Boylan wrote:<br>
> Hello everyone,<br>
><br>
> Back at the Portland summit the Infra team committed to archiving six months<br>
> of test logs for Openstack. Since then we have managed to do just that.<br>
> However, more recently we have seen the growth rate on those logs continue<br>
> to grow beyond what is a currently sustainable level.<br>
><br>
> For reasons, we currently store logs on a filesystem backed by cinder<br>
> volumes. Rackspace limits the size and number of volumes attached to a<br>
> single host meaning the upper bound on the log archive filesystem is ~12TB<br>
> and we are almost there. You can see real numbers and pretty graphs on our<br>
> cacti server [0].<br>
><br>
> Long term we are trying to move to putting all of the logs in swift, but it<br>
> turns out there are some use case issues we need to sort out around that<br>
> before we can do so (but this is being worked on so should happen). Until<br>
> that day arrives we need to work on logging more smartly, and if we can't do<br>
> that we will have to reduce the log retention period.<br>
><br>
> So what can you do? Well it appears that our log files may need a diet. I<br>
> have listed the worst offenders below (after a small sampling, there may be<br>
> more) and it would be great if we could go through those with a comb and<br>
> figure out if we are logging actually useful data. The great thing about<br>
> doing this is it will make lives better for deployers of Openstack too.<br>
><br>
> Some initial checking indicates a lot of this noise may be related to<br>
> ceilometer. It looks like it is logging AMQP stuff frequently and inflating<br>
> the logs of individual services as it polls them.<br>
><br>
> Offending files from tempest tests:<br>
> screen-n-cond.txt.gz 7.3M<br>
> screen-ceilometer-collector.txt.gz 6.0M<br>
> screen-n-api.txt.gz 3.7M<br>
> screen-n-cpu.txt.gz 3.6M<br>
> tempest.txt.gz 2.7M<br>
> screen-ceilometer-anotification.txt.gz 1.9M<br>
> subunit_log.txt.gz 1.5M<br>
> screen-g-api.txt.gz 1.4M<br>
> screen-ceilometer-acentral.txt.gz 1.4M<br>
> screen-n-net.txt.gz 1.4M<br>
> from: <a href="http://logs.openstack.org/52/81252/2/gate/gate-tempest-dsvm-full/488bc4e/logs/?C=S;O=D" target="_blank">http://logs.openstack.org/52/81252/2/gate/gate-tempest-dsvm-full/488bc4e/logs/?C=S;O=D</a><br>
><br>
> Unittest offenders:<br>
> Nova subunit_log.txt.gz 14M<br>
> Neutron subunit_log.txt.gz 7.8M<br>
> Keystone subunit_log.txt.gz 4.8M<br>
><br>
> Note all of the above files are compressed with gzip -9 and the filesizes<br>
> above reflect compressed file sizes.<br>
><br>
> Debug logs are important to you guys when dealing with Jenkins results. We<br>
> want your feedback on how we can make this better for everyone.<br>
><br>
> [0] <a href="http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all" target="_blank">http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all</a><br>
><br>
> Thank you,<br>
> Clark Boylan<br>
><br>
> _______________________________________________<br>
> OpenStack-dev mailing list<br>
> <a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
><br>
<br>
<br>
</div></div><span><font color="#888888">--<br>
Sean Dague<br>
Samsung Research America<br>
<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a> / <a href="mailto:sean.dague@samsung.com" target="_blank">sean.dague@samsung.com</a><br>
<a href="http://dague.net" target="_blank">http://dague.net</a><br>
<br>
</font></span><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>