<div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, Mar 26, 2014 at 11:15 AM, Vishvananda Ishaya <span dir="ltr"><<a href="mailto:vishvananda@gmail.com" target="_blank">vishvananda@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><br><div><div class=""><div>On Mar 24, 2014, at 2:09 PM, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com" target="_blank">joe.gordon0@gmail.com</a>> wrote:</div>
<br><blockquote type="cite"><div dir="ltr" style="font-family:Menlo-Regular;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div class="gmail_extra"><br><br><br><div class="gmail_quote">On Mon, Mar 24, 2014 at 3:49 AM, Sean Dague<span> </span><span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span><span> </span>wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">Here is some preliminary views (it currently ignores the ceilometer<br>
logs, I haven't had a chance to dive in there yet).<br><br>It actually looks like a huge part of the issue is olso.messaging, the<br>bulk of screen-n-cond is oslo.messaging debug errors. It seems that in<br>debug mode oslo.messaging is basically a 100% trace mode, which include<br>
logging every time a UUID is created and every payload.<br><br>I'm not convinced why that's a useful. We don't log every sql statement<br>we run (with full payload).<br><br></blockquote><div><br></div><div>Agreed. I turned off oslo.messaging logs [1] and the file sizes in a check-tempest-dsvm-full dropped drastically to [2]. nova-conductor logs dropped way down from 7.3MB to 214K.</div>
<div><br></div><div>[1] <a href="https://review.openstack.org/#/c/82255/" target="_blank">https://review.openstack.org/#/c/82255/</a></div><div>[2] <a href="http://logs.openstack.org/55/82255/1/check/check-tempest-dsvm-full/88d1e36/logs/?C=S;O=D" target="_blank">http://logs.openstack.org/55/82255/1/check/check-tempest-dsvm-full/88d1e36/logs/?C=S;O=D</a></div>
<div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">The recent integration of oslo.messaging would also explain the new<br>
growth of logs.<br><br>Other issues include other oslo utils that have really verbose debug<br>modes. Like lockutils emitting 4 DEBUG messages for every lock acquired.<br><br>Part of the challenge is turning off DEBUG is currently embedded in code<br>
in oslo log, which makes it kind of awkward to set sane log levels for<br>included libraries because it requires an oslo round trip with code to<br>all the projects to do it.<br></blockquote><div><br></div><div>++</div><div>
<br></div><div>One possible solution is to start using the log_config_append and load the config from a logging.conf file. But we don't even copy over the sample file in devstack. So for icehouse we may want to do a cherry-pick from oslo-incubator to disable oslo.messaging</div>
</div></div></div></blockquote><div><br></div></div>Can’t we just specify a reasonable default_log_levels in *.conf in devstack? That would cut down the log chatter for integration tests, and wouldn’t be a breaking change.</div>
</div></blockquote><div><br></div><div>If we are having problems in gate with verbose and useless logs, others will to ... so I don't think we should sidestep the problem via devstack otherwise every deployer will have to do the same. This fits in with the *sane defaults* mantra.</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div style="word-wrap:break-word"><div><br></div><div>Vish</div><div><div class="h5"><div><br></div><div><blockquote type="cite">
<div dir="ltr" style="font-family:Menlo-Regular;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<br> <span> </span>-Sean<br><div><br>On 03/21/2014 07:23 PM, Clark Boylan wrote:<br>> Hello everyone,<br>><br>> Back at the Portland summit the Infra team committed to archiving six months<br>> of test logs for Openstack. Since then we have managed to do just that.<br>
> However, more recently we have seen the growth rate on those logs continue<br>> to grow beyond what is a currently sustainable level.<br>><br>> For reasons, we currently store logs on a filesystem backed by cinder<br>
> volumes. Rackspace limits the size and number of volumes attached to a<br>> single host meaning the upper bound on the log archive filesystem is ~12TB<br>> and we are almost there. You can see real numbers and pretty graphs on our<br>
> cacti server [0].<br>><br>> Long term we are trying to move to putting all of the logs in swift, but it<br>> turns out there are some use case issues we need to sort out around that<br>> before we can do so (but this is being worked on so should happen). Until<br>
> that day arrives we need to work on logging more smartly, and if we can't do<br>> that we will have to reduce the log retention period.<br>><br>> So what can you do? Well it appears that our log files may need a diet. I<br>
> have listed the worst offenders below (after a small sampling, there may be<br>> more) and it would be great if we could go through those with a comb and<br>> figure out if we are logging actually useful data. The great thing about<br>
> doing this is it will make lives better for deployers of Openstack too.<br>><br>> Some initial checking indicates a lot of this noise may be related to<br>> ceilometer. It looks like it is logging AMQP stuff frequently and inflating<br>
> the logs of individual services as it polls them.<br>><br>> Offending files from tempest tests:<br>> screen-n-cond.txt.gz 7.3M<br>> screen-ceilometer-collector.txt.gz 6.0M<br>> screen-n-api.txt.gz 3.7M<br>
> screen-n-cpu.txt.gz 3.6M<br>> tempest.txt.gz 2.7M<br>> screen-ceilometer-anotification.txt.gz 1.9M<br>> subunit_log.txt.gz 1.5M<br>> screen-g-api.txt.gz 1.4M<br>> screen-ceilometer-acentral.txt.gz 1.4M<br>
> screen-n-net.txt.gz 1.4M<br>> from:<span> </span><a href="http://logs.openstack.org/52/81252/2/gate/gate-tempest-dsvm-full/488bc4e/logs/?C=S;O=D" target="_blank">http://logs.openstack.org/52/81252/2/gate/gate-tempest-dsvm-full/488bc4e/logs/?C=S;O=D</a><br>
><br>> Unittest offenders:<br>> Nova subunit_log.txt.gz 14M<br>> Neutron subunit_log.txt.gz 7.8M<br>> Keystone subunit_log.txt.gz 4.8M<br>><br>> Note all of the above files are compressed with gzip -9 and the filesizes<br>
> above reflect compressed file sizes.<br>><br>> Debug logs are important to you guys when dealing with Jenkins results. We<br>> want your feedback on how we can make this better for everyone.<br>><br>> [0]<span> </span><a href="http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all" target="_blank">http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all</a><br>
><br>> Thank you,<br>> Clark Boylan<br>><br>> _______________________________________________<br>> OpenStack-dev mailing list<br>><span> </span><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
><span> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>><br><br><br></div><span><font color="#888888">--<br>
Sean Dague<br>Samsung Research America<br><a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a><span> </span>/<span> </span><a href="mailto:sean.dague@samsung.com" target="_blank">sean.dague@samsung.com</a><br>
<a href="http://dague.net/" target="_blank">http://dague.net</a><br><br></font></span><br>_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br><br></blockquote></div><br></div></div><span style="font-family:Menlo-Regular;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">_______________________________________________</span><br style="font-family:Menlo-Regular;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<span style="font-family:Menlo-Regular;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px;float:none;display:inline!important">OpenStack-dev mailing list</span><br style="font-family:Menlo-Regular;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<a href="mailto:OpenStack-dev@lists.openstack.org" style="font-family:Menlo-Regular;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">OpenStack-dev@lists.openstack.org</a><br style="font-family:Menlo-Regular;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px">
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" style="font-family:Menlo-Regular;font-size:12px;font-style:normal;font-variant:normal;font-weight:normal;letter-spacing:normal;line-height:normal;text-align:start;text-indent:0px;text-transform:none;white-space:normal;word-spacing:0px" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></blockquote>
</div><br></div></div></div><br>_______________________________________________<br>
OpenStack-dev mailing list<br>
<a href="mailto:OpenStack-dev@lists.openstack.org">OpenStack-dev@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br></blockquote></div><br></div></div>