<html><head><meta http-equiv="Content-Type" content="text/html charset=windows-1252"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div>On Mar 24, 2014, at 2:09 PM, Joe Gordon <<a href="mailto:joe.gordon0@gmail.com">joe.gordon0@gmail.com</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div dir="ltr" style="font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class="gmail_extra"><br class="Apple-interchange-newline"><br><br><div class="gmail_quote">On Mon, Mar 24, 2014 at 3:49 AM, Sean Dague<span class="Apple-converted-space"> </span><span dir="ltr"><<a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a>></span><span class="Apple-converted-space"> </span>wrote:<br><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">Here is some preliminary views (it currently ignores the ceilometer<br>logs, I haven't had a chance to dive in there yet).<br><br>It actually looks like a huge part of the issue is olso.messaging, the<br>bulk of screen-n-cond is oslo.messaging debug errors. It seems that in<br>debug mode oslo.messaging is basically a 100% trace mode, which include<br>logging every time a UUID is created and every payload.<br><br>I'm not convinced why that's a useful. We don't log every sql statement<br>we run (with full payload).<br><br></blockquote><div><br></div><div>Agreed. I turned off oslo.messaging logs [1] and the file sizes in a check-tempest-dsvm-full dropped drastically to [2]. nova-conductor logs dropped way down from 7.3MB to 214K.</div><div><br></div><div>[1] <a href="https://review.openstack.org/#/c/82255/" target="_blank">https://review.openstack.org/#/c/82255/</a></div><div>[2] <a href="http://logs.openstack.org/55/82255/1/check/check-tempest-dsvm-full/88d1e36/logs/?C=S;O=D" target="_blank">http://logs.openstack.org/55/82255/1/check/check-tempest-dsvm-full/88d1e36/logs/?C=S;O=D</a></div><div><br></div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;">The recent integration of oslo.messaging would also explain the new<br>growth of logs.<br><br>Other issues include other oslo utils that have really verbose debug<br>modes. Like lockutils emitting 4 DEBUG messages for every lock acquired.<br><br>Part of the challenge is turning off DEBUG is currently embedded in code<br>in oslo log, which makes it kind of awkward to set sane log levels for<br>included libraries because it requires an oslo round trip with code to<br>all the projects to do it.<br></blockquote><div><br></div><div>++</div><div><br></div><div>One possible solution is to start using the log_config_append and load the config from a logging.conf file. But we don't even copy over the sample file in devstack. So for icehouse we may want to do a cherry-pick from oslo-incubator to disable oslo.messaging</div></div></div></div></blockquote><div><br></div>Can’t we just specify a reasonable default_log_levels in *.conf in devstack? That would cut down the log chatter for integration tests, and wouldn’t be a breaking change.</div><div><br></div><div>Vish</div><div><br></div><div><blockquote type="cite"><div dir="ltr" style="font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><div class="gmail_extra"><div class="gmail_quote"><div> </div><blockquote class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left-width: 1px; border-left-color: rgb(204, 204, 204); border-left-style: solid; padding-left: 1ex;"><br> <span class="Apple-converted-space"> </span>-Sean<br><div><br>On 03/21/2014 07:23 PM, Clark Boylan wrote:<br>> Hello everyone,<br>><br>> Back at the Portland summit the Infra team committed to archiving six months<br>> of test logs for Openstack. Since then we have managed to do just that.<br>> However, more recently we have seen the growth rate on those logs continue<br>> to grow beyond what is a currently sustainable level.<br>><br>> For reasons, we currently store logs on a filesystem backed by cinder<br>> volumes. Rackspace limits the size and number of volumes attached to a<br>> single host meaning the upper bound on the log archive filesystem is ~12TB<br>> and we are almost there. You can see real numbers and pretty graphs on our<br>> cacti server [0].<br>><br>> Long term we are trying to move to putting all of the logs in swift, but it<br>> turns out there are some use case issues we need to sort out around that<br>> before we can do so (but this is being worked on so should happen). Until<br>> that day arrives we need to work on logging more smartly, and if we can't do<br>> that we will have to reduce the log retention period.<br>><br>> So what can you do? Well it appears that our log files may need a diet. I<br>> have listed the worst offenders below (after a small sampling, there may be<br>> more) and it would be great if we could go through those with a comb and<br>> figure out if we are logging actually useful data. The great thing about<br>> doing this is it will make lives better for deployers of Openstack too.<br>><br>> Some initial checking indicates a lot of this noise may be related to<br>> ceilometer. It looks like it is logging AMQP stuff frequently and inflating<br>> the logs of individual services as it polls them.<br>><br>> Offending files from tempest tests:<br>> screen-n-cond.txt.gz 7.3M<br>> screen-ceilometer-collector.txt.gz 6.0M<br>> screen-n-api.txt.gz 3.7M<br>> screen-n-cpu.txt.gz 3.6M<br>> tempest.txt.gz 2.7M<br>> screen-ceilometer-anotification.txt.gz 1.9M<br>> subunit_log.txt.gz 1.5M<br>> screen-g-api.txt.gz 1.4M<br>> screen-ceilometer-acentral.txt.gz 1.4M<br>> screen-n-net.txt.gz 1.4M<br>> from:<span class="Apple-converted-space"> </span><a href="http://logs.openstack.org/52/81252/2/gate/gate-tempest-dsvm-full/488bc4e/logs/?C=S;O=D" target="_blank">http://logs.openstack.org/52/81252/2/gate/gate-tempest-dsvm-full/488bc4e/logs/?C=S;O=D</a><br>><br>> Unittest offenders:<br>> Nova subunit_log.txt.gz 14M<br>> Neutron subunit_log.txt.gz 7.8M<br>> Keystone subunit_log.txt.gz 4.8M<br>><br>> Note all of the above files are compressed with gzip -9 and the filesizes<br>> above reflect compressed file sizes.<br>><br>> Debug logs are important to you guys when dealing with Jenkins results. We<br>> want your feedback on how we can make this better for everyone.<br>><br>> [0]<span class="Apple-converted-space"> </span><a href="http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all" target="_blank">http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=717&rra_id=all</a><br>><br>> Thank you,<br>> Clark Boylan<br>><br>> _______________________________________________<br>> OpenStack-dev mailing list<br>><span class="Apple-converted-space"> </span><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br>><span class="Apple-converted-space"> </span><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>><br><br><br></div><span><font color="#888888">--<br>Sean Dague<br>Samsung Research America<br><a href="mailto:sean@dague.net" target="_blank">sean@dague.net</a><span class="Apple-converted-space"> </span>/<span class="Apple-converted-space"> </span><a href="mailto:sean.dague@samsung.com" target="_blank">sean.dague@samsung.com</a><br><a href="http://dague.net/" target="_blank">http://dague.net</a><br><br></font></span><br>_______________________________________________<br>OpenStack-dev mailing list<br><a href="mailto:OpenStack-dev@lists.openstack.org" target="_blank">OpenStack-dev@lists.openstack.org</a><br><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br><br></blockquote></div><br></div></div><span style="font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">_______________________________________________</span><br style="font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><span style="font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;">OpenStack-dev mailing list</span><br style="font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><a href="mailto:OpenStack-dev@lists.openstack.org" style="font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">OpenStack-dev@lists.openstack.org</a><br style="font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" style="font-family: Menlo-Regular; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></blockquote></div><br></body></html>