Open Stack

On Fri, Aug 28, 2020, at 3:04 AM, Slawek Kaplonski wrote:
> Hi,
> 
> On Tue, Aug 18, 2020 at 02:10:40PM -0700, melanie witt wrote:
> > On 8/18/20 08:00, Slawek Kaplonski wrote:
> > > Hi,
> > > 
> > > I proposed patch [1] which seems that decreased size of the neutron-server log
> > > a bit - see [2] but it's still about 40M :/
> > > 
> > > [1] https://review.opendev.org/#/c/730879/
> > > [2] https://48dcf568cd222acfbfb6-11d92d8452a346ca231ad13d26a55a7d.ssl.cf2.rackcdn.com/746714/1/check/tempest-full-py3/5c1399c/controller/logs/
> > 
> > Thanks for jumping in to help, Slawek! Indeed your proposed patch improves things from 60M-70M => 40M (good!).
> > 
> > With your patch applied, the most frequent potential log message I see now is like this:
> > 
> > Aug 18 14:40:21.294549 ubuntu-bionic-rax-iad-0019321276 neutron-server[5829]: DEBUG neutron_lib.callbacks.manager [None req-eadfbe92-eaee-4e3e-a5c0-f18aa8ba9772 None None] Notify callbacks ['neutron.services.segments.db._update_segment_host_mapping_for_agent-8764691834039', 'neutron.plugins.ml2.plugin.Ml2Plugin._retry_binding_revived_agents-4033733'] for agent, after_update {{(pid=6206) _notify_loop /opt/stack/neutron-lib/neutron_lib/callbacks/manager.py:193}}
> > 
> > with the line count difference being with and without:
> > 
> > $ wc -l "screen-q-svc.txt"
> > 102493 screen-q-svc.txt
> > 
> > $ grep -v "neutron_lib.callbacks.manager" "screen-q-svc.txt" |wc -l
> > 83261
> > 
> > so I suppose we could predict a decrease in file size of about 40M => 32M if we were able to remove the neutron_lib.callbacks.manager output.
> 
> I was looking at this again today but I'm really not sure if we should get rid
> of those messages from the log.
> For now I think that indexing of screen-q-svc.txt file is disabled so this size
> of the log shouldn't be big problem (I hope) and I would like to not remove any
> other debug messages from it if that will not be really necessary.

Maybe as an option we split this into two log files. One that is INFO and above and the other that includes everything with DEBUG? Then we can index the INFO and above contents only.

One thing to keep in mind here is that this system tends to act like a canary for when our logs would create problems for people in production. The q-svc logs here are significantly more chatty than the other services. Not necessarily a problem, but don't be surprised if people notice after they upgrade and start complaining.

> 
> > 
> > But I'm not sure whether that's a critical debugging element or not.
> > 
> > -melanie
> > 
> 
> -- 
> Slawek Kaplonski
> Principal software engineer
> Red Hat
> 
> 
>