[ops][nova]Logging in nova and other openstack projects
Hi, We’ve been modifying our login habits for Nova on our Openstack setup to try to send only warning level and up logs to our log servers. To do so, I’ve created a logging.conf and configured logging according to the logging module documentation. While what I’ve done works, it seems to be a very convoluted process for something as simple as changing the logging level to warning. We worry that if we upgrade and the syntax for this configuration file changes, we may have to push more changes through ansible than we would like to. Is there an easier way to set the nova logs to warning level and up than making an additional config file for the python logging module? Best regards, Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.
On 05/06/2019 05:56 PM, Jean-Philippe Méthot wrote:
Hi,
We’ve been modifying our login habits for Nova on our Openstack setup to try to send only warning level and up logs to our log servers. To do so, I’ve created a logging.conf and configured logging according to the logging module documentation. While what I’ve done works, it seems to be a very convoluted process for something as simple as changing the logging level to warning. We worry that if we upgrade and the syntax for this configuration file changes, we may have to push more changes through ansible than we would like to.
It's unlikely that the syntax for the logging configuration file will change since it's upstream Python, not OpenStack or Nova that is the source of this syntax. That said, if all you want to do is change some or all package default logging levels, you can change the value of the CONF.default_log_levels option. The default_log_levels CONF option is actually derived from the oslo_log package that is used by all OpenStack service projects. It's default value is here: https://github.com/openstack/oslo.log/blob/29671ef2bfacb416d397abc57170bb090... So, if you don't want to mess with the standard Python logging conf, you can just change that CONF.default_log_levels option. Note that if you do specify a logging config file using a non-None CONF.log_config_append value, then all other logging configuration options (like default_log_levels) are ignored). Best, -jay
Hi, I’ve just tried setting everything to warn through the nova.conf option default_log_levels, as suggested. However, I’m still getting info level logs from the resource tracker like this : INFO nova.compute.resource_tracker Could the compute resource tracker logs be managed by another parameter than what’s in the default list for that configuration option? Best regards, Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.
Le 7 mai 2019 à 09:02, Jay Pipes <jaypipes@gmail.com> a écrit :
On 05/06/2019 05:56 PM, Jean-Philippe Méthot wrote:
Hi, We’ve been modifying our login habits for Nova on our Openstack setup to try to send only warning level and up logs to our log servers. To do so, I’ve created a logging.conf and configured logging according to the logging module documentation. While what I’ve done works, it seems to be a very convoluted process for something as simple as changing the logging level to warning. We worry that if we upgrade and the syntax for this configuration file changes, we may have to push more changes through ansible than we would like to.
It's unlikely that the syntax for the logging configuration file will change since it's upstream Python, not OpenStack or Nova that is the source of this syntax.
That said, if all you want to do is change some or all package default logging levels, you can change the value of the CONF.default_log_levels option.
The default_log_levels CONF option is actually derived from the oslo_log package that is used by all OpenStack service projects. It's default value is here:
https://github.com/openstack/oslo.log/blob/29671ef2bfacb416d397abc57170bb090...
So, if you don't want to mess with the standard Python logging conf, you can just change that CONF.default_log_levels option. Note that if you do specify a logging config file using a non-None CONF.log_config_append value, then all other logging configuration options (like default_log_levels) are ignored).
Best, -jay
As mentioned in my original response, if you have CONF.log_config_append set to anything, then the other conf options related to logging will be ignored. Best, -jay On Tue, May 7, 2019, 11:15 AM Jean-Philippe Méthot < jp.methot@planethoster.info> wrote:
Hi,
I’ve just tried setting everything to warn through the nova.conf option default_log_levels, as suggested. However, I’m still getting info level logs from the resource tracker like this :
INFO nova.compute.resource_tracker
Could the compute resource tracker logs be managed by another parameter than what’s in the default list for that configuration option?
Best regards,
Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.
Le 7 mai 2019 à 09:02, Jay Pipes <jaypipes@gmail.com> a écrit :
On 05/06/2019 05:56 PM, Jean-Philippe Méthot wrote:
Hi, We’ve been modifying our login habits for Nova on our Openstack setup to try to send only warning level and up logs to our log servers. To do so, I’ve created a logging.conf and configured logging according to the logging module documentation. While what I’ve done works, it seems to be a very convoluted process for something as simple as changing the logging level to warning. We worry that if we upgrade and the syntax for this configuration file changes, we may have to push more changes through ansible than we would like to.
It's unlikely that the syntax for the logging configuration file will change since it's upstream Python, not OpenStack or Nova that is the source of this syntax.
That said, if all you want to do is change some or all package default logging levels, you can change the value of the CONF.default_log_levels option.
The default_log_levels CONF option is actually derived from the oslo_log package that is used by all OpenStack service projects. It's default value is here:
https://github.com/openstack/oslo.log/blob/29671ef2bfacb416d397abc57170bb090...
So, if you don't want to mess with the standard Python logging conf, you can just change that CONF.default_log_levels option. Note that if you do specify a logging config file using a non-None CONF.log_config_append value, then all other logging configuration options (like default_log_levels) are ignored).
Best, -jay
Indeed, this is what was written in your original response as well as in the documentation. As a result, it was fairly difficult to miss and I did comment it out before restarting the service. Additionally, as per the configuration I had set up, had the log-config-append option be set, I wouldn’t have any INFO level log in my logs. Hence why I believe it is strange that I have info level logs, when I’ve set default_log_levels like this: default_log_levels = amqp=WARN,amqplib=WARN,boto=WARN,qpid=WARN,sqlalchemy=WARN,suds=WARN,oslo.messaging=WARN,iso8601=WARN,requests.packages.urllib3.connectionpool=WARN,urllib3.connectionpool=WARN,websocket=WARN,requests.packages.urllib3.util.retry=WARN,urllib3.util.retry=WARN,keystonemiddleware=WARN,routes.middleware=WARN,stevedore=WARN,taskflow=WARN,keystoneauth=WARN,oslo.cache=WARN Please understand that I am not doubting that your previous answer normally works. I have seen your presentations at past Openstack summit and know that you are a brilliant individual. However, I can only answer here that, from my observations, this is not working as intended. I’ll also add that this is on Pike, but we are slated to upgrade to Queens in the coming weeks. Best regards, Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.
Le 7 mai 2019 à 11:39, Jay Pipes <jaypipes@gmail.com> a écrit :
As mentioned in my original response, if you have CONF.log_config_append set to anything, then the other conf options related to logging will be ignored.
Best, -jay
On Tue, May 7, 2019, 11:15 AM Jean-Philippe Méthot <jp.methot@planethoster.info <mailto:jp.methot@planethoster.info>> wrote: Hi,
I’ve just tried setting everything to warn through the nova.conf option default_log_levels, as suggested. However, I’m still getting info level logs from the resource tracker like this :
INFO nova.compute.resource_tracker
Could the compute resource tracker logs be managed by another parameter than what’s in the default list for that configuration option?
Best regards,
Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.
Le 7 mai 2019 à 09:02, Jay Pipes <jaypipes@gmail.com <mailto:jaypipes@gmail.com>> a écrit :
On 05/06/2019 05:56 PM, Jean-Philippe Méthot wrote:
Hi, We’ve been modifying our login habits for Nova on our Openstack setup to try to send only warning level and up logs to our log servers. To do so, I’ve created a logging.conf and configured logging according to the logging module documentation. While what I’ve done works, it seems to be a very convoluted process for something as simple as changing the logging level to warning. We worry that if we upgrade and the syntax for this configuration file changes, we may have to push more changes through ansible than we would like to.
It's unlikely that the syntax for the logging configuration file will change since it's upstream Python, not OpenStack or Nova that is the source of this syntax.
That said, if all you want to do is change some or all package default logging levels, you can change the value of the CONF.default_log_levels option.
The default_log_levels CONF option is actually derived from the oslo_log package that is used by all OpenStack service projects. It's default value is here:
https://github.com/openstack/oslo.log/blob/29671ef2bfacb416d397abc57170bb090... <https://github.com/openstack/oslo.log/blob/29671ef2bfacb416d397abc57170bb090b116f68/oslo_log/_options.py#L19-L31>
So, if you don't want to mess with the standard Python logging conf, you can just change that CONF.default_log_levels option. Note that if you do specify a logging config file using a non-None CONF.log_config_append value, then all other logging configuration options (like default_log_levels) are ignored).
Best, -jay
On 07.05.2019 23:31, Jean-Philippe Méthot wrote:
Indeed, this is what was written in your original response as well as in the documentation. As a result, it was fairly difficult to miss and I did comment it out before restarting the service. Additionally, as per
There is also a deprecated (but still working) log_config [0]. So please double-check you don't have that configuration leftover. Another caveat might be that SIGHUP does not propagate to all of the child processes/threads/whatever to update its logging configs with the new default_log_levels and removed log_config(_append) ones... But you said you are restarting not reloading, so prolly can't be a problem here. [0] https://opendev.org/openstack/oslo.log/src/branch/master/oslo_log/_options.p...
the configuration I had set up, had the log-config-append option be set, I wouldn’t have any INFO level log in my logs. Hence why I believe it is strange that I have info level logs, when I’ve set default_log_levels like this:
default_log_levels = amqp=WARN,amqplib=WARN,boto=WARN,qpid=WARN,sqlalchemy=WARN,suds=WARN,oslo.messaging=WARN,iso8601=WARN,requests.packages.urllib3.connectionpool=WARN,urllib3.connectionpool=WARN,websocket=WARN,requests.packages.urllib3.util.retry=WARN,urllib3.util.retry=WARN,keystonemiddleware=WARN,routes.middleware=WARN,stevedore=WARN,taskflow=WARN,keystoneauth=WARN,oslo.cache=WARN
Please understand that I am not doubting that your previous answer normally works. I have seen your presentations at past Openstack summit and know that you are a brilliant individual. However, I can only answer here that, from my observations, this is not working as intended.
I’ll also add that this is on Pike, but we are slated to upgrade to Queens in the coming weeks.
Best regards,
Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.
Le 7 mai 2019 à 11:39, Jay Pipes <jaypipes@gmail.com <mailto:jaypipes@gmail.com>> a écrit :
As mentioned in my original response, if you have CONF.log_config_append set to anything, then the other conf options related to logging will be ignored.
Best, -jay
On Tue, May 7, 2019, 11:15 AM Jean-Philippe Méthot <jp.methot@planethoster.info <mailto:jp.methot@planethoster.info>> wrote:
Hi,
I’ve just tried setting everything to warn through the nova.conf option default_log_levels, as suggested. However, I’m still getting info level logs from the resource tracker like this :
INFO nova.compute.resource_tracker
Could the compute resource tracker logs be managed by another parameter than what’s in the default list for that configuration option?
Best regards,
Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.
Le 7 mai 2019 à 09:02, Jay Pipes <jaypipes@gmail.com <mailto:jaypipes@gmail.com>> a écrit :
On 05/06/2019 05:56 PM, Jean-Philippe Méthot wrote:
Hi, We’ve been modifying our login habits for Nova on our Openstack setup to try to send only warning level and up logs to our log servers. To do so, I’ve created a logging.conf and configured logging according to the logging module documentation. While what I’ve done works, it seems to be a very convoluted process for something as simple as changing the logging level to warning. We worry that if we upgrade and the syntax for this configuration file changes, we may have to push more changes through ansible than we would like to.
It's unlikely that the syntax for the logging configuration file will change since it's upstream Python, not OpenStack or Nova that is the source of this syntax.
That said, if all you want to do is change some or all package default logging levels, you can change the value of the CONF.default_log_levels option.
The default_log_levels CONF option is actually derived from the oslo_log package that is used by all OpenStack service projects. It's default value is here:
https://github.com/openstack/oslo.log/blob/29671ef2bfacb416d397abc57170bb090...
So, if you don't want to mess with the standard Python logging conf, you can just change that CONF.default_log_levels option. Note that if you do specify a logging config file using a non-None CONF.log_config_append value, then all other logging configuration options (like default_log_levels) are ignored).
Best, -jay
-- Best regards, Bogdan Dobrelya, Irc #bogdando
Sorry for delayed response... comments inline. On 05/07/2019 05:31 PM, Jean-Philippe Méthot wrote:
Indeed, this is what was written in your original response as well as in the documentation. As a result, it was fairly difficult to miss and I did comment it out before restarting the service. Additionally, as per the configuration I had set up, had the log-config-append option be set, I wouldn’t have any INFO level log in my logs. Hence why I believe it is strange that I have info level logs, when I’ve set default_log_levels like this:
default_log_levels = amqp=WARN,amqplib=WARN,boto=WARN,qpid=WARN,sqlalchemy=WARN,suds=WARN,oslo.messaging=WARN,iso8601=WARN,requests.packages.urllib3.connectionpool=WARN,urllib3.connectionpool=WARN,websocket=WARN,requests.packages.urllib3.util.retry=WARN,urllib3.util.retry=WARN,keystonemiddleware=WARN,routes.middleware=WARN,stevedore=WARN,taskflow=WARN,keystoneauth=WARN,oslo.cache=WARN
Do you see any of the above modules logging with INFO level, though? Or are you just seeing other modules (e.g. nova.*) logging at INFO level? If you are only seeing nova modules logging at INFO level, try adding: ,nova=WARN to the default_log_levels CONF option. Let us know if this works :) Best, -jay
Hi, Indeed, the remaining info messages were coming from the nova-compute resource tracker. Adding nova=WARN in the list did remove these messages. Thank you very much for your help. Best regards, Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.
Le 8 mai 2019 à 09:35, Jay Pipes <jaypipes@gmail.com> a écrit :
Sorry for delayed response... comments inline.
On 05/07/2019 05:31 PM, Jean-Philippe Méthot wrote:
Indeed, this is what was written in your original response as well as in the documentation. As a result, it was fairly difficult to miss and I did comment it out before restarting the service. Additionally, as per the configuration I had set up, had the log-config-append option be set, I wouldn’t have any INFO level log in my logs. Hence why I believe it is strange that I have info level logs, when I’ve set default_log_levels like this: default_log_levels = amqp=WARN,amqplib=WARN,boto=WARN,qpid=WARN,sqlalchemy=WARN,suds=WARN,oslo.messaging=WARN,iso8601=WARN,requests.packages.urllib3.connectionpool=WARN,urllib3.connectionpool=WARN,websocket=WARN,requests.packages.urllib3.util.retry=WARN,urllib3.util.retry=WARN,keystonemiddleware=WARN,routes.middleware=WARN,stevedore=WARN,taskflow=WARN,keystoneauth=WARN,oslo.cache=WARN
Do you see any of the above modules logging with INFO level, though? Or are you just seeing other modules (e.g. nova.*) logging at INFO level?
If you are only seeing nova modules logging at INFO level, try adding:
,nova=WARN
to the default_log_levels CONF option.
Let us know if this works :)
Best, -jay
Sweet! :) Glad it worked! Best, -jay On 05/08/2019 10:43 AM, Jean-Philippe Méthot wrote:
Hi,
Indeed, the remaining info messages were coming from the nova-compute resource tracker. Adding nova=WARN in the list did remove these messages. Thank you very much for your help.
Best regards,
Jean-Philippe Méthot Openstack system administrator Administrateur système Openstack PlanetHoster inc.
Le 8 mai 2019 à 09:35, Jay Pipes <jaypipes@gmail.com <mailto:jaypipes@gmail.com>> a écrit :
Sorry for delayed response... comments inline.
On 05/07/2019 05:31 PM, Jean-Philippe Méthot wrote:
Indeed, this is what was written in your original response as well as in the documentation. As a result, it was fairly difficult to miss and I did comment it out before restarting the service. Additionally, as per the configuration I had set up, had the log-config-append option be set, I wouldn’t have any INFO level log in my logs. Hence why I believe it is strange that I have info level logs, when I’ve set default_log_levels like this: default_log_levels = amqp=WARN,amqplib=WARN,boto=WARN,qpid=WARN,sqlalchemy=WARN,suds=WARN,oslo.messaging=WARN,iso8601=WARN,requests.packages.urllib3.connectionpool=WARN,urllib3.connectionpool=WARN,websocket=WARN,requests.packages.urllib3.util.retry=WARN,urllib3.util.retry=WARN,keystonemiddleware=WARN,routes.middleware=WARN,stevedore=WARN,taskflow=WARN,keystoneauth=WARN,oslo.cache=WARN
Do you see any of the above modules logging with INFO level, though? Or are you just seeing other modules (e.g. nova.*) logging at INFO level?
If you are only seeing nova modules logging at INFO level, try adding:
,nova=WARN
to the default_log_levels CONF option.
Let us know if this works :)
Best, -jay
participants (3)
-
Bogdan Dobrelya
-
Jay Pipes
-
Jean-Philippe Méthot