[oslo] run outlasted interval
Hello, I have an openstack queens installation on centos 7 and starting from last night I got some warnings in several service: nova conductor reports: 2019-09-19 17:37:54.091 161110 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.092 161108 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.093 161118 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.095 161116 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.096 161109 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.097 161107 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec openvswitch agent reports: 019-09-19 17:16:14.499 26115 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 3.57 sec 2019-09-19 17:23:44.200 26115 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 29.71 sec 2019-09-19 17:37:54.077 26115 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 9.91 sec Please, anyone could help me ? Regards Ignazio
Adding Nova and Neutron tags as I don't think this is an Oslo problem. What I believe those log messages are saying is that the thing from Nova/Neutron that Oslo called took longer to run than it should have. On 9/19/19 10:42 AM, Ignazio Cassano wrote:
Hello, I have an openstack queens installation on centos 7 and starting from last night I got some warnings in several service:
nova conductor reports:
2019-09-19 17:37:54.091 161110 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.092 161108 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.093 161118 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.095 161116 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.096 161109 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.097 161107 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec
openvswitch agent reports: 019-09-19 17:16:14.499 26115 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 3.57 sec 2019-09-19 17:23:44.200 26115 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 29.71 sec 2019-09-19 17:37:54.077 26115 WARNING oslo.service.loopingcall [-] Function 'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state' run outlasted interval by 9.91 sec
Please, anyone could help me ? Regards Ignazio
On 9/19/2019 10:48 AM, Ben Nemec wrote:
Adding Nova and Neutron tags as I don't think this is an Oslo problem. What I believe those log messages are saying is that the thing from Nova/Neutron that Oslo called took longer to run than it should have.
Right, check the usage on the hosts running those services to see if something like CPU is maxed out. -- Thanks, Matt
Hello, I'll check tomorrow. Regards Ignazio Il gio 19 set 2019, 18:05 Matt Riedemann <mriedemos@gmail.com> ha scritto:
On 9/19/2019 10:48 AM, Ben Nemec wrote:
Adding Nova and Neutron tags as I don't think this is an Oslo problem. What I believe those log messages are saying is that the thing from Nova/Neutron that Oslo called took longer to run than it should have.
Right, check the usage on the hosts running those services to see if something like CPU is maxed out.
--
Thanks,
Matt
Hello, no cpu problems on my 3 controllers. The warning appears on all controllers. I tried to reboot one controller at time , but he warning continues to appear :-( Ignazio Il giorno gio 19 set 2019 alle ore 18:05 Matt Riedemann <mriedemos@gmail.com> ha scritto:
On 9/19/2019 10:48 AM, Ben Nemec wrote:
Adding Nova and Neutron tags as I don't think this is an Oslo problem. What I believe those log messages are saying is that the thing from Nova/Neutron that Oslo called took longer to run than it should have.
Right, check the usage on the hosts running those services to see if something like CPU is maxed out.
--
Thanks,
Matt
Hello, thanks for your answr. I have another installatiom with same release, number of nodes and controllers and the warning does not appear. Insering debug=true in nova.conf did not help. Ignazio Il gio 19 set 2019, 17:48 Ben Nemec <openstack@nemebean.com> ha scritto:
Adding Nova and Neutron tags as I don't think this is an Oslo problem. What I believe those log messages are saying is that the thing from Nova/Neutron that Oslo called took longer to run than it should have.
On 9/19/19 10:42 AM, Ignazio Cassano wrote:
Hello, I have an openstack queens installation on centos 7 and starting from last night I got some warnings in several service:
nova conductor reports:
2019-09-19 17:37:54.091 161110 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.092 161108 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.093 161118 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.095 161116 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.096 161109 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec 2019-09-19 17:37:54.097 161107 WARNING oslo.service.loopingcall [-] Function 'nova.servicegroup.drivers.db.DbDriver._report_state' run outlasted interval by 29.88 sec
openvswitch agent reports: 019-09-19 17:16:14.499 26115 WARNING oslo.service.loopingcall [-] Function
'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state'
run outlasted interval by 3.57 sec 2019-09-19 17:23:44.200 26115 WARNING oslo.service.loopingcall [-] Function
'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state'
run outlasted interval by 29.71 sec 2019-09-19 17:37:54.077 26115 WARNING oslo.service.loopingcall [-] Function
'neutron.plugins.ml2.drivers.openvswitch.agent.ovs_neutron_agent.OVSNeutronAgent._report_state'
run outlasted interval by 9.91 sec
Please, anyone could help me ? Regards Ignazio
participants (3)
-
Ben Nemec
-
Ignazio Cassano
-
Matt Riedemann