Open Stack

On 04/11/2017 06:35 PM, Daniel Russell wrote:
> This is somewhat unrelated to the original email, but more of a question of an issue I have recently come across that seems tangentially related.
>
> When run_one_command is executed and the message is sent to rootwrap via that pipe, if rootwrap errors for some reason, I get " Unserializable message: ('#ERROR',  ValueError('I/O operation on closed file',))" returned with a stack trace.
>
> In this case it appears the problem was that conntrack wasn't installed.  In my case it was that the rootwrap filters denied the request to kill the process.  I only tracked it down because I configured syslog on the rootwrap daemon and killed the process (so l3-agent would respawn it, retry the command, and error again).  The error then appeared very clearly in /var/log/messages, so rootwrap does understand what is going on but it doesn't seem to feed it back to the caller.
>
> Is this a known listed bug?  It would make diagnosis a lot easier if an easy to understand error message came back instead of the above.

Hi Dan,

https://bugs.launchpad.net/neutron/+bug/1677742 was the bug I opened for 
the original rootwrap error, and the change for that merged upstream. 
That will catch the error and print the command being tried, which is at 
least a bread crumb and finding the failure.  Could be there is more 
info we can gather, if you can try that change and add info to the bug 
it would be great.

-Brian


> -----Original Message-----
> From: Brian Haley [mailto:haleyb.dev at gmail.com]
> Sent: Tuesday, 11 April 2017 6:24 AM
> To: openstack at lists.openstack.org
> Subject: Re: [Openstack] LinuxBridge Agent Error
>
> On 04/10/2017 04:15 PM, Georgios Dimitrakakis wrote:
>> Installing the "conntrack-tools" package from the
>> "centos-openstack-ocata" repository seems to have fixed the problem.
>>
>> My question is if this should have happened automatically during
>> installation and if yes why it didn't happen?
>
>  From your comment it was a manual install.  Looking at the neutron-sanity-check tool it doesn't seem to check for conntrack.  Fix proposed at https://review.openstack.org/455450
>
> Thanks!
>
> -Brian
>
>
>> On Fri, 07 Apr 2017 19:26:02 +0300, Georgios Dimitrakakis wrote:
>>> Hello,
>>>
>>> I am on CentOS 7 and this is a manual Ocata installation as described
>>> on the deployment guide.
>>>
>>> My cloud seems to be operating normally even with these messages. The
>>> thing is that I don't like all these (there are a whole bunch of them
>>> every second) in my log files.
>>>
>>> I will try to see if I am missing the "conntrack" and try to install
>>> the patch that will provide more info..
>>>
>>>
>>> Best,
>>>
>>> G.
>>>
>>>> Is this a packstack installation?
>>>>
>>>> Inviato da iPhone
>>>>
>>>> Il giorno 07 apr 2017, alle ore di 07:19, Kevin Benton  ha scritto:
>>>>
>>>>> You may be missing whatever package in your distro provides the
>>>>> 'conntrack' util.
>>>>>
>>>>> On Apr 7, 2017 03:52, "Georgios Dimitrakakis" wrote:
>>>>>
>>>>>> Hello,
>>>>>>
>>>>>> I am having an Ocata installation and on my Compute Node the
>>>>>> "linuxbridge-agent.log" file is constantly filling out with
>>>>>> messages like this:
>>>>>>
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [1]ent._common_agent Traceback (most recent call last):
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [2]ent._common_agent File
>>>>>>
>>>>>
>>>>
>>>> "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
>>>> _common_agent.py",
>>>>
>>>>>> line 453, in daemon_loop
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [3]ent._common_agent sync =
>>>>>> self.process_network_devices(device_info)
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [4]ent._common_agent File
>>>>>> "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line
>>>>>> 153, in wrapper
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [5]ent._common_agent return f(*args, **kwargs)
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [6]ent._common_agent File
>>>>>>
>>>>>
>>>>
>>>> "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
>>>> _common_agent.py",
>>>>
>>>>>> line 213, in process_network_devices
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [7]ent._common_agent resync_b =
>>>>>> self.treat_devices_removed(device_info['removed'])
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [8]ent._common_agent File
>>>>>> "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line
>>>>>> 153, in wrapper
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [9]ent._common_agent return f(*args, **kwargs)
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [10]ent._common_agent File
>>>>>>
>>>>>
>>>>
>>>> "/usr/lib/python2.7/site-packages/neutron/plugins/ml2/drivers/agent/
>>>> _common_agent.py",
>>>>
>>>>>> line 331, in treat_devices_removed
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [11]ent._common_agent self.sg_agent.remove_devices_filter(devices)
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [12]ent._common_agent File
>>>>>>
>>>>>
>>>>
>>>> "/usr/lib/python2.7/site-packages/neutron/agent/securitygroups_rpc.p
>>>> y",
>>>>>> line 238, in remove_devices_filter
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [13]ent._common_agent self.firewall.remove_port_filter(device)
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [14]ent._common_agent File
>>>>>>
>>>>>
>>>>
>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firew
>>>> all.py",
>>>>
>>>>>> line 222, in remove_port_filter
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [15]ent._common_agent
>>>>>> self._remove_conntrack_entries_from_port_deleted(port)
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [16]ent._common_agent File
>>>>>>
>>>>>
>>>>
>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/iptables_firew
>>>> all.py",
>>>>
>>>>>> line 194, in _remove_conntrack_entries_from_port_deleted
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [17]ent._common_agent [device_info], ethertype, set())
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [18]ent._common_agent File
>>>>>>
>>>>>
>>>>
>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.p
>>>> y",
>>>>>> line 121, in delete_conntrack_state_by_remote_ips
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [19]ent._common_agent
>>>>>> self._delete_conntrack_state(device_info_list, rule)
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [20]ent._common_agent File
>>>>>>
>>>>>
>>>>
>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/ip_conntrack.p
>>>> y",
>>>>>> line 103, in _delete_conntrack_state
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [21]ent._common_agent extra_ok_codes=[1])
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [22]ent._common_agent File
>>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py",
>>>>>> line 128, in execute
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [23]ent._common_agent execute_rootwrap_daemon(cmd, process_input,
>>>>>> addl_env))
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [24]ent._common_agent File
>>>>>> "/usr/lib/python2.7/site-packages/neutron/agent/linux/utils.py",
>>>>>> line 115, in execute_rootwrap_daemon
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [25]ent._common_agent return client.execute(cmd, process_input)
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [26]ent._common_agent File
>>>>>> "/usr/lib/python2.7/site-packages/oslo_rootwrap/client.py", line
>>>>>> 128, in execute
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [27]ent._common_agent res = proxy.run_one_command(cmd, stdin)
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [28]ent._common_agent File "", line 2, in run_one_command
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [29]ent._common_agent File
>>>>>> "/usr/lib64/python2.7/multiprocessing/managers.py", line 773, in
>>>>>> _callmethod
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [30]ent._common_agent raise convert_to_error(kind, result)
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [31]ent._common_agent RemoteError:
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [32]ent._common_agent
>>>>>>
>>>>>
>>>>
>>>> --------------------------------------------------------------------
>>>> -------
>>>>
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [33]ent._common_agent Unserializable message: ('#ERROR',
>>>>>> ValueError('I/O operation on closed file',))
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [34]ent._common_agent
>>>>>>
>>>>>
>>>>
>>>> --------------------------------------------------------------------
>>>> -------
>>>>
>>>>>> 2017-04-07 13:37:47.709 1397 ERROR neutron.plugins.ml2.drivers.ag
>>>>>> [35]ent._common_agent
>>>>>> 2017-04-07 13:37:47.710 1397 INFO neutron.plugins.ml2.drivers.ag
>>>>>> [36]ent._common_agent [req-93157f5e-9f2d-4547-9002-c473211df719 -
>>>>>> - - - -] Linux bridge agent Agent out of sync with plugin!
>>>>>>
>>>>>> Any idea what might be the problem???
>>>>>>
>>>>>> The full log of a total circle of messages can be found here:
>>>>>> https://pastebin.com/6Yzn0FuY [37]
>>>>>>
>>>>>> Looking forward for an answer!
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> G.
>>>>>>
>>>>>> _______________________________________________
>>>>>> Mailing list:
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [38]
>>>>>> Post to : openstack at lists.openstack.org [39] Unsubscribe :
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [40]
>>>>> !DSPAM:1,58e777a158574020391806!
>>>>
>>>>> _______________________________________________
>>>>> Mailing list:
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [42]
>>>>> Post to : openstack at lists.openstack.org [43] Unsubscribe :
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack [44]
>>>>>
>>>>> !DSPAM:1,58e777a158574020391806!
>>>>
>>>>
>>>> Links:
>>>> ------
>>>> [1] http://neutron.plugins.ml2.drivers.ag
>>>> [2] http://neutron.plugins.ml2.drivers.ag
>>>> [3] http://neutron.plugins.ml2.drivers.ag
>>>> [4] http://neutron.plugins.ml2.drivers.ag
>>>> [5] http://neutron.plugins.ml2.drivers.ag
>>>> [6] http://neutron.plugins.ml2.drivers.ag
>>>> [7] http://neutron.plugins.ml2.drivers.ag
>>>> [8] http://neutron.plugins.ml2.drivers.ag
>>>> [9] http://neutron.plugins.ml2.drivers.ag
>>>> [10] http://neutron.plugins.ml2.drivers.ag
>>>> [11] http://neutron.plugins.ml2.drivers.ag
>>>> [12] http://neutron.plugins.ml2.drivers.ag
>>>> [13] http://neutron.plugins.ml2.drivers.ag
>>>> [14] http://neutron.plugins.ml2.drivers.ag
>>>> [15] http://neutron.plugins.ml2.drivers.ag
>>>> [16] http://neutron.plugins.ml2.drivers.ag
>>>> [17] http://neutron.plugins.ml2.drivers.ag
>>>> [18] http://neutron.plugins.ml2.drivers.ag
>>>> [19] http://neutron.plugins.ml2.drivers.ag
>>>> [20] http://neutron.plugins.ml2.drivers.ag
>>>> [21] http://neutron.plugins.ml2.drivers.ag
>>>> [22] http://neutron.plugins.ml2.drivers.ag
>>>> [23] http://neutron.plugins.ml2.drivers.ag
>>>> [24] http://neutron.plugins.ml2.drivers.ag
>>>> [25] http://neutron.plugins.ml2.drivers.ag
>>>> [26] http://neutron.plugins.ml2.drivers.ag
>>>> [27] http://neutron.plugins.ml2.drivers.ag
>>>> [28] http://neutron.plugins.ml2.drivers.ag
>>>> [29] http://neutron.plugins.ml2.drivers.ag
>>>> [30] http://neutron.plugins.ml2.drivers.ag
>>>> [31] http://neutron.plugins.ml2.drivers.ag
>>>> [32] http://neutron.plugins.ml2.drivers.ag
>>>> [33] http://neutron.plugins.ml2.drivers.ag
>>>> [34] http://neutron.plugins.ml2.drivers.ag
>>>> [35] http://neutron.plugins.ml2.drivers.ag
>>>> [36] http://neutron.plugins.ml2.drivers.ag
>>>> [37] https://pastebin.com/6Yzn0FuY
>>>> [38] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>> [39] mailto:openstack at lists.openstack.org
>>>> [40] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>> [41] mailto:giorgis at acmac.uoc.gr
>>>> [42] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>> [43] mailto:openstack at lists.openstack.org
>>>> [44] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>>> [45] mailto:kevin at benton.pub
>>>
>>>
>>> _______________________________________________
>>> Mailing list:
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     : openstack at lists.openstack.org
>>> Unsubscribe :
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>
>>
>> _______________________________________________
>> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe :
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>
>
> _______________________________________________
> Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
> Post to     : openstack at lists.openstack.org
> Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>