Open Stack

Guys, I figured out part of the problem.

The problem is a wireless TP-link router with the OpenWRT firmware 
configured with bridge.

When I connect this wireless router to the switch with the OpenStack 
cloud servers, the Linux bridge agent starts to make an error and I lose 
access to the VMs.

It is not duplicate IP or DHCP.

Does anyone have any idea what it is?




Em 27-07-2018 08:32, Marcio Prado escreveu:
> Thanks for the help Eugen,
> 
> This log is from the linuxbridge of the controller node. Compute nodes
> are not logging errors.
> 
> Follows the output of the "openstack network agent list"
> 
> +--------------------------------------+--------------------+------------+-------------------+-------+-------+---------------------------+
> | ID                                   | Agent Type         | Host
>   | Availability Zone | Alive | State | Binary                    |
> +--------------------------------------+--------------------+------------+-------------------+-------+-------+---------------------------+
> | 590f5a6d-379b-4e8d-87ec-f1060cecf230 | Linux bridge agent |
> controller | None              | True  | UP    |
> neutron-linuxbridge-agent |
> | 88fb87c9-4c03-4faa-8286-95be3586fc94 | DHCP agent         |
> controller | nova              | True  | UP    | neutron-dhcp-agent
>     |
> | b982382e-438c-46a9-8d4e-d58d554150fd | Linux bridge agent | compute1
>   | None              | True  | UP    | neutron-linuxbridge-agent |
> | c7a9ba41-1fae-46cd-b61f-30bcacb0a4e8 | L3 agent           |
> controller | nova              | True  | UP    | neutron-l3-agent
>     |
> | c9a1ea4b-2d5d-4bda-9849-cd6e302a2917 | Metadata agent     |
> controller | None              | True  | UP    |
> neutron-metadata-agent    |
> | e690d4b9-9285-4ddd-a87a-f28ea99d9a73 | Linux bridge agent | compute3
>   | None              | False | UP    | neutron-linuxbridge-agent |
> | fdd8f615-f5d6-4100-826e-59f8270df715 | Linux bridge agent | compute2
>   | None              | False | UP    | neutron-linuxbridge-agent |
> +--------------------------------------+--------------------+------------+-------------------+-------+-------+---------------------------+
> 
> compute2 and compute3 are turned off intentionally.
> 
> Log compute1
> 
> /var/log/neutron/neutron-linuxbridge-agent.log
> 
> 2018-07-27 07:43:57.242 1895 INFO neutron.common.config [-]
> /usr/bin/neutron-linuxbridge-agent version 10.0.0
> 2018-07-27 07:43:57.243 1895 INFO
> neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent
> [-] Interface mappings: {'provider': 'eno3'}
> 2018-07-27 07:43:57.243 1895 INFO
> neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent
> [-] Bridge mappings: {}
> 2018-07-27 07:44:00.954 1895 INFO
> neutron.plugins.ml2.drivers.linuxbridge.agent.linuxbridge_neutron_agent
> [-] Agent initialized successfully, now running...
> 2018-07-27 07:44:01.582 1895 INFO
> neutron.plugins.ml2.drivers.agent._common_agent
> [req-3a8a42dc-32fc-40fc-8a4f-ddbb4d8c5f5b - - - - -] RPC agent_id:
> lb525400d52f59
> 2018-07-27 07:44:01.589 1895 INFO
> neutron.agent.agent_extensions_manager
> [req-3a8a42dc-32fc-40fc-8a4f-ddbb4d8c5f5b - - - - -] Loaded agent
> extensions: []
> 2018-07-27 07:44:01.716 1895 INFO
> neutron.plugins.ml2.drivers.agent._common_agent [-] Linux bridge agent
> Agent has just been revived. Doing a full sync.
> 2018-07-27 07:44:01.778 1895 INFO
> neutron.plugins.ml2.drivers.agent._common_agent
> [req-3a8a42dc-32fc-40fc-8a4f-ddbb4d8c5f5b - - - - -] Linux bridge
> agent Agent RPC Daemon Started!
> 2018-07-27 07:44:01.779 1895 INFO
> neutron.plugins.ml2.drivers.agent._common_agent
> [req-3a8a42dc-32fc-40fc-8a4f-ddbb4d8c5f5b - - - - -] Linux bridge
> agent Agent out of sync with plugin!
> 2018-07-27 07:44:02.418 1895 INFO
> neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect
> [req-3a8a42dc-32fc-40fc-8a4f-ddbb4d8c5f5b - - - - -] Clearing orphaned
> ARP spoofing entries for devices []
> 
> 
> I'm using this OpenStack cloud to run my master's experiment. I turned
> off all nodes, and after a few days I called again and from that the
> VMs were not remotely accessible.
> 
> So I delete existing networks and re-create. It was in an attempt to
> solve the problem.
> 
> Here is an attached image. Neutron is creating multiple interfaces on
> the 10.0.0.0 network on the router.
> 
> 
> Em 27-07-2018 05:05, Eugen Block escreveu:
>> Hi,
>> 
>> is there anything in the linuxbridge-agent logs on control and/or
>> compute node(s)?
>> Which neutron services don't start? Can you paste "openstack network
>> agent list" output?
>> 
>> The important question is: what was the cause of "neutron stopped
>> working" and why did you delete the existing networks? It probably
>> would be helpful knowing the reaseon to be able to prevent such
>> problemes in the future. Or are the provided logs from before?
>> 
>> We experience network/neutron troubles from time to time, and
>> sometimes the only way to fix it is a reboot.
>> 
>> Regards,
>> Eugen
>> 
>> 
>> Zitat von Marcio Prado <marcioprado at marcioprado.eti.br>:
>> 
>>> Good afternoon,
>>> 
>>> For no apparent reason my Neutron stopped working.
>>> 
>>> I deleted the networks, subnets and routers, created everything 
>>> again.
>>> 
>>> But it does not work. The logs are:
>>> 
>>> 
>>> 2018-07-26 11:29:16.101 3272 INFO  
>>> neutron.plugins.ml2.drivers.agent._common_agent  
>>> [req-9ba0ca9f-aeaf-44b2-ba24-c08556aae0ac - - - - -] Linux bridge  
>>> agent Agent out of sync with plugin!
>>> 2018-07-26 11:29:16.101 3272 INFO neutron.agent.securitygroups_rpc  
>>> [req-9ba0ca9f-aeaf-44b2-ba24-c08556aae0ac - - - - -] Preparing  
>>> filters for devices set(['tap69feb7be-2b', 'tap0efd5228-b0',  
>>> 'tap83a57ce5-a8', 'tapd50d137f-f6'])
>>> 2018-07-26 11:29:18.218 3272 INFO  
>>> neutron.plugins.ml2.drivers.agent._common_agent  
>>> [req-9ba0ca9f-aeaf-44b2-ba24-c08556aae0ac - - - - -] Port  
>>> tap69feb7be-2b updated. Details: {u'profile': {},  
>>> u'network_qos_policy_id': None, u'qos_policy_id': None,  
>>> u'allowed_address_pairs': [], u'admin_state_up': True,  
>>> u'network_id': u'0f293447-ad01-465e-a034-fdaa136a4488',  
>>> u'segmentation_id': None, u'device_owner':  
>>> u'network:router_gateway', u'physical_network': u'provider',  
>>> u'mac_address': u'fa:16:3e:a3:be:5c', u'device': u'tap69feb7be-2b',  
>>> u'port_security_enabled': False, u'port_id':  
>>> u'69feb7be-2b9c-4604-a078-32c984d7075a', u'fixed_ips':  
>>> [{u'subnet_id': u'5ef3df97-d88a-4c60-969c-5a862f04c1e0',  
>>> u'ip_address': u'192.168.0.14'}], u'network_type': u'flat'}
>>> 2018-07-26 11:29:18.871 3272 INFO  
>>> neutron.plugins.ml2.drivers.linuxbridge.agent.arp_protect  
>>> [req-9ba0ca9f-aeaf-44b2-ba24-c08556aae0ac - - - - -] Skipping ARP  
>>> spoofing rules for port 'tap69feb7be-2b' because it has port  
>>> security disabled
>>> 2018-07-26 11:29:20.208 3272 ERROR neutron.agent.linux.utils  
>>> [req-9ba0ca9f-aeaf-44b2-ba24-c08556aae0ac - - - - -] Exit code: 2;  
>>> Stdin: ; Stdout: ; Stderr: RTNETLINK answers: File exists
>>> 
>>> 2018-07-26 11:29:20.219 3272 ERROR  
>>> neutron.plugins.ml2.drivers.agent._common_agent  
>>> [req-9ba0ca9f-aeaf-44b2-ba24-c08556aae0ac - - - - -] Error in agent  
>>> loop. Devices info: {'current': set(['tap69feb7be-2b',  
>>> 'tap0efd5228-b0', 'tap83a57ce5-a8', 'tapd50d137f-f6']),  
>>> 'timestamps': {'tap0efd5228-b0': 9, 'tap69feb7be-2b': 13,  
>>> 'tap83a57ce5-a8': 10, 'tapd50d137f-f6': 8}, 'removed': set([]),  
>>> 'added': set(['tap69feb7be-2b', 'tap0efd5228-b0', 'tap83a57ce5-a8',  
>>> 'tapd50d137f-f6']), 'updated': set([])}
>>> 2018-07-26 11:29:20.219 3272 ERROR  
>>> neutron.plugins.ml2.drivers.agent._common_agent Traceback (most  
>>> recent call last):
>>> 2018-07-26 11:29:20.219 3272 ERROR  
>>> neutron.plugins.ml2.drivers.agent._common_agent   File  
>>> "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", 
>>> line 453, in  daemon_loop
>>> 2018-07-26 11:29:20.219 3272 ERROR  
>>> neutron.plugins.ml2.drivers.agent._common_agent     sync =  
>>> self.process_network_devices(device_info)
>>> 2018-07-26 11:29:20.219 3272 ERROR  
>>> neutron.plugins.ml2.drivers.agent._common_agent   File  
>>> "/usr/lib/python2.7/dist-packages/osprofiler/profiler.py", line 153,  
>>> in wrapper
>>> 2018-07-26 11:29:20.219 3272 ERROR  
>>> neutron.plugins.ml2.drivers.agent._common_agent     return f(*args,  
>>> **kwargs)
>>> 2018-07-26 11:29:20.219 3272 ERROR  
>>> neutron.plugins.ml2.drivers.agent._common_agent   File  
>>> "/usr/lib/python2.7/dist-packages/neutron/plugins/ml2/drivers/agent/_common_agent.py", 
>>> line 210, in  process_network_devices
>>> 
>>> Has anyone had similar experience?
>>> 
>>> -- Marcio Prado
>>> Analista de TI - Infraestrutura e Redes
>>> Fone: (35) 9.9821-3561
>>> www.marcioprado.eti.br
>>> 
>>> _______________________________________________
>>> Mailing list: 
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>>> Post to     : openstack at lists.openstack.org
>>> Unsubscribe : 
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> 
>> 
>> 
>> 
>> _______________________________________________
>> Mailing list: 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
>> Post to     : openstack at lists.openstack.org
>> Unsubscribe : 
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

-- 
Marcio Prado
Analista de TI - Infraestrutura e Redes
Fone: (35) 9.9821-3561
www.marcioprado.eti.br