[Openstack-operators] Mitaka to Newton networking issues

Grant Morley grant at absolutedevops.io
Tue Dec 6 12:59:28 UTC 2016


They both appear to be "ACTIVE" which is what I would expect:

root at management-1-utility-container-f1222d05:~# neutron port-show 
8cd027f1-9f8c-4077-9c8a-92abc62fadd4
+-----------------------+--------------------------------------------------------------------------------------+
| Field                 | Value |
+-----------------------+--------------------------------------------------------------------------------------+
| admin_state_up        | True |
| allowed_address_pairs | |
| binding:host_id       | network-1-neutron-agents-container-11d47568 |
| binding:profile       | {} |
| binding:vif_details   | {"port_filter": true} |
| binding:vif_type      | bridge |
| binding:vnic_type     | normal |
| created_at            | 2016-12-05T10:58:01Z |
| description | |
| device_id             | a8a10308-d62f-420f-99cf-f3727ef2b784 |
| device_owner          | network:router_ha_interface |
| extra_dhcp_opts | |
| fixed_ips             | {"subnet_id": 
"6495d542-4b78-40df-84af-31500aaa0bf8", "ip_address": "169.254.192.5"} |
| id                    | 8cd027f1-9f8c-4077-9c8a-92abc62fadd4 |
| mac_address           | fa:16:3e:58:a1:a4 |
| name                  | HA port tenant e0ffdeb1e910469d9e625b95f2fa6c54 |
| network_id            | 2b04fc3a-5c0d-4f55-996f-8888d8bd1e1d |
| port_security_enabled | False |
| project_id | |
| revision_number       | 23 |
| security_groups | |
| status                | ACTIVE |
| tenant_id | |
| updated_at            | 2016-12-06T10:18:00Z |
+-----------------------+--------------------------------------------------------------------------------------+
root at management-1-utility-container-f1222d05:~# neutron port-show 
bda1f324-3178-46e5-8638-0f454ba09cab
+-----------------------+--------------------------------------------------------------------------------------+
| Field                 | Value |
+-----------------------+--------------------------------------------------------------------------------------+
| admin_state_up        | True |
| allowed_address_pairs | |
| binding:host_id       | network-2-neutron-agents-container-40906bfc |
| binding:profile       | {} |
| binding:vif_details   | {"port_filter": true} |
| binding:vif_type      | bridge |
| binding:vnic_type     | normal |
| created_at            | 2016-12-05T10:58:01Z |
| description | |
| device_id             | a8a10308-d62f-420f-99cf-f3727ef2b784 |
| device_owner          | network:router_ha_interface |
| extra_dhcp_opts | |
| fixed_ips             | {"subnet_id": 
"6495d542-4b78-40df-84af-31500aaa0bf8", "ip_address": "169.254.192.1"} |
| id                    | bda1f324-3178-46e5-8638-0f454ba09cab |
| mac_address           | fa:16:3e:c3:8a:14 |
| name                  | HA port tenant e0ffdeb1e910469d9e625b95f2fa6c54 |
| network_id            | 2b04fc3a-5c0d-4f55-996f-8888d8bd1e1d |
| port_security_enabled | False |
| project_id | |
| revision_number       | 15 |
| security_groups | |
| status                | ACTIVE |
| tenant_id | |
| updated_at            | 2016-12-05T14:35:16Z |
+-----------------------+--------------------------------------------------------------------------------------+



On 06/12/16 12:53, Kevin Benton wrote:
> Can you do a 'neutron port-show' for both of those HA ports to check 
> their status field?
>
> On Tue, Dec 6, 2016 at 2:29 AM, Grant Morley <grant at absolutedevops.io 
> <mailto:grant at absolutedevops.io>> wrote:
>
>     Hi Kevin & Neil,
>
>     Many thanks for the reply. I have attached a screen shot showing
>     that we cannot ping between the L3 HA nodes on the router name
>     spaces. This was previously working fine with Mitaka, and has only
>     stopped working since the upgrade to Newton.
>
>     From the packet captures and TCP dumps, the traffic doesn't seem
>     to be even leaving the namespace.
>
>     On the attachment, the left hand side shows the state of
>     keepalived showing both HA agents as master and the ring hand side
>     is the ping attempt.
>
>     Regards,
>
>     On 06/12/16 10:14, Kevin Benton wrote:
>>     Yes, that is a misleading warning. What is happening is that it's
>>     trying to load the interface driver as an alias first, which
>>     results in a stevedore warning that you see and then it falls
>>     back to loading it by the class path, which is what you have
>>     configured. We will need to see if there is a way we can suppress
>>     that warning somehow when we make the call to load by an alias
>>     and it fails.
>>
>>     If you switch your interface to just 'linuxbridge', that should
>>     get rid of the warning.
>>
>>
>>     For both L3 HA nodes becoming master, we need a little more info
>>     to figure out the root cause. Can you try switching into the
>>     router namespace on one of the L3 HA nodes and see if you can
>>     ping the other router instance across the L3 HA network for that
>>     router?
>>
>>     On Mon, Dec 5, 2016 at 7:54 AM, Neil Jerram <neil at tigera.io
>>     <mailto:neil at tigera.io>> wrote:
>>
>>         I have also recently been seeing 'Could not load
>>         <whatever>InterfaceDriver' warnings from the DHCP agent, and
>>         haven't yet understood that - although I'm pretty sure that
>>         my interface driver is being loaded really - or else none of
>>         my networking function would work at all.
>>
>>         So it's possible that that part of your report is benign, and
>>         just a misleading warning.  That said, I am still worried
>>         about it too, and would like to understand it properly.
>>
>>         I'm not aware of seeing the other symptoms you mentioned.
>>
>>              Neil
>>
>>
>>         On Mon, Dec 5, 2016 at 3:14 PM Grant Morley
>>         <grant at absolutedevops.io <mailto:grant at absolutedevops.io>> wrote:
>>
>>             Hi All,
>>
>>             We have just upgraded from Mitaka to Newton. We are
>>             running OSA and we seem to have come across some weird
>>             networking issues since the upgrade. Basically network
>>             access to instances is very intermittent and seems to
>>             randomly stop working.
>>
>>             We are running neutron in HA and it appears that both of
>>             the neutron nodes are now trying to be master and are
>>             both trying to bring up the gateway IP addresses which
>>             would be causing conflicts.
>>
>>             We are also seeing a lot of the following in the
>>             "neutron-dhcp-agent" log files:
>>
>>             2016-12-05 14:42:24.837 2020 WARNING stevedore.named
>>             [req-1955d0a1-1453-4c65-a93a-54e8ea39b230
>>             1ac995c0729142289f7237222f335806
>>             3cc95dbe91c84e3e8ebbb9893ee54d20 - - -] Could not load
>>             neutron.agent.linux.interface.BridgeInterfaceDriver
>>             2016-12-05 14:42:42.803 2020 INFO
>>             neutron.agent.dhcp.agent
>>             [req-fad7d2bb-9d3c-4192-868a-0164b382aecf
>>             1ac995c0729142289f7237222f335806
>>             3cc95dbe91c84e3e8ebbb9893ee54d20 - - -] Trigger
>>             reload_allocations for port admin_state_up=True,
>>             allowed_address_pairs=[], binding:host_id=,
>>             binding:profile=, binding:vif_details=,
>>             binding:vif_type=unbound, binding:vnic_type=normal,
>>             created_at=2016-12-05T14:42:42Z, description=,
>>             device_id=8752effa-2ff2-4ce1-be70-e9f2243612cb,
>>             device_owner=network:floatingip, extra_dhcp_opts=[],
>>             fixed_ips=[{u'subnet_id':
>>             u'4ca7db2d-544a-4a97-b5a4-3cbf2467a4b7', u'ip_address':
>>             u'XXX.XXX.XXX.XXX'}],
>>             id=b3cf223d-8e76-484a-a649-d8a7dd435124,
>>             mac_address=fa:16:3e:ff:0d:50, name=,
>>             network_id=af5db886-0178-4f8d-9189-f55f773b37fa,
>>             port_security_enabled=False, project_id=,
>>             revision_number=4, security_groups=[], status=N/A,
>>             tenant_id=, updated_at=2016-12-05T14:42:42Z
>>
>>             I am a bit concerned about neutron not being able to load
>>             the Bridge interface driver.
>>
>>             Has anyone else come across this at all or have any
>>             pointers? This was working fine in Mitaka it just seems
>>             since the upgrade to Newton, we have these issues.
>>
>>             I am able to provide more logs if they are needed.
>>
>>             Regards,
>>
>>             -- 
>>             Grant Morley
>>             Cloud Lead
>>             Absolute DevOps Ltd
>>             Units H, J & K, Gateway 1000, Whittle Way, Stevenage,
>>             Herts, SG1 2FP
>>             www.absolutedevops.io <http://www.absolutedevops.io>
>>             grant at absolutedevops.io <mailto:grant at absolutedevops.io>
>>             0845 874 0580
>>             _______________________________________________
>>             OpenStack-operators mailing list
>>             OpenStack-operators at lists.openstack.org
>>             <mailto:OpenStack-operators at lists.openstack.org>
>>             http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>             <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators>
>>
>>
>>         _______________________________________________
>>         OpenStack-operators mailing list
>>         OpenStack-operators at lists.openstack.org
>>         <mailto:OpenStack-operators at lists.openstack.org>
>>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
>>         <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators>
>>
>>
>
>     -- 
>     Grant Morley
>     Cloud Lead
>     Absolute DevOps Ltd
>     Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP
>     www.absolutedevops.io <http://www.absolutedevops.io>
>     grant at absolutedevops.io <mailto:grant at absolutedevops.io> 0845 874
>     0580
>
>

-- 
Grant Morley
Cloud Lead
Absolute DevOps Ltd
Units H, J & K, Gateway 1000, Whittle Way, Stevenage, Herts, SG1 2FP
www.absolutedevops.io <http://www.absolutedevops.io/> 
grant at absolutedevops.io <mailto:grant at absolutedevops.i> 0845 874 0580
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20161206/b91ad77c/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: image/png
Size: 4369 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20161206/b91ad77c/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ado_new.png
Type: image/png
Size: 4369 bytes
Desc: not available
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20161206/b91ad77c/attachment-0001.png>


More information about the OpenStack-operators mailing list