Open Stack

I'm trying to assign floating ips but they are not accessible on some instances.

Instance overview lists floating ip under IP Addresses, but when I take a look at the port on the subnet it has a status of N/A and Host = None, Profile = None, VIF Type = unbound, VIF Details = none, VNIC Type normal.


So I started investigating keepalived.conf for the HA router and noticed that it seems to be unable to bind to the virtual interfaces.


OpenStack Kilo

Ubuntu 14.04

2 Network Nodes

1 Controller

4 Compute Nodes​


Network 1

Feb  3 12:15:20 network2 Keepalived[16193]: Starting VRRP child process, pid=16194
Feb  3 12:15:20 network2 Keepalived_vrrp[16194]: Interface queue is empty
Feb  3 12:15:20 network2 Keepalived_vrrp[16194]: No such interface, qg-18b1e1bc-10
Feb  3 12:15:20 network2 Keepalived_vrrp[16194]: Registering Kernel netlink reflector
Feb  3 12:15:20 network2 Keepalived_vrrp[16194]: Registering Kernel netlink command channel
Feb  3 12:15:20 network2 Keepalived_vrrp[16194]: Registering gratuitous ARP shared channel
Feb  3 12:15:20 network2 Keepalived_vrrp[16194]: Initializing ipvs 2.6
Feb  3 12:15:20 network2 Keepalived_vrrp[16194]: Opening file '/var/lib/neutron/ha_confs/c19a7d8b-5f93-4b1c-a507-6b3f380bd468/keepalived.conf'.
Feb  3 12:15:20 network2 Keepalived_vrrp[16194]: Configuration is using : 62880 Bytes
Feb  3 12:15:20 network2 Keepalived_vrrp[16194]: Using LinkWatch kernel netlink reflector...
Feb  3 12:15:20 network2 Keepalived_vrrp[16194]: VRRP_Instance(VR_1) Entering BACKUP STATE
Feb  3 12:15:23 network2 Keepalived_vrrp[16194]: Netlink: filter function error
Feb  3 12:15:23 network2 Keepalived_vrrp[16194]: Netlink: filter function error
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: Interface queue is empty
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: No such interface, qg-18b1e1bc-10
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: No such interface, qr-01280b44-f7
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: Registering Kernel netlink reflector
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: Registering Kernel netlink command channel
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: Registering gratuitous ARP shared channel
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: Initializing ipvs 2.6
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: Opening file '/var/lib/neutron/ha_confs/c19a7d8b-5f93-4b1c-a507-6b3f380bd468/keepalived.conf'.
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: Configuration is using : 62190 Bytes
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: Using LinkWatch kernel netlink reflector...
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: cant do IP_DROP_MEMBERSHIP errno=Bad file descriptor (9)
Feb  3 12:15:24 network2 Keepalived_vrrp[16194]: VRRP_Instance(VR_1) Entering BACKUP STATE
Feb  3 12:15:31 network2 Keepalived_vrrp[16194]: VRRP_Instance(VR_1) Transition to MASTER STATE
Feb  3 12:15:33 network2 Keepalived_vrrp[16194]: VRRP_Instance(VR_1) Entering MASTER STATE
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: Interface queue is empty
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: No such interface, qg-18b1e1bc-10
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: No such interface, qr-01280b44-f7
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: Registering Kernel netlink reflector
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: Registering Kernel netlink command channel
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: Registering gratuitous ARP shared channel
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: Initializing ipvs 2.6
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: Opening file '/var/lib/neutron/ha_confs/c19a7d8b-5f93-4b1c-a507-6b3f380bd468/keepalived.conf'.
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: Configuration is using : 62302 Bytes
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: Using LinkWatch kernel netlink reflector...
Feb  3 12:21:30 network2 Keepalived_vrrp[16194]: cant do IP_DROP_MEMBERSHIP errno=Bad file descriptor (9)
Feb  3 12:21:32 network2 Keepalived_vrrp[16194]: VRRP_Instance(VR_1) Transition to MASTER STATE

/var/lib/neutron/ha_confs/c19a7d8b-5f93-4b1c-a507-6b3f380bd468/keepalived.conf
vrrp_instance VR_1 {
    state BACKUP
    interface ha-4c437c18-33
    virtual_router_id 1
    priority 50
    garp_master_repeat 5
    garp_master_refresh 10
    nopreempt
    advert_int 2
    track_interface {
        ha-4c437c18-33
    }
    virtual_ipaddress {
        169.254.0.1/24 dev ha-4c437c18-33
    }
    virtual_ipaddress_excluded {
        10.10.21.103/24 dev qr-01280b44-f7
        157.157.9.118/24 dev qg-18b1e1bc-10
        157.157.9.119/32 dev qg-18b1e1bc-10
        fe80::f816:3eff:fe3f:d0b9/64 dev qr-01280b44-f7 scope link
        fe80::f816:3eff:fe91:5199/64 dev qg-18b1e1bc-10 scope link
    }
    virtual_routes {
        0.0.0.0/0 via 157.157.9.1 dev qg-18b1e1bc-10
    }
}


ovs-vsctl show | grep -a1 qr-01280b44-f7
                type: internal
        Port "qr-01280b44-f7"
            tag: 52
            Interface "qr-01280b44-f7"
                type: internal

ovs-vsctl show | grep -a1 qg-18b1e1bc-10
                type: internal
        Port "qg-18b1e1bc-10"
            tag: 2
            Interface "qg-18b1e1bc-10"
                type: internal





Network 2

Feb  3 12:15:20 network3 Keepalived[16329]: Starting VRRP child process, pid=16330
Feb  3 12:15:20 network3 Keepalived_vrrp[16330]: Interface queue is empty
Feb  3 12:15:20 network3 Keepalived_vrrp[16330]: No such interface, qg-18b1e1bc-10
Feb  3 12:15:20 network3 Keepalived_vrrp[16330]: Registering Kernel netlink reflector
Feb  3 12:15:20 network3 Keepalived_vrrp[16330]: Registering Kernel netlink command channel
Feb  3 12:15:20 network3 Keepalived_vrrp[16330]: Registering gratuitous ARP shared channel
Feb  3 12:15:20 network3 Keepalived_vrrp[16330]: Initializing ipvs 2.6
Feb  3 12:15:20 network3 Keepalived_vrrp[16330]: Opening file '/var/lib/neutron/ha_confs/c19a7d8b-5f93-4b1c-a507-6b3f380bd468/keepalived.conf'.
Feb  3 12:15:20 network3 Keepalived_vrrp[16330]: Configuration is using : 62880 Bytes
Feb  3 12:15:20 network3 Keepalived_vrrp[16330]: Using LinkWatch kernel netlink reflector...
Feb  3 12:15:20 network3 Keepalived_vrrp[16330]: VRRP_Instance(VR_1) Entering BACKUP STATE
Feb  3 12:15:23 network3 Keepalived_vrrp[16330]: Netlink: filter function error
Feb  3 12:15:23 network3 Keepalived_vrrp[16330]: Netlink: filter function error
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: Interface queue is empty
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: No such interface, qg-18b1e1bc-10
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: No such interface, qr-01280b44-f7
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: Registering Kernel netlink reflector
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: Registering Kernel netlink command channel
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: Registering gratuitous ARP shared channel
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: Initializing ipvs 2.6
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: Opening file '/var/lib/neutron/ha_confs/c19a7d8b-5f93-4b1c-a507-6b3f380bd468/keepalived.conf'.
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: Configuration is using : 62190 Bytes
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: Using LinkWatch kernel netlink reflector...
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: cant do IP_DROP_MEMBERSHIP errno=Bad file descriptor (9)
Feb  3 12:15:24 network3 Keepalived_vrrp[16330]: VRRP_Instance(VR_1) Entering BACKUP STATE
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: Interface queue is empty
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: No such interface, qg-18b1e1bc-10
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: No such interface, qr-01280b44-f7
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: Registering Kernel netlink reflector
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: Registering Kernel netlink command channel
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: Registering gratuitous ARP shared channel
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: Initializing ipvs 2.6
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: Opening file '/var/lib/neutron/ha_confs/c19a7d8b-5f93-4b1c-a507-6b3f380bd468/keepalived.conf'.
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: Configuration is using : 62302 Bytes
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: Using LinkWatch kernel netlink reflector...
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: cant do IP_DROP_MEMBERSHIP errno=Bad file descriptor (9)
Feb  3 12:21:30 network3 Keepalived_vrrp[16330]: VRRP_Instance(VR_1) Entering BACKUP STATE


/var/lib/neutron/ha_confs/c19a7d8b-5f93-4b1c-a507-6b3f380bd468/keepalived.conf​

vrrp_instance VR_1 {
    state BACKUP
    interface ha-bd7e248e-96
    virtual_router_id 1
    priority 50
    garp_master_repeat 5
    garp_master_refresh 10
    nopreempt
    advert_int 2
    track_interface {
        ha-bd7e248e-96
    }
    virtual_ipaddress {
        169.254.0.1/24 dev ha-bd7e248e-96
    }
    virtual_ipaddress_excluded {
        10.10.21.103/24 dev qr-01280b44-f7
        157.157.9.118/24 dev qg-18b1e1bc-10
        157.157.9.119/32 dev qg-18b1e1bc-10
        fe80::f816:3eff:fe3f:d0b9/64 dev qr-01280b44-f7 scope link
        fe80::f816:3eff:fe91:5199/64 dev qg-18b1e1bc-10 scope link
    }
    virtual_routes {
        0.0.0.0/0 via 157.157.9.1 dev qg-18b1e1bc-10
    }
}


ovs-vsctl show | grep -a1 qr-01280b44-f7
        Port "qr-01280b44-f7"
            tag: 51
            Interface "qr-01280b44-f7"
                type: internal

ovs-vsctl show | grep -a1 qg-18b1e1bc-10
                type: internal
        Port "qg-18b1e1bc-10"
            tag: 1
            Interface "qg-18b1e1bc-10"
                type: internal​​​

​



-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20160203/63a24915/attachment.html>