Open Stack

Running latest EPEL Folsom packages on RHEL 6.3.  Three nodes right now,
one controller, one network node, one compute node.  The network node has
three NICs, one for external net, one for management net, one for VM
network traffic.  It has been a miserable journey so far.

The lastest calamity began with a failed spawn of the Cirros test image.  I
booted it like this:

# nova --os-username demo --os-password demo --os-tenant-name demoProject
boot --image aefa581f-47b0-4d46-8dbc-1a1f7f02dfa0 --flavor 2  --nic
net-id=3de1e780-07d1-42af-89cc-0feaf1ece6e9 server-01

This succeeded but went directly into an ERROR state.  The compute node's
/var/log/nova/compute.log showed this:

ProcessExecutionError: Unexpected error while running command.
Command: sudo nova-rootwrap /etc/nova/rootwrap.conf brctl addbr
qbr2218b8c4-7d
Exit code: 1
Stdout: ''
Stderr: 'add bridge failed: Package not installed\n'

Hrm.  So then I ran this:

# brctl show
bridge name bridge id STP enabled interfaces
br-eth1 /sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
/sys/class/net/br-eth1/bridge: No such file or directory
0000.bc305befedd1 no
br-int /sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
/sys/class/net/br-int/bridge: No such file or directory
0000.7e1636f42c4b no

GAH!   What!!! First of all, bridge capability is set by default in the
RHEL 6.3 kernel.  Secondly, nova knows that it's supposed to be using
openvswitch.  The ProcessExecutionError's trace showed that the offending
code came from /usr/lib/python2.6/site-packages/nova/virt/libvirt/vif.py
line 216 which has this comment:

    def plug(self, instance, vif):
        """Plug using hybrid strategy

        Create a per-VIF linux bridge, then link that bridge to the OVS
        integration bridge via a veth device, setting up the other end
        of the veth device just like a normal OVS port.  Then boot the
        VIF on the linux bridge using standard libvirt mechanisms
        """

Thirdly, ovs-vsctrl is happy:

# ovs-vsctl show
44435595-8cc8-469c-ace4-ded76a7b864d
    Bridge "br-eth1"
        Port "br-eth1"
            Interface "br-eth1"
                type: internal
        Port "phy-br-eth1"
            Interface "phy-br-eth1"
        Port "eth1"
            Interface "eth1"
    Bridge br-int
        Port "int-br-eth1"
            Interface "int-br-eth1"
        Port br-int
            Interface br-int
                type: internal
    ovs_version: "1.7.3"

Final note, my network node fails the same way, but the controller does not.

I hope so much that somebody knows what is going on here.  This is very
terrible for me as I am struggling to achieve minimal functionality.
 Thanks.

-- 
\*..+.-
--Greg Chavez
+//..;};
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20130211/a890e56a/attachment.html>