[openstack-dev] [Quantum] security groups now enforced w/devstack
Yoshihiro Kaneko
ykaneko0929 at gmail.com
Mon Sep 24 07:44:04 UTC 2012
Hi Dan,
2012/9/21 Dan Wendlandt <dan at nicira.com>:
> On Fri, Sep 21, 2012 at 12:49 AM, Yoshihiro Kaneko
> <ykaneko0929 at gmail.com> wrote:
>> Hi Akihiro,
>>
>> Does Security Groups function?
>> VM came to be able to obtain an IP address by DHCP, but Security
>> Groups still does not work. There is nova-compute chains for an
>> instance, but it does not match any packet.
>
> It certainly works in my setup, so the trick is figuring out the
> difference I guess. How are you ping to reach the VM? directly via a
> fixed IP (e.g., pinging from within the namespace?) or via a floating
> ip? One possible issue is that the latter requires open vswitch to be
> version 1.4.3, due to a bug in OVS. We're still waiting on this code
> to work its way through the precise updates (I believe), but you can
> access it directly via the main folsom testing PPA:
> https://launchpad.net/~openstack-ubuntu-testing/+archive/folsom-trunk-testing
Sorry. It is my error. Security Groups works.
Last time, I did ping to a floating IP on the host. In that case, it seems that
these packets do not match iptables rules.
When I ping from remote host, it worked correctly.
Pinging from within the namespace was accepted by the following rule of
the nova-compute-inst-N chain.
0 0 ACCEPT all -- * * 10.0.0.0/24 0.0.0.0/0
I tried version 1.4.0-1ubuntu1.2 and 1.4.0-1ubuntu1.3~ppa of OVS. Both
versions were Ok.
>
>>
>> Security Groups makes iptables rules in the root name space. That
>> rule has fixed IP address of a target instance as a destination.
>> On the other, Quantum L3 agent makes SNAT rules in qrouter name
>> space, so I think fixed IP address is not found in the root name space.
>
> I don't believe this is the cause of the issue, as the vifs that
> represent the VM are actually in the root namespace, as are the
> iptables rules that filter traffic to/from them.
>
>> In addition, I think that Security Groups does not work correctly when
>> there is more than one VM with the same fixed IP address. If so, we
>> should choose between Security Groups or Overlapping IP range.
>
> That is correct. We're already doing some brainstorming to see if
> there's any non-invasive change we might be able to make here to allow
> both to co-exist.
I see.
Is it expected to be realized in Grizzly?
Thanks,
Kaneko
>
> Dan
>
>
>>
>> Thanks,
>> Kaneko
>>
>> 2012/9/20 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
>>> Hi Akihiro,
>>>
>>> I submitted bug report.
>>> https://bugs.launchpad.net/nova/+bug/1053312
>>> Please add comment for the details.
>>>
>>> Thanks,
>>> Kaneko
>>>
>>> 2012/9/20 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
>>>> Hi Akihiro,
>>>>
>>>> 2012/9/20 Akihiro MOTOKI <amotoki at gmail.com>:
>>>>> Hi Yoshihiro,
>>>>>
>>>>> Thank for testing.
>>>>> Have you already filed a bug for this problem? If not yet, could you
>>>>> file it as a nova bug?
>>>>> It is a "folsom-rc-potential" bug and needs to be fixed I think.
>>>>
>>>> Sure, will do.
>>>>
>>>>>
>>>>> Thanks,
>>>>>
>>>>> 2012/9/20 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
>>>>>> 2012/9/20 Akihiro MOTOKI <motoki at da.jp.nec.com>:
>>>>>>> Hi Dan, Yoshihiro,
>>>>>>>
>>>>>>> I investigated the problem Yoshihiro reported more.
>>>>>>>
>>>>>>> When nova-compute launches an instance on KVM, vif driver plug() is called
>>>>>>> twice
>>>>>>> accorrding to the nova-compute log [1].
>>>>>>>
>>>>>>> LibvirtHybridOVSBridgeDriver plug() calles
>>>>>>> nova.network.linux_net._create_veth_pair()
>>>>>>> to create veth pair. _create_veth_pair() checks whether the specified device
>>>>>>> exists
>>>>>>> and in case the device exists it first deletes the device and (re)creates
>>>>>>> it.
>>>>>>> It may be a cause of the problem.
>>>>>>>
>>>>>>> After the following change [2], this problem seems to be addressed.
>>>>>>> If this direction is OK, I will post a patch to gerrit.
>>>>>>>
>>>>>>>> Yoshihiro,
>>>>>>> Could you test my patch on your environment?
>>>>>>
>>>>>> Great!
>>>>>> qvo was listed in brctl show and ovs-ofctl show and VM obtained an IP
>>>>>> address by DHCP.
>>>>>>
>>>>>> $ sudo ovs-vsctl show
>>>>>> 6d7929f6-8b09-41bc-b775-1ef50915eb7b
>>>>>> Bridge br-int
>>>>>> Port "tap94dddc19-c7"
>>>>>> tag: 1
>>>>>> Interface "tap94dddc19-c7"
>>>>>> type: internal
>>>>>> Port "qvo4307567c-49"
>>>>>> tag: 1
>>>>>> Interface "qvo4307567c-49"
>>>>>> Port "qr-56b7fd42-93"
>>>>>> tag: 1
>>>>>> Interface "qr-56b7fd42-93"
>>>>>> type: internal
>>>>>> Port br-int
>>>>>> Interface br-int
>>>>>> type: internal
>>>>>> Bridge br-ex
>>>>>> Port br-ex
>>>>>> Interface br-ex
>>>>>> type: internal
>>>>>> Port "eth1"
>>>>>> Interface "eth1"
>>>>>> Port "qg-e399dae7-66"
>>>>>> Interface "qg-e399dae7-66"
>>>>>> type: internal
>>>>>> ovs_version: "1.4.0+build0"
>>>>>> $ brctl show
>>>>>> bridge name bridge id STP enabled interfaces
>>>>>> br-ex 0000.ae454d673043 no qg-e399dae7-66
>>>>>> br-int 0000.a6c5c5495143 no qr-56b7fd42-93
>>>>>> qvo4307567c-49
>>>>>> tap94dddc19-c7
>>>>>> qbr4307567c-49 8000.3ab31e81960e no qvb4307567c-49
>>>>>> vnet0
>>>>>> virbr0 8000.000000000000 yes
>>>>>> virbr1 8000.525400d8a792 yes virbr1-nic
>>>>>> $ sudo ovs-ofctl show br-int
>>>>>> OFPT_FEATURES_REPLY (xid=0x1): ver:0x1, dpid:0000a6c5c5495143
>>>>>> n_tables:255, n_buffers:256
>>>>>> features: capabilities:0xc7, actions:0xfff
>>>>>> 1(tap94dddc19-c7): addr:78:00:00:00:00:00
>>>>>> config: PORT_DOWN
>>>>>> state: LINK_DOWN
>>>>>> 2(qr-56b7fd42-93): addr:78:00:00:00:00:00
>>>>>> config: PORT_DOWN
>>>>>> state: LINK_DOWN
>>>>>> 3(qvo4307567c-49): addr:fe:9f:c2:68:9b:7a
>>>>>> config: 0
>>>>>> state: 0
>>>>>> current: 10GB-FD COPPER
>>>>>> LOCAL(br-int): addr:a6:c5:c5:49:51:43
>>>>>> config: PORT_DOWN
>>>>>> state: LINK_DOWN
>>>>>> OFPT_GET_CONFIG_REPLY (xid=0x3): frags=normal miss_send_len=0
>>>>>>
>>>>>> Thanks,
>>>>>> Kaneko
>>>>>>
>>>>>>>
>>>>>>> [1]
>>>>>>> $ grep "qvo612b2502-6e" ~/logs/screen-n-cpu.log | grep ^Running
>>>>>>> Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip link
>>>>>>> show dev qvo612b2502-6e
>>>>>>> Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip link
>>>>>>> add qvb612b2502-6e type veth peer name qvo612b2502-6e
>>>>>>> Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip link
>>>>>>> set qvo612b2502-6e up
>>>>>>> Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip link
>>>>>>> set qvo612b2502-6e promisc on
>>>>>>> Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
>>>>>>> ovs-vsctl -- --may-exist add-port br-int qvo612b2502-6e -- set Interface
>>>>>>> qvo612b2502-6e external-ids:iface-id=612b2502-6e8c-4a78-aedd-c5cb08738564
>>>>>>> external-ids:iface-status=active external-ids:attached-mac=fa:16:3e:97:e4:aa
>>>>>>> external-ids:vm-uuid=f0321c00-8ef0-4a18-9921-8452236877f2
>>>>>>> Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip link
>>>>>>> show dev qvo612b2502-6e
>>>>>>> Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip link
>>>>>>> add qvb612b2502-6e type veth peer name qvo612b2502-6e
>>>>>>> Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip link
>>>>>>> set qvo612b2502-6e up
>>>>>>> Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf ip link
>>>>>>> set qvo612b2502-6e promisc on
>>>>>>> Running cmd (subprocess): sudo nova-rootwrap /etc/nova/rootwrap.conf
>>>>>>> ovs-vsctl -- --may-exist add-port br-int qvo612b2502-6e -- set Interface
>>>>>>> qvo612b2502-6e external-ids:iface-id=612b2502-6e8c-4a78-aedd-c5cb08738564
>>>>>>> external-ids:iface-status=active external-ids:attached-mac=fa:16:3e:97:e4:aa
>>>>>>> external-ids:vm-uuid=f0321c00-8ef0-4a18-9921-8452236877f2
>>>>>>>
>>>>>>> [2]
>>>>>>> diff --git a/nova/virt/libvirt/vif.py b/nova/virt/libvirt/vif.py
>>>>>>> index 1a64765..ea0834d 100644
>>>>>>> --- a/nova/virt/libvirt/vif.py
>>>>>>> +++ b/nova/virt/libvirt/vif.py
>>>>>>> @@ -212,15 +212,15 @@ class
>>>>>>> LibvirtHybridOVSBridgeDriver(LibvirtBridgeDriver,
>>>>>>> br_name = self.get_br_name(iface_id)
>>>>>>> v1_name, v2_name = self.get_veth_pair_names(iface_id)
>>>>>>>
>>>>>>> - linux_net._create_veth_pair(v1_name, v2_name)
>>>>>>> -
>>>>>>> if not linux_net._device_exists(br_name):
>>>>>>> utils.execute('brctl', 'addbr', br_name, run_as_root=True)
>>>>>>>
>>>>>>> - utils.execute('ip', 'link', 'set', br_name, 'up', run_as_root=True)
>>>>>>> - utils.execute('brctl', 'addif', br_name, v1_name, run_as_root=True)
>>>>>>> - self.create_ovs_vif_port(v2_name, iface_id, mapping['mac'],
>>>>>>> - instance['uuid'])
>>>>>>> + if not linux_net._device_exists(v2_name):
>>>>>>> + linux_net._create_veth_pair(v1_name, v2_name)
>>>>>>> + utils.execute('ip', 'link', 'set', br_name, 'up',
>>>>>>> run_as_root=True)
>>>>>>> + utils.execute('brctl', 'addif', br_name, v1_name,
>>>>>>> run_as_root=True)
>>>>>>> + self.create_ovs_vif_port(v2_name, iface_id, mapping['mac'],
>>>>>>> + instance['uuid'])
>>>>>>>
>>>>>>> network['bridge'] = br_name
>>>>>>> return self._get_configurations(instance, network, mapping)
>>>>>>>
>>>>>>>
>>>>>>> (2012/09/20 13:13), Yoshihiro Kaneko wrote:
>>>>>>>>
>>>>>>>> Has nobody encountered this problem? Only me?
>>>>>>>> I will try this in other fresh environment.
>>>>>>>>
>>>>>>>> Kaneko
>>>>>>>>
>>>>>>>> 2012/9/19 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
>>>>>>>>>
>>>>>>>>> I found that qvo was not "internal". Therefore I set "type=internal"
>>>>>>>>> to qvo manually.
>>>>>>>>> As a result, qvo appeared in the output of "brctl show" and VM was
>>>>>>>>> able to obtain
>>>>>>>>> IP address by DHCP.
>>>>>>>>> Is this correct?
>>>>>>>>>
>>>>>>>>> Before set interface type
>>>>>>>>> ----------
>>>>>>>>> $ sudo ovs-vsctl show
>>>>>>>>> 6d7929f6-8b09-41bc-b775-1ef50915eb7b
>>>>>>>>> Bridge br-int
>>>>>>>>> Port "tap8c462a80-f6"
>>>>>>>>> tag: 1
>>>>>>>>> Interface "tap8c462a80-f6"
>>>>>>>>> type: internal
>>>>>>>>> Port "qvo076911a5-59"
>>>>>>>>> tag: 1
>>>>>>>>> Interface "qvo076911a5-59"
>>>>>>>>> Port br-int
>>>>>>>>> Interface br-int
>>>>>>>>> type: internal
>>>>>>>>> Port "qr-6dd16ab9-d6"
>>>>>>>>> tag: 1
>>>>>>>>> Interface "qr-6dd16ab9-d6"
>>>>>>>>> type: internal
>>>>>>>>> Bridge br-ex
>>>>>>>>> Port br-ex
>>>>>>>>> Interface br-ex
>>>>>>>>> type: internal
>>>>>>>>> Port "qg-de1db666-3d"
>>>>>>>>> Interface "qg-de1db666-3d"
>>>>>>>>> type: internal
>>>>>>>>> Port "eth1"
>>>>>>>>> Interface "eth1"
>>>>>>>>> ovs_version: "1.4.0+build0"
>>>>>>>>> $ brctl show
>>>>>>>>> bridge name bridge id STP enabled interfaces
>>>>>>>>> br-ex 0000.ae454d673043 no qg-de1db666-3d
>>>>>>>>> br-int 0000.c6d8e9064848 no qr-6dd16ab9-d6
>>>>>>>>> tap8c462a80-f6
>>>>>>>>> qbr076911a5-59 8000.26397c9d0d15 no
>>>>>>>>> qvb076911a5-59
>>>>>>>>> vnet0
>>>>>>>>> virbr0 8000.000000000000 yes
>>>>>>>>> virbr1 8000.525400d8a792 yes virbr1-nic
>>>>>>>>> ----------
>>>>>>>>>
>>>>>>>>> Set interface type to qvo
>>>>>>>>> ----------
>>>>>>>>> $ sudo ovs-vsctl set Interface qvo076911a5-59 type=internal
>>>>>>>>> $ sudo ovs-vsctl show
>>>>>>>>> 6d7929f6-8b09-41bc-b775-1ef50915eb7b
>>>>>>>>> Bridge br-int
>>>>>>>>> Port "tap8c462a80-f6"
>>>>>>>>> tag: 1
>>>>>>>>> Interface "tap8c462a80-f6"
>>>>>>>>> type: internal
>>>>>>>>> Port "qvo076911a5-59"
>>>>>>>>> tag: 1
>>>>>>>>> Interface "qvo076911a5-59"
>>>>>>>>> type: internal
>>>>>>>>> Port br-int
>>>>>>>>> Interface br-int
>>>>>>>>> type: internal
>>>>>>>>> Port "qr-6dd16ab9-d6"
>>>>>>>>> tag: 1
>>>>>>>>> Interface "qr-6dd16ab9-d6"
>>>>>>>>> type: internal
>>>>>>>>> Bridge br-ex
>>>>>>>>> Port br-ex
>>>>>>>>> Interface br-ex
>>>>>>>>> type: internal
>>>>>>>>> Port "qg-de1db666-3d"
>>>>>>>>> Interface "qg-de1db666-3d"
>>>>>>>>> type: internal
>>>>>>>>> Port "eth1"
>>>>>>>>> Interface "eth1"
>>>>>>>>> ovs_version: "1.4.0+build0"
>>>>>>>>> $ brctl show
>>>>>>>>> bridge name bridge id STP enabled interfaces
>>>>>>>>> br-ex 0000.ae454d673043 no qg-de1db666-3d
>>>>>>>>> br-int 0000.c6d8e9064848 no qr-6dd16ab9-d6
>>>>>>>>> qvo076911a5-59
>>>>>>>>> tap8c462a80-f6
>>>>>>>>> qbr076911a5-59 8000.26397c9d0d15 no
>>>>>>>>> qvb076911a5-59
>>>>>>>>> vnet0
>>>>>>>>> virbr0 8000.000000000000 yes
>>>>>>>>> virbr1 8000.525400d8a792 yes virbr1-nic
>>>>>>>>> ----------
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>>
>>>>>>>>> Kaneko
>>>>>>>>>
>>>>>>>>> 2012/9/19 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
>>>>>>>>>>
>>>>>>>>>> I re-added qvo to br-int manually, then it appeared in the output of
>>>>>>>>>> "brctl show".
>>>>>>>>>> And VM was able to obtain an IP address by DHCP.
>>>>>>>>>> What I did is as follows.
>>>>>>>>>> sudo ovs-vsctl del-port qvoXXXX
>>>>>>>>>> sudo ovs-vsctl add-port br-int qvoXXXX
>>>>>>>>>> sudo ovs-vsctl set port qvoXXXX tag=1
>>>>>>>>>>
>>>>>>>>>> I restarted devstack without re-add qvo, the problem was reproduced.
>>>>>>>>>> VM could not obtain IP address from DHCP server.
>>>>>>>>>>
>>>>>>>>>> And I re-added qvo manually again. In this time, I did a thing same as
>>>>>>>>>> vif-driver recorded in a syslog. Then, VM obtains an IP address by DHCP
>>>>>>>>>> again.
>>>>>>>>>>
>>>>>>>>>> sudo ovs-vsctl del-port qvoXXXX
>>>>>>>>>> sudo ovs-vsctl -- --may-exist add-port br-int qvoXXX -- set Interface
>>>>>>>>>> qvoXXXX
>>>>>>>>>> external-ids:iface-id=XXXX external-ids:iface-status=active
>>>>>>>>>> external-ids:attached-mac=XXXX external-ids:vm-uuid=XXXX
>>>>>>>>>> (VLAN tag was added automatically)
>>>>>>>>>>
>>>>>>>>>> What should I do next?
>>>>>>>>>>
>>>>>>>>>> Thanks,
>>>>>>>>>>
>>>>>>>>>> Kaneko
>>>>>>>>>>
>>>>>>>>>> 2012/9/18 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
>>>>>>>>>>>
>>>>>>>>>>> 2012/9/18 Dan Wendlandt <dan at nicira.com>:
>>>>>>>>>>>>
>>>>>>>>>>>> On Mon, Sep 17, 2012 at 11:12 PM, Yoshihiro Kaneko
>>>>>>>>>>>> <ykaneko0929 at gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks for the reply.
>>>>>>>>>>>>>
>>>>>>>>>>>>> This is the result of "brctl show".
>>>>>>>>>>>>> ----------
>>>>>>>>>>>>> $ brctl show
>>>>>>>>>>>>> bridge name bridge id STP enabled interfaces
>>>>>>>>>>>>> br-ex 0000.6e47647aee44 no
>>>>>>>>>>>>> qg-8c3a652b-f1
>>>>>>>>>>>>> br-int 0000.4af0ded34a40 no
>>>>>>>>>>>>> qr-f9a49be5-c1
>>>>>>>>>>>>>
>>>>>>>>>>>>> tap5a418ee0-53
>>>>>>>>>>>>> qbr99eea189-fc 8000.5e15f4c7d4f2 no
>>>>>>>>>>>>> qvb99eea189-fc
>>>>>>>>>>>>> vnet0
>>>>>>>>>>>>> virbr0 8000.000000000000 yes
>>>>>>>>>>>>> virbr1 8000.525400d8a792 yes virbr1-nic
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> can you send the output of
>>>>>>>>>>>>
>>>>>>>>>>>> ovs-vsctl list-ports br-int (or ovs-vsctl show)?
>>>>>>>>>>>>
>>>>>>>>>>>> Its odd that the above output does not include qvo99eea189-fc, which
>>>>>>>>>>>> should be attached to br-int.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> This is the output. I restarted devstack, so the interface name is
>>>>>>>>>>> different from above.
>>>>>>>>>>> There is not "qvo" in the output of "brctl show". But it exists in
>>>>>>>>>>> "ovs-vsctl list-ports br-int"
>>>>>>>>>>> (and "ovs-vsctl show"). And not found in "ovs-dpctl show br-int".
>>>>>>>>>>>
>>>>>>>>>>> ----------
>>>>>>>>>>> $ sudo brctl show
>>>>>>>>>>> bridge name bridge id STP enabled interfaces
>>>>>>>>>>> br-ex 0000.2a93351db24b no qg-669c94f9-41
>>>>>>>>>>> br-int 0000.363636554a4e no qr-185216c4-b7
>>>>>>>>>>> tap4061a128-21
>>>>>>>>>>> qbra5838945-25 8000.d639643f11ef no
>>>>>>>>>>> qvba5838945-25
>>>>>>>>>>> vnet0
>>>>>>>>>>> virbr0 8000.000000000000 yes
>>>>>>>>>>> virbr1 8000.525400d8a792 yes virbr1-nic
>>>>>>>>>>> $
>>>>>>>>>>>
>>>>>>>>>>> $ sudo ovs-vsctl list-ports br-int
>>>>>>>>>>> qr-185216c4-b7
>>>>>>>>>>> qvoa5838945-25
>>>>>>>>>>> tap4061a128-21
>>>>>>>>>>> $
>>>>>>>>>>> $ sudo ovs-dpctl show br-int
>>>>>>>>>>> system at br-int:
>>>>>>>>>>> lookups: hit:7 missed:14 lost:0
>>>>>>>>>>> flows: 0
>>>>>>>>>>> port 0: br-int (internal)
>>>>>>>>>>> Sep 18
>>>>>>>>>>> 16:35:45|00001|netdev_linux|WARN|/sys/class/net/tap4061a128-21/carrier:
>>>>>>>>>>> open failed: No such file or directory
>>>>>>>>>>> port 1: tap4061a128-21 (internal)
>>>>>>>>>>> Sep 18
>>>>>>>>>>> 16:35:45|00002|netdev_linux|WARN|/sys/class/net/qr-185216c4-b7/carrier:
>>>>>>>>>>> open failed: No such file or directory
>>>>>>>>>>> port 2: qr-185216c4-b7 (internal)
>>>>>>>>>>> ----------
>>>>>>>>>>>
>>>>>>>>>>> Kaneko
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> dan
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> ----------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Following is the result that was running tcpdump on each terminal
>>>>>>>>>>>>> concurrently.
>>>>>>>>>>>>> I don't know why each DHCP packet was received twice on vnet0. And
>>>>>>>>>>>>> the tap
>>>>>>>>>>>>> device used by dnsmasq did not receive udp packet in qdhcp-
>>>>>>>>>>>>> namespace.
>>>>>>>>>>>>> ----------
>>>>>>>>>>>>> # tcpdump -ni vnet0 udp
>>>>>>>>>>>>> tcpdump: WARNING: vnet0: no IPv4 address assigned
>>>>>>>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>>>>>>>>>> decode
>>>>>>>>>>>>> listening on vnet0, link-type EN10MB (Ethernet), capture size 65535
>>>>>>>>>>>>> bytes
>>>>>>>>>>>>> 12:00:56.619340 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>> 12:00:56.619423 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>> 12:00:59.624796 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>> 12:00:59.624858 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>> 12:01:02.630474 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>> 12:01:02.630564 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>>
>>>>>>>>>>>>> # tcpdump -ni qvb99eea189-fc udp
>>>>>>>>>>>>> tcpdump: WARNING: qvb99eea189-fc: no IPv4 address assigned
>>>>>>>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>>>>>>>>>> decode
>>>>>>>>>>>>> listening on qvb99eea189-fc, link-type EN10MB (Ethernet), capture
>>>>>>>>>>>>> size
>>>>>>>>>>>>> 65535 bytes
>>>>>>>>>>>>> 12:00:56.619436 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>> 12:00:59.624872 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>> 12:01:02.630586 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>>
>>>>>>>>>>>>> # tcpdump -ni qvo99eea189-fc udp
>>>>>>>>>>>>> tcpdump: WARNING: qvo99eea189-fc: no IPv4 address assigned
>>>>>>>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>>>>>>>>>> decode
>>>>>>>>>>>>> listening on qvo99eea189-fc, link-type EN10MB (Ethernet), capture
>>>>>>>>>>>>> size
>>>>>>>>>>>>> 65535 bytes
>>>>>>>>>>>>> 12:00:56.619467 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>> 12:00:59.624904 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>> 12:01:02.630633 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>>>>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>>>>>>>>
>>>>>>>>>>>>> # tcpdump -ni tap5a418ee0-53 udp
>>>>>>>>>>>>> tcpdump: tap5a418ee0-53: No such device exists
>>>>>>>>>>>>> (SIOCGIFHWADDR: No such device)
>>>>>>>>>>>>> # ip netns exec qdhcp-c95edcb8-3df6-4e92-a367-4fce5bce6e63 tcpdump
>>>>>>>>>>>>> -ni
>>>>>>>>>>>>> tap5a418ee0-53 udp
>>>>>>>>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol
>>>>>>>>>>>>> decode
>>>>>>>>>>>>> listening on tap5a418ee0-53, link-type EN10MB (Ethernet), capture
>>>>>>>>>>>>> size
>>>>>>>>>>>>> 65535 bytes
>>>>>>>>>>>>> ----------
>>>>>>>>>>>>>
>>>>>>>>>>>>> When using
>>>>>>>>>>>>> "libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver",
>>>>>>>>>>>>> VM can obtain an IP address by DHCP.
>>>>>>>>>>>>> ----------
>>>>>>>>>>>>> $ grep libvirt_vif_driver /etc/nova/nova.conf
>>>>>>>>>>>>> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
>>>>>>>>>>>>> $ nova console-log vm1
>>>>>>>>>>>>> <snip>
>>>>>>>>>>>>> Starting network...
>>>>>>>>>>>>> udhcpc (v1.18.5) started
>>>>>>>>>>>>> Sending discover...
>>>>>>>>>>>>> Sending select for 10.0.0.3...
>>>>>>>>>>>>> Lease of 10.0.0.3 obtained, lease time 120
>>>>>>>>>>>>> deleting routers
>>>>>>>>>>>>> route: SIOCDELRT: No such process
>>>>>>>>>>>>> adding dns 10.0.0.2
>>>>>>>>>>>>> <snip>
>>>>>>>>>>>>> ----------
>>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Kaneko
>>>>>>>>>>>>>
>>>>>>>>>>>>> 2012/9/17 Dan Wendlandt <dan at nicira.com>:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Kaneko,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks for the detailed report. At a glance, things look correct.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> I'd suggest running tcpdump on various interfaces to figure how far
>>>>>>>>>>>>>> the DHCP requests/responses are reaching.
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> tcpdump -ni vnetX udp (replace vnetX with correct device... can't
>>>>>>>>>>>>>> tell
>>>>>>>>>>>>>> from your output. try running brctl show).
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> tcpdump -ni qvb99eea189-fc udp (veth device on VIF-specific
>>>>>>>>>>>>>> bridge)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> tcpdump -ni qvo99eea189-fc udp (veth device on main OVS bridge)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> tcpdump -ni tap5a418ee0-53 udp (tap device used by dnsmasq)
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Another interesting data point would be whether you only see these
>>>>>>>>>>>>>> issues when using
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> "libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver",
>>>>>>>>>>>>>> or whether you also see it with
>>>>>>>>>>>>>> "libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Dan
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Fri, Sep 14, 2012 at 3:13 AM, Yoshihiro Kaneko
>>>>>>>>>>>>>> <ykaneko0929 at gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Dan,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I tried it on Ubuntu 12.04. However VM could not obtain an IP
>>>>>>>>>>>>>>> address from
>>>>>>>>>>>>>>> a DHCP server.
>>>>>>>>>>>>>>> Any advice?
>>>>>>>>>>>>>>> If more information is needed, please let me know.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> $ git clone git://github.com/openstack-dev/devstack
>>>>>>>>>>>>>>> Cloning into 'devstack'...
>>>>>>>>>>>>>>> remote: Counting objects: 6415, done.
>>>>>>>>>>>>>>> remote: Compressing objects: 100% (2186/2186), done.
>>>>>>>>>>>>>>> remote: Total 6415 (delta 4352), reused 6145 (delta 4151)
>>>>>>>>>>>>>>> Receiving objects: 100% (6415/6415), 1.10 MiB | 470 KiB/s, done.
>>>>>>>>>>>>>>> Resolving deltas: 100% (4352/4352), done.
>>>>>>>>>>>>>>> $ cd devstack/
>>>>>>>>>>>>>>> $ git fetch https://review.openstack.org/openstack-dev/devstack
>>>>>>>>>>>>>>> refs/changes/50/11650/8 && git checkout FETCH_HEAD
>>>>>>>>>>>>>>> remote: Counting objects: 5, done
>>>>>>>>>>>>>>> remote: Finding sources: 100% (3/3)
>>>>>>>>>>>>>>> remote: Total 3 (delta 2), reused 3 (delta 2)
>>>>>>>>>>>>>>> Unpacking objects: 100% (3/3), done.
>>>>>>>>>>>>>>> From https://review.openstack.org/openstack-dev/devstack
>>>>>>>>>>>>>>> * branch refs/changes/50/11650/8 -> FETCH_HEAD
>>>>>>>>>>>>>>> Note: checking out 'FETCH_HEAD'.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> You are in 'detached HEAD' state. You can look around, make
>>>>>>>>>>>>>>> experimental
>>>>>>>>>>>>>>> changes and commit them, and you can discard any commits you make
>>>>>>>>>>>>>>> in this
>>>>>>>>>>>>>>> state without impacting any branches by performing another
>>>>>>>>>>>>>>> checkout.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> If you want to create a new branch to retain commits you create,
>>>>>>>>>>>>>>> you may
>>>>>>>>>>>>>>> do so (now or later) by using -b with the checkout command again.
>>>>>>>>>>>>>>> Example:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> git checkout -b new_branch_name
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> HEAD is now at cea6c51... Quantum enhancements
>>>>>>>>>>>>>>> $ vi localrc
>>>>>>>>>>>>>>> $ cat localrc
>>>>>>>>>>>>>>> disable_service n-net
>>>>>>>>>>>>>>> enable_service q-svc q-agt q-dhcp q-l3 quantum
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> $ cp samples/local.sh .
>>>>>>>>>>>>>>> $ ./stack.sh
>>>>>>>>>>>>>>> <snip>
>>>>>>>>>>>>>>> $ . ./openrc demo demo
>>>>>>>>>>>>>>> $ nova image-list
>>>>>>>>>>>>>>> <snip>
>>>>>>>>>>>>>>> $ quantum net-list
>>>>>>>>>>>>>>> <snip>
>>>>>>>>>>>>>>> $ nova boot --flavor 6 --image 7777d0ed-294c-4634-9304-769f64a52c81
>>>>>>>>>>>>>>> --nic net-id=c95edcb8-3df6-4e92-a367-4fce5bce6e63 vm1
>>>>>>>>>>>>>>> <snip>
>>>>>>>>>>>>>>> $ nova list
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +--------------------------------------+------+--------+---------------+
>>>>>>>>>>>>>>> | ID | Name | Status | Networks
>>>>>>>>>>>>>>> |
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +--------------------------------------+------+--------+---------------+
>>>>>>>>>>>>>>> | 723d76c4-4dea-4fe4-a4d8-6ca8d08bb936 | vm1 | ACTIVE |
>>>>>>>>>>>>>>> net1=10.0.0.3 |
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> +--------------------------------------+------+--------+---------------+
>>>>>>>>>>>>>>> $ nova console-log vm1
>>>>>>>>>>>>>>> <snip>
>>>>>>>>>>>>>>> Starting network...
>>>>>>>>>>>>>>> udhcpc (v1.18.5) started
>>>>>>>>>>>>>>> Sending discover...
>>>>>>>>>>>>>>> Sending discover...
>>>>>>>>>>>>>>> Sending discover...
>>>>>>>>>>>>>>> No lease, failing
>>>>>>>>>>>>>>> WARN: /etc/rc3.d/S40-network failed
>>>>>>>>>>>>>>> <snip>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> $ sudo ovs-vsctl show
>>>>>>>>>>>>>>> 6d7929f6-8b09-41bc-b775-1ef50915eb7b
>>>>>>>>>>>>>>> Bridge br-ex
>>>>>>>>>>>>>>> Port "qg-8c3a652b-f1"
>>>>>>>>>>>>>>> Interface "qg-8c3a652b-f1"
>>>>>>>>>>>>>>> type: internal
>>>>>>>>>>>>>>> Port br-ex
>>>>>>>>>>>>>>> Interface br-ex
>>>>>>>>>>>>>>> type: internal
>>>>>>>>>>>>>>> Bridge br-int
>>>>>>>>>>>>>>> Port "tap5a418ee0-53"
>>>>>>>>>>>>>>> tag: 1
>>>>>>>>>>>>>>> Interface "tap5a418ee0-53"
>>>>>>>>>>>>>>> type: internal
>>>>>>>>>>>>>>> Port "qr-f9a49be5-c1"
>>>>>>>>>>>>>>> tag: 1
>>>>>>>>>>>>>>> Interface "qr-f9a49be5-c1"
>>>>>>>>>>>>>>> type: internal
>>>>>>>>>>>>>>> Port "qvo99eea189-fc"
>>>>>>>>>>>>>>> tag: 1
>>>>>>>>>>>>>>> Interface "qvo99eea189-fc"
>>>>>>>>>>>>>>> Port br-int
>>>>>>>>>>>>>>> Interface br-int
>>>>>>>>>>>>>>> type: internal
>>>>>>>>>>>>>>> ovs_version: "1.4.0+build0"
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> $ ip netns
>>>>>>>>>>>>>>> qrouter-aed60aac-288b-41ee-bc4a-9ed5a356a16d
>>>>>>>>>>>>>>> qdhcp-c95edcb8-3df6-4e92-a367-4fce5bce6e63
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> $ ps ax|grep dnsmasq
>>>>>>>>>>>>>>> 6107 ? S 0:00 dnsmasq --no-hosts --no-resolv
>>>>>>>>>>>>>>> --strict-order --bind-interfaces --interface=tap5a418ee0-53
>>>>>>>>>>>>>>> --except-interface=lo --domain=openstacklocal
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --pid-file=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/pid
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --dhcp-hostsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/host
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --dhcp-optsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/opts
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --dhcp-script=/opt/stack/quantum/bin/quantum-dhcp-agent-dnsmasq-lease-update
>>>>>>>>>>>>>>> --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s
>>>>>>>>>>>>>>> 6108 ? S 0:00 dnsmasq --no-hosts --no-resolv
>>>>>>>>>>>>>>> --strict-order --bind-interfaces --interface=tap5a418ee0-53
>>>>>>>>>>>>>>> --except-interface=lo --domain=openstacklocal
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --pid-file=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/pid
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --dhcp-hostsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/host
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --dhcp-optsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/opts
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --dhcp-script=/opt/stack/quantum/bin/quantum-dhcp-agent-dnsmasq-lease-update
>>>>>>>>>>>>>>> --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s
>>>>>>>>>>>>>>> 28913 pts/1 S+ 0:00 grep --color=auto dnsmasq
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>> Kaneko
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> 2012/9/14 Dan Wendlandt <dan at nicira.com>:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi quantum hackers,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> We're pushing a change to devstack to use a new vif-driver for
>>>>>>>>>>>>>>>> Quantum
>>>>>>>>>>>>>>>> with Open vSwitch (https://review.openstack.org/#/c/11650/). The
>>>>>>>>>>>>>>>> benefit of this driver is that it is compatible with Nova's
>>>>>>>>>>>>>>>> security
>>>>>>>>>>>>>>>> group filtering. This is "a good thing", since it more closely
>>>>>>>>>>>>>>>> maps
>>>>>>>>>>>>>>>> to how real users will deploy Quantum.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> However, this may catch developers by surprise who are suddenly
>>>>>>>>>>>>>>>> unable
>>>>>>>>>>>>>>>> to ping or SSH to instances because the security groups drop
>>>>>>>>>>>>>>>> traffic
>>>>>>>>>>>>>>>> by default.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Preferred method of dealing with this is to add the following
>>>>>>>>>>>>>>>> lines to
>>>>>>>>>>>>>>>> local.sh in your devstack directory, which open up your VMs for
>>>>>>>>>>>>>>>> ping
>>>>>>>>>>>>>>>> and SSH for the 'demo' user:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
>>>>>>>>>>>>>>>> nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Another work around is to disable nova security groups by adding
>>>>>>>>>>>>>>>> 'LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver' to
>>>>>>>>>>>>>>>> your localrc
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Dan
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>>>>> Dan Wendlandt
>>>>>>>>>>>>>>>> Nicira, Inc: www.nicira.com
>>>>>>>>>>>>>>>> twitter: danwendlandt
>>>>>>>>>>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> --
>>>>>>>>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>>> Dan Wendlandt
>>>>>>>>>>>>>> Nicira, Inc: www.nicira.com
>>>>>>>>>>>>>> twitter: danwendlandt
>>>>>>>>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> --
>>>>>>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>> Dan Wendlandt
>>>>>>>>>>>> Nicira, Inc: www.nicira.com
>>>>>>>>>>>> twitter: danwendlandt
>>>>>>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Akihiro MOTOKI <amotoki at gmail.com>
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>> _______________________________________________
>> OpenStack-dev mailing list
>> OpenStack-dev at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
> --
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
> Dan Wendlandt
> Nicira, Inc: www.nicira.com
> twitter: danwendlandt
> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list