[openstack-dev] [Quantum] security groups now enforced w/devstack

Akihiro MOTOKI amotoki at gmail.com
Thu Sep 20 05:51:24 UTC 2012


Hi Yoshihiro,

I have encountered the same problem with OVS plugin.
It usually fails, but sometimes succeeds. ("fail" means that qvo****
is listed in ovs-vsctl show and is not listed in brctl show /
ovs-ofctl show br-int)
I have not identified what is the difference between two cases.

btw, calling "add-port" with "type=internal" is not a valid solution
for the problem since qvo*** netdev is a peer of veth pairs.
If my understanding is correct, when "type=internal" is specified OVS
creates a network device with a name specified.

Thanks,
Akihiro

2012/9/20 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
> Has nobody encountered this problem? Only me?
> I will try this in other fresh environment.
>
> Kaneko
>
> 2012/9/19 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
>> I found that qvo was not "internal". Therefore I set "type=internal"
>> to qvo manually.
>> As a result, qvo appeared in the output of "brctl show" and VM was
>> able to obtain
>> IP address by DHCP.
>> Is this correct?
>>
>> Before set interface type
>> ----------
>> $ sudo ovs-vsctl show
>> 6d7929f6-8b09-41bc-b775-1ef50915eb7b
>>     Bridge br-int
>>         Port "tap8c462a80-f6"
>>             tag: 1
>>             Interface "tap8c462a80-f6"
>>                 type: internal
>>         Port "qvo076911a5-59"
>>             tag: 1
>>             Interface "qvo076911a5-59"
>>         Port br-int
>>             Interface br-int
>>                 type: internal
>>         Port "qr-6dd16ab9-d6"
>>             tag: 1
>>             Interface "qr-6dd16ab9-d6"
>>                 type: internal
>>     Bridge br-ex
>>         Port br-ex
>>             Interface br-ex
>>                 type: internal
>>         Port "qg-de1db666-3d"
>>             Interface "qg-de1db666-3d"
>>                 type: internal
>>         Port "eth1"
>>             Interface "eth1"
>>     ovs_version: "1.4.0+build0"
>> $ brctl show
>> bridge name     bridge id               STP enabled     interfaces
>> br-ex           0000.ae454d673043       no              qg-de1db666-3d
>> br-int          0000.c6d8e9064848       no              qr-6dd16ab9-d6
>>                                                         tap8c462a80-f6
>> qbr076911a5-59          8000.26397c9d0d15       no              qvb076911a5-59
>>                                                         vnet0
>> virbr0          8000.000000000000       yes
>> virbr1          8000.525400d8a792       yes             virbr1-nic
>> ----------
>>
>> Set interface type to qvo
>> ----------
>> $ sudo ovs-vsctl set Interface qvo076911a5-59 type=internal
>> $ sudo ovs-vsctl show
>> 6d7929f6-8b09-41bc-b775-1ef50915eb7b
>>     Bridge br-int
>>         Port "tap8c462a80-f6"
>>             tag: 1
>>             Interface "tap8c462a80-f6"
>>                 type: internal
>>         Port "qvo076911a5-59"
>>             tag: 1
>>             Interface "qvo076911a5-59"
>>                 type: internal
>>         Port br-int
>>             Interface br-int
>>                 type: internal
>>         Port "qr-6dd16ab9-d6"
>>             tag: 1
>>             Interface "qr-6dd16ab9-d6"
>>                 type: internal
>>     Bridge br-ex
>>         Port br-ex
>>             Interface br-ex
>>                 type: internal
>>         Port "qg-de1db666-3d"
>>             Interface "qg-de1db666-3d"
>>                 type: internal
>>         Port "eth1"
>>             Interface "eth1"
>>     ovs_version: "1.4.0+build0"
>> $ brctl show
>> bridge name     bridge id               STP enabled     interfaces
>> br-ex           0000.ae454d673043       no              qg-de1db666-3d
>> br-int          0000.c6d8e9064848       no              qr-6dd16ab9-d6
>>                                                         qvo076911a5-59
>>                                                         tap8c462a80-f6
>> qbr076911a5-59          8000.26397c9d0d15       no              qvb076911a5-59
>>                                                         vnet0
>> virbr0          8000.000000000000       yes
>> virbr1          8000.525400d8a792       yes             virbr1-nic
>> ----------
>>
>> Thanks,
>>
>> Kaneko
>>
>> 2012/9/19 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
>>> I re-added qvo to br-int manually, then it appeared in the output of
>>> "brctl show".
>>> And VM was able to obtain an IP address by DHCP.
>>> What I did is as follows.
>>> sudo ovs-vsctl del-port qvoXXXX
>>> sudo ovs-vsctl add-port br-int qvoXXXX
>>> sudo ovs-vsctl set port qvoXXXX tag=1
>>>
>>> I restarted devstack without re-add qvo, the problem was reproduced.
>>> VM could not obtain IP address from DHCP server.
>>>
>>> And I re-added qvo manually again. In this time, I did a thing same as
>>> vif-driver recorded in a syslog. Then, VM obtains an IP address by DHCP
>>> again.
>>>
>>> sudo ovs-vsctl del-port qvoXXXX
>>> sudo ovs-vsctl -- --may-exist add-port br-int qvoXXX -- set Interface qvoXXXX
>>> external-ids:iface-id=XXXX external-ids:iface-status=active
>>> external-ids:attached-mac=XXXX external-ids:vm-uuid=XXXX
>>> (VLAN tag was added automatically)
>>>
>>> What should I do next?
>>>
>>> Thanks,
>>>
>>> Kaneko
>>>
>>> 2012/9/18 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
>>>> 2012/9/18 Dan Wendlandt <dan at nicira.com>:
>>>>> On Mon, Sep 17, 2012 at 11:12 PM, Yoshihiro Kaneko
>>>>> <ykaneko0929 at gmail.com> wrote:
>>>>>> Hi Dan,
>>>>>>
>>>>>> Thanks for the reply.
>>>>>>
>>>>>> This is the result of "brctl show".
>>>>>> ----------
>>>>>> $ brctl show
>>>>>> bridge name     bridge id               STP enabled     interfaces
>>>>>> br-ex           0000.6e47647aee44       no              qg-8c3a652b-f1
>>>>>> br-int          0000.4af0ded34a40       no              qr-f9a49be5-c1
>>>>>>                                                         tap5a418ee0-53
>>>>>> qbr99eea189-fc          8000.5e15f4c7d4f2       no              qvb99eea189-fc
>>>>>>                                                         vnet0
>>>>>> virbr0          8000.000000000000       yes
>>>>>> virbr1          8000.525400d8a792       yes             virbr1-nic
>>>>>
>>>>> can you send the output of
>>>>>
>>>>> ovs-vsctl list-ports br-int (or ovs-vsctl show)?
>>>>>
>>>>> Its odd that the above output does not include qvo99eea189-fc, which
>>>>> should be attached to br-int.
>>>>
>>>> This is the output. I restarted devstack, so the interface name is
>>>> different from above.
>>>> There is not "qvo" in the output of "brctl show". But it exists in
>>>> "ovs-vsctl list-ports br-int"
>>>> (and "ovs-vsctl show"). And not found in "ovs-dpctl show br-int".
>>>>
>>>> ----------
>>>> $ sudo brctl show
>>>> bridge name     bridge id               STP enabled     interfaces
>>>> br-ex           0000.2a93351db24b       no              qg-669c94f9-41
>>>> br-int          0000.363636554a4e       no              qr-185216c4-b7
>>>>                                                         tap4061a128-21
>>>> qbra5838945-25          8000.d639643f11ef       no              qvba5838945-25
>>>>                                                         vnet0
>>>> virbr0          8000.000000000000       yes
>>>> virbr1          8000.525400d8a792       yes             virbr1-nic
>>>> $
>>>>
>>>> $ sudo ovs-vsctl list-ports br-int
>>>> qr-185216c4-b7
>>>> qvoa5838945-25
>>>> tap4061a128-21
>>>> $
>>>> $ sudo ovs-dpctl show br-int
>>>> system at br-int:
>>>>         lookups: hit:7 missed:14 lost:0
>>>>         flows: 0
>>>>         port 0: br-int (internal)
>>>> Sep 18 16:35:45|00001|netdev_linux|WARN|/sys/class/net/tap4061a128-21/carrier:
>>>> open failed: No such file or directory
>>>>         port 1: tap4061a128-21 (internal)
>>>> Sep 18 16:35:45|00002|netdev_linux|WARN|/sys/class/net/qr-185216c4-b7/carrier:
>>>> open failed: No such file or directory
>>>>         port 2: qr-185216c4-b7 (internal)
>>>> ----------
>>>>
>>>> Kaneko
>>>>
>>>>>
>>>>> dan
>>>>>
>>>>>
>>>>>
>>>>>> ----------
>>>>>>
>>>>>> Following is the result that was running tcpdump on each terminal concurrently.
>>>>>> I don't know why each DHCP packet was received twice on vnet0. And the tap
>>>>>> device used by dnsmasq did not receive udp packet in qdhcp- namespace.
>>>>>> ----------
>>>>>> # tcpdump -ni vnet0 udp
>>>>>> tcpdump: WARNING: vnet0: no IPv4 address assigned
>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>>>>> listening on vnet0, link-type EN10MB (Ethernet), capture size 65535 bytes
>>>>>> 12:00:56.619340 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>> 12:00:56.619423 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>> 12:00:59.624796 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>> 12:00:59.624858 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>> 12:01:02.630474 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>> 12:01:02.630564 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>
>>>>>> # tcpdump -ni qvb99eea189-fc udp
>>>>>> tcpdump: WARNING: qvb99eea189-fc: no IPv4 address assigned
>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>>>>> listening on qvb99eea189-fc, link-type EN10MB (Ethernet), capture size
>>>>>> 65535 bytes
>>>>>> 12:00:56.619436 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>> 12:00:59.624872 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>> 12:01:02.630586 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>
>>>>>> # tcpdump -ni qvo99eea189-fc udp
>>>>>> tcpdump: WARNING: qvo99eea189-fc: no IPv4 address assigned
>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>>>>> listening on qvo99eea189-fc, link-type EN10MB (Ethernet), capture size
>>>>>> 65535 bytes
>>>>>> 12:00:56.619467 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>> 12:00:59.624904 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>> 12:01:02.630633 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>>>
>>>>>> # tcpdump -ni tap5a418ee0-53 udp
>>>>>> tcpdump: tap5a418ee0-53: No such device exists
>>>>>> (SIOCGIFHWADDR: No such device)
>>>>>> # ip netns exec qdhcp-c95edcb8-3df6-4e92-a367-4fce5bce6e63 tcpdump -ni
>>>>>> tap5a418ee0-53 udp
>>>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>>>>> listening on tap5a418ee0-53, link-type EN10MB (Ethernet), capture size
>>>>>> 65535 bytes
>>>>>> ----------
>>>>>>
>>>>>> When using "libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver",
>>>>>> VM can obtain an IP address by DHCP.
>>>>>> ----------
>>>>>> $ grep libvirt_vif_driver /etc/nova/nova.conf
>>>>>> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
>>>>>> $ nova console-log vm1
>>>>>> <snip>
>>>>>> Starting network...
>>>>>> udhcpc (v1.18.5) started
>>>>>> Sending discover...
>>>>>> Sending select for 10.0.0.3...
>>>>>> Lease of 10.0.0.3 obtained, lease time 120
>>>>>> deleting routers
>>>>>> route: SIOCDELRT: No such process
>>>>>> adding dns 10.0.0.2
>>>>>> <snip>
>>>>>> ----------
>>>>>>
>>>>>> Thanks,
>>>>>>
>>>>>> Kaneko
>>>>>>
>>>>>> 2012/9/17 Dan Wendlandt <dan at nicira.com>:
>>>>>>> Hi Kaneko,
>>>>>>>
>>>>>>> Thanks for the detailed report.  At a glance, things look correct.
>>>>>>>
>>>>>>> I'd suggest running tcpdump on various interfaces to figure how far
>>>>>>> the DHCP requests/responses are reaching.
>>>>>>>
>>>>>>> tcpdump -ni vnetX udp (replace vnetX with correct device... can't tell
>>>>>>> from your output.  try running brctl show).
>>>>>>>
>>>>>>> tcpdump -ni qvb99eea189-fc udp   (veth device on VIF-specific bridge)
>>>>>>>
>>>>>>> tcpdump -ni qvo99eea189-fc udp   (veth device on main OVS bridge)
>>>>>>>
>>>>>>> tcpdump -ni tap5a418ee0-53 udp (tap device used by dnsmasq)
>>>>>>>
>>>>>>> Another interesting data point would be whether you only see these
>>>>>>> issues when using
>>>>>>> "libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver",
>>>>>>> or whether you also see it with
>>>>>>> "libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Dan
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Sep 14, 2012 at 3:13 AM, Yoshihiro Kaneko <ykaneko0929 at gmail.com> wrote:
>>>>>>>> Hi Dan,
>>>>>>>>
>>>>>>>> I tried it on Ubuntu 12.04. However VM could not obtain an IP address from
>>>>>>>> a DHCP server.
>>>>>>>> Any advice?
>>>>>>>> If more information is needed, please let me know.
>>>>>>>>
>>>>>>>> $ git clone git://github.com/openstack-dev/devstack
>>>>>>>> Cloning into 'devstack'...
>>>>>>>> remote: Counting objects: 6415, done.
>>>>>>>> remote: Compressing objects: 100% (2186/2186), done.
>>>>>>>> remote: Total 6415 (delta 4352), reused 6145 (delta 4151)
>>>>>>>> Receiving objects: 100% (6415/6415), 1.10 MiB | 470 KiB/s, done.
>>>>>>>> Resolving deltas: 100% (4352/4352), done.
>>>>>>>> $ cd devstack/
>>>>>>>> $ git fetch https://review.openstack.org/openstack-dev/devstack
>>>>>>>> refs/changes/50/11650/8 && git checkout FETCH_HEAD
>>>>>>>> remote: Counting objects: 5, done
>>>>>>>> remote: Finding sources: 100% (3/3)
>>>>>>>> remote: Total 3 (delta 2), reused 3 (delta 2)
>>>>>>>> Unpacking objects: 100% (3/3), done.
>>>>>>>> From https://review.openstack.org/openstack-dev/devstack
>>>>>>>>  * branch            refs/changes/50/11650/8 -> FETCH_HEAD
>>>>>>>> Note: checking out 'FETCH_HEAD'.
>>>>>>>>
>>>>>>>> You are in 'detached HEAD' state. You can look around, make experimental
>>>>>>>> changes and commit them, and you can discard any commits you make in this
>>>>>>>> state without impacting any branches by performing another checkout.
>>>>>>>>
>>>>>>>> If you want to create a new branch to retain commits you create, you may
>>>>>>>> do so (now or later) by using -b with the checkout command again. Example:
>>>>>>>>
>>>>>>>>   git checkout -b new_branch_name
>>>>>>>>
>>>>>>>> HEAD is now at cea6c51... Quantum enhancements
>>>>>>>> $ vi localrc
>>>>>>>> $ cat localrc
>>>>>>>> disable_service n-net
>>>>>>>> enable_service q-svc q-agt q-dhcp q-l3 quantum
>>>>>>>>
>>>>>>>> $ cp samples/local.sh .
>>>>>>>> $ ./stack.sh
>>>>>>>> <snip>
>>>>>>>> $ . ./openrc demo demo
>>>>>>>> $ nova image-list
>>>>>>>> <snip>
>>>>>>>> $ quantum net-list
>>>>>>>> <snip>
>>>>>>>> $ nova boot --flavor 6 --image 7777d0ed-294c-4634-9304-769f64a52c81
>>>>>>>> --nic net-id=c95edcb8-3df6-4e92-a367-4fce5bce6e63 vm1
>>>>>>>> <snip>
>>>>>>>> $ nova list
>>>>>>>> +--------------------------------------+------+--------+---------------+
>>>>>>>> | ID                                   | Name | Status | Networks      |
>>>>>>>> +--------------------------------------+------+--------+---------------+
>>>>>>>> | 723d76c4-4dea-4fe4-a4d8-6ca8d08bb936 | vm1  | ACTIVE | net1=10.0.0.3 |
>>>>>>>> +--------------------------------------+------+--------+---------------+
>>>>>>>> $ nova console-log vm1
>>>>>>>> <snip>
>>>>>>>> Starting network...
>>>>>>>> udhcpc (v1.18.5) started
>>>>>>>> Sending discover...
>>>>>>>> Sending discover...
>>>>>>>> Sending discover...
>>>>>>>> No lease, failing
>>>>>>>> WARN: /etc/rc3.d/S40-network failed
>>>>>>>> <snip>
>>>>>>>>
>>>>>>>> $ sudo ovs-vsctl show
>>>>>>>> 6d7929f6-8b09-41bc-b775-1ef50915eb7b
>>>>>>>>     Bridge br-ex
>>>>>>>>         Port "qg-8c3a652b-f1"
>>>>>>>>             Interface "qg-8c3a652b-f1"
>>>>>>>>                 type: internal
>>>>>>>>         Port br-ex
>>>>>>>>             Interface br-ex
>>>>>>>>                 type: internal
>>>>>>>>     Bridge br-int
>>>>>>>>         Port "tap5a418ee0-53"
>>>>>>>>             tag: 1
>>>>>>>>             Interface "tap5a418ee0-53"
>>>>>>>>                 type: internal
>>>>>>>>         Port "qr-f9a49be5-c1"
>>>>>>>>             tag: 1
>>>>>>>>             Interface "qr-f9a49be5-c1"
>>>>>>>>                 type: internal
>>>>>>>>         Port "qvo99eea189-fc"
>>>>>>>>             tag: 1
>>>>>>>>             Interface "qvo99eea189-fc"
>>>>>>>>         Port br-int
>>>>>>>>             Interface br-int
>>>>>>>>                 type: internal
>>>>>>>>     ovs_version: "1.4.0+build0"
>>>>>>>>
>>>>>>>> $ ip netns
>>>>>>>> qrouter-aed60aac-288b-41ee-bc4a-9ed5a356a16d
>>>>>>>> qdhcp-c95edcb8-3df6-4e92-a367-4fce5bce6e63
>>>>>>>>
>>>>>>>> $ ps ax|grep dnsmasq
>>>>>>>>  6107 ?        S      0:00 dnsmasq --no-hosts --no-resolv
>>>>>>>> --strict-order --bind-interfaces --interface=tap5a418ee0-53
>>>>>>>> --except-interface=lo --domain=openstacklocal
>>>>>>>> --pid-file=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/pid
>>>>>>>> --dhcp-hostsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/host
>>>>>>>> --dhcp-optsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/opts
>>>>>>>> --dhcp-script=/opt/stack/quantum/bin/quantum-dhcp-agent-dnsmasq-lease-update
>>>>>>>> --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s
>>>>>>>>  6108 ?        S      0:00 dnsmasq --no-hosts --no-resolv
>>>>>>>> --strict-order --bind-interfaces --interface=tap5a418ee0-53
>>>>>>>> --except-interface=lo --domain=openstacklocal
>>>>>>>> --pid-file=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/pid
>>>>>>>> --dhcp-hostsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/host
>>>>>>>> --dhcp-optsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/opts
>>>>>>>> --dhcp-script=/opt/stack/quantum/bin/quantum-dhcp-agent-dnsmasq-lease-update
>>>>>>>> --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s
>>>>>>>> 28913 pts/1    S+     0:00 grep --color=auto dnsmasq
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Kaneko
>>>>>>>>
>>>>>>>> 2012/9/14 Dan Wendlandt <dan at nicira.com>:
>>>>>>>>> Hi quantum hackers,
>>>>>>>>>
>>>>>>>>> We're pushing a change to devstack to use a new vif-driver for Quantum
>>>>>>>>> with Open vSwitch (https://review.openstack.org/#/c/11650/).  The
>>>>>>>>> benefit of this driver is that it is compatible with Nova's security
>>>>>>>>> group filtering.  This is "a good thing", since it more closely maps
>>>>>>>>> to how real users will deploy Quantum.
>>>>>>>>>
>>>>>>>>> However, this may catch developers by surprise who are suddenly unable
>>>>>>>>> to ping or SSH to instances because the security groups drop traffic
>>>>>>>>> by default.
>>>>>>>>>
>>>>>>>>> Preferred method of dealing with this is to add the following lines to
>>>>>>>>> local.sh in your devstack directory, which open up your VMs for ping
>>>>>>>>> and SSH for the 'demo' user:
>>>>>>>>>
>>>>>>>>> nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
>>>>>>>>> nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
>>>>>>>>>
>>>>>>>>> Another work around is to disable nova security groups by adding
>>>>>>>>> 'LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver' to
>>>>>>>>> your localrc
>>>>>>>>>
>>>>>>>>> Dan
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>> Dan Wendlandt
>>>>>>>>> Nicira, Inc: www.nicira.com
>>>>>>>>> twitter: danwendlandt
>>>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> OpenStack-dev mailing list
>>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> OpenStack-dev mailing list
>>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>> Dan Wendlandt
>>>>>>> Nicira, Inc: www.nicira.com
>>>>>>> twitter: danwendlandt
>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>> Dan Wendlandt
>>>>> Nicira, Inc: www.nicira.com
>>>>> twitter: danwendlandt
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



-- 
Akihiro MOTOKI <amotoki at gmail.com>



More information about the OpenStack-dev mailing list