[openstack-dev] [Quantum] security groups now enforced w/devstack
Yoshihiro Kaneko
ykaneko0929 at gmail.com
Wed Sep 19 12:03:58 UTC 2012
I found that qvo was not "internal". Therefore I set "type=internal"
to qvo manually.
As a result, qvo appeared in the output of "brctl show" and VM was
able to obtain
IP address by DHCP.
Is this correct?
Before set interface type
----------
$ sudo ovs-vsctl show
6d7929f6-8b09-41bc-b775-1ef50915eb7b
Bridge br-int
Port "tap8c462a80-f6"
tag: 1
Interface "tap8c462a80-f6"
type: internal
Port "qvo076911a5-59"
tag: 1
Interface "qvo076911a5-59"
Port br-int
Interface br-int
type: internal
Port "qr-6dd16ab9-d6"
tag: 1
Interface "qr-6dd16ab9-d6"
type: internal
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port "qg-de1db666-3d"
Interface "qg-de1db666-3d"
type: internal
Port "eth1"
Interface "eth1"
ovs_version: "1.4.0+build0"
$ brctl show
bridge name bridge id STP enabled interfaces
br-ex 0000.ae454d673043 no qg-de1db666-3d
br-int 0000.c6d8e9064848 no qr-6dd16ab9-d6
tap8c462a80-f6
qbr076911a5-59 8000.26397c9d0d15 no qvb076911a5-59
vnet0
virbr0 8000.000000000000 yes
virbr1 8000.525400d8a792 yes virbr1-nic
----------
Set interface type to qvo
----------
$ sudo ovs-vsctl set Interface qvo076911a5-59 type=internal
$ sudo ovs-vsctl show
6d7929f6-8b09-41bc-b775-1ef50915eb7b
Bridge br-int
Port "tap8c462a80-f6"
tag: 1
Interface "tap8c462a80-f6"
type: internal
Port "qvo076911a5-59"
tag: 1
Interface "qvo076911a5-59"
type: internal
Port br-int
Interface br-int
type: internal
Port "qr-6dd16ab9-d6"
tag: 1
Interface "qr-6dd16ab9-d6"
type: internal
Bridge br-ex
Port br-ex
Interface br-ex
type: internal
Port "qg-de1db666-3d"
Interface "qg-de1db666-3d"
type: internal
Port "eth1"
Interface "eth1"
ovs_version: "1.4.0+build0"
$ brctl show
bridge name bridge id STP enabled interfaces
br-ex 0000.ae454d673043 no qg-de1db666-3d
br-int 0000.c6d8e9064848 no qr-6dd16ab9-d6
qvo076911a5-59
tap8c462a80-f6
qbr076911a5-59 8000.26397c9d0d15 no qvb076911a5-59
vnet0
virbr0 8000.000000000000 yes
virbr1 8000.525400d8a792 yes virbr1-nic
----------
Thanks,
Kaneko
2012/9/19 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
> I re-added qvo to br-int manually, then it appeared in the output of
> "brctl show".
> And VM was able to obtain an IP address by DHCP.
> What I did is as follows.
> sudo ovs-vsctl del-port qvoXXXX
> sudo ovs-vsctl add-port br-int qvoXXXX
> sudo ovs-vsctl set port qvoXXXX tag=1
>
> I restarted devstack without re-add qvo, the problem was reproduced.
> VM could not obtain IP address from DHCP server.
>
> And I re-added qvo manually again. In this time, I did a thing same as
> vif-driver recorded in a syslog. Then, VM obtains an IP address by DHCP
> again.
>
> sudo ovs-vsctl del-port qvoXXXX
> sudo ovs-vsctl -- --may-exist add-port br-int qvoXXX -- set Interface qvoXXXX
> external-ids:iface-id=XXXX external-ids:iface-status=active
> external-ids:attached-mac=XXXX external-ids:vm-uuid=XXXX
> (VLAN tag was added automatically)
>
> What should I do next?
>
> Thanks,
>
> Kaneko
>
> 2012/9/18 Yoshihiro Kaneko <ykaneko0929 at gmail.com>:
>> 2012/9/18 Dan Wendlandt <dan at nicira.com>:
>>> On Mon, Sep 17, 2012 at 11:12 PM, Yoshihiro Kaneko
>>> <ykaneko0929 at gmail.com> wrote:
>>>> Hi Dan,
>>>>
>>>> Thanks for the reply.
>>>>
>>>> This is the result of "brctl show".
>>>> ----------
>>>> $ brctl show
>>>> bridge name bridge id STP enabled interfaces
>>>> br-ex 0000.6e47647aee44 no qg-8c3a652b-f1
>>>> br-int 0000.4af0ded34a40 no qr-f9a49be5-c1
>>>> tap5a418ee0-53
>>>> qbr99eea189-fc 8000.5e15f4c7d4f2 no qvb99eea189-fc
>>>> vnet0
>>>> virbr0 8000.000000000000 yes
>>>> virbr1 8000.525400d8a792 yes virbr1-nic
>>>
>>> can you send the output of
>>>
>>> ovs-vsctl list-ports br-int (or ovs-vsctl show)?
>>>
>>> Its odd that the above output does not include qvo99eea189-fc, which
>>> should be attached to br-int.
>>
>> This is the output. I restarted devstack, so the interface name is
>> different from above.
>> There is not "qvo" in the output of "brctl show". But it exists in
>> "ovs-vsctl list-ports br-int"
>> (and "ovs-vsctl show"). And not found in "ovs-dpctl show br-int".
>>
>> ----------
>> $ sudo brctl show
>> bridge name bridge id STP enabled interfaces
>> br-ex 0000.2a93351db24b no qg-669c94f9-41
>> br-int 0000.363636554a4e no qr-185216c4-b7
>> tap4061a128-21
>> qbra5838945-25 8000.d639643f11ef no qvba5838945-25
>> vnet0
>> virbr0 8000.000000000000 yes
>> virbr1 8000.525400d8a792 yes virbr1-nic
>> $
>>
>> $ sudo ovs-vsctl list-ports br-int
>> qr-185216c4-b7
>> qvoa5838945-25
>> tap4061a128-21
>> $
>> $ sudo ovs-dpctl show br-int
>> system at br-int:
>> lookups: hit:7 missed:14 lost:0
>> flows: 0
>> port 0: br-int (internal)
>> Sep 18 16:35:45|00001|netdev_linux|WARN|/sys/class/net/tap4061a128-21/carrier:
>> open failed: No such file or directory
>> port 1: tap4061a128-21 (internal)
>> Sep 18 16:35:45|00002|netdev_linux|WARN|/sys/class/net/qr-185216c4-b7/carrier:
>> open failed: No such file or directory
>> port 2: qr-185216c4-b7 (internal)
>> ----------
>>
>> Kaneko
>>
>>>
>>> dan
>>>
>>>
>>>
>>>> ----------
>>>>
>>>> Following is the result that was running tcpdump on each terminal concurrently.
>>>> I don't know why each DHCP packet was received twice on vnet0. And the tap
>>>> device used by dnsmasq did not receive udp packet in qdhcp- namespace.
>>>> ----------
>>>> # tcpdump -ni vnet0 udp
>>>> tcpdump: WARNING: vnet0: no IPv4 address assigned
>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>>> listening on vnet0, link-type EN10MB (Ethernet), capture size 65535 bytes
>>>> 12:00:56.619340 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>> 12:00:56.619423 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>> 12:00:59.624796 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>> 12:00:59.624858 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>> 12:01:02.630474 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>> 12:01:02.630564 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>
>>>> # tcpdump -ni qvb99eea189-fc udp
>>>> tcpdump: WARNING: qvb99eea189-fc: no IPv4 address assigned
>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>>> listening on qvb99eea189-fc, link-type EN10MB (Ethernet), capture size
>>>> 65535 bytes
>>>> 12:00:56.619436 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>> 12:00:59.624872 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>> 12:01:02.630586 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>
>>>> # tcpdump -ni qvo99eea189-fc udp
>>>> tcpdump: WARNING: qvo99eea189-fc: no IPv4 address assigned
>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>>> listening on qvo99eea189-fc, link-type EN10MB (Ethernet), capture size
>>>> 65535 bytes
>>>> 12:00:56.619467 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>> 12:00:59.624904 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>> 12:01:02.630633 IP 0.0.0.0.68 > 255.255.255.255.67: BOOTP/DHCP,
>>>> Request from fa:16:3e:48:aa:85, length 280
>>>>
>>>> # tcpdump -ni tap5a418ee0-53 udp
>>>> tcpdump: tap5a418ee0-53: No such device exists
>>>> (SIOCGIFHWADDR: No such device)
>>>> # ip netns exec qdhcp-c95edcb8-3df6-4e92-a367-4fce5bce6e63 tcpdump -ni
>>>> tap5a418ee0-53 udp
>>>> tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
>>>> listening on tap5a418ee0-53, link-type EN10MB (Ethernet), capture size
>>>> 65535 bytes
>>>> ----------
>>>>
>>>> When using "libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver",
>>>> VM can obtain an IP address by DHCP.
>>>> ----------
>>>> $ grep libvirt_vif_driver /etc/nova/nova.conf
>>>> libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
>>>> $ nova console-log vm1
>>>> <snip>
>>>> Starting network...
>>>> udhcpc (v1.18.5) started
>>>> Sending discover...
>>>> Sending select for 10.0.0.3...
>>>> Lease of 10.0.0.3 obtained, lease time 120
>>>> deleting routers
>>>> route: SIOCDELRT: No such process
>>>> adding dns 10.0.0.2
>>>> <snip>
>>>> ----------
>>>>
>>>> Thanks,
>>>>
>>>> Kaneko
>>>>
>>>> 2012/9/17 Dan Wendlandt <dan at nicira.com>:
>>>>> Hi Kaneko,
>>>>>
>>>>> Thanks for the detailed report. At a glance, things look correct.
>>>>>
>>>>> I'd suggest running tcpdump on various interfaces to figure how far
>>>>> the DHCP requests/responses are reaching.
>>>>>
>>>>> tcpdump -ni vnetX udp (replace vnetX with correct device... can't tell
>>>>> from your output. try running brctl show).
>>>>>
>>>>> tcpdump -ni qvb99eea189-fc udp (veth device on VIF-specific bridge)
>>>>>
>>>>> tcpdump -ni qvo99eea189-fc udp (veth device on main OVS bridge)
>>>>>
>>>>> tcpdump -ni tap5a418ee0-53 udp (tap device used by dnsmasq)
>>>>>
>>>>> Another interesting data point would be whether you only see these
>>>>> issues when using
>>>>> "libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtHybridOVSBridgeDriver",
>>>>> or whether you also see it with
>>>>> "libvirt_vif_driver=nova.virt.libvirt.vif.LibvirtOpenVswitchDriver
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Dan
>>>>>
>>>>>
>>>>> On Fri, Sep 14, 2012 at 3:13 AM, Yoshihiro Kaneko <ykaneko0929 at gmail.com> wrote:
>>>>>> Hi Dan,
>>>>>>
>>>>>> I tried it on Ubuntu 12.04. However VM could not obtain an IP address from
>>>>>> a DHCP server.
>>>>>> Any advice?
>>>>>> If more information is needed, please let me know.
>>>>>>
>>>>>> $ git clone git://github.com/openstack-dev/devstack
>>>>>> Cloning into 'devstack'...
>>>>>> remote: Counting objects: 6415, done.
>>>>>> remote: Compressing objects: 100% (2186/2186), done.
>>>>>> remote: Total 6415 (delta 4352), reused 6145 (delta 4151)
>>>>>> Receiving objects: 100% (6415/6415), 1.10 MiB | 470 KiB/s, done.
>>>>>> Resolving deltas: 100% (4352/4352), done.
>>>>>> $ cd devstack/
>>>>>> $ git fetch https://review.openstack.org/openstack-dev/devstack
>>>>>> refs/changes/50/11650/8 && git checkout FETCH_HEAD
>>>>>> remote: Counting objects: 5, done
>>>>>> remote: Finding sources: 100% (3/3)
>>>>>> remote: Total 3 (delta 2), reused 3 (delta 2)
>>>>>> Unpacking objects: 100% (3/3), done.
>>>>>> From https://review.openstack.org/openstack-dev/devstack
>>>>>> * branch refs/changes/50/11650/8 -> FETCH_HEAD
>>>>>> Note: checking out 'FETCH_HEAD'.
>>>>>>
>>>>>> You are in 'detached HEAD' state. You can look around, make experimental
>>>>>> changes and commit them, and you can discard any commits you make in this
>>>>>> state without impacting any branches by performing another checkout.
>>>>>>
>>>>>> If you want to create a new branch to retain commits you create, you may
>>>>>> do so (now or later) by using -b with the checkout command again. Example:
>>>>>>
>>>>>> git checkout -b new_branch_name
>>>>>>
>>>>>> HEAD is now at cea6c51... Quantum enhancements
>>>>>> $ vi localrc
>>>>>> $ cat localrc
>>>>>> disable_service n-net
>>>>>> enable_service q-svc q-agt q-dhcp q-l3 quantum
>>>>>>
>>>>>> $ cp samples/local.sh .
>>>>>> $ ./stack.sh
>>>>>> <snip>
>>>>>> $ . ./openrc demo demo
>>>>>> $ nova image-list
>>>>>> <snip>
>>>>>> $ quantum net-list
>>>>>> <snip>
>>>>>> $ nova boot --flavor 6 --image 7777d0ed-294c-4634-9304-769f64a52c81
>>>>>> --nic net-id=c95edcb8-3df6-4e92-a367-4fce5bce6e63 vm1
>>>>>> <snip>
>>>>>> $ nova list
>>>>>> +--------------------------------------+------+--------+---------------+
>>>>>> | ID | Name | Status | Networks |
>>>>>> +--------------------------------------+------+--------+---------------+
>>>>>> | 723d76c4-4dea-4fe4-a4d8-6ca8d08bb936 | vm1 | ACTIVE | net1=10.0.0.3 |
>>>>>> +--------------------------------------+------+--------+---------------+
>>>>>> $ nova console-log vm1
>>>>>> <snip>
>>>>>> Starting network...
>>>>>> udhcpc (v1.18.5) started
>>>>>> Sending discover...
>>>>>> Sending discover...
>>>>>> Sending discover...
>>>>>> No lease, failing
>>>>>> WARN: /etc/rc3.d/S40-network failed
>>>>>> <snip>
>>>>>>
>>>>>> $ sudo ovs-vsctl show
>>>>>> 6d7929f6-8b09-41bc-b775-1ef50915eb7b
>>>>>> Bridge br-ex
>>>>>> Port "qg-8c3a652b-f1"
>>>>>> Interface "qg-8c3a652b-f1"
>>>>>> type: internal
>>>>>> Port br-ex
>>>>>> Interface br-ex
>>>>>> type: internal
>>>>>> Bridge br-int
>>>>>> Port "tap5a418ee0-53"
>>>>>> tag: 1
>>>>>> Interface "tap5a418ee0-53"
>>>>>> type: internal
>>>>>> Port "qr-f9a49be5-c1"
>>>>>> tag: 1
>>>>>> Interface "qr-f9a49be5-c1"
>>>>>> type: internal
>>>>>> Port "qvo99eea189-fc"
>>>>>> tag: 1
>>>>>> Interface "qvo99eea189-fc"
>>>>>> Port br-int
>>>>>> Interface br-int
>>>>>> type: internal
>>>>>> ovs_version: "1.4.0+build0"
>>>>>>
>>>>>> $ ip netns
>>>>>> qrouter-aed60aac-288b-41ee-bc4a-9ed5a356a16d
>>>>>> qdhcp-c95edcb8-3df6-4e92-a367-4fce5bce6e63
>>>>>>
>>>>>> $ ps ax|grep dnsmasq
>>>>>> 6107 ? S 0:00 dnsmasq --no-hosts --no-resolv
>>>>>> --strict-order --bind-interfaces --interface=tap5a418ee0-53
>>>>>> --except-interface=lo --domain=openstacklocal
>>>>>> --pid-file=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/pid
>>>>>> --dhcp-hostsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/host
>>>>>> --dhcp-optsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/opts
>>>>>> --dhcp-script=/opt/stack/quantum/bin/quantum-dhcp-agent-dnsmasq-lease-update
>>>>>> --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s
>>>>>> 6108 ? S 0:00 dnsmasq --no-hosts --no-resolv
>>>>>> --strict-order --bind-interfaces --interface=tap5a418ee0-53
>>>>>> --except-interface=lo --domain=openstacklocal
>>>>>> --pid-file=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/pid
>>>>>> --dhcp-hostsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/host
>>>>>> --dhcp-optsfile=/opt/stack/data/dhcp/c95edcb8-3df6-4e92-a367-4fce5bce6e63/opts
>>>>>> --dhcp-script=/opt/stack/quantum/bin/quantum-dhcp-agent-dnsmasq-lease-update
>>>>>> --leasefile-ro --dhcp-range=set:tag0,10.0.0.0,static,120s
>>>>>> 28913 pts/1 S+ 0:00 grep --color=auto dnsmasq
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Kaneko
>>>>>>
>>>>>> 2012/9/14 Dan Wendlandt <dan at nicira.com>:
>>>>>>> Hi quantum hackers,
>>>>>>>
>>>>>>> We're pushing a change to devstack to use a new vif-driver for Quantum
>>>>>>> with Open vSwitch (https://review.openstack.org/#/c/11650/). The
>>>>>>> benefit of this driver is that it is compatible with Nova's security
>>>>>>> group filtering. This is "a good thing", since it more closely maps
>>>>>>> to how real users will deploy Quantum.
>>>>>>>
>>>>>>> However, this may catch developers by surprise who are suddenly unable
>>>>>>> to ping or SSH to instances because the security groups drop traffic
>>>>>>> by default.
>>>>>>>
>>>>>>> Preferred method of dealing with this is to add the following lines to
>>>>>>> local.sh in your devstack directory, which open up your VMs for ping
>>>>>>> and SSH for the 'demo' user:
>>>>>>>
>>>>>>> nova secgroup-add-rule default icmp -1 -1 0.0.0.0/0
>>>>>>> nova secgroup-add-rule default tcp 22 22 0.0.0.0/0
>>>>>>>
>>>>>>> Another work around is to disable nova security groups by adding
>>>>>>> 'LIBVIRT_FIREWALL_DRIVER=nova.virt.firewall.NoopFirewallDriver' to
>>>>>>> your localrc
>>>>>>>
>>>>>>> Dan
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>> Dan Wendlandt
>>>>>>> Nicira, Inc: www.nicira.com
>>>>>>> twitter: danwendlandt
>>>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> OpenStack-dev mailing list
>>>>>>> OpenStack-dev at lists.openstack.org
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>>
>>>>>> _______________________________________________
>>>>>> OpenStack-dev mailing list
>>>>>> OpenStack-dev at lists.openstack.org
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>> Dan Wendlandt
>>>>> Nicira, Inc: www.nicira.com
>>>>> twitter: danwendlandt
>>>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>>>
>>>>> _______________________________________________
>>>>> OpenStack-dev mailing list
>>>>> OpenStack-dev at lists.openstack.org
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>> _______________________________________________
>>>> OpenStack-dev mailing list
>>>> OpenStack-dev at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>>
>>>
>>> --
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>> Dan Wendlandt
>>> Nicira, Inc: www.nicira.com
>>> twitter: danwendlandt
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~
>>>
>>> _______________________________________________
>>> OpenStack-dev mailing list
>>> OpenStack-dev at lists.openstack.org
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list