[Openstack] Openstack security - port-security, security-groups with added container
Farhad Sunavala
fsbiz at yahoo.com
Fri Apr 22 22:52:17 UTC 2016
Thank you Kevin.Works like a charm now.
Farhad.
On Friday, April 22, 2016 4:43 AM, Kevin Benton <kevin at benton.pub> wrote:
The status being DOWN is an issue. That means either the port binding failed because the host_id was not set on the port that matches the hostname where the openvswitch agent is running, or the openvswitch agent failed to identify the port from the container that was plugged in.So there are two things. First do a port-show on that port with admin credentials so you can see if binding:host_id is set to the hostname of the node the container is on. Second, I assume veth_h0 is the connection to the container? If that's the case, you will need to add the external ID to OVSDB that indicates the Neutron port it corresponds to. Here is an example command of how to add a port in a way that the openvswitch agent can correctly wire it up. Apologies for it being so hacky, it's just a quick script I had to test Neutron agent wiring without using VMs.port_body=$(neutron port-show $PORT_UUID)port_id=$(echo "$port_body" | grep "| id" | awk '{ print $4 }')port_mac=$(echo "$port_body" | grep "| mac_address" | awk '{ print $4 }')port_name=${port_id:0:11}port_name="tap${port_name}"sudo ovs-vsctl --timeout=120 -- --if-exists del-port $port_name -- add-port br-int $port_name -- set Interface $port_name external-ids:iface-id=$port_id external-ids:iface-status=active external-ids:attached-mac=$port_mac type=internal
The port_name may not need to be changed to match the partial port UUID format, but there used to be old code that depended on that name so it might be safest to first start with making your veth pair name match that format if you can.The giveaway is that your veth_h0 port is missing a tag on br-int. If it doesn't have a tag (or if it has a tag of 4095), that means the openvswitch agent hasn't correctly wired it.
On Apr 21, 2016 20:57, "Farhad Sunavala" <fsbiz at yahoo.com> wrote:
Thanks Kevin and Kaustubh. Yes, just 1 node (controller, network and compute).
SRC C DST| | | OpenvSwitch SRC VM = 192.168.0.3DST VM = 192.168.0.4Container C = 192.168.0.5
root at fs-10-145-105-64:~# neutron port-list+--------------------------------------+------+-------------------+-------------------------------------------------------+| id | name | mac_address | fixed_ips |+--------------------------------------+------+-------------------+-------------------------------------------------------+| 371b8923-efb7-499f-b344-a9f38d538a73 | | fa:16:3e:1e:bd:2c | {"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", || | | | "ip_address": "192.168.0.3"} || 86883e00-9d97-4bed-b05e-c942227fb268 | | fa:16:3e:2b:36:22 | {"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", || | | | "ip_address": "192.168.0.2"} || a1213dae-ab39-41c0-b2cd-99b20714aa8b | | fa:16:3e:c7:23:be | {"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", || | | | "ip_address": "192.168.0.5"} || b2339046-da71-43e9-a6f8-cbadf24be654 | | fa:16:3e:06:c5:23 | {"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", || | | | "ip_address": "192.168.0.1"} || b2b2c53a-c92e-47b6-88bc-db3d3e135b79 | | fa:16:3e:89:0d:fa | {"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", || | | | "ip_address": "192.168.0.4"} |+--------------------------------------+------+-------------------+-------------------------------------------------------+
Just limiting output to SRC (192.168.0.3) and Container C (192.168.0.5)
root at fs-10-145-105-64:~# neutron port-show 371b8923-efb7-499f-b344-a9f38d538a73+-----------------------+------------------------------------------------------------------------------------+| Field | Value |+-----------------------+------------------------------------------------------------------------------------+| admin_state_up | True || allowed_address_pairs | || binding:vnic_type | normal || created_at | 2016-04-21T20:39:37 || description | || device_id | b7995a2e-f5c7-4398-a801-8d71c9c05a54 || device_owner | compute:None || dns_name | || extra_dhcp_opts | || fixed_ips | {"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", "ip_address": "192.168.0.3"} || id | 371b8923-efb7-499f-b344-a9f38d538a73 || mac_address | fa:16:3e:1e:bd:2c || name | || network_id | b2be2b73-6e17-427e-9ec3-5e6d1a1ef98d || port_security_enabled | False || security_groups | || status | ACTIVE || tenant_id | ba13e4a3e7724db8889852b398988587 || updated_at | 2016-04-22T03:38:48 |+-----------------------+------------------------------------------------------------------------------------+
root at fs-10-145-105-64:~# neutron port-show a1213dae-ab39-41c0-b2cd-99b20714aa8b+-----------------------+------------------------------------------------------------------------------------+| Field | Value |+-----------------------+------------------------------------------------------------------------------------+| admin_state_up | True || allowed_address_pairs | || binding:vnic_type | normal || created_at | 2016-04-21T20:41:50 || description | || device_id | || device_owner | || dns_name | || extra_dhcp_opts | || fixed_ips | {"subnet_id": "762f6059-a06f-4b3f-a91d-d807bcf9f349", "ip_address": "192.168.0.5"} || id | a1213dae-ab39-41c0-b2cd-99b20714aa8b || mac_address | fa:16:3e:c7:23:be || name | || network_id | b2be2b73-6e17-427e-9ec3-5e6d1a1ef98d || port_security_enabled | False || security_groups | || status | DOWN || tenant_id | ba13e4a3e7724db8889852b398988587 || updated_at | 2016-04-21T20:41:50 |+-----------------------+------------------------------------------------------------------------------------+
The status=DOWN makes sense since the container is not under nova-control.
root at fs-10-145-105-64:~# docker psCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMESc8d99952695d ubuntu "/bin/bash" 7 hours ago Up 7 hours backstabbing_spenceroot at fs-10-145-105-64:~# docker inspect -f '{{.State.Pid}}' c8d99952695d105529
root at fs-10-145-105-64:~# ip netns105529qrouter-872d95f1-5ef7-4811-8232-8ced2322863bqdhcp-b2be2b73-6e17-427e-9ec3-5e6d1a1ef98d83376root at fs-10-145-105-64:~# ip netns exec 105529 ifconfigeth0 Link encap:Ethernet HWaddr fa:16:3e:c7:23:be inet addr:192.168.0.5 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fec7:23be/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:23908 errors:0 dropped:0 overruns:0 frame:0 TX packets:24514 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2303668 (2.3 MB) TX bytes:2329412 (2.3 MB)
As you can see the MAC address of container matches what neutron has given along withthe IP address of 192.168.0.5
root at fs-10-145-105-64:~# brctl showbridge name bridge id STP enabled interfacesroot at fs-10-145-1058.-
root at fs-10-145-105-64:~# iptables -LChain INPUT (policy ACCEPT)target prot opt source destinationnova-api-INPUT all -- anywhere anywhereACCEPT udp -- anywhere anywhere udp dpt:domainACCEPT tcp -- anywhere anywhere tcp dpt:domainACCEPT udp -- anywhere anywhere udp dpt:bootpsACCEPT tcp -- anywhere anywhere tcp dpt:bootps
Chain FORWARD (policy ACCEPT)target prot opt source destinationnova-filter-top all -- anywhere anywherenova-api-FORWARD all -- anywhere anywhereACCEPT all -- anywhere 192.168.122.0/24 ctstate RELATED,ESTABLISHEDACCEPT all -- 192.168.122.0/24 anywhereACCEPT all -- anywhere anywhereREJECT all -- anywhere anywhere reject-with icmp-port-unreachableREJECT all -- anywhere anywhere reject-with icmp-port-unreachable
Chain OUTPUT (policy ACCEPT)target prot opt source destinationnova-filter-top all -- anywhere anywherenova-api-OUTPUT all -- anywhere anywhereACCEPT udp -- anywhere anywhere udp dpt:bootpc
Chain nova-api-FORWARD (1 references)target prot opt source destination
Chain nova-api-INPUT (1 references)target prot opt source destinationACCEPT tcp -- anywhere fs-10.145.105.64 tcp dpt:8775
Chain nova-api-OUTPUT (1 references)target prot opt source destination
Chain nova-api-local (1 references)target prot opt source destination
Chain nova-filter-top (2 references)target prot opt source destinationnova-api-local all -- anywhere anywhere
======================================================================
Before sending 3 pings:
root at c8d99952695d:/# ifconfig eth0eth0 Link encap:Ethernet HWaddr fa:16:3e:c7:23:be inet addr:192.168.0.5 Bcast:192.168.0.255 Mask:255.255.255.0 inet6 addr: fe80::f816:3eff:fec7:23be/64 Scope:Link UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:23908 errors:0 dropped:0 overruns:0 frame:0 TX packets:24514 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:2303668 (2.3 MB) TX bytes:2329412 (2.3 MB)
root at fs-10-145-105-64:~# ovs-ofctl dump-flows br-intNXST_FLOW reply (xid=0x4): cookie=0xb67c96fbae779e37, duration=26464.927s, table=0, n_packets=0, n_bytes=0, idle_age=26464, priority=20,mpls actions=resubmit(,10) cookie=0xb67c96fbae779e37, duration=25823.697s, table=0, n_packets=0, n_bytes=0, idle_age=25823, priority=10,icmp6,in_port=7,icmp_type=136 actions=resubmit(,24) cookie=0xb67c96fbae779e37, duration=25803.737s, table=0, n_packets=0, n_bytes=0, idle_age=25803, priority=10,icmp6,in_port=8,icmp_type=136 actions=resubmit(,24) cookie=0xb67c96fbae779e37, duration=26465.387s, table=0, n_packets=734, n_bytes=39540, idle_age=668, priority=0 actions=NORMAL cookie=0xb67c96fbae779e37, duration=26464.921s, table=10, n_packets=0, n_bytes=0, idle_age=26464, priority=0 actions=drop cookie=0xb67c96fbae779e37, duration=26465.381s, table=23, n_packets=0, n_bytes=0, idle_age=26465, priority=0 actions=drop cookie=0xb67c96fbae779e37, duration=26465.375s, table=24, n_packets=0, n_bytes=0, idle_age=26465, priority=0 actions=drop
root at c8d99952695d:/# ping 192.168.0.3 -c3PING 192.168.0.3 (192.168.0.3) 56(84) bytes of data.From 192.168.0.5 icmp_seq=1 Destination Host UnreachableFrom 192.168.0.5 icmp_seq=2 Destination Host UnreachableFrom 192.168.0.5 icmp_seq=3 Destination Host Unreachable
--- 192.168.0.3 ping statistics ---3 packets transmitted, 0 received, +3 errors, 100% packet loss, time 2014ms
(TCPDUn veth_h0 - container's tap interface on br-int - Note: SRC MAC address matches one given by neutron)root at fs-10-145-105-64:~# tcpdump -i veth_h0 -nnXXetcpdump: WARNING: veth_h0: no IPv4 address assignedtcpdump: verbose output suppressed, use -v or -vv for full protocol decodelistening on veth_h0, link-type EN10MB (Ethernet), capture size 65535 bytes20:51:58.629048 fa:16:3e:c7:23:be > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.0.3 tell 192.168.0.5, length 28 0x0000: ffff ffff ffff fa16 3ec7 23be 0806 0001 ........>.#..... 0x0010: 0800 0604 0001 fa16 3ec7 23be c0a8 0005 ........>.#..... 0x0020: 0000 0000 0000 c0a8 0003 ..........20:51:59.625739 fa:16:3e:c7:23:be > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.0.3 tell 192.168.0.5, length 28 0x0000: ffff ffff ffff fa16 3ec7 23be 0806 0001 ........>.#..... 0x0010: 0800 0604 0001 fa16 3ec7 23be c0a8 0005 ........>.#..... 0x0020: 0000 0000 0000 c0a8 0003 ..........20:52:00.625741 fa:16:3e:c7:23:be > ff:ff:ff:ff:ff:ff, ethertype ARP (0x0806), length 42: Request who-has 192.168.0.3 tell 192.168.0.5, length 28 0x0000: ffff ffff ffff fa16 3ec7 23be 0806 0001 ........>.#..... 0x0010: 0800 0604 0001 fa16 3ec7 23be c0a8 0005 ........>.#..... 0x0020: 0000 0000 0000 c0a8 0003 ..........
root at fs-10-145-105-64:~# ovs-ofctl dump-flows br-intNXST_FLOW reply (xid=0x4): cookie=0xb67c96fbae779e37, duration=26703.277s, table=0, n_packets=0, n_bytes=0, idle_age=26703, priority=20,mpls actions=resubmit(,10) cookie=0xb67c96fbae779e37, duration=26062.047s, table=0, n_packets=0, n_bytes=0, idle_age=26062, priority=10,icmp6,in_port=7,icmp_type=136 actions=resubmit(,24) cookie=0xb67c96fbae779e37, duration=26042.087s, table=0, n_packets=0, n_bytes=0, idle_age=26042, priority=10,icmp6,in_port=8,icmp_type=136 actions=resubmit(,24) cookie=0xb67c96fbae779e37, duration=26703.737s, table=0, n_packets=737, n_bytes=39666, idle_age=123, priority=0 actions=NORMAL cookie=0xb67c96fbae779e37, duration=26703.271s, table=10, n_packets=0, n_bytes=0, idle_age=26703, priority=0 actions=drop cookie=0xb67c96fbae779e37, duration=26703.731s, table=23, n_packets=0, n_bytes=0, idle_age=26703, priority=0 actions=drop
The packet counter has incremented by 3 (737-734) on table=0, actions:NORMAL
But, the packets never make it out of the tap interface to SRC (192.168.0.3).
Totally stumped. It is reproducible easily.Just get the latest devstack from master (or mitaka stable) latest openvswitch (2.5.90) or OVS 2.4.1.
Thanks,Farhad.
On Thursday, April 21, 2016 3:52 PM, Kevin Benton <kevin at benton.pub> wrote:
I don't see port details for the container port (50686370). That would be the one that needs port_security_enabled=False if your container is not using the MAC address defined in the Neutron Port for that container.
On Thu, Apr 21, 2016 at 3:30 PM, Kaustubh Kelkar <kaustubh.kelkar at casa-systems.com> wrote:
Hi, Can you post the output of:iptables –Lbrctl showovs-ofctl show br-int I presume all the instances are on the same physical node? -Kaustubh From: Farhad Sunavala [mailto:fsbiz at yahoo.com]
Sent: Thursday, April 21, 2016 3:15 PM
To: openstack at lists.openstack.org
Subject: [Openstack] Openstack security - port-security, security-groups with added container Hi folks, My setup is as follows. SRC C DST| | | OpenvSwitch SRC VM = 192.168.0.3DST VM = 192.168.0.4 Container C = 192.168.0.5 Openstack version: latest from masterOpenvSwitch: 2.4.1 Openstack brings up SRC (192.168.0.3) and DST VM (192.168.0.5)I manually bringup container C using docker and connect it to br-int.I allocate a neutron port (192.168.0.5) and configure this IP address on the container C. SRC,DST VMs work fine.Now, my problem is that the container C is not able to pass traffic. Troubleshooting reveals it is sending packet to br-int, it hits the NORMAL action flows on br-int but then br-int does not transmit them forward.I have disabled security-groups and port-security on all ports. It is definitely something to do with security (anti-spoofing) but I am not able to narrow it down.Any pointers will be appreciated. Thanks,Farhad. Details of troubleshooting: Setup: SRC C DST| | | OpenvSwitch SRC VM = 192.168.0.3DST VM = 192.168.0.4 Container C = 192.168.0.5 root at fs-10-145-105-64:~# neutron port-list+--------------------------------------+------+-------------------+-------------------------------------------------------+| id | name | mac_address | fixed_ips |+--------------------------------------+------+-------------------+-------------------------------------------------------+| 31d7fef4-2689-40d5-886c-b0419295eabb | | fa:16:3e:e7:f8:47 | {"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", || | | | "ip_address": "192.168.0.4"} || 3466abf1-aa9a-420a-bb9b-c4ad55561f19 | | fa:16:3e:84:62:af | {"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", || | | | "ip_address": "192.168.0.2"} || 50686370-de40-4665-a5a1-39b5b795122b | | fa:16:3e:0d:ae:bc | {"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", || | | | "ip_address": "192.168.0.5"} || c6043a1e-ac4d-4138-af14-b91ae8e4d57e | | fa:16:3e:36:95:77 | {"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", || | | | "ip_address": "192.168.0.1"} || e8a167b1-8457-4bd7-a18b-82712e0c46dd | | fa:16:3e:85:92:ee | {"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", || | | | "ip_address": "192.168.0.3"} |+--------------------------------------+------+-------------------+-------------------------------------------------------+root at fs-10-145-105-64:~# neutron port-show e8a167b1-8457-4bd7-a18b-82712e0c46dd+-----------------------+------------------------------------------------------------------------------------+| Field | Value |+-----------------------+------------------------------------------------------------------------------------+| admin_state_up | True || allowed_address_pairs | || binding:host_id | fs-10-145-105-64 || binding:profile | {} || binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} || binding:vif_type | ovs || binding:vnic_type | normal || created_at | 2016-04-21T15:27:51 || description | || device_id | 1a65bcfb-4607-4529-a313-6927d86c9302 || device_owner | compute:nova || dns_name | || extra_dhcp_opts | || fixed_ips | {"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", "ip_address": "192.168.0.3"} || id | e8a167b1-8457-4bd7-a18b-82712e0c46dd || mac_address | fa:16:3e:85:92:ee || name | || network_id | e4c38c74-c9c2-47a7-802d-446488b82e5f || port_security_enabled | True || security_groups | fcf62b2a-1dd4-432a-b47b-5d9b79813da7 || status | ACTIVE || tenant_id | 62b9eff93cc74b65a2b52a0d41604e2c || updated_at | 2016-04-21T15:27:58 |+-----------------------+------------------------------------------------------------------------------------+ root at fs-10-145-105-64:~# neutron port-show 31d7fef4-2689-40d5-886c-b0419295eabb+-----------------------+------------------------------------------------------------------------------------+| Field | Value |+-----------------------+------------------------------------------------------------------------------------+| admin_state_up | True || allowed_address_pairs | || binding:host_id | fs-10-145-105-64 || binding:profile | {} || binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} || binding:vif_type | ovs || binding:vnic_type | normal || created_at | 2016-04-21T15:28:37 || description | || device_id | 9c8a5e3c-0867-4938-b26e-4834a24d0c5d || device_owner | compute:nova || dns_name | || extra_dhcp_opts | || fixed_ips | {"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", "ip_address": "192.168.0.4"} || id | 31d7fef4-2689-40d5-886c-b0419295eabb || mac_address | fa:16:3e:e7:f8:47 || name | || network_id | e4c38c74-c9c2-47a7-802d-446488b82e5f || port_security_enabled | False || security_groups | || status | ACTIVE || tenant_id | 62b9eff93cc74b65a2b52a0d41604e2c || updated_at | 2016-04-21T18:25:20 |+-----------------------+------------------------------------------------------------------------------------+ root at fs-10-145-105-64:~# neutron port-show 31d7fef4-2689-40d5-886c-b0419295eabb+-----------------------+------------------------------------------------------------------------------------+| Field | Value |+-----------------------+------------------------------------------------------------------------------------+| admin_state_up | True || allowed_address_pairs | || binding:host_id | fs-10-145-105-64 || binding:profile | {} || binding:vif_details | {"port_filter": true, "ovs_hybrid_plug": false} || binding:vif_type | ovs || binding:vnic_type | normal || created_at | 2016-04-21T15:28:37 || description | || device_id | 9c8a5e3c-0867-4938-b26e-4834a24d0c5d || device_owner | compute:nova || dns_name | || extra_dhcp_opts | || fixed_ips | {"subnet_id": "19c6dc03-b255-45e9-9bd7-366631b745b6", "ip_address": "192.168.0.4"} || id | 31d7fef4-2689-40d5-886c-b0419295eabb || mac_address | fa:16:3e:e7:f8:47 || name | || network_id | e4c38c74-c9c2-47a7-802d-446488b82e5f || port_security_enabled | False || security_groups | || status | ACTIVE || tenant_id | 62b9eff93cc74b65a2b52a0d41604e2c || updated_at | 2016-04-21T18:25:20 |+-----------------------+------------------------------------------------------------------------------------+ root at fs-10-145-105-64:~# ovs-vsctl showcd040c9d-427b-418b-943c-7ad5522e1d8f Bridge br-int fail_mode: secure Port "tap3466abf1-aa" tag: 2 Interface "tap3466abf1-aa" type: internal Port "tape8a167b1-84" tag: 2 Interface "tape8a167b1-84" Port br-int Interface br-int type: internal Port patch-tun
...
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack/attachments/20160422/bc7e42ab/attachment.html>
More information about the Openstack
mailing list