[openstack-dev] [networking-sfc] SFC flows issue
Ravi Sekhar Reddy Konda
ravisekhar.konda at oracle.com
Sat Sep 10 14:58:46 UTC 2016
Hi Mohan,
|--------------| |--------------| |--------------|
| SRC-VM | | SF-VM | | DST-VM |
|--------------| |--------------| |--------------|
192.168.0.9 P1 | 192.168.0.4 P2 | | p3 192.168.0.12 p4 | 192.168.0.8
fa:16:3e:48:88:96 | fa:16:3e:bb:be:c2 | | fa:16:3e:9a:29:f1 | fa:16:3e:9f:25:c8
| | | |
Net1-----------------------------------------------------------------------------------------------------------------
Actually I am getting the packet on the egress port tap interface, but the packet is getting dropped there itself instead of getting forwarded to the destination VM port
The reason I found is the Src MAC and Dst MAC are in the reverse order in the ICMP packet, because of which packet is getting dropped
stack at cloud-VirtualBox:~/test$ sudo tcpdump -n -e -i tap157bda28-a4 -v icmp
tcpdump: WARNING: tap157bda28-a4: no IPv4 address assigned
tcpdump: listening on tap157bda28-a4, link-type EN10MB (Ethernet), capture size 65535 bytes
192.168.0.9 > 192.168.0.8: ICMP echo request, id 17665, seq 0, length 64
19:27:56.812674 fa:16:3e:9a:29:f1 > fa:16:3e:9f:25:c8, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 44153, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.0.9 > 192.168.0.8: ICMP echo request, id 17665, seq 1, length 64
19:27:57.814347 fa:16:3e:9a:29:f1 > fa:16:3e:9f:25:c8, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 63, id 44254, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.0.9 > 192.168.0.8: ICMP echo request, id 17665, seq 2, length 64
stack at cloud-VirtualBox:~/test$ sudo ovs-ofctl dump-flows br-int -O openflow13 | grep in_port=17
2016-09-10T14:01:22Z|00001|ofp_actions|INFO|OFPAT_SET_MPLS_TTL is deprecated in OpenFlow13 (use Set-Field)
cookie=0x936171a2fbba22bd, duration=407.031s, table=0, n_packets=0, n_bytes=0, priority=30,ip,in_port=17,nw_src=192.168.0.9,nw_dst=192.168.0.8 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=2513.381s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=17,icmp_type=136 actions=resubmit(,24)
cookie=0x936171a2fbba22bd, duration=2513.307s, table=0, n_packets=224, n_bytes=9408, priority=10,arp,in_port=17 actions=resubmit(,24)
cookie=0x936171a2fbba22bd, duration=2513.401s, table=0, n_packets=716, n_bytes=141262, priority=9,in_port=17 actions=resubmit(,25)
cookie=0x936171a2fbba22bd, duration=2513.393s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=17,icmp_type=136,nd_target=fe80::f816:3eff:fe9a:29f1 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=2513.347s, table=24, n_packets=5, n_bytes=210, priority=2,arp,in_port=17,arp_spa=192.168.0.12 actions=resubmit(,25)
cookie=0x936171a2fbba22bd, duration=2513.415s, table=25, n_packets=50, n_bytes=4684, priority=2,in_port=17,dl_src=fa:16:3e:9a:29:f1 actions=NORMAL
In the last rule we are looking for in_port=17, and src MAC "fa:16:3e:9a:29:f1", but what the packet header contains Dst MAC in place of Src MAC
I brought up "trusty-server-cloudimg-amd64-disk1.img" as ServiceFunction VM. As Ubuntu VM contains a single interface, I added other interface manually with following steps
=> Added /etc/network/interfaces.d/eth1.cfg
=> ifup eth1
Then added default route for the dst VM IpAddress
=> ip route add 192.168.0.8/32 via 192.168.0.12 dev eth1
also enabled packet forwarding
=> echo 1 > /proc/sys/net/ipv4/ip_forward
eth0 Link encap:Ethernet HWaddr fa:16:3e:bb:be:c2
inet addr:192.168.0.4 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:febb:bec2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:1241 errors:0 dropped:0 overruns:0 frame:0
TX packets:704 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:155099 (155.0 KB) TX bytes:91291 (91.2 KB)
eth1 Link encap:Ethernet HWaddr fa:16:3e:9a:29:f1
inet addr:192.168.0.12 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::f816:3eff:fe9a:29f1/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1
RX packets:229 errors:0 dropped:0 overruns:0 frame:0
TX packets:68 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:66984 (66.9 KB) TX bytes:6272 (6.2 KB)
root at sfvm:/home/ubuntu# route -n
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 192.168.0.1 0.0.0.0 UG 0 0 0 eth0
169.254.169.254 192.168.0.1 255.255.255.255 UGH 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth0
192.168.0.0 0.0.0.0 255.255.255.0 U 0 0 0 eth1
192.168.0.8 192.168.0.12 255.255.255.255 UGH 0 0 0 eth1
root at sfvm:/home/ubuntu#
Just to verify, have I manually sent ping request from egress port in the ServiceFunction VM ( ping -S 192.168.0.12 192.168.0.8)
stack at cloud-VirtualBox:~/test$ sudo tcpdump -n -e -i tap157bda28-a4 -v icmp
tcpdump: WARNING: tap157bda28-a4: no IPv4 address assigned
tcpdump: listening on tap157bda28-a4, link-type EN10MB (Ethernet), capture size 65535 bytes
19:35:33.683969 fa:16:3e:9a:29:f1 > fa:16:3e:9f:25:c8, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 16699, offset 0, flags [DF], proto ICMP (1), length 84)
192.168.0.12 > 192.168.0.8: ICMP echo request, id 1351, seq 1, length 64
19:35:33.684715 fa:16:3e:9f:25:c8 > fa:16:3e:9a:29:f1, ethertype IPv4 (0x0800), length 98: (tos 0x0, ttl 64, id 30764, offset 0, flags [none], proto ICMP (1), length 84)
192.168.0.8 > 192.168.0.12: ICMP echo reply, id 1351, seq 1, length 64
Here MAC addresses are getting updated properly and I am getting response.
I am not able to figure out the reason for reverse MAC addresses when I send ping request using the port-chain
Below is the openflow dump
stack at cloud-VirtualBox:~/test$ sudo ovs-ofctl dump-flows br-int -O openflow13
2016-09-10T13:58:48Z|00001|ofp_actions|INFO|OFPAT_SET_MPLS_TTL is deprecated in OpenFlow13 (use Set-Field)
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x936171a2fbba22bd, duration=253.158s, table=0, n_packets=0, n_bytes=0, priority=30,ip,in_port=17,nw_src=192.168.0.9,nw_dst=192.168.0.8 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=253.155s, table=0, n_packets=18, n_bytes=1764, priority=30,ip,in_port=15,nw_src=192.168.0.9,nw_dst=192.168.0.8 actions=group:1
cookie=0x936171a2fbba22bd, duration=5521.152s, table=0, n_packets=0, n_bytes=0, priority=20,mpls actions=resubmit(,10)
cookie=0x936171a2fbba22bd, duration=2411.459s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=15,icmp_type=136 actions=resubmit(,24)
cookie=0x936171a2fbba22bd, duration=2361.237s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=16,icmp_type=136 actions=resubmit(,24)
cookie=0x936171a2fbba22bd, duration=2359.508s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=17,icmp_type=136 actions=resubmit(,24)
cookie=0x936171a2fbba22bd, duration=1506.783s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=18,icmp_type=136 actions=resubmit(,24)
cookie=0x936171a2fbba22bd, duration=2411.443s, table=0, n_packets=10, n_bytes=420, priority=10,arp,in_port=15 actions=resubmit(,24)
cookie=0x936171a2fbba22bd, duration=2361.219s, table=0, n_packets=40, n_bytes=1680, priority=10,arp,in_port=16 actions=resubmit(,24)
cookie=0x936171a2fbba22bd, duration=2359.434s, table=0, n_packets=224, n_bytes=9408, priority=10,arp,in_port=17 actions=resubmit(,24)
cookie=0x936171a2fbba22bd, duration=1506.767s, table=0, n_packets=9, n_bytes=378, priority=10,arp,in_port=18 actions=resubmit(,24)
cookie=0x936171a2fbba22bd, duration=5522.397s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=1 actions=drop
cookie=0x936171a2fbba22bd, duration=2411.475s, table=0, n_packets=638, n_bytes=127072, priority=9,in_port=15 actions=resubmit(,25)
cookie=0x936171a2fbba22bd, duration=2361.254s, table=0, n_packets=1596, n_bytes=263939, priority=9,in_port=16 actions=resubmit(,25)
cookie=0x936171a2fbba22bd, duration=2359.528s, table=0, n_packets=662, n_bytes=130522, priority=9,in_port=17 actions=resubmit(,25)
cookie=0x936171a2fbba22bd, duration=1506.798s, table=0, n_packets=414, n_bytes=81165, priority=9,in_port=18 actions=resubmit(,25)
cookie=0x936171a2fbba22bd, duration=5484.887s, table=0, n_packets=2482, n_bytes=242710, priority=3,in_port=1,vlan_tci=0x0000/0x1fff actions=push_vlan:0x8100,set_field:4098->vlan_vid,NORMAL
cookie=0x936171a2fbba22bd, duration=5522.826s, table=0, n_packets=5918, n_bytes=769504, priority=0 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=253.202s, table=5, n_packets=18, n_bytes=1764, priority=0,ip,dl_dst=fa:16:3e:bb:be:c2 actions=push_mpls:0x8847,set_field:65791->mpls_label,set_mpls_ttl(255),push_vlan:0x8100,set_field:4100->vlan_vid,resubmit(,10)
cookie=0x936171a2fbba22bd, duration=253.126s, table=10, n_packets=18, n_bytes=1764, priority=1,mpls,dl_vlan=4,dl_dst=fa:16:3e:bb:be:c2,mpls_label=65791 actions=pop_vlan,pop_mpls:0x0800,output:16
cookie=0x936171a2fbba22bd, duration=5521.146s, table=10, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x936171a2fbba22bd, duration=5522.820s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x936171a2fbba22bd, duration=2411.467s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=15,icmp_type=136,nd_target=fe80::f816:3eff:fe48:8896 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=2361.245s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=16,icmp_type=136,nd_target=fe80::f816:3eff:febb:bec2 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=2359.520s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=17,icmp_type=136,nd_target=fe80::f816:3eff:fe9a:29f1 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=1506.790s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=18,icmp_type=136,nd_target=fe80::f816:3eff:fe9f:25c8 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=2411.451s, table=24, n_packets=10, n_bytes=420, priority=2,arp,in_port=15,arp_spa=192.168.0.9 actions=resubmit(,25)
cookie=0x936171a2fbba22bd, duration=2361.229s, table=24, n_packets=40, n_bytes=1680, priority=2,arp,in_port=16,arp_spa=192.168.0.4 actions=resubmit(,25)
cookie=0x936171a2fbba22bd, duration=2359.474s, table=24, n_packets=5, n_bytes=210, priority=2,arp,in_port=17,arp_spa=192.168.0.12 actions=resubmit(,25)
cookie=0x936171a2fbba22bd, duration=1506.775s, table=24, n_packets=9, n_bytes=378, priority=2,arp,in_port=18,arp_spa=192.168.0.8 actions=resubmit(,25)
cookie=0x936171a2fbba22bd, duration=5522.815s, table=24, n_packets=219, n_bytes=9198, priority=0 actions=drop
cookie=0x936171a2fbba22bd, duration=2411.489s, table=25, n_packets=33, n_bytes=2960, priority=2,in_port=15,dl_src=fa:16:3e:48:88:96 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=2361.270s, table=25, n_packets=1014, n_bytes=138605, priority=2,in_port=16,dl_src=fa:16:3e:bb:be:c2 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=2359.543s, table=25, n_packets=50, n_bytes=4684, priority=2,in_port=17,dl_src=fa:16:3e:9a:29:f1 actions=NORMAL
cookie=0x936171a2fbba22bd, duration=1506.817s, table=25, n_packets=30, n_bytes=2716, priority=2,in_port=18,dl_src=fa:16:3e:9f:25:c8 actions=NORMAL
stack at cloud-VirtualBox:~/test$
Thanks in advance,
Ravi
----- Original Message -----
From: all
To: openstack-dev at lists.openstack.org
Sent: Tuesday, September 6, 2016 1:17:11 PM GMT +05:30 Chennai, Kolkata, Mumbai, New Delhi
Subject: Re: [openstack-dev] [networking-sfc] SFC flows issue
Hi Ravi ,
Could you share tcpdump on P3 as well ( sudo tcpdump -n -e -i "P3 tab-interface" )
IMO , the possible reasons would be
[1] port-security on SF port may block the packet , since it destined to different ip address.
[2] The SF program may alter the IP Header (Destination field )
Flow rules are looks okay , and it shows packets are dropped in between p2 and p3 .
Thanks.,
Mohankumar.N
On Sat, Sep 3, 2016 at 10:07 AM, Ravi Sekhar Reddy Konda < ravisekhar.konda at oracle.com > wrote:
Hi Networking SFC team
I am trying out networking-SFC on the AllInOne Devstack(master branch) VM brought up on the VirtualBox.
I am trying out the following scenario
|--------------| |--------------| |--------------|
| SRC-VM | | SF-VM | | DST-VM |
|--------------| |--------------| |--------------|
p1 | p2 | |p3 p4 |
| | | |
| | | |
Net1---------------------------------------------------------------------------------------------
P1 -> 192.168.0.6 "fa:16:3e:d8:8f:28"
P2 -> 192.168.0.10 "fa:16:3e:12:7e:50"
P3 -> 192.168.0.5 "fa:16:3e:1e:f8:c2"
P4 -> 192.168.0.14 "fa:16:3e:4d:e8:c3"
On SF-VM, I have written a simple application to capture the packets on eth0 and forward to eth1
The issue I am facing is packets are getting sent from Port-1 (SRC-VM) to Port-2 (SF-VM), but I am not seeing packets getting sent from Port-3 (SF-VM) to Port-4 (DST-VM)
Here is the Classifier I created
neutron flow-classifier-create --ethertype IPv4 --source-ip-prefix 192.168.0.6/32 --logical-source-port 6c700c35-6505-4304-89cd-b48513884cf5 --destination-ip-prefix 192.168.0.14/32 --logical-destination-port d0b218c9-57f4-4d38-b363-2086b8dbc8b9 --protocol icmp fc1
==> Can you please let me know are the flows as expected in this scenario
OFPST_FLOW reply (OF1.3) (xid=0x2):
cookie=0x9118e6e52dd3dfa0, duration=34624.734s, table=0, n_packets=399, n_bytes=39102, priority=30,icmp,in_port=9,nw_src=192.168.0.6,nw_dst=192.168.0.14 actions=group:1
cookie=0x9118e6e52dd3dfa0, duration=34624.696s, table=0, n_packets=0, n_bytes=0, priority=30,icmp,in_port=13,nw_src=192.168.0.6,nw_dst=192.168.0.14 actions=NORMAL
cookie=0x9118e6e52dd3dfa0, duration=45300.986s, table=0, n_packets=0, n_bytes=0, priority=20,mpls actions=resubmit(,10)
cookie=0x9118e6e52dd3dfa0, duration=42858.783s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=9,icmp_type=136 actions=resubmit(,24)
cookie=0x9118e6e52dd3dfa0, duration=42250.745s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=10,icmp_type=136 actions=resubmit(,24)
cookie=0x9118e6e52dd3dfa0, duration=37845.792s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=12,icmp_type=136 actions=resubmit(,24)
cookie=0x9118e6e52dd3dfa0, duration=37617.801s, table=0, n_packets=0, n_bytes=0, priority=10,icmp6,in_port=13,icmp_type=136 actions=resubmit(,24)
cookie=0x9118e6e52dd3dfa0, duration=42858.758s, table=0, n_packets=73, n_bytes=3066, priority=10,arp,in_port=9 actions=resubmit(,24)
cookie=0x9118e6e52dd3dfa0, duration=42250.725s, table=0, n_packets=151, n_bytes=6342, priority=10,arp,in_port=10 actions=resubmit(,24)
cookie=0x9118e6e52dd3dfa0, duration=37845.769s, table=0, n_packets=666039, n_bytes=27973638, priority=10,arp,in_port=12 actions=resubmit(,24)
cookie=0x9118e6e52dd3dfa0, duration=37617.782s, table=0, n_packets=666069, n_bytes=27974898, priority=10,arp,in_port=13 actions=resubmit(,24)
cookie=0x9118e6e52dd3dfa0, duration=45302.016s, table=0, n_packets=0, n_bytes=0, priority=2,in_port=1 actions=drop
cookie=0x9118e6e52dd3dfa0, duration=42858.804s, table=0, n_packets=10993, n_bytes=2219925, priority=9,in_port=9 actions=resubmit(,25)
cookie=0x9118e6e52dd3dfa0, duration=42250.774s, table=0, n_packets=22275, n_bytes=4028172, priority=9,in_port=10 actions=resubmit(,25)
cookie=0x9118e6e52dd3dfa0, duration=37845.822s, table=0, n_packets=9967, n_bytes=1989597, priority=9,in_port=12 actions=resubmit(,25)
cookie=0x9118e6e52dd3dfa0, duration=37617.828s, table=0, n_packets=11154, n_bytes=2154155, priority=9,in_port=13 actions=resubmit(,25)
cookie=0x9118e6e52dd3dfa0, duration=45268.699s, table=0, n_packets=14048, n_bytes=1104503, priority=3,in_port=1,vlan_tci=0x0000/0x1fff actions=push_vlan:0x8100,set_field:4098->vlan_vid,NORMAL
cookie=0x9118e6e52dd3dfa0, duration=45302.314s, table=0, n_packets=24842, n_bytes=2954751, priority=0 actions=NORMAL
cookie=0x9118e6e52dd3dfa0, duration=34624.792s, table=5, n_packets=399, n_bytes=39102, priority=0,ip,dl_dst=fa:16:3e:12:7e:50 actions=push_mpls:0x8847,set_field:65791->mpls_label,set_mpls_ttl(255),push_vlan:0x8100,set_field:4099->vlan_vid,resubmit(,10)
cookie=0x9118e6e52dd3dfa0, duration=34624.649s, table=10, n_packets=399, n_bytes=39102, priority=1,mpls,dl_vlan=3,dl_dst=fa:16:3e:12:7e:50,mpls_label=65791 actions=pop_vlan,pop_mpls:0x0800,output:10
cookie=0x9118e6e52dd3dfa0, duration=45300.979s, table=10, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x9118e6e52dd3dfa0, duration=45302.309s, table=23, n_packets=0, n_bytes=0, priority=0 actions=drop
cookie=0x9118e6e52dd3dfa0, duration=42858.796s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=9,icmp_type=136,nd_target=fe80::f816:3eff:fed8:8f28 actions=NORMAL
cookie=0x9118e6e52dd3dfa0, duration=42250.752s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=10,icmp_type=136,nd_target=fe80::f816:3eff:fe12:7e50 actions=NORMAL
cookie=0x9118e6e52dd3dfa0, duration=37845.802s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=12,icmp_type=136,nd_target=fe80::f816:3eff:fe4d:e8c3 actions=NORMAL
cookie=0x9118e6e52dd3dfa0, duration=37617.815s, table=24, n_packets=0, n_bytes=0, priority=2,icmp6,in_port=13,icmp_type=136,nd_target=fe80::f816:3eff:fe1e:f8c2 actions=NORMAL
cookie=0x9118e6e52dd3dfa0, duration=42858.773s, table=24, n_packets=73, n_bytes=3066, priority=2,arp,in_port=9,arp_spa=192.168.0.6 actions=resubmit(,25)
cookie=0x9118e6e52dd3dfa0, duration=42250.735s, table=24, n_packets=139, n_bytes=5838, priority=2,arp,in_port=10,arp_spa=192.168.0.10 actions=resubmit(,25)
cookie=0x9118e6e52dd3dfa0, duration=37845.783s, table=24, n_packets=666039, n_bytes=27973638, priority=2,arp,in_port=12,arp_spa=192.168.0.14 actions=resubmit(,25)
cookie=0x9118e6e52dd3dfa0, duration=37617.791s, table=24, n_packets=666022, n_bytes=27972924, priority=2,arp,in_port=13,arp_spa=192.168.0.5 actions=resubmit(,25)
cookie=0x9118e6e52dd3dfa0, duration=45302.304s, table=24, n_packets=59, n_bytes=2478, priority=0 actions=drop
cookie=0x9118e6e52dd3dfa0, duration=42858.823s, table=25, n_packets=99, n_bytes=5900, priority=2,in_port=9,dl_src=fa:16:3e:d8:8f:28 actions=NORMAL
cookie=0x9118e6e52dd3dfa0, duration=42250.791s, table=25, n_packets=11603, n_bytes=1849532, priority=2,in_port=10,dl_src=fa:16:3e:12:7e:50 actions=NORMAL
cookie=0x9118e6e52dd3dfa0, duration=37845.841s, table=25, n_packets=666304, n_bytes=27999888, priority=2,in_port=12,dl_src=fa:16:3e:4d:e8:c3 actions=NORMAL
cookie=0x9118e6e52dd3dfa0, duration=37617.845s, table=25, n_packets=667436, n_bytes=28153632, priority=2,in_port=13,dl_src=fa:16:3e:1e:f8:c2 actions=NORMAL
When I manually send ICMP packets from eth1 interface(P3) on SF-VM, packets are going as expected
Thanks in Advance
Ravi
__________________________________________________________________________
OpenStack Development Mailing List (not for usage questions)
Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160910/770c94c1/attachment.html>
More information about the OpenStack-dev
mailing list