<div dir="ltr">Hi, Michael,<div><br></div><div>Thanks a lot for your reply.</div><div><br></div><div>I can make sure that there is no router or multiple dhcp services in my environment. </div><div><br></div><div>As shown in my first mail, the haproxy in the amphora tries to find the gateway ip 10.0.1.1 that does not exist in the environment. </div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">ubuntu@amphora-a0621f0e-d27f-</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">4f22-a4ee-05b695e2b71f:~$ sudo ip netns exec amphora-haproxy tcpdump -i eth1 -nn</span><br></div><div><div style="font-size:12.8px">sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-<wbr>a4ee-05b695e2b71f</div><div style="font-size:12.8px">tcpdump: verbose output suppressed, use -v or -vv for full protocol decode</div><div style="font-size:12.8px">listening on eth1, link-type EN10MB (Ethernet), capture size 262144 bytes</div><div style="font-size:12.8px">^C07:25:24.225614 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601, win 28200, options [mss 1410,sackOK,TS val 30692602 ecr 0,nop,wscale 7], length 0</div><div style="font-size:12.8px">07:25:24.237854 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28</div><div style="font-size:12.8px">07:25:25.224801 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28</div><div style="font-size:12.8px">07:25:25.228610 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601, win 28200, options [mss 1410,sackOK,TS val 30692853 ecr 0,nop,wscale 7], length 0</div><div style="font-size:12.8px">07:25:26.224533 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28</div><div style="font-size:12.8px">07:25:27.230911 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28</div><div style="font-size:12.8px">07:25:27.250858 IP 10.0.1.2.55294 > 10.0.1.4.80: Flags [S], seq 1637781601, win 28200, options [mss 1410,sackOK,TS val 30693359 ecr 0,nop,wscale 7], length 0</div><div style="font-size:12.8px">07:25:28.228796 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28</div><div style="font-size:12.8px">07:25:29.228697 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28</div><div style="font-size:12.8px">07:25:29.290551 ARP, Request who-has 10.0.1.4 tell 10.0.1.2, length 28</div><div style="font-size:12.8px">07:25:29.290985 ARP, Reply 10.0.1.4 is-at fa:16:3e:be:5a:d5, length 28</div><div style="font-size:12.8px">07:25:31.251122 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28</div><div style="font-size:12.8px">07:25:32.248737 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28</div><div style="font-size:12.8px">07:25:33.250309 ARP, Request who-has 10.0.1.1 tell 10.0.1.4, length 28</div></div><div><br></div><div><b>So if the subnet is not attached to any router, why does haproxy try to find the gateway ip that does not exist at all? Maybe that is the reason why haproxy receives the packet from curl but fail to respond. </b></div><div><br></div><div>I think the gateway ip (10.0.1.10) confuses you. Actually, in my environment octavia and tricircle (<a href="https://wiki.openstack.org/wiki/Tricircle">https://wiki.openstack.org/wiki/Tricircle</a>) are installed together. Because of the cross-neutron mechanism of tricircle, the gateway ip of subnet in that region is 10.0.1.10. But I can make sure that gateway ip (10.0.1.1 or 10.0.1.10) does not exist in the network, since there is no router at all. This error also happens in my another environment where octavia is installed alone. The environment is installed on Oct. 6, and all the repos are the latest at that time. </div><div><br></div><div>Best regards,</div><div>Yipei</div><div><br><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Nov 9, 2017 at 2:50 PM, Yipei Niu <span dir="ltr"><<a href="mailto:newypei@gmail.com" target="_blank">newypei@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>Hi, Michael,</div><div><br></div><div>Based on your mail, the information is as follows.</div><div><br></div><div>1. The version of Octavia I used is Queens, and the latest commit message is</div><div><div>commit 2ab2836d0ebdd0fd5bc32d3adcc44a<wbr>92557c8c1d</div><div>Author: OpenStack Proposal Bot <<a href="mailto:openstack-infra@lists.openstack.org" target="_blank">openstack-infra@lists.opensta<wbr>ck.org</a>></div><div>Date: Fri Nov 3 17:58:59 2017 +0000</div><div><br></div><div> Updated from global requirements</div><div> </div><div> Change-Id: I9047e289b8a3c931156da480b3f9f<wbr>676c54a8358</div></div><div><br></div><div>2. The info of the amphora and other VMs is as follows.</div><div>+-----------------------------<wbr>---------+--------------------<wbr>--------------------------+---<wbr>-----+------------+-----------<wbr>--+---------------------------<wbr>------------------------------<wbr>+</div><div>| ID | Name | Status | Task State | Power State | Networks |</div><div>+-----------------------------<wbr>---------+--------------------<wbr>--------------------------+---<wbr>-----+------------+-----------<wbr>--+---------------------------<wbr>------------------------------<wbr>+</div><div>| 33bd02cb-f853-404d-a705-99bc1b<wbr>04a178 | amphora-a0621f0e-d27f-4f22-a4e<wbr>e-05b695e2b71f | ACTIVE | - | Running | lb-mgmt-net1=192.168.1.4; net1=10.0.1.8 |</div><div>| dd046fc9-e2bf-437d-8c51-c397bc<wbr>cc3dc1 | client1 | ACTIVE | - | Running | net1=10.0.1.3 |</div><div>| 50446c75-7cb7-43eb-b057-4b6b89<wbr>a926bc | client3 | ACTIVE | - | Running | net4=10.0.4.3 |</div><div>+-----------------------------<wbr>---------+--------------------<wbr>--------------------------+---<wbr>-----+------------+-----------<wbr>--+---------------------------<wbr>------------------------------<wbr>+</div><div><br></div><div>3. The info of the load balancer is as follows.</div><div>+---------------------+-------<wbr>------------------------------<wbr>-----------+</div><div>| Field | Value |</div><div>+---------------------+-------<wbr>------------------------------<wbr>-----------+</div><div>| admin_state_up | True |</div><div>| description | |</div><div>| id | 51cba1d5-cc3c-48ff-b41e-839619<wbr>093334 |</div><div>| listeners | {"id": "b20ad920-c6cd-4e71-a9b9-c134e<wbr>57ecd20"} |</div><div>| name | lb1 |</div><div>| operating_status | ONLINE |</div><div>| pools | {"id": "d0042605-da50-4048-b298-66042<wbr>0b0a1d2"} |</div><div>| provider | octavia |</div><div>| provisioning_status | ACTIVE |</div><div>| tenant_id | c2a97a04cb6d4f25bdcb8b3f263c86<wbr>9e |</div><div>| vip_address | 10.0.1.4 |</div><div>| vip_port_id | 2209a819-0ac8-4211-b878-f0b41a<wbr>c4727b |</div><div>| vip_subnet_id | cbcf4f04-da6d-4800-8b40-4b1419<wbr>72c2bf |</div><div>+---------------------+-------<wbr>------------------------------<wbr>-----------+</div><div><br></div><div>4. The info of the listener is as follows.</div><div>+---------------------------+-<wbr>------------------------------<wbr>-----------------+</div><div>| Field | Value |</div><div>+---------------------------+-<wbr>------------------------------<wbr>-----------------+</div><div>| admin_state_up | True |</div><div>| connection_limit | -1 |</div><div>| default_pool_id | d0042605-da50-4048-b298-660420<wbr>b0a1d2 |</div><div>| default_tls_container_ref | |</div><div>| description | |</div><div>| id | b20ad920-c6cd-4e71-a9b9-c134e5<wbr>7ecd20 |</div><div>| loadbalancers | {"id": "51cba1d5-cc3c-48ff-b41e-83961<wbr>9093334"} |</div><div>| name | listener1 |</div><div>| protocol | HTTP |</div><div>| protocol_port | 80 |</div><div>| sni_container_refs | |</div><div>| tenant_id | c2a97a04cb6d4f25bdcb8b3f263c86<wbr>9e |</div><div>+---------------------------+-<wbr>------------------------------<wbr>-----------------+</div><div><br></div><div>5. The members of the load balancer lb1 are as follows.</div><div>+-----------------------------<wbr>---------+------+-------------<wbr>---------------------+--------<wbr>--+---------------+--------+--<wbr>------------------------------<wbr>------+----------------+</div><div>| id | name | tenant_id | address | protocol_port | weight | subnet_id | admin_state_up |</div><div>+-----------------------------<wbr>---------+------+-------------<wbr>---------------------+--------<wbr>--+---------------+--------+--<wbr>------------------------------<wbr>------+----------------+</div><div>| 420c905c-1077-46c9-8b04-526a59<wbr>d93376 | | c2a97a04cb6d4f25bdcb8b3f263c86<wbr>9e | 10.0.1.3 | 80 | 1 | cbcf4f04-da6d-4800-8b40-4b1419<wbr>72c2bf | True |</div><div>+-----------------------------<wbr>---------+------+-------------<wbr>---------------------+--------<wbr>--+---------------+--------+--<wbr>------------------------------<wbr>------+----------------+</div><div><br></div><div>6. Since the VIP and the members reside in the same subnet, only two subnets are listed as follows.</div><div>+-----------------------------<wbr>---------+-----------------+--<wbr>------------------------------<wbr>--+----------------+----------<wbr>------------------------------<wbr>-----------+</div><div>| id | name | tenant_id | cidr | allocation_pools |</div><div>+-----------------------------<wbr>---------+-----------------+--<wbr>------------------------------<wbr>--+----------------+----------<wbr>------------------------------<wbr>-----------+</div><div>| 752f865d-89e4-4284-9e91-8617a5<wbr>a21da1 | lb-mgmt-subnet1 | c2a97a04cb6d4f25bdcb8b3f263c86<wbr>9e | <a href="http://192.168.1.0/24" target="_blank">192.168.1.0/24</a> | {"start": "192.168.1.10", "end": "192.168.1.254"} |</div><div>| | | | | {"start": "192.168.1.1", "end": "192.168.1.8"} |</div><div>| cbcf4f04-da6d-4800-8b40-4b1419<wbr>72c2bf | subnet1 | c2a97a04cb6d4f25bdcb8b3f263c86<wbr>9e | <a href="http://10.0.1.0/24" target="_blank">10.0.1.0/24</a> | {"start": "10.0.1.1", "end": "10.0.1.9"} |</div><div>| | | | | {"start": "10.0.1.11", "end": "10.0.1.254"} |</div><div>+-----------------------------<wbr>---------+-----------------+--<wbr>------------------------------<wbr>--+----------------+----------<wbr>------------------------------<wbr>-----------+</div><div><br></div><div>7. The detailed info of subnet1 and lb-mgmt-subnet is listed as follows, respectively.</div><div>lb-mgmt-subnet1</div><div>+-------------------+---------<wbr>------------------------------<wbr>------------+</div><div>| Field | Value |</div><div>+-------------------+---------<wbr>------------------------------<wbr>------------+</div><div>| allocation_pools | {"start": "192.168.1.1", "end": "192.168.1.8"} |</div><div>| | {"start": "192.168.1.10", "end": "192.168.1.254"} |</div><div>| cidr | <a href="http://192.168.1.0/24" target="_blank">192.168.1.0/24</a> |</div><div>| created_at | 2017-11-05T12:14:45Z |</div><div>| description | |</div><div>| dns_nameservers | |</div><div>| enable_dhcp | True |</div><div>| gateway_ip | 192.168.1.9 |</div><div>| host_routes | |</div><div>| id | 752f865d-89e4-4284-9e91-8617a5<wbr>a21da1 |</div><div>| ip_version | 4 |</div><div>| ipv6_address_mode | |</div><div>| ipv6_ra_mode | |</div><div>| name | lb-mgmt-subnet1 |</div><div>| network_id | b4261144-3342-4605-8ca6-146e5b<wbr>84c4ea |</div><div>| project_id | c2a97a04cb6d4f25bdcb8b3f263c86<wbr>9e |</div><div>| revision_number | 0 |</div><div>| service_types | |</div><div>| subnetpool_id | |</div><div>| tags | |</div><div>| tenant_id | c2a97a04cb6d4f25bdcb8b3f263c86<wbr>9e |</div><div>| updated_at | 2017-11-05T12:14:45Z |</div><div>+-------------------+---------<wbr>------------------------------<wbr>------------+</div><div><br></div><div>subnet1</div><div>+-------------------+---------<wbr>------------------------------<wbr>------+</div><div>| Field | Value |</div><div>+-------------------+---------<wbr>------------------------------<wbr>------+</div><div>| allocation_pools | {"start": "10.0.1.1", "end": "10.0.1.9"} |</div><div>| | {"start": "10.0.1.11", "end": "10.0.1.254"} |</div><div>| cidr | <a href="http://10.0.1.0/24" target="_blank">10.0.1.0/24</a> |</div><div>| created_at | 2017-11-05T12:37:56Z |</div><div>| description | |</div><div>| dns_nameservers | |</div><div>| enable_dhcp | True |</div><div>| gateway_ip | 10.0.1.10 |</div><div>| host_routes | |</div><div>| id | cbcf4f04-da6d-4800-8b40-4b1419<wbr>72c2bf |</div><div>| ip_version | 4 |</div><div>| ipv6_address_mode | |</div><div>| ipv6_ra_mode | |</div><div>| name | subnet1 |</div><div>| network_id | 310fea4b-36ae-4617-b499-5936e8<wbr>eda842 |</div><div>| project_id | c2a97a04cb6d4f25bdcb8b3f263c86<wbr>9e |</div><div>| revision_number | 0 |</div><div>| service_types | |</div><div>| subnetpool_id | |</div><div>| tags | |</div><div>| tenant_id | c2a97a04cb6d4f25bdcb8b3f263c86<wbr>9e |</div><div>| updated_at | 2017-11-05T12:37:56Z |</div><div>+-------------------+---------<wbr>------------------------------<wbr>------+</div><div><br></div><div>8. The info of interfaces in the default and amphora-haproxy network namespace of the amphora are as follows.</div><div>ubuntu@amphora-a0621f0e-d27f-4<wbr>f22-a4ee-05b695e2b71f:~$ ifconfig</div><div>ens3 Link encap:Ethernet HWaddr fa:16:3e:9e:6b:77 </div><div> inet addr:192.168.1.4 Bcast:192.168.1.255 Mask:255.255.255.0</div><div> inet6 addr: fe80::f816:3eff:fe9e:6b77/64 Scope:Link</div><div> UP BROADCAST RUNNING MULTICAST MTU:1450 Metric:1</div><div> RX packets:13112 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:41491 errors:0 dropped:0 overruns:0 carrier:0</div><div> collisions:0 txqueuelen:1000 </div><div> RX bytes:775372 (775.3 KB) TX bytes:9653389 (9.6 MB)</div><div><br></div><div>lo Link encap:Local Loopback </div><div> inet addr:127.0.0.1 Mask:255.0.0.0</div><div> inet6 addr: ::1/128 Scope:Host</div><div> UP LOOPBACK RUNNING MTU:65536 Metric:1</div><div> RX packets:128 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:128 errors:0 dropped:0 overruns:0 carrier:0</div><div> collisions:0 txqueuelen:1 </div><div> RX bytes:11424 (11.4 KB) TX bytes:11424 (11.4 KB)</div><div><br></div><div>ubuntu@amphora-a0621f0e-d27f-4<wbr>f22-a4ee-05b695e2b71f:~$ sudo ip netns exec amphora-haproxy ifconfig</div><span class="gmail-"><div>sudo: unable to resolve host amphora-a0621f0e-d27f-4f22-a4e<wbr>e-05b695e2b71f</div></span><div>eth1 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5 </div><div> inet addr:10.0.1.8 Bcast:10.0.1.255 Mask:255.255.255.0</div><div> inet6 addr: fe80::f816:3eff:febe:5ad5/64 Scope:Link</div><div> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1</div><div> RX packets:107 errors:0 dropped:0 overruns:0 frame:0</div><div> TX packets:218 errors:0 dropped:0 overruns:0 carrier:0</div><div> collisions:0 txqueuelen:1000 </div><div> RX bytes:6574 (6.5 KB) TX bytes:9468 (9.4 KB)</div><div><br></div><div>eth1:0 Link encap:Ethernet HWaddr fa:16:3e:be:5a:d5 </div><div> inet addr:10.0.1.4 Bcast:10.0.1.255 Mask:255.255.255.0</div><div> UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1</div><div><br></div><div>9. When curl the VIP from the host, it does not respond and finally return a timeout error.</div><div><div>stack@devstack-1:/opt/stack/oc<wbr>tavia$ sudo ip netns exec qdhcp-310fea4b-36ae-4617-b499-<wbr>5936e8eda842 curl 10.0.1.4</div><div>curl: (7) Failed to connect to 10.0.1.4 port 80: Connection timed out</div></div><div><br></div><div>10. Results of running "netstat -rn" on the host are as follows.</div><div>Kernel IP routing table</div><div>Destination Gateway Genmask Flags MSS Window irtt Iface</div><div>0.0.0.0 192.168.1.9 0.0.0.0 UG 0 0 0 o-hm0</div><div>0.0.0.0 10.0.2.2 0.0.0.0 UG 0 0 0 enp0s3</div><div>10.0.2.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s3</div><div>169.254.0.0 0.0.0.0 255.255.0.0 U 0 0 0 enp0s10</div><div>192.168.1.0 0.0.0.0 255.255.255.0 U 0 0 0 o-hm0</div><div>192.168.56.0 0.0.0.0 255.255.255.0 U 0 0 0 enp0s10</div><div>192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0 virbr0</div><div> <br></div><div>11. In the amphora, the first two commit message of amphora-agent are as follows.</div><div><div><br></div><div>commit 2ab2836d0ebdd0fd5bc32d3adcc44a<wbr>92557c8c1d</div><div>Author: OpenStack Proposal Bot <<a href="mailto:openstack-infra@lists.openstack.org" target="_blank">openstack-infra@lists.opensta<wbr>ck.org</a>></div><div>Date: Fri Nov 3 17:58:59 2017 +0000</div><div><br></div><div> Updated from global requirements</div><div> </div><div> Change-Id: I9047e289b8a3c931156da480b3f9f<wbr>676c54a8358</div><div><br></div><div>commit 504cb6c682e4779b5889c0eb68705d<wbr>0ab12e2c81</div><div>Merge: e983508 b8ebbe9</div><div>Author: Zuul <<a href="mailto:zuul@review.openstack.org" target="_blank">zuul@review.openstack.org</a>></div><div>Date: Wed Nov 1 19:46:39 2017 +0000</div><div><br></div><div> Merge "Add cached_zone to the amphora record"</div></div><div><br></div><div>Best regards,</div><div>Yipei</div><div><br></div></div>
</blockquote></div><br></div></div></div>