<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote">On Sat, Oct 12, 2013 at 8:59 AM, Nick Maslov <span dir="ltr"><<a href="mailto:azpekt@gmail.com" target="_blank">azpekt@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Hi,<br>
<br>
I have following setup:<br>
<br>
1) infrastructure node, IP in bond, hosting following KVM guests:<br>
1.1) Postgres KVM guest<br>
1.2) MQ KVM guest<br>
1.3) DNS KVM guest<br>
1.4) Control node with Nova API, Cinder API, Quantum Server, etc.<br>
...<br>
1.8) Quantum network node with quantum agents<br>
<br>
Agents on this network node are always dying and starting up again:<br>
<br>
# quantum agent-list<br>
+--------------------------------------+--------------------+-----------------------------+-------+----------------+<br>
| id | agent_type | host | alive | admin_state_up |<br>
+--------------------------------------+--------------------+-----------------------------+-------+----------------+<br>
| 5656392b-b6fe-4570-802f-97d2154acf31 | L3 agent | <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> | xxx | True |<br>
| 1093fb73-6622-448e-8dad-558a36cca306 | DHCP agent | <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> | xxx | True |<br>
| 4518830d-e112-439f-a629-7defa7bd29e9 | Open vSwitch agent | <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> | xxx | True |<br>
| 86ee6d24-2e6a-4f58-addb-290fefc26401 | Open vSwitch agent | nova05 | :-) | True |<br>
| b67697bb-3ec1-49fc-8f3c-7e4e7892e83a | Open vSwitch agent | nova04 | :-) | True |<br>
+--------------------------------------+--------------------+-----------------------------+-------+----------------+<br>
<br>
Few minutes after, those agents will be up again, one may die - while others not.<br>
<br>
ping net01-001<br>
PING <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34) 56(84) bytes of data.<br>
64 bytes from <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34): icmp_req=1 ttl=64 time=0.912 ms<br>
64 bytes from <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34): icmp_req=2 ttl=64 time=0.273 ms<br>
64 bytes from <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34): icmp_req=2 ttl=64 time=0.319 ms (DUP!)<br>
64 bytes from <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34): icmp_req=3 ttl=64 time=0.190 ms<br>
64 bytes from <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34): icmp_req=4 ttl=64 time=0.230 ms<br>
64 bytes from <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34): icmp_req=4 ttl=64 time=0.305 ms (DUP!)<br>
64 bytes from <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34): icmp_req=5 ttl=64 time=0.199 ms<br>
64 bytes from <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34): icmp_req=7 ttl=64 time=0.211 ms<br>
64 bytes from <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34): icmp_req=8 ttl=64 time=0.322 ms<br>
64 bytes from <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> (10.10.146.34): icmp_req=8 ttl=64 time=0.409 ms (DUP!)<br>
^C<br>
--- <a href="http://net01-001.int.net.net" target="_blank">net01-001.int.net.net</a> ping statistics ---<br>
8 packets transmitted, 7 received, +3 duplicates, 12% packet loss, time 7017ms<br>
<br>
SSH`ing to network node is also difficult - constant freezes. Nothing suspicious in the logs.<br></blockquote><div><br></div><div>Those DUP!'s are suspicious, since you aren't pinging a broadcast domain. That might indicate there's something up with the OVS GRE mesh.<br>
</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
Since DHCP agent may be down, spawning a VM may end in "waiting for network device" state. Then, it might get the internal IP and then floating - but accessing it also proves to be very troublesome - I believe because of L3 agent flapping.<br>
<br>
My OpenStack was set up under this manual - <a href="https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst" target="_blank">https://github.com/mseknibilel/OpenStack-Grizzly-Install-Guide/blob/OVS_MultiNode/OpenStack_Grizzly_Install_Guide.rst</a><br>
<br><blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote">
Only thing I did - I added HAproxy/keepalived on top of it, balancing API requests on control nodes. But this shouldn`t impact networking...<br></blockquote></blockquote><div><br></div><div>Agreed, it should not affect network connectivity for the network node.<br>
<br></div><div>Not sure what the issue is. Perhaps you might try following through Darragh's excellent tutorial on debugging L3 issues in OVS/Quantum here:<br><br><a href="http://techbackground.blogspot.com/2013/05/the-quantum-l3-router-and-floating-ips.html">http://techbackground.blogspot.com/2013/05/the-quantum-l3-router-and-floating-ips.html</a><br>
<br></div><div>DId you manually set up KVM instances for all these nodes, or are you using something like Triple-O?<br></div><div><br></div><div>Best,<br></div><div>-jay<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<blockquote style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex" class="gmail_quote"></blockquote>
<br>
Anyone have any thoughts about this?<br>
<br>
Cheers,<br>
NM<br>
<br>
_______________________________________________<br>
Mailing list: <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a><br>
Post to : <a href="mailto:openstack@lists.openstack.org">openstack@lists.openstack.org</a><br>
Unsubscribe : <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack</a> </blockquote></div><br></div></div>