<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif;">
<div>
<div>It's been a while since I used keepalived. However, can you confirm that on failover that the new master sends out a garp (gratuitous arp) for the VIP that it took over? This garp should update the switches arp tables. (Which is what your outbound connection
from a vm -> google is essentially doing for you).</div>
<div><br>
</div>
<div>This should be configured by garp_master_delay. The default if I remember right is 5 seconds.</div>
<div>
<div>
<div><br>
</div>
<div>Also, you should double check that you don’t have port security enabled on the switch port. At least on older cisco ios devices, if you had port security enabled to permit x number of dynamic mac-addresses, it kept track of the number off allowed mac
addresses by adding static mac entries to the switch, which will completely screw up the type of clustering you are doing.</div>
<div>____________________________________________</div>
<div> </div>
<div>Kris Lindgren</div>
<div>Senior Linux Systems Engineer</div>
<div>GoDaddy, LLC.</div>
<div><br>
</div>
</div>
</div>
</div>
<div><br>
</div>
<span id="OLK_SRC_BODY_SECTION">
<div style="font-family:Calibri; font-size:11pt; text-align:left; color:black; BORDER-BOTTOM: medium none; BORDER-LEFT: medium none; PADDING-BOTTOM: 0in; PADDING-LEFT: 0in; PADDING-RIGHT: 0in; BORDER-TOP: #b5c4df 1pt solid; BORDER-RIGHT: medium none; PADDING-TOP: 3pt">
<span style="font-weight:bold">From: </span>Pedro Sousa <<a href="mailto:pgsousa@gmail.com">pgsousa@gmail.com</a>><br>
<span style="font-weight:bold">Date: </span>Tuesday, January 13, 2015 at 1:18 PM<br>
<span style="font-weight:bold">To: </span>Jesse Keating <<a href="mailto:jlk@bluebox.net">jlk@bluebox.net</a>><br>
<span style="font-weight:bold">Cc: </span>"<a href="mailto:OpenStack-operators@lists.openstack.org">OpenStack-operators@lists.openstack.org</a>" <<a href="mailto:openstack-operators@lists.openstack.org">openstack-operators@lists.openstack.org</a>><br>
<span style="font-weight:bold">Subject: </span>Re: [Openstack-operators] Fwd: HAPROXY 504 errors in HA conf<br>
</div>
<div><br>
</div>
<div>
<div>
<div dir="ltr">As expected If I reboot the Keepalived MASTER node, I get timeouts again, so my understanding is that this happens when the VIP fails over to another node. Anyone has explanation for this?
<div><br>
</div>
<div>Thanks</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jan 13, 2015 at 8:08 PM, Pedro Sousa <span dir="ltr">
<<a href="mailto:pgsousa@gmail.com" target="_blank">pgsousa@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>Hi,</div>
<div><br>
</div>
I think I found out the issue, as I have all the 3 nodes running Keepalived as MASTER, when I reboot one of the servers, one of the VIPS failsover to it, causing the timeout issues. So I left only one server as MASTER and the other 2 as BACKUP, and If I reboot
the BACKUP servers everything will work fine.
<div><br>
</div>
<div>As a note aside, I don't know if this is some ARP issue because I have a similar problem with Neutron L3 running in HA Mode. If I reboot the server that is running as MASTER I loose connection to my floating IPS because the switch doesn't know yet that
the Mac Addr has changed. To everything start working I have to ping an outside host like google from an instance.<br>
</div>
<div>
<div><br>
</div>
<div>Maybe someone could share some experience on this,</div>
<div><br>
</div>
<div>Thank you for your help.</div>
<div><br>
<div><br>
</div>
<div><br>
</div>
</div>
</div>
</div>
<div class="HOEnZb">
<div class="h5">
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jan 13, 2015 at 7:18 PM, Pedro Sousa <span dir="ltr">
<<a href="mailto:pgsousa@gmail.com" target="_blank">pgsousa@gmail.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">
<div>Jesse,</div>
<div><br>
</div>
I see a lot of these messages in glance-api:
<div><br>
</div>
<div>
<div>2015-01-13 19:16:29.084 29269 DEBUG glance.api.middleware.version_negotiation [29d94a9a-135b-4bf2-a97b-f23b0704ee15 eb7ff2b5f0f34f51ac9ea0f75b60065d 2524b02b63994749ad1fed6f3a825c15 - - -] Unknown version. Returning version choices. process_request /usr/lib/python2.7/site-packages/glance/api/middleware/version_negotiation.py:64</div>
</div>
<div><br>
</div>
<div>While running openstack-status (glance image-list)</div>
<div>
<div><br>
</div>
<div>== Glance images ==</div>
<div>Error finding address for <a href="http://172.16.21.20:9292/v1/images/detail?sort_key=name&sort_dir=asc&limit=20" target="_blank">
http://172.16.21.20:9292/v1/images/detail?sort_key=name&sort_dir=asc&limit=20</a>: HTTPConnectionPool(host='172.16.21.20', port=9292): Max retries exceeded with url: /v1/images/detail?sort_key=name&sort_dir=asc&limit=20 (Caused by <class 'httplib.BadStatusLine'>:
'')</div>
</div>
<div><br>
</div>
<div><br>
</div>
<div>Thanks</div>
<div><br>
</div>
</div>
<div>
<div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Jan 13, 2015 at 6:52 PM, Jesse Keating <span dir="ltr">
<<a href="mailto:jlk@bluebox.net" target="_blank">jlk@bluebox.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span>On 1/13/15 10:42 AM, Pedro Sousa wrote:<br>
</span>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span>Hi<br>
<br>
<br>
I've changed some haproxy confs, now I'm getting a different error:<br>
<br>
</span> *== Nova networks ==*<br>
*ERROR (ConnectionError): HTTPConnectionPool(host='172.<u></u>16.21.20',<span><br>
port=8774): Max retries exceeded with url:<br>
/v2/<u></u>2524b02b63994749ad1fed6f3a825c<u></u>15/os-networks (Caused by <class<br>
</span> 'httplib.BadStatusLine'>: '')*<br>
*== Nova instance flavors ==*<span><br>
<br>
If I restart my openstack services everything will start working.<br>
<br>
I'm attaching my new haproxy conf.<br>
<br>
<br>
Thanks<br>
<br>
</span></blockquote>
<br>
Sounds like your services are losing access to something, like rabbit or the database. What do your service logs show prior to restart? Are they throwing any errors?<span><font color="#888888"><br>
<br>
<br>
-- <br>
-jlk</font></span>
<div>
<div><br>
<br>
______________________________<u></u>_________________<br>
OpenStack-operators mailing list<br>
<a href="mailto:OpenStack-operators@lists.openstack.org" target="_blank">OpenStack-operators@lists.<u></u>openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators" target="_blank">http://lists.openstack.org/<u></u>cgi-bin/mailman/listinfo/<u></u>openstack-operators</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</span>
</body>
</html>