<br><br><div class="gmail_quote">---------- Forwarded message ----------<br>From: <b class="gmail_sendername">Roman Sokolkov</b> <span dir="ltr"><<a href="mailto:rsokolkov@gmail.com">rsokolkov@gmail.com</a>></span><br>
Date: 2011/12/28<br>Subject: Re: [Openstack] reconnecting to RabbitMQ<br>To: Ask Solem <<a href="mailto:ask@rabbitmq.com">ask@rabbitmq.com</a>><br><br><br>Hi!<div><br></div><div>I tried to make Rabbit HA for Openstack.</div>
<div><br></div><div>My first e-mail was little innacurate. </div><div><br></div><div>I use two nodes with active/active rabbits with mirrored queues. Additionally I use Virtual IP. And all it managed by Pacemaker. Openstack uses this remote rabbitmq <span style="font-family:arial,sans-serif;font-size:13px;white-space:nowrap;background-color:rgb(255,255,255)">via VIP.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px;white-space:nowrap;background-color:rgb(255,255,255)"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px;white-space:nowrap;background-color:rgb(255,255,255)">I hardly reboot node with VIP and VIP moves to other node. In this case, connections on Openstack host staying forever in ESTABLISHED state(in py-amqlib 0.6.1 no tcp keepalive options).</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px;white-space:nowrap;background-color:rgb(255,255,255)">Openstack think that all good and doesn't try to reconnect. But I if correctly stop RabbitMQ, connections closed correctly and Openstack will try to reconnect.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px;white-space:nowrap;background-color:rgb(255,255,255)"><br></span></div><div><font face="arial, sans-serif"><span style="white-space:nowrap">I solve this issue by adding tcp_keepalive options in py-amqlib(transport.py). </span></font></div>
<div><font face="arial, sans-serif"><span style="white-space:nowrap"><br></span></font></div><div><font face="arial, sans-serif"><span style="white-space:nowrap">It's very interesting what implemented in KAMQP. And how heartbeats work?</span></font></div>
<div><font face="arial, sans-serif"><span style="white-space:nowrap"><br></span></font></div><div><font face="arial, sans-serif"><span style="white-space:nowrap"><br>
</span></font></div><div><div><div class="h5"><br><div class="gmail_quote">2011/12/28 Ask Solem <span dir="ltr"><<a href="mailto:ask@rabbitmq.com" target="_blank">ask@rabbitmq.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Not sure how I can help until the problem is identified.<br>
Kombu does not do automatic reconnecting by default, but provides<br>
the primitives necessary to implement it.<br>
<br>
amqplib does not support heartbeats so there may be cases<br>
where the client/server does not properly detect that the socket<br>
has been closed, this usually only a problem with some particular intermediate<br>
network components like routers and firewalls. Support for heartbeats<br>
is currently in the works, and I already have a rudimentary implementation<br>
at <a href="http://github.com/ask/kamqp/" target="_blank">http://github.com/ask/kamqp/</a> if someone wants to try it out (though that<br>
probably requires some hacking).<br>
<div><div><br>
<br>
On 28 Dec 2011, at 10:51, Alexis Richardson wrote:<br>
<br>
> Hi all,<br>
><br>
> I am bcc'ing the author of Kombu in case he can help.<br>
><br>
> alexis<br>
><br>
><br>
> On Sun, Dec 25, 2011 at 2:11 AM, Chris Behrens<br>
> <<a href="mailto:chris.behrens@rackspace.com" target="_blank">chris.behrens@rackspace.com</a>> wrote:<br>
>> Hi Roman,<br>
>><br>
>> Are you using diablo or trunk (both of which use kombu by default and should support reconnecting)?<br>
>><br>
>> If so, can you 'egrep' for the following strings in logs and provide the output? You can reply to me directly so we're not spamming this list.<br>
>><br>
>> "Unable to connect to AMQP server"<br>
>> "Connected to AMQP server"<br>
>> "Re-established AMQP queues"<br>
>> "AMQP server on.*unreachable"<br>
>> 'Failed to consume"<br>
>> "Failed to publish"<br>
>><br>
>> Thanks,<br>
>><br>
>> - Chris<br>
>><br>
>><br>
>> On Dec 24, 2011, at 1:56 PM, Roman Sokolkov wrote:<br>
>><br>
>>> Hello!<br>
>>><br>
>>> If for some reason rabbitmq has been down some time or connection was aborted, nova couldn't reconnect automatically and I need to restart all nova services manually.<br>
>>><br>
>>> Additionally after restarting rabbit, If I make some call to API. For example, "nova floating-ip-list" only<br>
>>> API reconnects automatically. Other nova services staying disconnected.<br>
>>><br>
>>> Are there mechanisms to monitor establising of connections? And it difficult to add them?<br>
>>><br>
>>> --<br>
>>> Best regards, Roman Sokolkov<br>
>>> _______________________________________________<br>
>>> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
>>> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>>> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Mailing list: <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>> Post to : <a href="mailto:openstack@lists.launchpad.net" target="_blank">openstack@lists.launchpad.net</a><br>
>> Unsubscribe : <a href="https://launchpad.net/~openstack" target="_blank">https://launchpad.net/~openstack</a><br>
>> More help : <a href="https://help.launchpad.net/ListHelp" target="_blank">https://help.launchpad.net/ListHelp</a><br>
<br>
</div></div></blockquote></div><br><br clear="all"><div><br></div></div></div><span class="HOEnZb"><font color="#888888">-- <br>Regards, Roman Sokolkov<br>
</font></span></div>
</div><br><br clear="all"><div><br></div>-- <br>Regards, Roman Sokolkov<br>