Open Stack

Seems like I'm hitting following bug, although the underlying networking
issue could be different.
>
> https://bugs.launchpad.net/nova/+bug/856764 (RabbitMQ connections lack
> heartbeat or TCP keepalives)


Will try to upgrade amqblib / kombu version and check if it can fix the
problem. Thanks!

--
Qiu Yu
http://www.unicell.info


On Wed, Dec 19, 2012 at 9:39 PM, unicell <unicell at gmail.com> wrote:

> Just got luck to reproduce it again！.
>
> All Recv-Q/Send-Q are empty, both controller side and compute node side.
> --
> root at 10.81.44.230:~# netstat -ant | grep -E 'Recv-Q|5672'
>
> Proto Recv-Q Send-Q Local Address           Foreign Address         State
>
> tcp        0      0 0.0.0.0:55672           0.0.0.0:*
> LISTEN
> tcp        0      0 127.0.0.1:57497         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:57499         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:56775         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:56877         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:56866         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:57507         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:56778         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:56779         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:56491         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:57508         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:56777         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:56776         127.0.0.1:5672
>  ESTABLISHED
> tcp        0      0 127.0.0.1:56493         127.0.0.1:5672
>  ESTABLISHED
> tcp6       0      0 :::5672                 :::*                    LISTEN
>
> tcp6       0      0 10.81.44.230:5672       10.46.139.31:59842
>  ESTABLISHED
> tcp6       0      0 10.81.44.230:5672       10.46.178.20:51570
>  ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:57508
> ESTABLISHED
> tcp6       0      0 10.81.44.230:5672       10.46.139.31:59843
>  ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:56775
> ESTABLISHED
> tcp6       0      0 10.81.44.230:5672       10.46.178.20:59565
>  ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:57497
> ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:56779
> ESTABLISHED
> tcp6       0      0 10.81.44.230:5672       10.46.139.171:54251
> ESTABLISHED
> tcp6       0      0 10.81.44.230:5672       10.46.178.20:54857
>  ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:57499
> ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:57507
> ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:56877
> ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:56866
> ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:56778
> ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:56777
> ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:56491
> ESTABLISHED
> tcp6       0      0 10.81.44.230:5672       10.46.139.171:54250
> ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:56493
> ESTABLISHED
> tcp6       0      0 127.0.0.1:5672          127.0.0.1:56776
> ESTABLISHED
>
>
> [root at 10.46.178.20 log]# netstat -ant | grep -E 'Recv-Q|5672'
> Proto Recv-Q Send-Q Local Address               Foreign Address
>   State
> tcp        0      0 10.46.178.20:59565          10.81.44.230:5672
>   ESTABLISHED
> tcp        0      0 10.46.178.20:59441          10.81.44.230:5672
>   ESTABLISHED
>
> --
> Qiu Yu
> http://www.unicell.info
>
>
>
> On Wed, Dec 19, 2012 at 8:59 PM, unicell <unicell at gmail.com> wrote:
>
>> Thank you Wen Jian. I don't possess the environment anymore. :-(  So
>> I'll try to follow your advice whenever it is reproducible.
>>
>> Posting this message here trying to collect more ideas, in case it
>> happens again.
>>
>> --
>> Qiu Yu
>>
>>
>> On Wed, Dec 19, 2012 at 8:41 PM, Jian Wen <wenjian at canonical.com> wrote:
>>
>>> , the TCP connection is
>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20121220/11dd8589/attachment.html>