<div dir="ltr"><div><div><div><div>I confirm that it happened again. I started all the designate processes and after an hour (approximately), the designate-api process died with the same stack trace.<br></div><br>Restarting the designate-api does not help because API requests are not replied any more (timeouts):<br>2015-07-21 10:12:53.463 4403 ERROR designate.api.middleware [req-281e9665-c49b-43e0-a5d0-9a48e5f52aa1 noauth-user noauth-project - - -] Timed out waiting for a reply to message ID 19cda11c089f43e2a43a75a5851926c8<br><br></div>When the designate-api process dies, I need to restart all the designate processes. Then the api works correctly until the process dies again.<br><br></div>I consider that it's normal the hex dump for qpid (it is debug level) although I noticed it was different in rabbitmq.<br><br></div>We have deployed designate in an Ubuntu host, as installation instructions recommended, and I don't think there is any security issue to stop the service. In fact, the trace is really strange because the api was already bound on port 9001. Our Openstack platform is supported by RedHat and that's why we need to integrate with qpid.<br><div><div><div><br></div><div>I will try a couple of different scenarios:<br></div><div>a) Use a qpid local instance (instead of OST qpid instance)<br></div><div>b) Use a rabbitmq local instance<br></div><div><br></div><div><br></div></div></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jul 20, 2015 at 5:20 PM, Kiall Mac Innes <span dir="ltr"><<a href="mailto:kiall@macinnes.ie" target="_blank">kiall@macinnes.ie</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Side Question: Is is normal for QPid to log all the<br>
"\x0f\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00"'s etc?<br>
<br>
I'm guessing that, since you're using qpid, you're also on RedHat. Could<br>
RH's SELinux policies be preventing the service from binding to tcp/9001?<br>
<br>
If you start as root, do you see similar issues?<br>
<br>
Thanks,<br>
Kiall<br>
<span class=""><br>
On 20/07/15 15:48, Jaime Fernández wrote:<br>
> Hi Tim,<br>
><br>
> I only start one api process. In fact, when I say that the api process<br>
> dies, I don't have any designate-api process and there is no process<br>
> listening on the 9001 port.<br>
><br>
> When I started all the designate processes, the API worked correctly<br>
> because I had tested it. But after some inactivity period (a couple of<br>
> hours, or a day), then the designate-api process died. It is not<br>
> possible that the process has been restarted during this time.<br>
><br>
> I've just started the process again and now it works. I will check if it<br>
> dies again and report it.<br>
><br>
><br>
> Thanks<br>
><br>
</span><span class="">> On Mon, Jul 20, 2015 at 4:24 PM, Tim Simmons <<a href="mailto:tim.simmons@rackspace.com">tim.simmons@rackspace.com</a><br>
> <mailto:<a href="mailto:tim.simmons@rackspace.com">tim.simmons@rackspace.com</a>>> wrote:<br>
><br>
> Jaime,<br>
><br>
><br>
> Usually that's the error you see if you're trying to start up<br>
> multiple API processes. They all try and bind to port 9001, so that<br>
> error is saying the API can't bind. So something else (I suspect<br>
> another designate-api process, or some other type of API) is already<br>
> listening on that port.<br>
><br>
><br>
> Hope that helps,<br>
><br>
> Tim Simmons<br>
><br>
><br>
</span>> ------------------------------------------------------------------------<br>
> *From:* Jaime Fernández <<a href="mailto:jjjaime@gmail.com">jjjaime@gmail.com</a> <mailto:<a href="mailto:jjjaime@gmail.com">jjjaime@gmail.com</a>>><br>
> *Sent:* Monday, July 20, 2015 8:54 AM<br>
> *To:* OpenStack Development Mailing List (not for usage questions)<br>
> *Subject:* [openstack-dev] [designate] The designate API service is<br>
<span class="">> stopped<br>
><br>
> I've followed instructions to install Designate in Dev environment:<br>
> <a href="http://docs.openstack.org/developer/designate/install/ubuntu-dev.html" rel="noreferrer" target="_blank">http://docs.openstack.org/developer/designate/install/ubuntu-dev.html</a><br>
><br>
</span><div><div class="h5">> I've made some slight modifications to use qpid (instead of<br>
> rabbitmq) and to integrate with Infoblox.<br>
><br>
> What I've seen is that designate-api process dies (the other<br>
> processes are running correctly). I'm not sure if the problem could<br>
> be a network issue between designate-api and qpid.<br>
><br>
> Here it is the output for last traces of designate-api process:<br>
><br>
> 2015-07-20 14:43:37.728 727 DEBUG qpid.messaging.io.raw [-]<br>
> READ[3f383f8]:<br>
> '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00'<br>
> readable<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411<br>
> 2015-07-20 14:43:37.729 727 DEBUG qpid.messaging.io.ops [-]<br>
> RCVD[3f383f8]: ConnectionHeartbeat() write<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651<br>
> 2015-07-20 14:43:37.729 727 DEBUG qpid.messaging.io.ops [-]<br>
> SENT[3f383f8]: ConnectionHeartbeat() write_op<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683<br>
> 2015-07-20 14:43:37.730 727 DEBUG qpid.messaging.io.raw [-]<br>
> SENT[3f383f8]:<br>
> '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00'<br>
> writeable<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475<br>
> Traceback (most recent call last):<br>
> File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/hub.py",<br>
> line 457, in fire_timers<br>
> timer()<br>
> File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/timer.py",<br>
> line 58, in __call__<br>
> cb(*args, **kw)<br>
> File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py",<br>
> line 214, in main<br>
> result = function(*args, **kwargs)<br>
> File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/service.py",<br>
> line 623, in run_service<br>
> service.start()<br>
> File "/home/sysadmin/openstack/designate/designate/service.py",<br>
> line 173, in start<br>
> socket = self._wsgi_get_socket()<br>
> File "/home/sysadmin/openstack/designate/designate/service.py",<br>
> line 209, in _wsgi_get_socket<br>
> 'port': self._service_config.api_port})<br>
</div></div>> RuntimeError: Could not bind to <a href="http://0.0.0.0:9001" rel="noreferrer" target="_blank">0.0.0.0:9001</a> <<a href="http://0.0.0.0:9001" rel="noreferrer" target="_blank">http://0.0.0.0:9001</a>><br>
<span class="">> after trying for 30 seconds<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup [-] Could<br>
</span>> not bind to <a href="http://0.0.0.0:9001" rel="noreferrer" target="_blank">0.0.0.0:9001</a> <<a href="http://0.0.0.0:9001" rel="noreferrer" target="_blank">http://0.0.0.0:9001</a>> after trying for 30<br>
<div><div class="h5">> seconds<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup Traceback<br>
> (most recent call last):<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/threadgroup.py",<br>
> line 154, in wait<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup x.wait()<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/threadgroup.py",<br>
> line 51, in wait<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup<br>
> return self.thread.wait()<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py",<br>
> line 175, in wait<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup<br>
> return self._exit_event.wait()<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/event.py",<br>
> line 121, in wait<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup<br>
> return hubs.get_hub().switch()<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/hub.py",<br>
> line 294, in switch<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup<br>
> return self.greenlet.switch()<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py",<br>
> line 214, in main<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup<br>
> result = function(*args, **kwargs)<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File<br>
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/service.py",<br>
> line 623, in run_service<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup<br>
> service.start()<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File<br>
> "/home/sysadmin/openstack/designate/designate/service.py", line 173,<br>
> in start<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup<br>
> socket = self._wsgi_get_socket()<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File<br>
> "/home/sysadmin/openstack/designate/designate/service.py", line 209,<br>
> in _wsgi_get_socket<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup<br>
> 'port': self._service_config.api_port})<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup<br>
</div></div>> RuntimeError: Could not bind to <a href="http://0.0.0.0:9001" rel="noreferrer" target="_blank">0.0.0.0:9001</a> <<a href="http://0.0.0.0:9001" rel="noreferrer" target="_blank">http://0.0.0.0:9001</a>><br>
<div><div class="h5">> after trying for 30 seconds<br>
> 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup<br>
> 2015-07-20 14:43:55.222 727 INFO designate.service [-] Stopping api<br>
> service<br>
> 2015-07-20 14:43:55.222 727 DEBUG qpid.messaging.io.ops [-]<br>
> SENT[3f383f8]:<br>
> SessionDetach(name='08d49368-a86c-4736-bf1c-bd04d09ad579:31')<br>
> write_op<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683<br>
> 2015-07-20 14:43:55.223 727 DEBUG qpid.messaging.io.raw [-]<br>
> SENT[3f383f8]:<br>
> "\x0f\x00\x009\x00\x00\x00\x00\x00\x00\x00\x00\x02\x03\x01\x00\x00'08d49368-a86c-4736-bf1c-bd04d09ad579:31"<br>
> writeable<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475<br>
> 2015-07-20 14:43:55.235 727 DEBUG qpid.messaging.io.raw [-]<br>
> READ[3f383f8]:<br>
> "\x0f\x00\x00:\x00\x00\x00\x00\x00\x00\x00\x00\x02\x04\x03\x00\x00'08d49368-a86c-4736-bf1c-bd04d09ad579:31\x00"<br>
> readable<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411<br>
> 2015-07-20 14:43:55.235 727 DEBUG qpid.messaging.io.ops [-]<br>
> RCVD[3f383f8]:<br>
> SessionDetached(name='08d49368-a86c-4736-bf1c-bd04d09ad579:31',<br>
> code=0) write<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651<br>
> 2015-07-20 14:43:55.236 727 DEBUG qpid.messaging.io.ops [-]<br>
> SENT[3f383f8]: ConnectionClose(reply_code=200) write_op<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683<br>
> 2015-07-20 14:43:55.236 727 DEBUG qpid.messaging.io.raw [-]<br>
> SENT[3f383f8]:<br>
> '\x0f\x00\x00\x12\x00\x00\x00\x00\x00\x00\x00\x00\x01\x0b\x01\x00\x00\xc8'<br>
> writeable<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475<br>
> 2015-07-20 14:43:55.249 727 DEBUG qpid.messaging.io.raw [-]<br>
> READ[3f383f8]:<br>
> '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\x0c\x00\x00'<br>
> readable<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411<br>
> 2015-07-20 14:43:55.249 727 DEBUG qpid.messaging.io.ops [-]<br>
> RCVD[3f383f8]: ConnectionCloseOk() write<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651<br>
> 2015-07-20 14:43:55.639 727 DEBUG qpid.messaging.io.ops [-]<br>
> SENT[3ea43f8]: MessageCancel(destination='0', id=serial(71))<br>
> write_op<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683<br>
> 2015-07-20 14:43:55.640 727 DEBUG qpid.messaging.io.ops [-]<br>
> SENT[3ea43f8]:<br>
> QueueDelete(queue='reply_0b3c96ac8b0c4eabbce54791d02166ec',<br>
> id=serial(72), sync=True) write_op<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683<br>
> 2015-07-20 14:43:55.641 727 DEBUG qpid.messaging.io.raw [-]<br>
> SENT[3ea43f8]:<br>
> '\x0f\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x04\x08\x01\x00\x01\x00\x010\x0f\x01\x009\x00\x01\x00\x00\x00\x00\x00\x00\x08\x02\x01\x01\x01\x00&reply_0b3c96ac8b0c4eabbce54791d02166ec'<br>
> writeable<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475<br>
> 2015-07-20 14:43:55.653 727 DEBUG qpid.messaging.io.raw [-]<br>
> READ[3ea43f8]:<br>
> '\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00H'<br>
> readable<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411<br>
> 2015-07-20 14:43:55.654 727 DEBUG qpid.messaging.io.ops [-]<br>
> RCVD[3ea43f8]: SessionCompleted(commands=[0-72]) write<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651<br>
> 2015-07-20 14:43:55.655 727 DEBUG qpid.messaging.io.ops [-]<br>
> SENT[3ea43f8]:<br>
> SessionDetach(name='303ecf97-cc51-4aba-84a0-4923f5ea50d4:0')<br>
> write_op<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683<br>
> 2015-07-20 14:43:55.656 727 DEBUG qpid.messaging.io.raw [-]<br>
> SENT[3ea43f8]:<br>
> '\x0f\x00\x008\x00\x00\x00\x00\x00\x00\x00\x00\x02\x03\x01\x00\x00&303ecf97-cc51-4aba-84a0-4923f5ea50d4:0'<br>
> writeable<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475<br>
> 2015-07-20 14:43:55.668 727 DEBUG qpid.messaging.io.raw [-]<br>
> READ[3ea43f8]:<br>
> '\x0f\x00\x009\x00\x00\x00\x00\x00\x00\x00\x00\x02\x04\x03\x00\x00&303ecf97-cc51-4aba-84a0-4923f5ea50d4:0\x00'<br>
> readable<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411<br>
> 2015-07-20 14:43:55.668 727 DEBUG qpid.messaging.io.ops [-]<br>
> RCVD[3ea43f8]:<br>
> SessionDetached(name='303ecf97-cc51-4aba-84a0-4923f5ea50d4:0',<br>
> code=0) write<br>
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651<br>
><br>
> Have you experienced this issue? Do you know the source of the problem?<br>
><br>
> Thanks.<br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe:<br>
</div></div>> <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<span class="">><br>
><br>
><br>
><br>
> __________________________________________________________________________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
</span>> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<span class="">><br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
</span>Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</blockquote></div><br></div>