[openstack-dev] [designate] The designate API service is stopped
Jaime Fernández
jjjaime at gmail.com
Tue Jul 21 08:24:53 UTC 2015
I confirm that it happened again. I started all the designate processes and
after an hour (approximately), the designate-api process died with the same
stack trace.
Restarting the designate-api does not help because API requests are not
replied any more (timeouts):
2015-07-21 10:12:53.463 4403 ERROR designate.api.middleware
[req-281e9665-c49b-43e0-a5d0-9a48e5f52aa1 noauth-user noauth-project - - -]
Timed out waiting for a reply to message ID 19cda11c089f43e2a43a75a5851926c8
When the designate-api process dies, I need to restart all the designate
processes. Then the api works correctly until the process dies again.
I consider that it's normal the hex dump for qpid (it is debug level)
although I noticed it was different in rabbitmq.
We have deployed designate in an Ubuntu host, as installation instructions
recommended, and I don't think there is any security issue to stop the
service. In fact, the trace is really strange because the api was already
bound on port 9001. Our Openstack platform is supported by RedHat and
that's why we need to integrate with qpid.
I will try a couple of different scenarios:
a) Use a qpid local instance (instead of OST qpid instance)
b) Use a rabbitmq local instance
On Mon, Jul 20, 2015 at 5:20 PM, Kiall Mac Innes <kiall at macinnes.ie> wrote:
> Side Question: Is is normal for QPid to log all the
> "\x0f\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00"'s etc?
>
> I'm guessing that, since you're using qpid, you're also on RedHat. Could
> RH's SELinux policies be preventing the service from binding to tcp/9001?
>
> If you start as root, do you see similar issues?
>
> Thanks,
> Kiall
>
> On 20/07/15 15:48, Jaime Fernández wrote:
> > Hi Tim,
> >
> > I only start one api process. In fact, when I say that the api process
> > dies, I don't have any designate-api process and there is no process
> > listening on the 9001 port.
> >
> > When I started all the designate processes, the API worked correctly
> > because I had tested it. But after some inactivity period (a couple of
> > hours, or a day), then the designate-api process died. It is not
> > possible that the process has been restarted during this time.
> >
> > I've just started the process again and now it works. I will check if it
> > dies again and report it.
> >
> >
> > Thanks
> >
> > On Mon, Jul 20, 2015 at 4:24 PM, Tim Simmons <tim.simmons at rackspace.com
> > <mailto:tim.simmons at rackspace.com>> wrote:
> >
> > Jaime,
> >
> >
> > Usually that's the error you see if you're trying to start up
> > multiple API processes. They all try and bind to port 9001, so that
> > error is saying the API can't bind. So something else (I suspect
> > another designate-api process, or some other type of API) is already
> > listening on that port.
> >
> >
> > Hope that helps,
> >
> > Tim Simmons
> >
> >
> >
> ------------------------------------------------------------------------
> > *From:* Jaime Fernández <jjjaime at gmail.com <mailto:jjjaime at gmail.com
> >>
> > *Sent:* Monday, July 20, 2015 8:54 AM
> > *To:* OpenStack Development Mailing List (not for usage questions)
> > *Subject:* [openstack-dev] [designate] The designate API service is
> > stopped
> >
> > I've followed instructions to install Designate in Dev environment:
> >
> http://docs.openstack.org/developer/designate/install/ubuntu-dev.html
> >
> > I've made some slight modifications to use qpid (instead of
> > rabbitmq) and to integrate with Infoblox.
> >
> > What I've seen is that designate-api process dies (the other
> > processes are running correctly). I'm not sure if the problem could
> > be a network issue between designate-api and qpid.
> >
> > Here it is the output for last traces of designate-api process:
> >
> > 2015-07-20 14:43:37.728 727 DEBUG qpid.messaging.io.raw [-]
> > READ[3f383f8]:
> > '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00'
> > readable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411
> > 2015-07-20 14:43:37.729 727 DEBUG qpid.messaging.io.ops [-]
> > RCVD[3f383f8]: ConnectionHeartbeat() write
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651
> > 2015-07-20 14:43:37.729 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3f383f8]: ConnectionHeartbeat() write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:37.730 727 DEBUG qpid.messaging.io.raw [-]
> > SENT[3f383f8]:
> > '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00'
> > writeable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475
> > Traceback (most recent call last):
> > File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/hub.py",
> > line 457, in fire_timers
> > timer()
> > File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/timer.py",
> > line 58, in __call__
> > cb(*args, **kw)
> > File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py",
> > line 214, in main
> > result = function(*args, **kwargs)
> > File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/service.py",
> > line 623, in run_service
> > service.start()
> > File "/home/sysadmin/openstack/designate/designate/service.py",
> > line 173, in start
> > socket = self._wsgi_get_socket()
> > File "/home/sysadmin/openstack/designate/designate/service.py",
> > line 209, in _wsgi_get_socket
> > 'port': self._service_config.api_port})
> > RuntimeError: Could not bind to 0.0.0.0:9001 <http://0.0.0.0:9001>
> > after trying for 30 seconds
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup [-] Could
> > not bind to 0.0.0.0:9001 <http://0.0.0.0:9001> after trying for 30
> > seconds
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup Traceback
> > (most recent call last):
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/threadgroup.py",
> > line 154, in wait
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> x.wait()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/threadgroup.py",
> > line 51, in wait
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > return self.thread.wait()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py",
> > line 175, in wait
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > return self._exit_event.wait()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/event.py",
> > line 121, in wait
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > return hubs.get_hub().switch()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/hub.py",
> > line 294, in switch
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > return self.greenlet.switch()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py",
> > line 214, in main
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > result = function(*args, **kwargs)
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/service.py",
> > line 623, in run_service
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > service.start()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> > "/home/sysadmin/openstack/designate/designate/service.py", line 173,
> > in start
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > socket = self._wsgi_get_socket()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> > "/home/sysadmin/openstack/designate/designate/service.py", line 209,
> > in _wsgi_get_socket
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > 'port': self._service_config.api_port})
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > RuntimeError: Could not bind to 0.0.0.0:9001 <http://0.0.0.0:9001>
> > after trying for 30 seconds
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > 2015-07-20 14:43:55.222 727 INFO designate.service [-] Stopping api
> > service
> > 2015-07-20 14:43:55.222 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3f383f8]:
> > SessionDetach(name='08d49368-a86c-4736-bf1c-bd04d09ad579:31')
> > write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:55.223 727 DEBUG qpid.messaging.io.raw [-]
> > SENT[3f383f8]:
> >
> "\x0f\x00\x009\x00\x00\x00\x00\x00\x00\x00\x00\x02\x03\x01\x00\x00'08d49368-a86c-4736-bf1c-bd04d09ad579:31"
> > writeable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475
> > 2015-07-20 14:43:55.235 727 DEBUG qpid.messaging.io.raw [-]
> > READ[3f383f8]:
> >
> "\x0f\x00\x00:\x00\x00\x00\x00\x00\x00\x00\x00\x02\x04\x03\x00\x00'08d49368-a86c-4736-bf1c-bd04d09ad579:31\x00"
> > readable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411
> > 2015-07-20 14:43:55.235 727 DEBUG qpid.messaging.io.ops [-]
> > RCVD[3f383f8]:
> > SessionDetached(name='08d49368-a86c-4736-bf1c-bd04d09ad579:31',
> > code=0) write
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651
> > 2015-07-20 14:43:55.236 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3f383f8]: ConnectionClose(reply_code=200) write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:55.236 727 DEBUG qpid.messaging.io.raw [-]
> > SENT[3f383f8]:
> >
> '\x0f\x00\x00\x12\x00\x00\x00\x00\x00\x00\x00\x00\x01\x0b\x01\x00\x00\xc8'
> > writeable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475
> > 2015-07-20 14:43:55.249 727 DEBUG qpid.messaging.io.raw [-]
> > READ[3f383f8]:
> > '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\x0c\x00\x00'
> > readable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411
> > 2015-07-20 14:43:55.249 727 DEBUG qpid.messaging.io.ops [-]
> > RCVD[3f383f8]: ConnectionCloseOk() write
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651
> > 2015-07-20 14:43:55.639 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3ea43f8]: MessageCancel(destination='0', id=serial(71))
> > write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:55.640 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3ea43f8]:
> > QueueDelete(queue='reply_0b3c96ac8b0c4eabbce54791d02166ec',
> > id=serial(72), sync=True) write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:55.641 727 DEBUG qpid.messaging.io.raw [-]
> > SENT[3ea43f8]:
> >
> '\x0f\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x04\x08\x01\x00\x01\x00\x010\x0f\x01\x009\x00\x01\x00\x00\x00\x00\x00\x00\x08\x02\x01\x01\x01\x00&reply_0b3c96ac8b0c4eabbce54791d02166ec'
> > writeable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475
> > 2015-07-20 14:43:55.653 727 DEBUG qpid.messaging.io.raw [-]
> > READ[3ea43f8]:
> >
> '\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00H'
> > readable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411
> > 2015-07-20 14:43:55.654 727 DEBUG qpid.messaging.io.ops [-]
> > RCVD[3ea43f8]: SessionCompleted(commands=[0-72]) write
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651
> > 2015-07-20 14:43:55.655 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3ea43f8]:
> > SessionDetach(name='303ecf97-cc51-4aba-84a0-4923f5ea50d4:0')
> > write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:55.656 727 DEBUG qpid.messaging.io.raw [-]
> > SENT[3ea43f8]:
> >
> '\x0f\x00\x008\x00\x00\x00\x00\x00\x00\x00\x00\x02\x03\x01\x00\x00&303ecf97-cc51-4aba-84a0-4923f5ea50d4:0'
> > writeable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475
> > 2015-07-20 14:43:55.668 727 DEBUG qpid.messaging.io.raw [-]
> > READ[3ea43f8]:
> >
> '\x0f\x00\x009\x00\x00\x00\x00\x00\x00\x00\x00\x02\x04\x03\x00\x00&303ecf97-cc51-4aba-84a0-4923f5ea50d4:0\x00'
> > readable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411
> > 2015-07-20 14:43:55.668 727 DEBUG qpid.messaging.io.ops [-]
> > RCVD[3ea43f8]:
> > SessionDetached(name='303ecf97-cc51-4aba-84a0-4923f5ea50d4:0',
> > code=0) write
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651
> >
> > Have you experienced this issue? Do you know the source of the
> problem?
> >
> > Thanks.
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > <
> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150721/a8e8affc/attachment.html>
More information about the OpenStack-dev
mailing list