[openstack-dev] [designate] The designate API service is stopped
Kiall Mac Innes
kiall at macinnes.ie
Tue Jul 21 09:31:06 UTC 2015
Inline.
Thanks,
Kiall
On 21/07/15 09:24, Jaime Fernández wrote:
> I confirm that it happened again. I started all the designate processes
> and after an hour (approximately), the designate-api process died with
> the same stack trace.
>
> Restarting the designate-api does not help because API requests are not
> replied any more (timeouts):
> 2015-07-21 10:12:53.463 4403 ERROR designate.api.middleware
> [req-281e9665-c49b-43e0-a5d0-9a48e5f52aa1 noauth-user noauth-project - -
> -] Timed out waiting for a reply to message ID
> 19cda11c089f43e2a43a75a5851926c8
This suggests the API service is back up and running, but either your
message broker (qpid), or the target service (designate-central) is down.
>
> When the designate-api process dies, I need to restart all the designate
> processes. Then the api works correctly until the process dies again.
This leads me to believe qpid is somehow responsible, but, I don't have
any concrete reasons to believe that. Simply a gut feeling! If all
services need to be restarted, then it's most likely a shared resource
that's failing, mis-configured, or somehow incompatible with Designate.
I've never seen anything like this before :/
>
> I consider that it's normal the hex dump for qpid (it is debug level)
> although I noticed it was different in rabbitmq.
>
> We have deployed designate in an Ubuntu host, as installation
> instructions recommended, and I don't think there is any security issue
> to stop the service. In fact, the trace is really strange because the
> api was already bound on port 9001. Our Openstack platform is supported
> by RedHat and that's why we need to integrate with qpid.
>
> I will try a couple of different scenarios:
> a) Use a qpid local instance (instead of OST qpid instance)
> b) Use a rabbitmq local instance
Can you try with RabbitMQ? It's really the only one we test, but it's
all hidden behind the oslo.messaging library, so I can't think of any
good reason why one would work and the other doesn't. If this works,
I'll test out with qpid and see if there's anything obvious.
>
> On Mon, Jul 20, 2015 at 5:20 PM, Kiall Mac Innes <kiall at macinnes.ie
> <mailto:kiall at macinnes.ie>> wrote:
>
> Side Question: Is is normal for QPid to log all the
> "\x0f\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00"'s etc?
>
> I'm guessing that, since you're using qpid, you're also on RedHat. Could
> RH's SELinux policies be preventing the service from binding to
> tcp/9001?
>
> If you start as root, do you see similar issues?
>
> Thanks,
> Kiall
>
> On 20/07/15 15:48, Jaime Fernández wrote:
> > Hi Tim,
> >
> > I only start one api process. In fact, when I say that the api process
> > dies, I don't have any designate-api process and there is no process
> > listening on the 9001 port.
> >
> > When I started all the designate processes, the API worked correctly
> > because I had tested it. But after some inactivity period (a couple of
> > hours, or a day), then the designate-api process died. It is not
> > possible that the process has been restarted during this time.
> >
> > I've just started the process again and now it works. I will check if it
> > dies again and report it.
> >
> >
> > Thanks
> >
> > On Mon, Jul 20, 2015 at 4:24 PM, Tim Simmons <tim.simmons at rackspace.com <mailto:tim.simmons at rackspace.com>
> > <mailto:tim.simmons at rackspace.com <mailto:tim.simmons at rackspace.com>>> wrote:
> >
> > Jaime,
> >
> >
> > Usually that's the error you see if you're trying to start up
> > multiple API processes. They all try and bind to port 9001, so that
> > error is saying the API can't bind. So something else (I suspect
> > another designate-api process, or some other type of API) is already
> > listening on that port.
> >
> >
> > Hope that helps,
> >
> > Tim Simmons
> >
> >
> >
> ------------------------------------------------------------------------
> > *From:* Jaime Fernández <jjjaime at gmail.com
> <mailto:jjjaime at gmail.com> <mailto:jjjaime at gmail.com
> <mailto:jjjaime at gmail.com>>>
> > *Sent:* Monday, July 20, 2015 8:54 AM
> > *To:* OpenStack Development Mailing List (not for usage questions)
> > *Subject:* [openstack-dev] [designate] The designate API
> service is
> > stopped
> >
> > I've followed instructions to install Designate in Dev environment:
> > http://docs.openstack.org/developer/designate/install/ubuntu-dev.html
> >
> > I've made some slight modifications to use qpid (instead of
> > rabbitmq) and to integrate with Infoblox.
> >
> > What I've seen is that designate-api process dies (the other
> > processes are running correctly). I'm not sure if the problem
> could
> > be a network issue between designate-api and qpid.
> >
> > Here it is the output for last traces of designate-api process:
> >
> > 2015-07-20 14:43:37.728 727 DEBUG qpid.messaging.io.raw [-]
> > READ[3f383f8]:
> > '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00'
> > readable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411
> > 2015-07-20 14:43:37.729 727 DEBUG qpid.messaging.io.ops [-]
> > RCVD[3f383f8]: ConnectionHeartbeat() write
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651
> > 2015-07-20 14:43:37.729 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3f383f8]: ConnectionHeartbeat() write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:37.730 727 DEBUG qpid.messaging.io.raw [-]
> > SENT[3f383f8]:
> > '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\n\x00\x00'
> > writeable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475
> > Traceback (most recent call last):
> > File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/hub.py",
> > line 457, in fire_timers
> > timer()
> > File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/timer.py",
> > line 58, in __call__
> > cb(*args, **kw)
> > File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py",
> > line 214, in main
> > result = function(*args, **kwargs)
> > File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/service.py",
> > line 623, in run_service
> > service.start()
> > File "/home/sysadmin/openstack/designate/designate/service.py",
> > line 173, in start
> > socket = self._wsgi_get_socket()
> > File "/home/sysadmin/openstack/designate/designate/service.py",
> > line 209, in _wsgi_get_socket
> > 'port': self._service_config.api_port})
> > RuntimeError: Could not bind to 0.0.0.0:9001
> <http://0.0.0.0:9001> <http://0.0.0.0:9001>
> > after trying for 30 seconds
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup [-] Could
> > not bind to 0.0.0.0:9001 <http://0.0.0.0:9001>
> <http://0.0.0.0:9001> after trying for 30
> > seconds
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> Traceback
> > (most recent call last):
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/threadgroup.py",
> > line 154, in wait
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> x.wait()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/threadgroup.py",
> > line 51, in wait
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > return self.thread.wait()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py",
> > line 175, in wait
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > return self._exit_event.wait()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/event.py",
> > line 121, in wait
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > return hubs.get_hub().switch()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/hubs/hub.py",
> > line 294, in switch
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > return self.greenlet.switch()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/eventlet-0.17.4-py2.7.egg/eventlet/greenthread.py",
> > line 214, in main
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > result = function(*args, **kwargs)
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> >
> "/home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/oslo.service-0.4.0-py2.7.egg/oslo_service/service.py",
> > line 623, in run_service
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > service.start()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> > "/home/sysadmin/openstack/designate/designate/service.py",
> line 173,
> > in start
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > socket = self._wsgi_get_socket()
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup File
> > "/home/sysadmin/openstack/designate/designate/service.py",
> line 209,
> > in _wsgi_get_socket
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > 'port': self._service_config.api_port})
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > RuntimeError: Could not bind to 0.0.0.0:9001
> <http://0.0.0.0:9001> <http://0.0.0.0:9001>
> > after trying for 30 seconds
> > 2015-07-20 14:43:55.221 727 ERROR oslo_service.threadgroup
> > 2015-07-20 14:43:55.222 727 INFO designate.service [-]
> Stopping api
> > service
> > 2015-07-20 14:43:55.222 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3f383f8]:
> > SessionDetach(name='08d49368-a86c-4736-bf1c-bd04d09ad579:31')
> > write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:55.223 727 DEBUG qpid.messaging.io.raw [-]
> > SENT[3f383f8]:
> >
> "\x0f\x00\x009\x00\x00\x00\x00\x00\x00\x00\x00\x02\x03\x01\x00\x00'08d49368-a86c-4736-bf1c-bd04d09ad579:31"
> > writeable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475
> > 2015-07-20 14:43:55.235 727 DEBUG qpid.messaging.io.raw [-]
> > READ[3f383f8]:
> >
> "\x0f\x00\x00:\x00\x00\x00\x00\x00\x00\x00\x00\x02\x04\x03\x00\x00'08d49368-a86c-4736-bf1c-bd04d09ad579:31\x00"
> > readable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411
> > 2015-07-20 14:43:55.235 727 DEBUG qpid.messaging.io.ops [-]
> > RCVD[3f383f8]:
> > SessionDetached(name='08d49368-a86c-4736-bf1c-bd04d09ad579:31',
> > code=0) write
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651
> > 2015-07-20 14:43:55.236 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3f383f8]: ConnectionClose(reply_code=200) write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:55.236 727 DEBUG qpid.messaging.io.raw [-]
> > SENT[3f383f8]:
> >
> '\x0f\x00\x00\x12\x00\x00\x00\x00\x00\x00\x00\x00\x01\x0b\x01\x00\x00\xc8'
> > writeable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475
> > 2015-07-20 14:43:55.249 727 DEBUG qpid.messaging.io.raw [-]
> > READ[3f383f8]:
> > '\x0f\x00\x00\x10\x00\x00\x00\x00\x00\x00\x00\x00\x01\x0c\x00\x00'
> > readable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411
> > 2015-07-20 14:43:55.249 727 DEBUG qpid.messaging.io.ops [-]
> > RCVD[3f383f8]: ConnectionCloseOk() write
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651
> > 2015-07-20 14:43:55.639 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3ea43f8]: MessageCancel(destination='0', id=serial(71))
> > write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:55.640 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3ea43f8]:
> > QueueDelete(queue='reply_0b3c96ac8b0c4eabbce54791d02166ec',
> > id=serial(72), sync=True) write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:55.641 727 DEBUG qpid.messaging.io.raw [-]
> > SENT[3ea43f8]:
> >
> '\x0f\x01\x00\x14\x00\x01\x00\x00\x00\x00\x00\x00\x04\x08\x01\x00\x01\x00\x010\x0f\x01\x009\x00\x01\x00\x00\x00\x00\x00\x00\x08\x02\x01\x01\x01\x00&reply_0b3c96ac8b0c4eabbce54791d02166ec'
> > writeable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475
> > 2015-07-20 14:43:55.653 727 DEBUG qpid.messaging.io.raw [-]
> > READ[3ea43f8]:
> >
> '\x0f\x00\x00\x1a\x00\x00\x00\x00\x00\x00\x00\x00\x02\n\x01\x00\x00\x08\x00\x00\x00\x00\x00\x00\x00H'
> > readable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411
> > 2015-07-20 14:43:55.654 727 DEBUG qpid.messaging.io.ops [-]
> > RCVD[3ea43f8]: SessionCompleted(commands=[0-72]) write
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651
> > 2015-07-20 14:43:55.655 727 DEBUG qpid.messaging.io.ops [-]
> > SENT[3ea43f8]:
> > SessionDetach(name='303ecf97-cc51-4aba-84a0-4923f5ea50d4:0')
> > write_op
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:683
> > 2015-07-20 14:43:55.656 727 DEBUG qpid.messaging.io.raw [-]
> > SENT[3ea43f8]:
> >
> '\x0f\x00\x008\x00\x00\x00\x00\x00\x00\x00\x00\x02\x03\x01\x00\x00&303ecf97-cc51-4aba-84a0-4923f5ea50d4:0'
> > writeable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:475
> > 2015-07-20 14:43:55.668 727 DEBUG qpid.messaging.io.raw [-]
> > READ[3ea43f8]:
> >
> '\x0f\x00\x009\x00\x00\x00\x00\x00\x00\x00\x00\x02\x04\x03\x00\x00&303ecf97-cc51-4aba-84a0-4923f5ea50d4:0\x00'
> > readable
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:411
> > 2015-07-20 14:43:55.668 727 DEBUG qpid.messaging.io.ops [-]
> > RCVD[3ea43f8]:
> > SessionDetached(name='303ecf97-cc51-4aba-84a0-4923f5ea50d4:0',
> > code=0) write
> >
> /home/sysadmin/openstack/designate/.venv/local/lib/python2.7/site-packages/qpid/messaging/driver.py:651
> >
> > Have you experienced this issue? Do you know the source of the
> problem?
> >
> > Thanks.
> >
> >
> >
> __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> >
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
> >
> >
> >
> > __________________________________________________________________________
> > OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> >
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
More information about the OpenStack-dev
mailing list