[Openstack-operators] Cannot launch instances on Ocata.
Andy Wojnarek
andy.wojnarek at theatsgroup.com
Wed May 17 22:35:04 UTC 2017
Thanks!
I see my endpoints had controller:8778/placement, and that appears to be wrong… I think they must have updated the Ocata installation guide as such – because I see it correct in the guide now. I also see some people having the same issue as me, where the placement API is returning a 404.
I added the new endpoints in, and restarted, but I’m still getting 404s. So now I’ll just troubleshoot and figure out why I’m getting 404’s in general.
Nova-placement-api.log
017-05-17 18:29:57.341 4941 DEBUG nova.api.openstack.placement.requestlog [req-ef132927-a597-4d6c-9255-b1558d919bda 30ba9e287aff4fe5b806c327901192dd 15f08d64f0ce4dab95448b40a45ff8dd - default default] Starting request: 192.168.241.115 "GET /placement/resource_providers/f4df986c-1a2c-4e0f-827e-9867f5b16b66" __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/placement/requestlog.py:38
2017-05-17 18:29:57.342 4941 INFO nova.api.openstack.placement.requestlog [req-ef132927-a597-4d6c-9255-b1558d919bda 30ba9e287aff4fe5b806c327901192dd 15f08d64f0ce4dab95448b40a45ff8dd - default default] 192.168.241.115 "GET /placement/resource_providers/f4df986c-1a2c-4e0f-827e-9867f5b16b66" status: 404 len: 52 microversion: 1.0
2017-05-17 18:29:57.348 4944 DEBUG nova.api.openstack.placement.requestlog [req-18a8653e-9a30-4015-a66d-c783e44b0310 30ba9e287aff4fe5b806c327901192dd 15f08d64f0ce4dab95448b40a45ff8dd - default default] Starting request: 192.168.241.115 "POST /placement/resource_providers" __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/placement/requestlog.py:38
2017-05-17 18:29:57.349 4944 INFO nova.api.openstack.placement.requestlog [req-18a8653e-9a30-4015-a66d-c783e44b0310 30ba9e287aff4fe5b806c327901192dd 15f08d64f0ce4dab95448b40a45ff8dd - default default] 192.168.241.115 "POST /placement/resource_providers" status: 404 len: 52 microversion: 1.0
2017-05-17 18:29:57.389 4942 DEBUG nova.api.openstack.placement.requestlog [req-2cc2fdae-af5d-499e-93cd-449503218458 30ba9e287aff4fe5b806c327901192dd 15f08d64f0ce4dab95448b40a45ff8dd - default default] Starting request: 192.168.241.115 "GET /placement/resource_providers/f4df986c-1a2c-4e0f-827e-9867f5b16b66/allocations" __call__ /usr/lib/python2.7/dist-packages/nova/api/openstack/placement/requestlog.py:38
2017-05-17 18:29:57.391 4942 INFO nova.api.openstack.placement.requestlog [req-2cc2fdae-af5d-499e-93cd-449503218458 30ba9e287aff4fe5b806c327901192dd 15f08d64f0ce4dab95448b40a45ff8dd - default default] 192.168.241.115 "GET /placement/resource_providers/f4df986c-1a2c-4e0f-827e-9867f5b16b66/allocations" status: 404 len: 52 microversion: 1.0
nova_placement_access.log
192.168.241.115 - - [17/May/2017:18:32:01 -0400] "POST /placement/resource_providers HTTP/1.1" 404 367 "-" "nova-compute keystoneauth1/2.18.0 python-requests/2.12.4 CPython/2.7.12"
192.168.241.115 - - [17/May/2017:18:32:01 -0400] "GET /placement/resource_providers/f4df986c-1a2c-4e0f-827e-9867f5b16b66/allocations HTTP/1.1" 404 367 "-" "nova-compute keystoneauth1/2.18.0 python-requests/2.12.4 CPython/2.7.12"
192.168.241.115 - - [17/May/2017:18:33:02 -0400] "GET /placement/resource_providers/f4df986c-1a2c-4e0f-827e-9867f5b16b66 HTTP/1.1" 404 368 "-" "nova-compute keystoneauth1/2.18.0 python-requests/2.12.4 CPython/2.7.12"
192.168.241.115 - - [17/May/2017:18:33:02 -0400] "POST /placement/resource_providers HTTP/1.1" 404 367 "-" "nova-compute keystoneauth1/2.18.0 python-requests/2.12.4 CPython/2.7.12"
192.168.241.115 - - [17/May/2017:18:33:02 -0400] "GET /placement/resource_providers/f4df986c-1a2c-4e0f-827e-9867f5b16b66/allocations HTTP/1.1" 404 367 "-" "nova-compute keystoneauth1/2.18.0 python-requests/2.12.4 CPython/2.7.12"
root at gvicopnstk01:/var/log/apache2# openstack endpoint list | grep -i 8778
| 4103a80eceb84e2cbdd1f75e1a34321c | RegionOne | placement | placement | True | internal | http://gvicopnstk01:8778/placement |
| 46df3838adbe4af3955cd0dc5a97e11c | RegionOne | placement | placement | True | public | http://gvicopnstk01:8778/placement |
| 5331ff425b384951b503e2dc07e38913 | RegionOne | placement | placement | True | internal | http://gvicopnstk01:8778 |
| a83c1c9c85eb489aa3dd687aa91381e8 | RegionOne | placement | placement | True | public | http://gvicopnstk01:8778 |
| a899bde2010641a192887c9b924de10a | RegionOne | placement | placement | True | admin | http://gvicopnstk01:8778 |
| d5dddc0923c14cda918a502a587e6320 | RegionOne | placement | placement | True | admin | http://gvicopnstk01:8778/placement |
Thanks,
Andrew Wojnarek | Sr. Systems Engineer | ATS Group, LLC
mobile 717.856.6901 | andy.wojnarek at TheATSGroup.com
Galileo Performance Explorer Blog <http://galileosuite.com/blog/> Offers Deep Insights for Server/Storage Systems
On 5/17/17, 5:32 PM, "Erik McCormick" <emccormick at cirrusseven.com> wrote:
I'm just spit-balling now because I haven't used Ocata yet, but your
placement API may be up, but it's returning a 404 so it seems to be
misconfigured somehow.
Perhaps there's some useful nuggets in here:
https://ask.openstack.org/en/question/102256/how-to-configure-placement-service-for-compute-node-on-ocata/
or here:
https://docs.openstack.org/developer/nova/placement.html
I'm assuming that service spits out its own log. IF so, is there anything in it?
-Erik
On Wed, May 17, 2017 at 5:23 PM, Andy Wojnarek
<andy.wojnarek at theatsgroup.com> wrote:
> I’m seeing the following on the controller:
>
>
> 2017-05-17 17:20:12.049 2212 ERROR nova.scheduler.client.report [req-2953a824-f607-4b9d-86bf-f0d585fba787 b07949d8ae7144049851c7abb39ac6db 4fd0307bf4b74c5a8718b180c24c7cff - - -] Failed to retrieve filtered list of resource providers from placement API for filters {'resources': 'DISK_GB:1,MEMORY_MB:512,VCPU:1'}. Got 404: 404 Not Found
>
> The resource could not be found.
>
> .
>
> I don’t see any scheduler log on the compute:
>
> root at gvicopnstk02:/var/log/nova# ls -ltr
> total 58584
> -rw-r--r-- 1 nova nova 1353724 May 14 06:25 nova-compute.log.4.gz
> -rw-rw-r-- 1 nova nova 20 May 14 06:25 nova-manage.log.4.gz
> -rw-r--r-- 1 nova nova 1349014 May 15 06:25 nova-compute.log.3.gz
> -rw-rw-r-- 1 nova nova 20 May 15 06:25 nova-manage.log.3.gz
> -rw-rw-r-- 1 nova nova 20 May 16 06:25 nova-manage.log.2.gz
> -rw-r--r-- 1 nova nova 1350172 May 16 06:25 nova-compute.log.2.gz
> -rw-r--r-- 1 nova nova 38318600 May 17 06:25 nova-compute.log.1
> -rw-rw-r-- 1 nova nova 0 May 17 06:25 nova-manage.log.1
> -rw-rw-r-- 1 nova nova 0 May 17 06:25 nova-manage.log
> -rw-r--r-- 1 nova nova 17588608 May 17 17:21 nova-compute.log
>
>
> 2017-05-17 17:20:46.483 1528 ERROR nova.scheduler.client.report [req-19cd6ce4-cb9c-4b7a-8cdb-0d3643f38701 - - - - -] Failed to create resource provider record in placement API for UUID f4df986c-1a2c-4e0f-827e-9867f5b16b66. Got 404: 404 Not Found
>
> The resource could not be found.
>
>
> So it looks like the placement API isn’t working?
>
>
> Placement looks up and running:
>
>
> root at gvicopnstk01:/var/log/nova# openstack service list
> +----------------------------------+-----------+-----------+
> | ID | Name | Type |
> +----------------------------------+-----------+-----------+
> | 018d4b8b185b4137be4a2fee14b361ee | glance | image |
> | 39d57b81f57140f9936bcc0a6f8ac244 | keystone | identity |
> | 626d6cf1c9c842a39283b5595e597af0 | placement | placement |
> | 6b90234efded4ed9b4344e8eb14f422b | neutron | network |
> | ebbcff558b904f21818a656bd177f51b | nova | compute |
>
>
>
>
>
> root at gvicopnstk01:/var/log/nova# openstack endpoint list | grep -i placement
> | 4103a80eceb84e2cbdd1f75e1a34321c | RegionOne | placement | placement | True | internal | http://gvicopnstk01:8778/placement |
> | 46df3838adbe4af3955cd0dc5a97e11c | RegionOne | placement | placement | True | public | http://gvicopnstk01:8778/placement |
> | d5dddc0923c14cda918a502a587e6320 | RegionOne | placement | placement | True | admin | http://gvicopnstk01:8778/placement |
>
>
> root at gvicopnstk01:/var/log/nova# netstat -an | grep -i 8778
> tcp6 0 0 :::8778 :::* LISTEN
> tcp6 0 0 192.168.241.114:8778 192.168.241.115:50734 TIME_WAIT
> tcp6 0 0 192.168.241.114:8778 192.168.241.115:50736 FIN_WAIT2
>
> This placement thing is new to Ocata right?
>
> Thanks,
> Andrew Wojnarek | Sr. Systems Engineer | ATS Group, LLC
> mobile 717.856.6901 | andy.wojnarek at TheATSGroup.com
> Galileo Performance Explorer Blog <http://galileosuite.com/blog/> Offers Deep Insights for Server/Storage Systems
>
>
> On 5/17/17, 5:19 PM, "Erik McCormick" <emccormick at cirrusseven.com> wrote:
>
> You'll want to check the nova-scheduler.log (controller) and the
> nova-compute.log (compute). You can look for your request ID and then
> go forward from there. Those should shed some more light on what the
> issue is
>
> -Erik
>
> On Wed, May 17, 2017 at 5:09 PM, Andy Wojnarek
> <andy.wojnarek at theatsgroup.com> wrote:
> > Hi,
> >
> >
> >
> > I have a new Openstack cloud running in our lab, but I am unable to launch
> > instances. This is Ocata running on Ubuntu 16.04.2
> >
> >
> >
> > Here are the errors I am getting when trying to launch an instance:
> >
> >
> >
> > On my controller node in log file /var/log/nova/nova-conductor.log
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> > [req-a9beeb33-9454-47a2-96e2-908d5b1e4c46 b07949d8ae7144049851c7abb39ac6db
> > 4fd0307bf4b74c5a8718b180c24c7cff - - -] Failed to schedule instances
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager Traceback (most
> > recent call last):
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 866, in
> > schedule_and_build_instances
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> > request_specs[0].to_legacy_filter_properties_dict())
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/nova/conductor/manager.py", line 597, in
> > _schedule_instances
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager hosts =
> > self.scheduler_client.select_destinations(context, spec_obj)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/nova/scheduler/utils.py", line 371, in
> > wrapped
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager return
> > func(*args, **kwargs)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line
> > 51, in select_destinations
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager return
> > self.queryclient.select_destinations(context, spec_obj)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/nova/scheduler/client/__init__.py", line
> > 37, in __run_method
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager return
> > getattr(self.instance, __name)(*args, **kwargs)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/nova/scheduler/client/query.py", line 32,
> > in select_destinations
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager return
> > self.scheduler_rpcapi.select_destinations(context, spec_obj)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/nova/scheduler/rpcapi.py", line 129, in
> > select_destinations
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager return
> > cctxt.call(ctxt, 'select_destinations', **msg_args)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/client.py", line 169,
> > in call
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> > retry=self.retry)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/oslo_messaging/transport.py", line 97, in
> > _send
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> > timeout=timeout, retry=retry)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
> > line 458, in send
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager retry=retry)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/amqpdriver.py",
> > line 449, in _send
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager raise result
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> > NoValidHost_Remote: No valid host was found. There are not enough hosts
> > available.
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager Traceback (most
> > recent call last):
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py", line 218,
> > in inner
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager return
> > func(*args, **kwargs)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/nova/scheduler/manager.py", line 98, in
> > select_destinations
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager dests =
> > self.driver.select_destinations(ctxt, spec_obj)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager File
> > "/usr/lib/python2.7/dist-packages/nova/scheduler/filter_scheduler.py", line
> > 79, in select_destinations
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager raise
> > exception.NoValidHost(reason=reason)
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager NoValidHost: No
> > valid host was found. There are not enough hosts available.
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> >
> > 2017-05-17 16:48:33.656 2654 ERROR nova.conductor.manager
> >
> > 2017-05-17 16:48:33.686 2654 DEBUG oslo_db.sqlalchemy.engines
> > [req-a9beeb33-9454-47a2-96e2-908d5b1e4c46 b07949d8ae7144049851c7abb39ac6db
> > 4fd0307bf4b74c5a8718b180c24c7cff - - -] MySQL server mode set to
> > STRICT_TRANS_TABLES,STRICT_ALL_TABLES,NO_ZERO_IN_DATE,NO_ZERO_DATE,ERROR_FOR_DIVISION_BY_ZERO,TRADITIONAL,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
> > _check_effective_sql_mode
> > /usr/lib/python2.7/dist-packages/oslo_db/sqlalchemy/engines.py:261
> >
> > 2017-05-17 16:48:36.013 2654 WARNING nova.scheduler.utils
> > [req-a9beeb33-9454-47a2-96e2-908d5b1e4c46 b07949d8ae7144049851c7abb39ac6db
> > 4fd0307bf4b74c5a8718b180c24c7cff - - -] Failed to
> > compute_task_build_instances: No valid host was found. There are not enough
> > hosts available.
> >
> >
> >
> >
> >
> > The hypervisor is up:
> >
> > root at gvicopnstk01:/var/log/nova# openstack hypervisor list
> >
> > +----+---------------------+-----------------+-----------------+-------+
> >
> > | ID | Hypervisor Hostname | Hypervisor Type | Host IP | State |
> >
> > +----+---------------------+-----------------+-----------------+-------+
> >
> > | 1 | gvicopnstk02 | QEMU | 192.168.241.115 | up |
> >
> >
> >
> > Services are up:
> >
> > root at gvicopnstk01:/var/log/nova# openstack compute service list
> >
> > +----+------------------+--------------+----------+---------+-------+----------------------------+
> >
> > | ID | Binary | Host | Zone | Status | State |
> > Updated At |
> >
> > +----+------------------+--------------+----------+---------+-------+----------------------------+
> >
> > | 6 | nova-consoleauth | gvicopnstk01 | internal | enabled | up |
> > 2017-05-17T21:07:00.000000 |
> >
> > | 7 | nova-scheduler | gvicopnstk01 | internal | enabled | up |
> > 2017-05-17T21:07:00.000000 |
> >
> > | 9 | nova-conductor | gvicopnstk01 | internal | enabled | up |
> > 2017-05-17T21:07:00.000000 |
> >
> > | 24 | nova-compute | gvicopnstk02 | nova | enabled | up |
> > 2017-05-17T21:07:07.000000 |
> >
> >
> >
> > I absolutely cannot figure out. It’s acting like there are no valid compute
> > nodes available, but all the Openstack commands are coming back as status is
> > up and running.
> >
> >
> >
> > Thanks,
> >
> > Andrew Wojnarek | Sr. Systems Engineer | ATS Group, LLC
> >
> > mobile 717.856.6901 | andy.wojnarek at TheATSGroup.com
> >
> > Galileo Performance Explorer Blog Offers Deep Insights for Server/Storage
> > Systems
> >
> >
> > _______________________________________________
> > OpenStack-operators mailing list
> > OpenStack-operators at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
> >
>
>
>
More information about the OpenStack-operators
mailing list