[openstack-dev] [stable/ocata] [nova] [devstack multi-node] nova-conductor complaining about "No cell mapping found for cell0"
Prashant Shetty
prashantshetty1985 at gmail.com
Thu Feb 23 09:04:11 UTC 2017
Thanks Matt, I found out there was issue in my nova.conf on controller.
[placement] section was missing on controller nova.conf.
Looks like devstack ignores configuring nova.conf if n-cpu is not running.
I have filed https://bugs.launchpad.net/devstack/+bug/1667219 and posted
fix https://review.openstack.org/#/c/437274/.
Let me know what you think.
Thanks,
Prashant
On Wed, Feb 22, 2017 at 8:19 PM, Matt Riedemann <mriedemos at gmail.com> wrote:
> On 2/22/2017 9:33 AM, Prashant Shetty wrote:
>
>> Thanks Matt.
>>
>> Here are the steps I have performed, I dont see any error related to
>> cell0 now but n-cond still not able to detect computes after discover as
>> well :(.
>>
>> Do we need any cell setting on nova-compute nodes as well?.
>>
>> vmware at cntr11:~/nsbu_cqe_openstack/devstack$ nova service-list
>> +----+------------------+---------------+----------+--------
>> -+-------+----------------------------+-----------------+
>> | Id | Binary | Host | Zone | Status | State |
>> Updated_at | Disabled Reason |
>> +----+------------------+---------------+----------+--------
>> -+-------+----------------------------+-----------------+
>> | 7 | nova-conductor | cntr11 | internal | enabled | up |
>> 2017-02-22T14:23:34.000000 | - |
>> | 9 | nova-scheduler | cntr11 | internal | enabled | up |
>> 2017-02-22T14:23:28.000000 | - |
>> | 10 | nova-consoleauth | cntr11 | internal | enabled | up |
>> 2017-02-22T14:23:33.000000 | - |
>> | 11 | nova-compute | esx-ubuntu-02 | nova | enabled | up |
>> 2017-02-22T14:23:35.000000 | - |
>> | 12 | nova-compute | esx-ubuntu-03 | nova | enabled | up |
>> 2017-02-22T14:23:35.000000 | - |
>> | 13 | nova-compute | esx-ubuntu-01 | nova | enabled | up |
>> 2017-02-22T14:23:28.000000 | - |
>> | 14 | nova-compute | kvm-3 | nova | enabled | up |
>> 2017-02-22T14:23:28.000000 | - |
>> | 15 | nova-compute | kvm-1 | nova | enabled | up |
>> 2017-02-22T14:23:32.000000 | - |
>> | 16 | nova-compute | kvm-2 | nova | enabled | up |
>> 2017-02-22T14:23:32.000000 | - |
>> +----+------------------+---------------+----------+--------
>> -+-------+----------------------------+-----------------+
>> vmware at cntr11:~/nsbu_cqe_openstack/devstack$
>> vmware at cntr11:~/nsbu_cqe_openstack/devstack$ nova-manage cell_v2
>> map_cell0 --database_connection
>> mysql+pymysql://root:vmware@127.0.0.1/nova?charset=utf8
>> <http://root:vmware@127.0.0.1/nova?charset=utf8>
>> vmware at cntr11:~/nsbu_cqe_openstack/devstack$ nova-manage cell_v2
>> simple_cell_setup --transport-url
>> rabbit://stackrabbit:vmware@60.0.24.49:5672/
>> <http://stackrabbit:vmware@60.0.24.49:5672/>
>>
>> Cell0 is already setup
>> vmware at cntr11:~/nsbu_cqe_openstack/devstack$ nova-manage cell_v2
>> list_cells
>> +-------+--------------------------------------+
>> | Name | UUID |
>> +-------+--------------------------------------+
>> | None | ea6bec24-058a-4ba2-8d21-57d34c01802c |
>> | cell0 | 00000000-0000-0000-0000-000000000000 |
>> +-------+--------------------------------------+
>> vmware at cntr11:~/nsbu_cqe_openstack/devstack$ nova-manage cell_v2
>> discover_hosts --verbose
>> Found 2 cell mappings.
>> Skipping cell0 since it does not contain hosts.
>> Getting compute nodes from cell: ea6bec24-058a-4ba2-8d21-57d34c01802c
>> Found 6 computes in cell: ea6bec24-058a-4ba2-8d21-57d34c01802c
>> Checking host mapping for compute host 'kvm-3':
>> a4b175d6-f5cc-45a8-9cf2-45726293b5c5
>> Checking host mapping for compute host 'esx-ubuntu-02':
>> 70281329-590c-4cb7-8839-fd84160345b7
>> Checking host mapping for compute host 'esx-ubuntu-03':
>> 04ea75a2-789e-483e-8d0e-4b0f79e012dc
>> Checking host mapping for compute host 'kvm-1':
>> dfabae3c-4ea9-4e8f-a496-8880dd9e89d9
>> Checking host mapping for compute host 'kvm-2':
>> d1cb30f5-822c-4c18-81fb-921ca676b834
>> Checking host mapping for compute host 'esx-ubuntu-01':
>> d00f8f16-af6b-437d-8136-bc744eb2472f
>> vmware at cntr11:~/nsbu_cqe_openstack/devstack$
>>
>> n-sch:
>> 2017-02-22 14:26:51.467 INFO nova.scheduler.host_manager
>> [req-56d1cefb-1dfb-481d-aaff-b7b6e05f83f0 None None] Successfully synced
>> instances from host 'kvm-2'.
>> 2017-02-22 14:26:51.608 INFO nova.scheduler.host_manager
>> [req-690b1a18-a709-49b2-bfad-2a6a75a3bee2 None None] Successfully synced
>> instances from host 'kvm-3'.
>> 2017-02-22 14:27:23.366 INFO nova.filters
>> [req-1085ec50-29f7-4946-81e2-03c1378e8077 alt_demo admin] Filter
>> RetryFilter returned 0 hosts
>> 2017-02-22 14:27:23.367 INFO nova.filters
>> [req-1085ec50-29f7-4946-81e2-03c1378e8077 alt_demo admin] Filtering
>> removed all hosts for the request with instance ID
>> 'c74f394f-c805-4b5c-ba42-507dfda2c5be'. Filter results: ['RetryFilter:
>> (start: 0, end: 0)']
>>
>>
>> n-cond:
>>
>> 2017-02-22 14:27:23.375 TRACE nova.conductor.manager File
>> "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 79, in
>> select_destinations
>> 2017-02-22 14:27:23.375 TRACE nova.conductor.manager raise
>> exception.NoValidHost(reason=reason)
>> 2017-02-22 14:27:23.375 TRACE nova.conductor.manager
>> 2017-02-22 14:27:23.375 TRACE nova.conductor.manager NoValidHost: No
>> valid host was found. There are not enough hosts available.
>> 2017-02-22 14:27:23.375 TRACE nova.conductor.manager
>> 2017-02-22 14:27:23.375 TRACE nova.conductor.manager
>> 2017-02-22 14:27:23.424 WARNING nova.scheduler.utils
>> [req-1085ec50-29f7-4946-81e2-03c1378e8077 alt_demo admin] Failed to
>> compute_task_build_instances: No valid host was found. There are not
>> enough hosts available.
>> Traceback (most recent call last):
>>
>> File
>> "/usr/local/lib/python2.7/dist-packages/oslo_messaging/rpc/server.py",
>> line 218, in inner
>> return func(*args, **kwargs)
>>
>> File "/opt/stack/nova/nova/scheduler/manager.py", line 98, in
>> select_destinations
>> dests = self.driver.select_destinations(ctxt, spec_obj)
>>
>> File "/opt/stack/nova/nova/scheduler/filter_scheduler.py", line 79, in
>> select_destinations
>> raise exception.NoValidHost(reason=reason)
>>
>> NoValidHost: No valid host was found. There are not enough hosts
>> available.
>>
>> 2017-02-22 14:27:23.425 WARNING nova.scheduler.utils
>> [req-1085ec50-29f7-4946-81e2-03c1378e8077 alt_demo admin] [instance:
>> c74f394f-c805-4b5c-ba42-507dfda2c5be] Setting instance to ERROR state.
>>
>> On Wed, Feb 22, 2017 at 5:44 PM, Matt Riedemann <mriedemos at gmail.com
>> <mailto:mriedemos at gmail.com>> wrote:
>>
>> On 2/21/2017 10:38 AM, Prashant Shetty wrote:
>>
>> Hi Mark,
>>
>> Thanks for your reply.
>>
>> I tried "nova-manage cell_v2 discover_hosts" and it returned
>> nothing and
>> still I have same issue on the node.
>>
>> Problem seems be the way devstack is getting configured,
>> As code suggest below we create cell0 on node where n-api and
>> n-cpu
>> runs. In my case compute is running only n-cpu and controller is
>> running
>> n-api service, due to this code there are no cell created in
>> controller
>> or compute.
>>
>>
>> The nova_cell0 database is created here:
>>
>> https://github.com/openstack-dev/devstack/blob/7a30c7fcabac1
>> cf28fd9baa39d05436680616aef/lib/nova#L680
>> <https://github.com/openstack-dev/devstack/blob/7a30c7fcabac
>> 1cf28fd9baa39d05436680616aef/lib/nova#L680>
>>
>> That's the same place that the nova_api database is created.
>>
>>
>> We will not have this problem in all-in-one-node setup.
>> --
>> # Do this late because it requires compute hosts to have started
>> if is_service_enabled n-api; then
>> if is_service_enabled n-cpu; then
>> create_cell
>> else
>> # Some CI systems like Hyper-V build the control plane on
>> # Linux, and join in non Linux Computes after setup. This
>> # allows them to delay the processing until after their
>> whole
>> # environment is up.
>> echo_summary "SKIPPING Cell setup because n-cpu is not
>> enabled.
>> You will have to do this manually before you have a working
>> environment."
>> fi
>> fi
>>
>>
>> You're correct that when stacking the control node where n-api is
>> running, you won't get to the create_cell call:
>>
>> https://github.com/openstack-dev/devstack/blob/7a30c7fcabac1
>> cf28fd9baa39d05436680616aef/stack.sh#L1371
>> <https://github.com/openstack-dev/devstack/blob/7a30c7fcabac
>> 1cf28fd9baa39d05436680616aef/stack.sh#L1371>
>>
>> The create_cell function is what creates the cell0 mapping in the
>> nova_api database and runs the simple_cell_setup command:
>>
>> https://github.com/openstack-dev/devstack/blob/7a30c7fcabac1
>> cf28fd9baa39d05436680616aef/lib/nova#L943
>> <https://github.com/openstack-dev/devstack/blob/7a30c7fcabac
>> 1cf28fd9baa39d05436680616aef/lib/nova#L943>
>>
>> You're running discover_hosts from the control node where the
>> nova_api database lives, so that looks correct.
>>
>> Can you run discover_hosts with the --verbose option to get some
>> more details, i.e. how many cell mappings are there, how many host
>> mappings and compute_nodes records are created?
>>
>> I think the issue is that you haven't run map_cell0 and
>> simple_cell_setup. In the gating multinode CI job, the create_cell
>> function in devstack is called because that's a 2-node job where
>> n-cpu is running on both nodes, but n-api is only running on the
>> control (primary) node. In your case you don't have that so you're
>> going to have to run these command manually.
>>
>> The docs here explain how to set this up and the commands to run:
>>
>> https://docs.openstack.org/developer/nova/cells.html#setup-
>> of-cells-v2
>> <https://docs.openstack.org/developer/nova/cells.html#setup-
>> of-cells-v2>
>> https://docs.openstack.org/developer/nova/cells.html#fresh-install
>>
>> <https://docs.openstack.org/developer/nova/cells.html#fresh-install>
>>
>>
>> ---
>>
>> vmware at cntr11:~$ nova-manage cell_v2 discover_hosts
>> vmware at cntr11:~$ nova service-list
>> +----+------------------+---------------+----------+--------
>> -+-------+----------------------------+-----------------+
>> | Id | Binary | Host | Zone | Status |
>> State |
>> Updated_at | Disabled Reason |
>> +----+------------------+---------------+----------+--------
>> -+-------+----------------------------+-----------------+
>> | 3 | nova-conductor | cntr11 | internal | enabled |
>> up |
>> 2017-02-21T15:34:13.000000 | - |
>> | 5 | nova-scheduler | cntr11 | internal | enabled |
>> up |
>> 2017-02-21T15:34:15.000000 | - |
>> | 6 | nova-consoleauth | cntr11 | internal | enabled |
>> up |
>> 2017-02-21T15:34:11.000000 | - |
>> | 7 | nova-compute | esx-ubuntu-02 | nova | enabled |
>> up |
>> 2017-02-21T15:34:14.000000 | - |
>> | 8 | nova-compute | esx-ubuntu-03 | nova | enabled |
>> up |
>> 2017-02-21T15:34:16.000000 | - |
>> | 9 | nova-compute | kvm-3 | nova | enabled |
>> up |
>> 2017-02-21T15:34:07.000000 | - |
>> | 10 | nova-compute | kvm-2 | nova | enabled |
>> up |
>> 2017-02-21T15:34:13.000000 | - |
>> | 11 | nova-compute | esx-ubuntu-01 | nova | enabled |
>> up |
>> 2017-02-21T15:34:14.000000 | - |
>> | 12 | nova-compute | kvm-1 | nova | enabled |
>> up |
>> 2017-02-21T15:34:09.000000 | - |
>> +----+------------------+---------------+----------+--------
>> -+-------+----------------------------+-----------------+
>> vmware at cntr11:~$
>> vmware at cntr11:~$ nova-manage cell_v2 list_cells
>> +------+------+
>> | Name | UUID |
>> +------+------+
>> +------+------+
>> vmware at cntr11:~$
>>
>>
>> Thanks,
>> Prashant
>>
>>
>> --
>>
>> Thanks,
>>
>> Matt Riedemann
>>
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>> >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>>
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> The scheduler failure is different. You have 2 cell mappings, one for
> cell0 and the other is your single-cell with the API and main nova
> database, with uuid ea6bec24-058a-4ba2-8d21-57d34c01802c.
>
> The discover_hosts output is showing that it's discovering the compute
> nodes in cell ea6bec24-058a-4ba2-8d21-57d34c01802c so those should all be
> mapped in the nova_api database (see the host_mappings table in the
> nova_api DB).
>
> The scheduler failure could just be due to resource limitations or the
> request for the server build or the filters you have enabled or any number
> of "normal" issues you can have when scheduling. You have to investigate
> that.
>
>
> --
>
> Thanks,
>
> Matt Riedemann
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170223/1a7ae5ae/attachment.html>
More information about the OpenStack-dev
mailing list