<div dir="ltr"><div style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)" class="gmail_default">Hi Matt,<br></div><div style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)" class="gmail_default">I have addressed your comment on patch and uploaded new patch to master branch.<br></div><div style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)" class="gmail_default">Could you please check <a href="https://review.openstack.org/#/c/437381">https://review.openstack.org/#/c/437381</a><br><br></div><div style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)" class="gmail_default">Thanks,<br></div><div style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)" class="gmail_default">Prashant<br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Feb 23, 2017 at 2:34 PM, Prashant Shetty <span dir="ltr"><<a href="mailto:prashantshetty1985@gmail.com" target="_blank">prashantshetty1985@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)">Thanks Matt, I found out there was issue in my nova.conf on controller. [placement] section was missing on controller nova.conf.<br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)">Looks like devstack ignores configuring nova.conf if n-cpu is not running. <br><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)">I have filed <a href="https://bugs.launchpad.net/devstack/+bug/1667219" target="_blank">https://bugs.launchpad.net/<wbr>devstack/+bug/1667219</a> and posted fix <a href="https://review.openstack.org/#/c/437274/" target="_blank">https://review.openstack.org/#<wbr>/c/437274/</a>.<br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)">Let me know what you think. <br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)"><br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)">Thanks,<br></div><div class="gmail_default" style="font-family:verdana,sans-serif;font-size:small;color:rgb(0,0,0)">Prashant<br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Feb 22, 2017 at 8:19 PM, Matt Riedemann <span dir="ltr"><<a href="mailto:mriedemos@gmail.com" target="_blank">mriedemos@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="m_659363294680715181HOEnZb"><div class="m_659363294680715181h5">On 2/22/2017 9:33 AM, Prashant Shetty wrote:<br>
</div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_659363294680715181h5">
Thanks Matt.<br>
<br>
Here are the steps I have performed, I dont see any error related to<br>
cell0 now but n-cond still not able to detect computes after discover as<br>
well :(.<br>
<br>
Do we need any cell setting on nova-compute nodes as well?.<br>
<br>
vmware@cntr11:~/nsbu_cqe_opens<wbr>tack/devstack$ nova service-list<br>
+----+------------------+-----<wbr>----------+----------+--------<wbr>-+-------+--------------------<wbr>--------+-----------------+<br>
| Id | Binary | Host | Zone | Status | State |<br>
Updated_at | Disabled Reason |<br>
+----+------------------+-----<wbr>----------+----------+--------<wbr>-+-------+--------------------<wbr>--------+-----------------+<br>
| 7 | nova-conductor | cntr11 | internal | enabled | up |<br>
2017-02-22T14:23:34.000000 | - |<br>
| 9 | nova-scheduler | cntr11 | internal | enabled | up |<br>
2017-02-22T14:23:28.000000 | - |<br>
| 10 | nova-consoleauth | cntr11 | internal | enabled | up |<br>
2017-02-22T14:23:33.000000 | - |<br>
| 11 | nova-compute | esx-ubuntu-02 | nova | enabled | up |<br>
2017-02-22T14:23:35.000000 | - |<br>
| 12 | nova-compute | esx-ubuntu-03 | nova | enabled | up |<br>
2017-02-22T14:23:35.000000 | - |<br>
| 13 | nova-compute | esx-ubuntu-01 | nova | enabled | up |<br>
2017-02-22T14:23:28.000000 | - |<br>
| 14 | nova-compute | kvm-3 | nova | enabled | up |<br>
2017-02-22T14:23:28.000000 | - |<br>
| 15 | nova-compute | kvm-1 | nova | enabled | up |<br>
2017-02-22T14:23:32.000000 | - |<br>
| 16 | nova-compute | kvm-2 | nova | enabled | up |<br>
2017-02-22T14:23:32.000000 | - |<br>
+----+------------------+-----<wbr>----------+----------+--------<wbr>-+-------+--------------------<wbr>--------+-----------------+<br>
vmware@cntr11:~/nsbu_cqe_opens<wbr>tack/devstack$<br>
vmware@cntr11:~/nsbu_cqe_opens<wbr>tack/devstack$ nova-manage cell_v2<br>
map_cell0 --database_connection<br>
mysql+pymysql://<a href="http://root:vmware@127.0.0.1/nova?charset=utf8" rel="noreferrer" target="_blank">root:vmware@12<wbr>7.0.0.1/nova?charset=utf8</a><br></div></div>
<<a href="http://root:vmware@127.0.0.1/nova?charset=utf8" rel="noreferrer" target="_blank">http://root:vmware@127.0.0.1/<wbr>nova?charset=utf8</a>><span><br>
vmware@cntr11:~/nsbu_cqe_opens<wbr>tack/devstack$ nova-manage cell_v2<br>
simple_cell_setup --transport-url<br>
rabbit://<a href="http://stackrabbit:vmware@60.0.24.49:5672/" rel="noreferrer" target="_blank">stackrabbit:vmware@60<wbr>.0.24.49:5672/</a><br></span>
<<a href="http://stackrabbit:vmware@60.0.24.49:5672/" rel="noreferrer" target="_blank">http://stackrabbit:vmware@60.<wbr>0.24.49:5672/</a>><div><div class="m_659363294680715181h5"><br>
Cell0 is already setup<br>
vmware@cntr11:~/nsbu_cqe_opens<wbr>tack/devstack$ nova-manage cell_v2 list_cells<br>
+-------+---------------------<wbr>-----------------+<br>
| Name | UUID |<br>
+-------+---------------------<wbr>-----------------+<br>
| None | ea6bec24-058a-4ba2-8d21-57d34c<wbr>01802c |<br>
| cell0 | 00000000-0000-0000-0000-000000<wbr>000000 |<br>
+-------+---------------------<wbr>-----------------+<br>
vmware@cntr11:~/nsbu_cqe_opens<wbr>tack/devstack$ nova-manage cell_v2<br>
discover_hosts --verbose<br>
Found 2 cell mappings.<br>
Skipping cell0 since it does not contain hosts.<br>
Getting compute nodes from cell: ea6bec24-058a-4ba2-8d21-57d34c<wbr>01802c<br>
Found 6 computes in cell: ea6bec24-058a-4ba2-8d21-57d34c<wbr>01802c<br>
Checking host mapping for compute host 'kvm-3':<br>
a4b175d6-f5cc-45a8-9cf2-457262<wbr>93b5c5<br>
Checking host mapping for compute host 'esx-ubuntu-02':<br>
70281329-590c-4cb7-8839-fd8416<wbr>0345b7<br>
Checking host mapping for compute host 'esx-ubuntu-03':<br>
04ea75a2-789e-483e-8d0e-4b0f79<wbr>e012dc<br>
Checking host mapping for compute host 'kvm-1':<br>
dfabae3c-4ea9-4e8f-a496-8880dd<wbr>9e89d9<br>
Checking host mapping for compute host 'kvm-2':<br>
d1cb30f5-822c-4c18-81fb-921ca6<wbr>76b834<br>
Checking host mapping for compute host 'esx-ubuntu-01':<br>
d00f8f16-af6b-437d-8136-bc744e<wbr>b2472f<br>
vmware@cntr11:~/nsbu_cqe_opens<wbr>tack/devstack$<br>
<br>
n-sch:<br>
2017-02-22 14:26:51.467 INFO nova.scheduler.host_manager<br>
[req-56d1cefb-1dfb-481d-aaff-b<wbr>7b6e05f83f0 None None] Successfully synced<br>
instances from host 'kvm-2'.<br>
2017-02-22 14:26:51.608 INFO nova.scheduler.host_manager<br>
[req-690b1a18-a709-49b2-bfad-2<wbr>a6a75a3bee2 None None] Successfully synced<br>
instances from host 'kvm-3'.<br>
2017-02-22 14:27:23.366 INFO nova.filters<br>
[req-1085ec50-29f7-4946-81e2-0<wbr>3c1378e8077 alt_demo admin] Filter<br>
RetryFilter returned 0 hosts<br>
2017-02-22 14:27:23.367 INFO nova.filters<br>
[req-1085ec50-29f7-4946-81e2-0<wbr>3c1378e8077 alt_demo admin] Filtering<br>
removed all hosts for the request with instance ID<br>
'c74f394f-c805-4b5c-ba42-507df<wbr>da2c5be'. Filter results: ['RetryFilter:<br>
(start: 0, end: 0)']<br>
<br>
<br>
n-cond:<br>
<br>
2017-02-22 14:27:23.375 TRACE nova.conductor.manager File<br>
"/opt/stack/nova/nova/schedule<wbr>r/filter_scheduler.py", line 79, in<br>
select_destinations<br>
2017-02-22 14:27:23.375 TRACE nova.conductor.manager raise<br>
exception.NoValidHost(reason=r<wbr>eason)<br>
2017-02-22 14:27:23.375 TRACE nova.conductor.manager<br>
2017-02-22 14:27:23.375 TRACE nova.conductor.manager NoValidHost: No<br>
valid host was found. There are not enough hosts available.<br>
2017-02-22 14:27:23.375 TRACE nova.conductor.manager<br>
2017-02-22 14:27:23.375 TRACE nova.conductor.manager<br>
2017-02-22 14:27:23.424 WARNING nova.scheduler.utils<br>
[req-1085ec50-29f7-4946-81e2-0<wbr>3c1378e8077 alt_demo admin] Failed to<br>
compute_task_build_instances: No valid host was found. There are not<br>
enough hosts available.<br>
Traceback (most recent call last):<br>
<br>
File<br>
"/usr/local/lib/python2.7/dist<wbr>-packages/oslo_messaging/rpc/s<wbr>erver.py",<br>
line 218, in inner<br>
return func(*args, **kwargs)<br>
<br>
File "/opt/stack/nova/nova/schedule<wbr>r/manager.py", line 98, in<br>
select_destinations<br>
dests = self.driver.select_destination<wbr>s(ctxt, spec_obj)<br>
<br>
File "/opt/stack/nova/nova/schedule<wbr>r/filter_scheduler.py", line 79, in<br>
select_destinations<br>
raise exception.NoValidHost(reason=r<wbr>eason)<br>
<br>
NoValidHost: No valid host was found. There are not enough hosts available.<br>
<br>
2017-02-22 14:27:23.425 WARNING nova.scheduler.utils<br>
[req-1085ec50-29f7-4946-81e2-0<wbr>3c1378e8077 alt_demo admin] [instance:<br>
c74f394f-c805-4b5c-ba42-507dfd<wbr>a2c5be] Setting instance to ERROR state.<br>
<br>
On Wed, Feb 22, 2017 at 5:44 PM, Matt Riedemann <<a href="mailto:mriedemos@gmail.com" target="_blank">mriedemos@gmail.com</a><br></div></div><div><div class="m_659363294680715181h5">
<mailto:<a href="mailto:mriedemos@gmail.com" target="_blank">mriedemos@gmail.com</a>>> wrote:<br>
<br>
On 2/21/2017 10:38 AM, Prashant Shetty wrote:<br>
<br>
Hi Mark,<br>
<br>
Thanks for your reply.<br>
<br>
I tried "nova-manage cell_v2 discover_hosts" and it returned<br>
nothing and<br>
still I have same issue on the node.<br>
<br>
Problem seems be the way devstack is getting configured,<br>
As code suggest below we create cell0 on node where n-api and n-cpu<br>
runs. In my case compute is running only n-cpu and controller is<br>
running<br>
n-api service, due to this code there are no cell created in<br>
controller<br>
or compute.<br>
<br>
<br>
The nova_cell0 database is created here:<br>
<br>
<a href="https://github.com/openstack-dev/devstack/blob/7a30c7fcabac1cf28fd9baa39d05436680616aef/lib/nova#L680" rel="noreferrer" target="_blank">https://github.com/openstack-d<wbr>ev/devstack/blob/7a30c7fcabac1<wbr>cf28fd9baa39d05436680616aef/li<wbr>b/nova#L680</a><br>
<<a href="https://github.com/openstack-dev/devstack/blob/7a30c7fcabac1cf28fd9baa39d05436680616aef/lib/nova#L680" rel="noreferrer" target="_blank">https://github.com/openstack-<wbr>dev/devstack/blob/7a30c7fcabac<wbr>1cf28fd9baa39d05436680616aef/l<wbr>ib/nova#L680</a>><br>
<br>
That's the same place that the nova_api database is created.<br>
<br>
<br>
We will not have this problem in all-in-one-node setup.<br>
--<br>
# Do this late because it requires compute hosts to have started<br>
if is_service_enabled n-api; then<br>
if is_service_enabled n-cpu; then<br>
create_cell<br>
else<br>
# Some CI systems like Hyper-V build the control plane on<br>
# Linux, and join in non Linux Computes after setup. This<br>
# allows them to delay the processing until after their<br>
whole<br>
# environment is up.<br>
echo_summary "SKIPPING Cell setup because n-cpu is not<br>
enabled.<br>
You will have to do this manually before you have a working<br>
environment."<br>
fi<br>
fi<br>
<br>
<br>
You're correct that when stacking the control node where n-api is<br>
running, you won't get to the create_cell call:<br>
<br>
<a href="https://github.com/openstack-dev/devstack/blob/7a30c7fcabac1cf28fd9baa39d05436680616aef/stack.sh#L1371" rel="noreferrer" target="_blank">https://github.com/openstack-d<wbr>ev/devstack/blob/7a30c7fcabac1<wbr>cf28fd9baa39d05436680616aef/st<wbr>ack.sh#L1371</a><br>
<<a href="https://github.com/openstack-dev/devstack/blob/7a30c7fcabac1cf28fd9baa39d05436680616aef/stack.sh#L1371" rel="noreferrer" target="_blank">https://github.com/openstack-<wbr>dev/devstack/blob/7a30c7fcabac<wbr>1cf28fd9baa39d05436680616aef/s<wbr>tack.sh#L1371</a>><br>
<br>
The create_cell function is what creates the cell0 mapping in the<br>
nova_api database and runs the simple_cell_setup command:<br>
<br>
<a href="https://github.com/openstack-dev/devstack/blob/7a30c7fcabac1cf28fd9baa39d05436680616aef/lib/nova#L943" rel="noreferrer" target="_blank">https://github.com/openstack-d<wbr>ev/devstack/blob/7a30c7fcabac1<wbr>cf28fd9baa39d05436680616aef/li<wbr>b/nova#L943</a><br>
<<a href="https://github.com/openstack-dev/devstack/blob/7a30c7fcabac1cf28fd9baa39d05436680616aef/lib/nova#L943" rel="noreferrer" target="_blank">https://github.com/openstack-<wbr>dev/devstack/blob/7a30c7fcabac<wbr>1cf28fd9baa39d05436680616aef/l<wbr>ib/nova#L943</a>><br>
<br>
You're running discover_hosts from the control node where the<br>
nova_api database lives, so that looks correct.<br>
<br>
Can you run discover_hosts with the --verbose option to get some<br>
more details, i.e. how many cell mappings are there, how many host<br>
mappings and compute_nodes records are created?<br>
<br>
I think the issue is that you haven't run map_cell0 and<br>
simple_cell_setup. In the gating multinode CI job, the create_cell<br>
function in devstack is called because that's a 2-node job where<br>
n-cpu is running on both nodes, but n-api is only running on the<br>
control (primary) node. In your case you don't have that so you're<br>
going to have to run these command manually.<br>
<br>
The docs here explain how to set this up and the commands to run:<br>
<br>
<a href="https://docs.openstack.org/developer/nova/cells.html#setup-of-cells-v2" rel="noreferrer" target="_blank">https://docs.openstack.org/dev<wbr>eloper/nova/cells.html#setup-o<wbr>f-cells-v2</a><br></div></div>
<<a href="https://docs.openstack.org/developer/nova/cells.html#setup-of-cells-v2" rel="noreferrer" target="_blank">https://docs.openstack.org/de<wbr>veloper/nova/cells.html#setup-<wbr>of-cells-v2</a>><br>
<a href="https://docs.openstack.org/developer/nova/cells.html#fresh-install" rel="noreferrer" target="_blank">https://docs.openstack.org/dev<wbr>eloper/nova/cells.html#fresh-i<wbr>nstall</a><div><div class="m_659363294680715181h5"><br>
<<a href="https://docs.openstack.org/developer/nova/cells.html#fresh-install" rel="noreferrer" target="_blank">https://docs.openstack.org/de<wbr>veloper/nova/cells.html#fresh-<wbr>install</a>><br>
<br>
<br>
---<br>
<br>
vmware@cntr11:~$ nova-manage cell_v2 discover_hosts<br>
vmware@cntr11:~$ nova service-list<br>
+----+------------------+-----<wbr>----------+----------+--------<wbr>-+-------+--------------------<wbr>--------+-----------------+<br>
| Id | Binary | Host | Zone | Status |<br>
State |<br>
Updated_at | Disabled Reason |<br>
+----+------------------+-----<wbr>----------+----------+--------<wbr>-+-------+--------------------<wbr>--------+-----------------+<br>
| 3 | nova-conductor | cntr11 | internal | enabled |<br>
up |<br>
2017-02-21T15:34:13.000000 | - |<br>
| 5 | nova-scheduler | cntr11 | internal | enabled |<br>
up |<br>
2017-02-21T15:34:15.000000 | - |<br>
| 6 | nova-consoleauth | cntr11 | internal | enabled |<br>
up |<br>
2017-02-21T15:34:11.000000 | - |<br>
| 7 | nova-compute | esx-ubuntu-02 | nova | enabled |<br>
up |<br>
2017-02-21T15:34:14.000000 | - |<br>
| 8 | nova-compute | esx-ubuntu-03 | nova | enabled |<br>
up |<br>
2017-02-21T15:34:16.000000 | - |<br>
| 9 | nova-compute | kvm-3 | nova | enabled |<br>
up |<br>
2017-02-21T15:34:07.000000 | - |<br>
| 10 | nova-compute | kvm-2 | nova | enabled |<br>
up |<br>
2017-02-21T15:34:13.000000 | - |<br>
| 11 | nova-compute | esx-ubuntu-01 | nova | enabled |<br>
up |<br>
2017-02-21T15:34:14.000000 | - |<br>
| 12 | nova-compute | kvm-1 | nova | enabled |<br>
up |<br>
2017-02-21T15:34:09.000000 | - |<br>
+----+------------------+-----<wbr>----------+----------+--------<wbr>-+-------+--------------------<wbr>--------+-----------------+<br>
vmware@cntr11:~$<br>
vmware@cntr11:~$ nova-manage cell_v2 list_cells<br>
+------+------+<br>
| Name | UUID |<br>
+------+------+<br>
+------+------+<br>
vmware@cntr11:~$<br>
<br>
<br>
Thanks,<br>
Prashant<br>
<br>
<br>
--<br>
<br>
Thanks,<br>
<br>
Matt Riedemann<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br></div></div><span>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@<wbr>lists.openstack.org?subject:un<wbr>subscribe</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cg<wbr>i-bin/mailman/listinfo/opensta<wbr>ck-dev</a>><br>
<br>
<br>
<br>
<br></span><span>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</span></blockquote>
<br>
The scheduler failure is different. You have 2 cell mappings, one for cell0 and the other is your single-cell with the API and main nova database, with uuid ea6bec24-058a-4ba2-8d21-57d34c<wbr>01802c.<br>
<br>
The discover_hosts output is showing that it's discovering the compute nodes in cell ea6bec24-058a-4ba2-8d21-57d34c<wbr>01802c so those should all be mapped in the nova_api database (see the host_mappings table in the nova_api DB).<br>
<br>
The scheduler failure could just be due to resource limitations or the request for the server build or the filters you have enabled or any number of "normal" issues you can have when scheduling. You have to investigate that.<div class="m_659363294680715181HOEnZb"><div class="m_659363294680715181h5"><br>
<br>
-- <br>
<br>
Thanks,<br>
<br>
Matt Riedemann<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>