<html><body>
<p><font size="2" face="sans-serif">Hi Tom,</font><br>
<br>
<br>
<font size="2" face="sans-serif">I did talked to infra, which I think it is resource issue, But they thought it is nova issue,</font><br>
<br>
<br>
<font size="2" face="sans-serif">For we boot k8s bay, we use baymodel with falvor m1.small, you can find devstack</font><br>
<br>
<br>
<br>
<font size="2" face="sans-serif">+-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+</font><br>
<font size="2" face="sans-serif">| ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |</font><br>
<font size="2" face="sans-serif">+-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+</font><br>
<font size="2" face="sans-serif">| 1 | m1.tiny | 512 | 1 | 0 | | 1 | 1.0 | True |</font><br>
<font size="2" face="sans-serif">| 2 | m1.small | 2048 |</font><font size="2" face="sans-serif"><b> 20 </b></font><font size="2" face="sans-serif"> | 0 | | 1 | 1.0 | True |</font><br>
<font size="2" face="sans-serif">| 3 | m1.medium | 4096 | 40 | 0 | | 2 | 1.0 | True |</font><br>
<font size="2" face="sans-serif">| 4 | m1.large | 8192 | 80 | 0 | | 4 | 1.0 | True |</font><br>
<font size="2" face="sans-serif">| 42 | m1.nano | 64 | 0 | 0 | | 1 | 1.0 | True |</font><br>
<font size="2" face="sans-serif">| 451 | m1.heat | 512 | 0 | 0 | | 1 | 1.0 | True |</font><br>
<font size="2" face="sans-serif">| 5 | m1.xlarge | 16384 | 160 | 0 | | 8 | 1.0 | True |</font><br>
<font size="2" face="sans-serif">| 84 | m1.micro | 128 | 0 | 0 | | 1 | 1.0 | True |</font><br>
<font size="2" face="sans-serif">+-----+-----------+-----------+------+-----------+------+-------+-------------+-----------+</font><br>
<br>
<br>
<br>
<font size="2" face="sans-serif">From logs below:</font><br>
<br>
<tt><font size="2">[req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] <br>
(devstack-trusty-rax-dfw-4299602, devstack-trusty-rax-dfw-4299602) <br>
ram:5172 disk:17408 io_ops:0 instances:1</font></tt><tt><font size="2"><b> does not have 20480 MB usable <br>
disk, it only has 17408.0 MB usable disk</b></font></tt><tt><font size="2">. host_passes <br>
/opt/stack/new/nova/nova/scheduler/filters/disk_filter.py:60<br>
2015-08-13 08:26:15.218 INFO nova.filters <br>
[req-e</font></tt><br>
<br>
<font size="2" face="sans-serif">It is 20GB disk space, so failed for that.</font><br>
<br>
<br>
<font size="2" face="sans-serif">I think it is related with this, the jenkins allocated VM disk space is not large.</font><br>
<font size="2" face="sans-serif">I am curious why it failed so often recently. Does os-infra changed something ?</font><br>
<br>
<br>
<br>
<br>
<font size="2" face="sans-serif">Thanks</font><br>
<br>
<br>
<br>
<br>
<font size="2" face="sans-serif">Best Wishes,</font><br>
<font size="2" face="sans-serif">--------------------------------------------------------------------------------</font><br>
<font size="2" face="sans-serif">Kai Qiang Wu (Î⿪ǿ Kennan£©<br>
IBM China System and Technology Lab, Beijing<br>
<br>
E-mail: wkqwu@cn.ibm.com<br>
Tel: 86-10-82451647<br>
Address: Building 28(Ring Building), ZhongGuanCun Software Park, <br>
No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193</font><br>
<font size="2" face="sans-serif">--------------------------------------------------------------------------------</font><br>
<font size="2" face="sans-serif">Follow your heart. You are miracle! </font><br>
<br>
<img width="16" height="16" src="cid:1__=C7BBF433DFA9DE718f9e8a93df938@cn.ibm.com" border="0" alt="Inactive hide details for Tom Cammann ---08/13/2015 06:24:19 PM---Hi Team, Wanted to let you know why we are having consistent "><font size="2" color="#424282" face="sans-serif">Tom Cammann ---08/13/2015 06:24:19 PM---Hi Team, Wanted to let you know why we are having consistent functional test</font><br>
<br>
<font size="1" color="#5F5F5F" face="sans-serif">From: </font><font size="1" face="sans-serif">Tom Cammann <tom.cammann@hp.com></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">To: </font><font size="1" face="sans-serif">"OpenStack Development Mailing List (not for usage questions)" <openstack-dev@lists.openstack.org></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Date: </font><font size="1" face="sans-serif">08/13/2015 06:24 PM</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Subject: </font><font size="1" face="sans-serif">[openstack-dev] [Magnum] Consistent functional test failures</font><br>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<tt><font size="2">Hi Team,<br>
<br>
Wanted to let you know why we are having consistent functional test <br>
failures in the gate.<br>
<br>
This is being caused by Nova returning "No valid host" to heat:<br>
<br>
2015-08-13 08:26:16.303 31543 INFO heat.engine.resource [-] CREATE: <br>
Server "kube_minion" [12ab45ef-0177-4118-9ba0-3fffbc3c1d1a] Stack <br>
"testbay-y366b2atg6mm-kube_minions-cdlfyvhaximr-0-dufsjliqfoet" <br>
[b40f0c9f-cb54-4d75-86c3-8a9f347a27a6]<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource Traceback (most <br>
recent call last):<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
"/opt/stack/new/heat/heat/engine/resource.py", line 625, in _action_recorder<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource yield<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
"/opt/stack/new/heat/heat/engine/resource.py", line 696, in _do_action<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource yield <br>
self.action_handler_task(action, args=handler_args)<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
"/opt/stack/new/heat/heat/engine/scheduler.py", line 320, in wrapper<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource step = <br>
next(subtask)<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
"/opt/stack/new/heat/heat/engine/resource.py", line 670, in <br>
action_handler_task<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource while not <br>
check(handler_data):<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
"/opt/stack/new/heat/heat/engine/resources/openstack/nova/server.py", <br>
line 759, in check_create_complete<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource return <br>
self.client_plugin()._check_active(server_id)<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
"/opt/stack/new/heat/heat/engine/clients/os/nova.py", line 232, in <br>
_check_active<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource 'code': <br>
fault.get('code', _('Unknown'))<br>
2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource <br>
ResourceInError: Went to status ERROR due to "Message: No valid host was <br>
found. There are not enough hosts available., Code: 500"<br>
<br>
And this in turn is being caused by the compute instance running out of <br>
disk space:<br>
<br>
2015-08-13 08:26:15.216 DEBUG nova.filters <br>
[req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] Starting with 1 <br>
host(s) get_filtered_objects /opt/stack/new/nova/nova/filters.py:70<br>
2015-08-13 08:26:15.217 DEBUG nova.filters <br>
[req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] Filter <br>
RetryFilter returned 1 host(s) get_filtered_objects <br>
/opt/stack/new/nova/nova/filters.py:84<br>
2015-08-13 08:26:15.217 DEBUG nova.filters <br>
[req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] Filter <br>
AvailabilityZoneFilter returned 1 host(s) get_filtered_objects <br>
/opt/stack/new/nova/nova/filters.py:84<br>
2015-08-13 08:26:15.217 DEBUG nova.filters <br>
[req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] Filter RamFilter <br>
returned 1 host(s) get_filtered_objects <br>
/opt/stack/new/nova/nova/filters.py:84<br>
2015-08-13 08:26:15.218 DEBUG nova.scheduler.filters.disk_filter <br>
[req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] <br>
(devstack-trusty-rax-dfw-4299602, devstack-trusty-rax-dfw-4299602) <br>
ram:5172 disk:17408 io_ops:0 instances:1 does not have 20480 MB usable <br>
disk, it only has 17408.0 MB usable disk. host_passes <br>
/opt/stack/new/nova/nova/scheduler/filters/disk_filter.py:60<br>
2015-08-13 08:26:15.218 INFO nova.filters <br>
[req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] Filter DiskFilter <br>
returned 0 hosts<br>
<br>
For now a recheck seems to work about 1 in 2, so we can still land patches.<br>
<br>
The fix for this could be to clean up our Magnum devstack install more <br>
aggressively, which might be as simple as cleaning up the images we use, <br>
or get infra to provide our tests with a larger disk size. I will <br>
probably test out a patch today which cleans up the images we use in <br>
devstack to see if that helps.<br>
<br>
If anyone can help progress this let me know.<br>
<br>
Cheers,<br>
Tom<br>
<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>
</font></tt><tt><font size="2"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></font></tt><tt><font size="2"><br>
<br>
</font></tt><br>
</body></html>