<html><body>
<p><font size="2" face="sans-serif">I have checked with infra team members. For two instances, 10GB each should be OK.</font><br>
<br>
<font size="2" face="sans-serif">So I add some steps to create magnum specific flavor(8 GB disk), instead of use the existed devstack flavors (m1.small needs 20GB, m1.tiny can not be used)</font><br>
<br>
<font size="2" face="sans-serif">Magnum creates one for jenkins job and delete it when tests finished.</font><br>
<br>
<br>
<font size="2" face="sans-serif">Thanks</font><br>
<br>
<font size="2" face="sans-serif">Best Wishes,</font><br>
<font size="2" face="sans-serif">--------------------------------------------------------------------------------</font><br>
<font size="2" face="sans-serif">Kai Qiang Wu (Î⿪ǿ Kennan£©<br>
IBM China System and Technology Lab, Beijing<br>
<br>
E-mail: wkqwu@cn.ibm.com<br>
Tel: 86-10-82451647<br>
Address: Building 28(Ring Building), ZhongGuanCun Software Park, <br>
No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193</font><br>
<font size="2" face="sans-serif">--------------------------------------------------------------------------------</font><br>
<font size="2" face="sans-serif">Follow your heart. You are miracle! </font><br>
<br>
<img width="16" height="16" src="cid:1__=C7BBF432DFA959B68f9e8a93df938@cn.ibm.com" border="0" alt="Inactive hide details for Clark Boylan ---08/14/2015 08:05:15 AM---On Thu, Aug 13, 2015, at 03:13 AM, Tom Cammann wrote: > Hi T"><font size="2" color="#424282" face="sans-serif">Clark Boylan ---08/14/2015 08:05:15 AM---On Thu, Aug 13, 2015, at 03:13 AM, Tom Cammann wrote: > Hi Team,</font><br>
<br>
<font size="1" color="#5F5F5F" face="sans-serif">From: </font><font size="1" face="sans-serif">Clark Boylan <cboylan@sapwetik.org></font><br>
<font size="1" color="#5F5F5F" face="sans-serif">To: </font><font size="1" face="sans-serif">openstack-dev@lists.openstack.org</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Date: </font><font size="1" face="sans-serif">08/14/2015 08:05 AM</font><br>
<font size="1" color="#5F5F5F" face="sans-serif">Subject: </font><font size="1" face="sans-serif">Re: [openstack-dev] [Magnum] Consistent functional test failures</font><br>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<tt><font size="2">On Thu, Aug 13, 2015, at 03:13 AM, Tom Cammann wrote:<br>
> Hi Team,<br>
> <br>
> Wanted to let you know why we are having consistent functional test <br>
> failures in the gate.<br>
> <br>
> This is being caused by Nova returning "No valid host" to heat:<br>
> <br>
> 2015-08-13 08:26:16.303 31543 INFO heat.engine.resource [-] CREATE: <br>
> Server "kube_minion" [12ab45ef-0177-4118-9ba0-3fffbc3c1d1a] Stack <br>
> "testbay-y366b2atg6mm-kube_minions-cdlfyvhaximr-0-dufsjliqfoet" <br>
> [b40f0c9f-cb54-4d75-86c3-8a9f347a27a6]<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource Traceback (most <br>
> recent call last):<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
> "/opt/stack/new/heat/heat/engine/resource.py", line 625, in<br>
> _action_recorder<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource yield<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
> "/opt/stack/new/heat/heat/engine/resource.py", line 696, in _do_action<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource yield <br>
> self.action_handler_task(action, args=handler_args)<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
> "/opt/stack/new/heat/heat/engine/scheduler.py", line 320, in wrapper<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource step = <br>
> next(subtask)<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
> "/opt/stack/new/heat/heat/engine/resource.py", line 670, in <br>
> action_handler_task<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource while not <br>
> check(handler_data):<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
> "/opt/stack/new/heat/heat/engine/resources/openstack/nova/server.py", <br>
> line 759, in check_create_complete<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource return <br>
> self.client_plugin()._check_active(server_id)<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource File <br>
> "/opt/stack/new/heat/heat/engine/clients/os/nova.py", line 232, in <br>
> _check_active<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource 'code': <br>
> fault.get('code', _('Unknown'))<br>
> 2015-08-13 08:26:16.303 31543 ERROR heat.engine.resource <br>
> ResourceInError: Went to status ERROR due to "Message: No valid host was <br>
> found. There are not enough hosts available., Code: 500"<br>
> <br>
> And this in turn is being caused by the compute instance running out of <br>
> disk space:<br>
> <br>
> 2015-08-13 08:26:15.216 DEBUG nova.filters <br>
> [req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] Starting with 1 <br>
> host(s) get_filtered_objects /opt/stack/new/nova/nova/filters.py:70<br>
> 2015-08-13 08:26:15.217 DEBUG nova.filters <br>
> [req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] Filter <br>
> RetryFilter returned 1 host(s) get_filtered_objects <br>
> /opt/stack/new/nova/nova/filters.py:84<br>
> 2015-08-13 08:26:15.217 DEBUG nova.filters <br>
> [req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] Filter <br>
> AvailabilityZoneFilter returned 1 host(s) get_filtered_objects <br>
> /opt/stack/new/nova/nova/filters.py:84<br>
> 2015-08-13 08:26:15.217 DEBUG nova.filters <br>
> [req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] Filter RamFilter <br>
> returned 1 host(s) get_filtered_objects <br>
> /opt/stack/new/nova/nova/filters.py:84<br>
> 2015-08-13 08:26:15.218 DEBUG nova.scheduler.filters.disk_filter <br>
> [req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] <br>
> (devstack-trusty-rax-dfw-4299602, devstack-trusty-rax-dfw-4299602) <br>
> ram:5172 disk:17408 io_ops:0 instances:1 does not have 20480 MB usable <br>
> disk, it only has 17408.0 MB usable disk. host_passes <br>
> /opt/stack/new/nova/nova/scheduler/filters/disk_filter.py:60<br>
> 2015-08-13 08:26:15.218 INFO nova.filters <br>
> [req-e5bb52cb-387e-4638-911e-8c72aa1b6400 admin admin] Filter DiskFilter <br>
> returned 0 hosts<br>
> <br>
> For now a recheck seems to work about 1 in 2, so we can still land<br>
> patches.<br>
> <br>
> The fix for this could be to clean up our Magnum devstack install more <br>
> aggressively, which might be as simple as cleaning up the images we use, <br>
> or get infra to provide our tests with a larger disk size. I will <br>
> probably test out a patch today which cleans up the images we use in <br>
> devstack to see if that helps.<br>
> <br>
It is not trivial to provide your tests with more disk as we are using<br>
the flavors appropriate for our RAM and CPU needs and are constrained by<br>
quotas in the clouds we use. Do you really need 20GB nested test<br>
instances? The VMs these jobs run on have ~13GB images which is almost<br>
half the size of the instances you are trying to boot there. I would<br>
definitely look into trimming the disk requirements for the nested VMs<br>
before anything else.<br>
<br>
As for working ~50% of the time hpcloud gives us more disk than<br>
rackspace which is likely why you see about half fail and half pass. The<br>
runs that pass probably run on hpcloud VMs.<br>
<br>
Clark<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: OpenStack-dev-request@lists.openstack.org?subject:unsubscribe<br>
</font></tt><tt><font size="2"><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a></font></tt><tt><font size="2"><br>
<br>
</font></tt><br>
</body></html>