Open Stack

Hi,

As Kai Qiang mentioned, magnum gate recently had a bunch of random failures, which occurred on creating a nova instance with 2G of RAM. According to the error message, it seems that the hypervisor tried to allocate memory to the nova instance but couldn’t find enough free memory in the host. However, by adding a few “nova hypervisor-show XX” before, during, and right after the test, it showed that the host has 6G of free RAM, which is far more than 2G. Here is a snapshot of the output [1]. You can find the full log here [2].

Another observation is that most of the failure happened on a node with name “devstack-trusty-ovh-*” (You can verify it by entering a query [3] at http://logstash.openstack.org/ ). It seems that the jobs will be fine if they are allocated to a node other than “ovh”.

Any hints to debug this issue further? Suggestions are greatly appreciated.

[1] http://paste.openstack.org/show/481746/
[2] http://logs.openstack.org/48/256748/1/check/gate-functional-dsvm-magnum-swarm/56d79c3/console.html
[3] https://review.openstack.org/#/c/254370/2/queries/1521237.yaml

Best regards,
Hongbin

From: Kai Qiang Wu [mailto:wkqwu at cn.ibm.com]
Sent: December-09-15 7:23 AM
To: openstack-dev at lists.openstack.org
Subject: [openstack-dev] [Infra][nova][magnum] Jenkins failed quite often for "Cannot set up guest memory 'pc.ram': Cannot allocate memory"


Hi All,

I am not sure what changes these days, We found quite often now, the Jenkins failed for:


http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz

2015-12-09 08:52:27.892<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_892>+0000: 22957: debug : qemuMonitorJSONCommandWithFd:264 : Send command '{"execute":"qmp_capabilities","id":"libvirt-1"}' for write with FD -1
2015-12-09 08:52:27.892<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_892>+0000: 22957: debug : qemuMonitorSend:959 : QEMU_MONITOR_SEND_MSG: mon=0x7fa66400c6f0 msg={"execute":"qmp_capabilities","id":"libvirt-1"}
 fd=-1
2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:347 : dispatching to max 0 clients, called from event watch 6
2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:360 : event not handled.
2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:347 : dispatching to max 0 clients, called from event watch 6
2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:360 : event not handled.
2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:347 : dispatching to max 0 clients, called from event watch 6
2015-12-09 08:52:27.941<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000: 22951: debug : virNetlinkEventCallback:360 : event not handled.
2015-12-09 08:52:28.070<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_28_070>+0000: 22951: error : qemuMonitorIORead:554 : Unable to read from monitor: Connection reset by peer
2015-12-09 08:52:28.070<http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_28_070>+0000: 22951: error : qemuMonitorIO:690 : internal error: early end of file from monitor: possible problem:
Cannot set up guest memory 'pc.ram': Cannot allocate memory




It not hit such resource issue before. I am not sure if Infra or nova guys know about it ?


Thanks

Best Wishes,
--------------------------------------------------------------------------------
Kai Qiang Wu (吴开强 Kennan）
IBM China System and Technology Lab, Beijing

E-mail: wkqwu at cn.ibm.com<mailto:wkqwu at cn.ibm.com>
Tel: 86-10-82451647
Address: Building 28(Ring Building), ZhongGuanCun Software Park,
No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193
--------------------------------------------------------------------------------
Follow your heart. You are miracle!
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20151212/9a2d5a5a/attachment.html>