Open Stack

Hi,

OVH is a new cloud provider for openstack-infra nodes:
http://www.openstack.org/blog/2015/12/announcing-a-new-cloud-provider-for-openstacks-ci-system-ovh/

It appears that selection of nodes on any cloud provider is a matter of 
luck:
"When a developer uploads a proposed change to an OpenStack project, 
available instances from any of our contributing cloud providers will be 
used interchangeably to test it."

You might want to ping people in #openstack-infra to find a point of 
contact for them (OVH) and/or to work with the infra folks directly to 
see about troubleshooting this further.


On 12/12/2015 02:16 PM, Hongbin Lu wrote:
> Hi,
>
> As Kai Qiang mentioned, magnum gate recently had a bunch of random
> failures, which occurred on creating a nova instance with 2G of RAM.
> According to the error message, it seems that the hypervisor tried to
> allocate memory to the nova instance but couldn’t find enough free
> memory in the host. However, by adding a few “nova hypervisor-show XX”
> before, during, and right after the test, it showed that the host has 6G
> of free RAM, which is far more than 2G. Here is a snapshot of the output
> [1]. You can find the full log here [2].
>
> Another observation is that most of the failure happened on a node with
> name “devstack-trusty-ovh-*” (You can verify it by entering a query [3]
> at http://logstash.openstack.org/ ). It seems that the jobs will be fine
> if they are allocated to a node other than “ovh”.
>
> Any hints to debug this issue further? Suggestions are greatly appreciated.
>
> <http://logstash.openstack.org/>
>
> [1] <http://logstash.openstack.org/>http://paste.openstack.org/show/481746/
>
> [2]
> http://logs.openstack.org/48/256748/1/check/gate-functional-dsvm-magnum-swarm/56d79c3/console.html
>
> [3] https://review.openstack.org/#/c/254370/2/queries/1521237.yaml
>
> Best regards,
>
> Hongbin
>
> *From:*Kai Qiang Wu [mailto:wkqwu at cn.ibm.com]
> *Sent:* December-09-15 7:23 AM
> *To:* openstack-dev at lists.openstack.org
> *Subject:* [openstack-dev] [Infra][nova][magnum] Jenkins failed quite
> often for "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
>
> Hi All,
>
> I am not sure what changes these days, We found quite often now, the
> Jenkins failed for:
>
>
> http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz
>
> 2015-12-09 08:52:27.892
> <http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_892>+0000:
> 22957: debug : qemuMonitorJSONCommandWithFd:264 : Send command
> '{"execute":"qmp_capabilities","id":"libvirt-1"}' for write with FD -1
> 2015-12-09 08:52:27.892
> <http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_892>+0000:
> 22957: debug : qemuMonitorSend:959 : QEMU_MONITOR_SEND_MSG:
> mon=0x7fa66400c6f0 msg={"execute":"qmp_capabilities","id":"libvirt-1"}
>   fd=-1
> 2015-12-09 08:52:27.941
> <http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000:
> 22951: debug : virNetlinkEventCallback:347 : dispatching to max 0
> clients, called from event watch 6
> 2015-12-09 08:52:27.941
> <http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000:
> 22951: debug : virNetlinkEventCallback:360 : event not handled.
> 2015-12-09 08:52:27.941
> <http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000:
> 22951: debug : virNetlinkEventCallback:347 : dispatching to max 0
> clients, called from event watch 6
> 2015-12-09 08:52:27.941
> <http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000:
> 22951: debug : virNetlinkEventCallback:360 : event not handled.
> 2015-12-09 08:52:27.941
> <http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000:
> 22951: debug : virNetlinkEventCallback:347 : dispatching to max 0
> clients, called from event watch 6
> 2015-12-09 08:52:27.941
> <http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_27_941>+0000:
> 22951: debug : virNetlinkEventCallback:360 : event not handled.
> 2015-12-09 08:52:28.070
> <http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_28_070>+0000:
> 22951: error : qemuMonitorIORead:554 : Unable to read from monitor:
> Connection reset by peer
> 2015-12-09 08:52:28.070
> <http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/libvirt/libvirtd.txt.gz#_2015-12-09_08_52_28_070>+0000:
> 22951: error : qemuMonitorIO:690 : internal error: early end of file
> from monitor: possible problem:
> Cannot set up guest memory 'pc.ram': Cannot allocate memory
>
>
>
>
> It not hit such resource issue before. I am not sure if Infra or nova
> guys know about it ?
>
>
> Thanks
>
> Best Wishes,
> --------------------------------------------------------------------------------
> Kai Qiang Wu (吴开强 Kennan）
> IBM China System and Technology Lab, Beijing
>
> E-mail: wkqwu at cn.ibm.com <mailto:wkqwu at cn.ibm.com>
> Tel: 86-10-82451647
> Address: Building 28(Ring Building), ZhongGuanCun Software Park,
> No.8 Dong Bei Wang West Road, Haidian District Beijing P.R.China 100193
> --------------------------------------------------------------------------------
> Follow your heart. You are miracle!
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>