[openstack-dev] [Infra][nova][magnum] Jenkins failed quite often for "Cannot set up guest memory 'pc.ram': Cannot allocate memory"

Clark Boylan cboylan at sapwetik.org
Tue Dec 15 22:37:23 UTC 2015


There is no ephemeral drive you will have to use the root disk. Grabbing
a random VM in OVH and running a quick df on it you should have about
60GB of free space for the job to use there. 

Clark

On Tue, Dec 15, 2015, at 02:00 PM, Egor Guz wrote:
> Clark,
> 
> 
> What about ephemeral storage at OVH vms? I see may storage related errors
> (see full output below) these days.
> Basically it  means Docker cannot create storage device at local drive
> 
> -- Logs begin at Mon 2015-12-14 06:40:09 UTC, end at Mon 2015-12-14
> 07:00:38 UTC. --
> Dec 14 06:45:50
> <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_45_50>
> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]: Stopped
> Docker Application Container Engine.
> Dec 14 06:47:54
> <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_47_54>
> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]:
> Starting Docker Application Container Engine...
> Dec 14 06:48:00
> <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_00>
> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 docker[1022]:
> Warning: '-d' is deprecated, it will be removed soon. See usage.
> Dec 14 06:48:00
> <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_00>
> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 docker[1022]:
> time="2015-12-14T06:48:00Z" level=warning msg="please use 'docker daemon'
> instead."
> Dec 14 06:48:03
> <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_03>
> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 docker[1022]:
> time="2015-12-14T06:48:03.447936206Z" level=info msg="Listening for HTTP
> on unix (/var/run/docker.sock)"
> Dec 14 06:48:06
> <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_06>
> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 docker[1022]:
> time="2015-12-14T06:48:06.280086735Z" level=fatal msg="Error starting
> daemon: error initializing graphdriver: Non existing device
> docker-docker--pool"
> Dec 14 06:48:06
> <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_06>
> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]:
> docker.service: main process exited, code=exited, status=1/FAILURE
> Dec 14 06:48:06
> <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_06>
> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]: Failed
> to start Docker Application Container Engine.
> Dec 14 06:48:06
> <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_06>
> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]: Unit
> docker.service entered failed state.
> Dec 14 06:48:06
> <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_06>
> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]:
> docker.service failed.
> 
> 
> http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz
> 
> 
>> Egor
> 
> 
> 
> 
> On 12/13/15, 10:51, "Clark Boylan" <cboylan at sapwetik.org> wrote:
> 
> >On Sat, Dec 12, 2015, at 02:16 PM, Hongbin Lu wrote:
> >> Hi,
> >> 
> >> As Kai Qiang mentioned, magnum gate recently had a bunch of random
> >> failures, which occurred on creating a nova instance with 2G of RAM.
> >> According to the error message, it seems that the hypervisor tried to
> >> allocate memory to the nova instance but couldn’t find enough free memory
> >> in the host. However, by adding a few “nova hypervisor-show XX” before,
> >> during, and right after the test, it showed that the host has 6G of free
> >> RAM, which is far more than 2G. Here is a snapshot of the output [1]. You
> >> can find the full log here [2].
> >If you look at the dstat log
> >http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/screen-dstat.txt.gz
> >the host has nowhere near 6GB free memory and less than 2GB. I think you
> >actually are just running out of memory.
> >> 
> >> Another observation is that most of the failure happened on a node with
> >> name “devstack-trusty-ovh-*” (You can verify it by entering a query [3]
> >> at http://logstash.openstack.org/ ). It seems that the jobs will be fine
> >> if they are allocated to a node other than “ovh”.
> >I have just done a quick spot check of the total memory on
> >devstack-trusty hosts across HPCloud, Rackspace, and OVH using `free -m`
> >and the results are 7480, 7732, and 6976 megabytes respectively. Despite
> >using 8GB flavors in each case there is variation and OVH comes in on
> >the low end for some reason. I am guessing that you fail here more often
> >because the other hosts give you just enough extra memory to boot these
> >VMs.
> >
> >We will have to look into why OVH has less memory despite using flavors
> >that should be roughly equivalent.
> >> 
> >> Any hints to debug this issue further? Suggestions are greatly
> >> appreciated.
> >> 
> >> [1] http://paste.openstack.org/show/481746/
> >> [2]
> >> http://logs.openstack.org/48/256748/1/check/gate-functional-dsvm-magnum-swarm/56d79c3/console.html
> >> [3] https://review.openstack.org/#/c/254370/2/queries/1521237.yaml
> >> 
> >> Best regards,
> >> Hongbin
> >
> >Clark
> >
> >__________________________________________________________________________
> >OpenStack Development Mailing List (not for usage questions)
> >Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list