Open Stack

Clark,


What about ephemeral storage at OVH vms? I see may storage related errors (see full output below) these days.
Basically it  means Docker cannot create storage device at local drive

-- Logs begin at Mon 2015-12-14 06:40:09 UTC, end at Mon 2015-12-14 07:00:38 UTC. --
Dec 14 06:45:50 <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_45_50> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]: Stopped Docker Application Container Engine.
Dec 14 06:47:54 <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_47_54> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]: Starting Docker Application Container Engine...
Dec 14 06:48:00 <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_00> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 docker[1022]: Warning: '-d' is deprecated, it will be removed soon. See usage.
Dec 14 06:48:00 <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_00> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 docker[1022]: time="2015-12-14T06:48:00Z" level=warning msg="please use 'docker daemon' instead."
Dec 14 06:48:03 <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_03> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 docker[1022]: time="2015-12-14T06:48:03.447936206Z" level=info msg="Listening for HTTP on unix (/var/run/docker.sock)"
Dec 14 06:48:06 <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_06> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 docker[1022]: time="2015-12-14T06:48:06.280086735Z" level=fatal msg="Error starting daemon: error initializing graphdriver: Non existing device docker-docker--pool"
Dec 14 06:48:06 <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_06> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]: docker.service: main process exited, code=exited, status=1/FAILURE
Dec 14 06:48:06 <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_06> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]: Failed to start Docker Application Container Engine.
Dec 14 06:48:06 <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_06> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]: Unit docker.service entered failed state.
Dec 14 06:48:06 <http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz#_Dec_14_06_48_06> te-egw4i5xthw-0-nmaiwpjhkqg6-kube-minion-5emvszmbwpi2 systemd[1]: docker.service failed.


http://logs.openstack.org/58/251158/3/check/gate-functional-dsvm-magnum-k8s/5ed0e01/logs/bay-nodes/worker-test_replication_controller_apis-172.24.5.11/docker.txt.gz


—
Egor




On 12/13/15, 10:51, "Clark Boylan" <cboylan at sapwetik.org> wrote:

>On Sat, Dec 12, 2015, at 02:16 PM, Hongbin Lu wrote:
>> Hi,
>> 
>> As Kai Qiang mentioned, magnum gate recently had a bunch of random
>> failures, which occurred on creating a nova instance with 2G of RAM.
>> According to the error message, it seems that the hypervisor tried to
>> allocate memory to the nova instance but couldn’t find enough free memory
>> in the host. However, by adding a few “nova hypervisor-show XX” before,
>> during, and right after the test, it showed that the host has 6G of free
>> RAM, which is far more than 2G. Here is a snapshot of the output [1]. You
>> can find the full log here [2].
>If you look at the dstat log
>http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnum-k8s/5305d7a/logs/screen-dstat.txt.gz
>the host has nowhere near 6GB free memory and less than 2GB. I think you
>actually are just running out of memory.
>> 
>> Another observation is that most of the failure happened on a node with
>> name “devstack-trusty-ovh-*” (You can verify it by entering a query [3]
>> at http://logstash.openstack.org/ ). It seems that the jobs will be fine
>> if they are allocated to a node other than “ovh”.
>I have just done a quick spot check of the total memory on
>devstack-trusty hosts across HPCloud, Rackspace, and OVH using `free -m`
>and the results are 7480, 7732, and 6976 megabytes respectively. Despite
>using 8GB flavors in each case there is variation and OVH comes in on
>the low end for some reason. I am guessing that you fail here more often
>because the other hosts give you just enough extra memory to boot these
>VMs.
>
>We will have to look into why OVH has less memory despite using flavors
>that should be roughly equivalent.
>> 
>> Any hints to debug this issue further? Suggestions are greatly
>> appreciated.
>> 
>> [1] http://paste.openstack.org/show/481746/
>> [2]
>> http://logs.openstack.org/48/256748/1/check/gate-functional-dsvm-magnum-swarm/56d79c3/console.html
>> [3] https://review.openstack.org/#/c/254370/2/queries/1521237.yaml
>> 
>> Best regards,
>> Hongbin
>
>Clark
>
>__________________________________________________________________________
>OpenStack Development Mailing List (not for usage questions)
>Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev