Open Stack

Looking at the dstat logs for a recent fail [0], it did help in that
more memory is available. You now have over 1GB available but still less
than 2GB. I would try using less memory. Can you use a 1GB flavor
instead of a 2GB flavor?

[0]
http://logs.openstack.org/58/251158/4/check/gate-functional-dsvm-magnum-swarm/6b022cc/logs/screen-dstat.txt.gz

On Sun, Dec 20, 2015, at 12:08 PM, Hongbin Lu wrote:
> Hi Clark,
> 
> Thanks for the fix. Unfortunately, it doesn't seem to help. The error
> still occurred [1] after you increased the memory restriction, and as
> before, most of them occurred in OVH host. Any further suggestion?
> 
> [1] http://status.openstack.org/elastic-recheck/#1521237
> 
> Best regards,
> Hongbin
> 
> -----Original Message-----
> From: Clark Boylan [mailto:cboylan at sapwetik.org] 
> Sent: December-15-15 5:41 PM
> To: openstack-dev at lists.openstack.org
> Subject: Re: [openstack-dev] [Infra][nova][magnum] Jenkins failed quite
> often for "Cannot set up guest memory 'pc.ram': Cannot allocate memory"
> 
> On Sun, Dec 13, 2015, at 10:51 AM, Clark Boylan wrote:
> > On Sat, Dec 12, 2015, at 02:16 PM, Hongbin Lu wrote:
> > > Hi,
> > > 
> > > As Kai Qiang mentioned, magnum gate recently had a bunch of random 
> > > failures, which occurred on creating a nova instance with 2G of RAM.
> > > According to the error message, it seems that the hypervisor tried 
> > > to allocate memory to the nova instance but couldn’t find enough 
> > > free memory in the host. However, by adding a few “nova 
> > > hypervisor-show XX” before, during, and right after the test, it 
> > > showed that the host has 6G of free RAM, which is far more than 2G. 
> > > Here is a snapshot of the output [1]. You can find the full log here [2].
> > If you look at the dstat log
> > http://logs.openstack.org/07/244907/5/check/gate-functional-dsvm-magnu
> > m-k8s/5305d7a/logs/screen-dstat.txt.gz
> > the host has nowhere near 6GB free memory and less than 2GB. I think 
> > you actually are just running out of memory.
> > > 
> > > Another observation is that most of the failure happened on a node 
> > > with name “devstack-trusty-ovh-*” (You can verify it by entering a 
> > > query [3] at http://logstash.openstack.org/ ). It seems that the 
> > > jobs will be fine if they are allocated to a node other than “ovh”.
> > I have just done a quick spot check of the total memory on 
> > devstack-trusty hosts across HPCloud, Rackspace, and OVH using `free 
> > -m` and the results are 7480, 7732, and 6976 megabytes respectively. 
> > Despite using 8GB flavors in each case there is variation and OVH 
> > comes in on the low end for some reason. I am guessing that you fail 
> > here more often because the other hosts give you just enough extra 
> > memory to boot these VMs.
> To follow up on this we seem to have tracked this down to how the linux
> kernel restricts memory at boot when you don't have a contiguous chunk of
> system memory. We have worked around this by increasing the memory
> restriction to 9023M at boot which gets OVH inline with Rackspace and
> slightly increases available memory on HPCloud (because it actually has
> more of it).
> 
> You should see this fix in action after image builds complete tomorrow
> (they start at 1400UTC ish).
> > 
> > We will have to look into why OVH has less memory despite using 
> > flavors that should be roughly equivalent.
> > > 
> > > Any hints to debug this issue further? Suggestions are greatly 
> > > appreciated.
> > > 
> > > [1] http://paste.openstack.org/show/481746/
> > > [2]
> > > http://logs.openstack.org/48/256748/1/check/gate-functional-dsvm-mag
> > > num-swarm/56d79c3/console.html [3] 
> > > https://review.openstack.org/#/c/254370/2/queries/1521237.yaml
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev