Open Stack

BTW the tracker link is http://tracker.ceph.com/issues/6142

This is an interesting issue, I'm definitely curious.

May I ask if this happened to you as well during recovery as is described in the tracker issue ?
Also, if you divide the amount of placement groups by the amount of OSDs - what number are you getting at ?

If this happens mostly during recovery, I'm curious if the amount of placement groups (other than the thread config) plays a role in the amount of threads required for healing and replication.

Thanks.
--
David Moreau Simard

De : "Fischer, Matt" <matthew.fischer at twcable.com<mailto:matthew.fischer at twcable.com>>
Date : Thu, 28 Aug 2014 16:51:18 -0400
À : Warren Wang <warren at wangspeed.com<mailto:warren at wangspeed.com>>, "openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>" <openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>>
Objet : Re: [Openstack-operators] Ceph crashes with larger clusters and denser hardware

What version of ceph was this seen on?

From: Warren Wang <warren at wangspeed.com<mailto:warren at wangspeed.com>>
Date: Thursday, August 28, 2014 10:38 AM
To: "openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>" <openstack-operators at lists.openstack.org<mailto:openstack-operators at lists.openstack.org>>
Subject: [Openstack-operators] Ceph crashes with larger clusters and denser hardware

One of my colleagues here at Comcast just returned from the Operators Summit and mentioned that multiple folks experienced Ceph instability with larger clusters. I wanted to send out a note and save headache for some folks.

If you up the number of threads per OSD, there are situations where many threads could be quickly spawned. You must up the max number of PIDs available to the OS, otherwise you essentially get fork bombed. Every single Ceph process with crash, and you might see a message in your shell about "Cannot allocate memory".

In your sysctl.conf:

# For Ceph
kernel.pid_max=4194303

Then run "sysctl -p". In 5 days on a lab Ceph box, we have mowed through nearly 2 million PIDs. There's a tracker about this to add it to the ceph.com<http://ceph.com> docs.

Warren
@comcastwarren

________________________________
This E-mail and any of its attachments may contain Time Warner Cable proprietary information, which is privileged, confidential, or subject to copyright belonging to Time Warner Cable. This E-mail is intended solely for the use of the individual or entity to which it is addressed. If you are not the intended recipient of this E-mail, you are hereby notified that any dissemination, distribution, copying, or action taken in relation to the contents of and attachments to this E-mail is strictly prohibited and may be unlawful. If you have received this E-mail in error, please notify the sender immediately and permanently delete the original and any copy of this E-mail and any printout.
_______________________________________________ OpenStack-operators mailing list OpenStack-operators at lists.openstack.org<mailto:OpenStack-operators at lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-operators/attachments/20140828/65ddb5d6/attachment.html>