[Openstack] [Swift] block I/O all disks
hkraemer at anynines.com
Mon Nov 9 10:39:09 UTC 2015
-----BEGIN PGP SIGNED MESSAGE-----
nothing can be logged because all disks are stale.
I see only an output on the IPMI
blocked for more than 120 seconds
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables
I found there could be a problem with the write caching of the disks.
So i have reduced the available dirty cache:
vm.dirty_background_ratio = 5
vm.dirty_ratio = 10
I hope this will solve the issue.
You're right only a hard reboot can solve the problem because SSH
login or other commands can't be executed because the whole system was
That's the problem i can't get any deeper informations was happened.
Thanks and cheers
On 07.11.2015 00:19, Mark Kirkwood wrote:
> Do you reboot the machine? It might be interesting just restart
> the swift storage services and see if that brings everything right
> Also check out the swift logs for what it is doing when things
> start to hang, and also what dmesg is saying at the time.
> A bit more info about your setup would be good - I'm guessing you
> have 12 swift object devices (on sata) and ? account and container
> ones (on ssd)?
> Approx how many containers and objects live on each server (or if
> easier tell us how many servers you have + replication level and
> how many accounts, containers and objects in total)?
> On 04/11/15 22:41, Heiko Krämer wrote:
>> Hi guys,
>> we notice on our Swift storage nodes some problems with our
>> disks. After some time they blocks all I/O requests to the disks.
>> Therefore the server isn't working suddenly and needs a reboot.
>> Serversetup: * Kernel 3.19.x * Ubuntu 14.04 * Swift (Kilo) * 12 x
>> 2TB SATA (JBOD) * 2 x 480GB SSD (Raid1) * 32GB RAM * 8 Cores CPU
>> The first try was an upgrade of the raid controller firmware and
>> drivers. The second one some tests of writes and reads to each
>> disk. I can't reproduce this issue but i heard on the last
>> summit, Swift can be the problem by this issue.
>> Do anyone solve this problem ?
>> - Heiko
>> _______________________________________________ Mailing list:
>> Post to : openstack at lists.openstack.org Unsubscribe :
> _______________________________________________ Mailing list:
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post
> to : openstack at lists.openstack.org Unsubscribe :
- - ----
Geschäftsführer: Alexander Faißt, Dipl.-Inf.(FH) Julian Fischer
Handelsregister: AG Saarbrücken HRB 17413, Ust-IdNr.: DE262633168
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.22 (GNU/Linux)
-----END PGP SIGNATURE-----
More information about the Openstack