Open Stack

Hello all,

I installed OpenStack with Glance + Ceph OSD with replication factor 2 and
now I can see the write operations are extremly slow.
For example, I can see only 0.04 MB/s write speed when I run rados bench
with 512b blocks:

rados bench -p test 60 write --no-cleanup -t 1 -b 512

 Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 0
objects
 Object prefix: benchmark_data_node-17.domain.tld_15862
   sec Cur ops   started  finished    avg MB/s     cur MB/s       last
lat          avg lat
     0       0         0         0              0
0                   -                   0
     1       1        83        82            0.0400341   0.0400391
0.008465       0.0120985
     2       1       169       168          0.0410111    0.0419922
0.080433       0.0118995
     3       1       240       239          0.0388959    0.034668
0.008052       0.0125385
     4       1       356       355          0.0433309   0.0566406
0.00837         0.0112662
     5       1       472       471          0.0459919   0.0566406
0.008343       0.0106034
     6       1       550       549          0.0446735   0.0380859
0.036639       0.0108791
     7       1       581       580          0.0404538   0.0151367
0.008614       0.0120654


My test environment configuration:
Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU cores per
node, HDDs WDC WD5003ABYX-01WERA0.
OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on separate
nodes).
CentOS 6.5, kernel 2.6.32-431.el6.x86_64.

I tested several config options for optimizations, like in
/etc/ceph/ceph.conf:

[default]
...
osd_pool_default_pg_num = 1024
osd_pool_default_pgp_num = 1024
osd_pool_default_flag_hashpspool = true
...
[osd]
osd recovery max active = 1
osd max backfills = 1
filestore max sync interval = 30
filestore min sync interval = 29
filestore flusher = false
filestore queue max ops = 10000
filestore op threads = 16
osd op threads = 16
...
[client]
rbd_cache = true
rbd_cache_writethrough_until_flush = true

and in /etc/cinder/cinder.conf:

[DEFAULT]
volume_tmp_dir=/tmp

but in the result performance was increased only on ~30 % and it not looks
like huge success.

Non-default mount options and TCP optimization increase the speed in about
1%:

[root at node-17 ~]# mount | grep ceph
/dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs
(rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)

[root at node-17 ~]# cat /etc/sysctl.conf
net.core.rmem_max = 16777216
net.core.wmem_max = 16777216
net.ipv4.tcp_rmem = 4096 87380 16777216
net.ipv4.tcp_wmem = 4096 65536 16777216
net.ipv4.tcp_window_scaling = 1
net.ipv4.tcp_timestamps = 1
net.ipv4.tcp_sack = 1


Do we have other ways to significantly improve CEPH storage performance?
Any feedback and comments are welcome!

Thank you!


-- 

Timur,
QA Engineer
OpenStack Projects
Mirantis Inc
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140927/1d4c1686/attachment.html>