Open Stack

I also have limited experience with Ceph and rados bench - but it looks
like you're setting the number of "worker threads" to only 1?  (-t 1)

I think the default is 16, and most storage distributed storage systems
designed for concurrency are going to do a bit better if you exercise more
concurrent workers... so you might try turning that up until you see some
diminishing returns.  Be sure to watch for resource contention on the load
generating server.

-Clay

On Mon, Sep 29, 2014 at 4:49 AM, Pasquale Porreca <
pasquale.porreca at dektech.com.au> wrote:

>  Hello
>
> I have no experience with Ceph and this specific benchmark tool, anyway I
> have experience with several other performance benchmark tools and file
> systems and I can say it always happen to have very very low performance
> results when the file size is too small (i.e. < 1MB).
>
> My suspect is that benchmark tools are not reliable for file size so
> small, since the time to write is so small that the overhead introduced by
> the test itself is not at all negligible.
>
> I saw that the default object size for rados is 4 MB, did you try your
> test without the option "-b 512"? I think the results should be different
> for several order of magnitude.
>
> BR
>
>
> On 09/27/14 17:14, Timur Nurlygayanov wrote:
>
>   Hello all,
>
>  I installed OpenStack with Glance + Ceph OSD with replication factor 2
> and now I can see the write operations are extremly slow.
> For example, I can see only 0.04 MB/s write speed when I run rados bench
> with 512b blocks:
>
>  rados bench -p test 60 write --no-cleanup -t 1 -b 512
>
>  Maintaining 1 concurrent writes of 512 bytes for up to 60 seconds or 0
> objects
>  Object prefix: benchmark_data_node-17.domain.tld_15862
>    sec Cur ops   started  finished    avg MB/s     cur MB/s       last
> lat          avg lat
>      0       0         0         0              0
> 0                   -                   0
>      1       1        83        82            0.0400341   0.0400391
> 0.008465       0.0120985
>      2       1       169       168          0.0410111    0.0419922
> 0.080433       0.0118995
>      3       1       240       239          0.0388959    0.034668
> 0.008052       0.0125385
>      4       1       356       355          0.0433309   0.0566406
> 0.00837         0.0112662
>      5       1       472       471          0.0459919   0.0566406
> 0.008343       0.0106034
>      6       1       550       549          0.0446735   0.0380859
> 0.036639       0.0108791
>      7       1       581       580          0.0404538   0.0151367
> 0.008614       0.0120654
>
>
> My test environment configuration:
>  Hardware servers with 1Gb network interfaces, 64Gb RAM and 16 CPU cores
> per node, HDDs WDC WD5003ABYX-01WERA0.
>  OpenStack with 1 controller, 1 compute and 2 ceph nodes (ceph on separate
> nodes).
> CentOS 6.5, kernel 2.6.32-431.el6.x86_64.
>
>  I tested several config options for optimizations, like in
> /etc/ceph/ceph.conf:
>
>  [default]
> ...
> osd_pool_default_pg_num = 1024
> osd_pool_default_pgp_num = 1024
> osd_pool_default_flag_hashpspool = true
> ...
> [osd]
> osd recovery max active = 1
> osd max backfills = 1
> filestore max sync interval = 30
> filestore min sync interval = 29
> filestore flusher = false
> filestore queue max ops = 10000
> filestore op threads = 16
> osd op threads = 16
> ...
> [client]
> rbd_cache = true
> rbd_cache_writethrough_until_flush = true
>
>  and in /etc/cinder/cinder.conf:
>
>  [DEFAULT]
>  volume_tmp_dir=/tmp
>
> but in the result performance was increased only on ~30 % and it not looks
> like huge success.
>
>  Non-default mount options and TCP optimization increase the speed in
> about 1%:
>
> [root at node-17 ~]# mount | grep ceph
> /dev/sda4 on /var/lib/ceph/osd/ceph-0 type xfs
> (rw,noexec,nodev,noatime,nodiratime,user_xattr,data=writeback,barrier=0)
>
> [root at node-17 ~]# cat /etc/sysctl.conf
> net.core.rmem_max = 16777216
> net.core.wmem_max = 16777216
> net.ipv4.tcp_rmem = 4096 87380 16777216
> net.ipv4.tcp_wmem = 4096 65536 16777216
> net.ipv4.tcp_window_scaling = 1
> net.ipv4.tcp_timestamps = 1
> net.ipv4.tcp_sack = 1
>
>
> Do we have other ways to significantly improve CEPH storage performance?
>  Any feedback and comments are welcome!
>
>  Thank you!
>
>
>  --
>
>  Timur,
> QA Engineer
> OpenStack Projects
> Mirantis Inc
>
>
> _______________________________________________
> OpenStack-dev mailing listOpenStack-dev at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> --
> Pasquale Porreca
>
> DEK Technologies
> Via dei Castelli Romani, 22
> 00040 Pomezia (Roma)
>
> Mobile +39 3394823805
> Skype paskporr
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20140929/54a86564/attachment.html>