[cinder] volume encryption performance impact
Hello, I've just started investigating Cinder volume encryption using Queens (RHOSP13) with a Ceph/RBD backend and the performance overhead is... surprising. Some naive bonnie++ numbers, comparing a plain vs encrypted volume: plain: write 1400MB/s, read 390MB/s encrypted: write 81MB/s, read 83MB/s The encryption was configured with: openstack volume type create --encryption-provider nova.volume.encryptors.luks.LuksEncryptor --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LuksEncryptor-Template-256 Does anyone have a similar setup, and can share their performance figures, or give me an idea of what percentage performance impact I should expect? Alternatively: is AES256 overkill, or, where should I start looking for a misconfiguration or bottleneck? Thanks in advance. Dave -- ** Dave Holland ** Systems Support -- Informatics Systems Group ** ** 01223 496923 ** Wellcome Sanger Institute, Hinxton, UK ** -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
On 19-01-09 15:13:29, Dave Holland wrote:
Hello,
I've just started investigating Cinder volume encryption using Queens (RHOSP13) with a Ceph/RBD backend and the performance overhead is... surprising. Some naive bonnie++ numbers, comparing a plain vs encrypted volume:
plain: write 1400MB/s, read 390MB/s encrypted: write 81MB/s, read 83MB/s
The encryption was configured with:
openstack volume type create --encryption-provider nova.volume.encryptors.luks.LuksEncryptor --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LuksEncryptor-Template-256
Does anyone have a similar setup, and can share their performance figures, or give me an idea of what percentage performance impact I should expect? Alternatively: is AES256 overkill, or, where should I start looking for a misconfiguration or bottleneck?
I haven't tested yet, but that doesn't sound right, it sounds like it's not using aes-ni (or tha amd equiv). 256 may be higher than is needed (256 aes has some attacks that 128 does not iirc as well) but should drop perf that much unless it's dropping back to sofware. -- Matthew Thode
Hi Dave, With the same key length and backend, we’ve done some quick checks at the time, but did not notice any significant performance impact (beyond a slight CPU increase). We did not test beyond the QoS limits we apply, though. Cheers, Arne
On 9 Jan 2019, at 16:13, Dave Holland <dh3@sanger.ac.uk> wrote:
Hello,
I've just started investigating Cinder volume encryption using Queens (RHOSP13) with a Ceph/RBD backend and the performance overhead is... surprising. Some naive bonnie++ numbers, comparing a plain vs encrypted volume:
plain: write 1400MB/s, read 390MB/s encrypted: write 81MB/s, read 83MB/s
The encryption was configured with:
openstack volume type create --encryption-provider nova.volume.encryptors.luks.LuksEncryptor --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LuksEncryptor-Template-256
Does anyone have a similar setup, and can share their performance figures, or give me an idea of what percentage performance impact I should expect? Alternatively: is AES256 overkill, or, where should I start looking for a misconfiguration or bottleneck?
Thanks in advance.
Dave -- ** Dave Holland ** Systems Support -- Informatics Systems Group ** ** 01223 496923 ** Wellcome Sanger Institute, Hinxton, UK **
-- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
-- Arne Wiebalck CERN IT
On 09-01-19 15:13:29, Dave Holland wrote:
Hello,
I've just started investigating Cinder volume encryption using Queens (RHOSP13) with a Ceph/RBD backend and the performance overhead is... surprising. Some naive bonnie++ numbers, comparing a plain vs encrypted volume:
plain: write 1400MB/s, read 390MB/s encrypted: write 81MB/s, read 83MB/s
The encryption was configured with:
openstack volume type create --encryption-provider nova.volume.encryptors.luks.LuksEncryptor --encryption-cipher aes-xts-plain64 --encryption-key-size 256 --encryption-control-location front-end LuksEncryptor-Template-256
Does anyone have a similar setup, and can share their performance figures, or give me an idea of what percentage performance impact I should expect? Alternatively: is AES256 overkill, or, where should I start looking for a misconfiguration or bottleneck?
What's the underlying version of QEMU being used here? FWIW I can't recall seeing any performance issues when working on and verifying this downstream with QEMU 2.10. Cheers, -- Lee Yarwood A5D1 9385 88CB 7E5F BE64 6618 BCA6 6E33 F672 2D76
Thanks Lee, Arne, Thomas for replying. On Thu, Jan 10, 2019 at 01:56:05PM +0000, Lee Yarwood wrote:
What's the underlying version of QEMU being used here?
It's qemu-kvm-rhev-2.10.0-21.el7_5.4.x86_64
FWIW I can't recall seeing any performance issues when working on and verifying this downstream with QEMU 2.10.
I had wondered about https://bugzilla.redhat.com/1500334 (LUKS driver buffer size) which fits the symptoms, but the fix apparently went in to qemu-kvm-rhev-2.10.0-11.el7 so shouldn't be affecting us. I have a case open with RH Support now and I am keeping my fingers crossed. We will be redeploying this system again shortly with the latest Queens/RHOSP13 package versions, so should end up with qemu-kvm-rhev-2.12.0-18.el7_6.1.x86_64 and I will re-test then. Cheers, Dave -- ** Dave Holland ** Systems Support -- Informatics Systems Group ** ** 01223 496923 ** Wellcome Sanger Institute, Hinxton, UK ** -- The Wellcome Sanger Institute is operated by Genome Research Limited, a charity registered in England with number 1021457 and a company registered in England with number 2742969, whose registered office is 215 Euston Road, London, NW1 2BE.
participants (4)
-
Arne Wiebalck
-
Dave Holland
-
Lee Yarwood
-
Matthew Thode