[osc] Very slow openstack commands in Stein Devstack running on VM
I run Stein Devstack in a virtual machine. Occasionally, openstack client commands take minutes to complete, which didn't happen at earlier releases, and which is rather annoying. The delay seems to take place at an early stage: Even when I forget to set the OS_* environment, it takes minutes until I am told that the request requires authentication. Interrupting the command, the stack trace confirms this (as a side remark, it also takes a long while until my ^C is processed): Traceback (most recent call last): File "/usr/local/bin/openstack", line 7, in <module> from openstackclient.shell import main File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 22, in <module> from osc_lib.api import auth File "/usr/local/lib/python2.7/dist-packages/osc_lib/api/auth.py", line 19, in <module> from keystoneauth1.loading import base File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/__init__.py", line 13, in <module> from keystoneauth1.loading import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/adapter.py", line 13, in <module> from keystoneauth1 import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 17, in <module> from keystoneauth1 import session File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 25, in <module> import requests File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 95, in <module> from urllib3.contrib import pyopenssl File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 48, in <module> from cryptography.hazmat.backends.openssl import backend as openssl_backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module> from cryptography.hazmat.backends.openssl.backend import backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 2419, in <module> backend = Backend() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 119, in __init__ self.activate_osrandom_engine() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 163, in activate_osrandom_engine with self._get_osurandom_engine() as e: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 146, in _get_osurandom_engine res = self._lib.ENGINE_init(e) KeyboardInterrupt My wild speculation is that the delay has something to do with obtaining a random number, which can be a problem on virtual machines due to their lack of entropy. My knowledge in this area is paper-thin, though. And I wonder what happens to all the Devstacks used in CI? Thus my questions: *Is the delay really caused by my running the cloud on a VM? What can I do to improve this?* Thanks, Bernd
After a little bit of self-education, I found that /proc/sys/kernel/random/entropy_avail was below 100, which is probably a good explanation for my problem. A good value would have four digits, it seems. My Devstack host runs Ubuntu 18, and I am now trying the rngd-tools5 package to improve the entropy situation. Since this is a test system, I am not so concerned about random number quality. In short, the problem seems to be solved, but if anybody has additional comments, I would be interested to hear them. Bernd On 7/28/2019 1:06 PM, Bernd Bausch wrote:
I run Stein Devstack in a virtual machine. Occasionally, openstack client commands take minutes to complete, which didn't happen at earlier releases, and which is rather annoying. The delay seems to take place at an early stage: Even when I forget to set the OS_* environment, it takes minutes until I am told that the request requires authentication.
Interrupting the command, the stack trace confirms this (as a side remark, it also takes a long while until my ^C is processed):
Traceback (most recent call last): File "/usr/local/bin/openstack", line 7, in <module> from openstackclient.shell import main File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 22, in <module> from osc_lib.api import auth File "/usr/local/lib/python2.7/dist-packages/osc_lib/api/auth.py", line 19, in <module> from keystoneauth1.loading import base File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/__init__.py", line 13, in <module> from keystoneauth1.loading import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/adapter.py", line 13, in <module> from keystoneauth1 import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 17, in <module> from keystoneauth1 import session File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 25, in <module> import requests File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 95, in <module> from urllib3.contrib import pyopenssl File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 48, in <module> from cryptography.hazmat.backends.openssl import backend as openssl_backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module> from cryptography.hazmat.backends.openssl.backend import backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 2419, in <module> backend = Backend() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 119, in __init__ self.activate_osrandom_engine() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 163, in activate_osrandom_engine with self._get_osurandom_engine() as e: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 146, in _get_osurandom_engine res = self._lib.ENGINE_init(e) KeyboardInterrupt
My wild speculation is that the delay has something to do with obtaining a random number, which can be a problem on virtual machines due to their lack of entropy. My knowledge in this area is paper-thin, though. And I wonder what happens to all the Devstacks used in CI?
Thus my questions: *Is the delay really caused by my running the cloud on a VM? What can I do to improve this?*
Thanks,
Bernd
On Sun, Jul 28, 2019 at 12:41 AM Bernd Bausch <berndbausch@gmail.com> wrote:
After a little bit of self-education, I found that /proc/sys/kernel/random/entropy_avail was below 100, which is probably a good explanation for my problem. A good value would have four digits, it seems.
My Devstack host runs Ubuntu 18, and I am now trying the rngd-tools5 package to improve the entropy situation. Since this is a test system, I am not so concerned about random number quality.
In short, the problem seems to be solved, but if anybody has additional comments, I would be interested to hear them.
I am running packstack (which IMHO for all practical purposes is the same) centos 7 stein in a ESXi vm guest (hope to move it to kvm as it is getting to my nerves). It is certainly not as fast as running on barebones but it is not minutes slow.
Bernd
On 7/28/2019 1:06 PM, Bernd Bausch wrote:
I run Stein Devstack in a virtual machine. Occasionally, openstack client commands take minutes to complete, which didn't happen at earlier releases, and which is rather annoying. The delay seems to take place at an early stage: Even when I forget to set the OS_* environment, it takes minutes until I am told that the request requires authentication.
Interrupting the command, the stack trace confirms this (as a side remark, it also takes a long while until my ^C is processed):
Traceback (most recent call last): File "/usr/local/bin/openstack", line 7, in <module> from openstackclient.shell import main File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 22, in <module> from osc_lib.api import auth File "/usr/local/lib/python2.7/dist-packages/osc_lib/api/auth.py", line 19, in <module> from keystoneauth1.loading import base File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/__init__.py", line 13, in <module> from keystoneauth1.loading import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/adapter.py", line 13, in <module> from keystoneauth1 import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 17, in <module> from keystoneauth1 import session File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 25, in <module> import requests File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 95, in <module> from urllib3.contrib import pyopenssl File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 48, in <module> from cryptography.hazmat.backends.openssl import backend as openssl_backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module> from cryptography.hazmat.backends.openssl.backend import backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 2419, in <module> backend = Backend() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 119, in __init__ self.activate_osrandom_engine() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 163, in activate_osrandom_engine with self._get_osurandom_engine() as e: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 146, in _get_osurandom_engine res = self._lib.ENGINE_init(e) KeyboardInterrupt
My wild speculation is that the delay has something to do with obtaining a random number, which can be a problem on virtual machines due to their lack of entropy. My knowledge in this area is paper-thin, though. And I wonder what happens to all the Devstacks used in CI?
Thus my questions: Is the delay really caused by my running the cloud on a VM? What can I do to improve this?
Thanks,
Bernd
On Sun, 2019-07-28 at 13:37 +0900, Bernd Bausch wrote:
After a little bit of self-education, I found that /proc/sys/kernel/random/entropy_avail was below 100, which is probably a good explanation for my problem. A good value would have four digits, it seems.
My Devstack host runs Ubuntu 18, and I am now trying the rngd-tools5 package to improve the entropy situation. Since this is a test system, I am not so concerned about random number quality.
In short, the problem seems to be solved, but if anybody has additional comments, I would be interested to hear them. am i have not encountered this on my own ububtu 18.04 devstack vms but i tend to enable the RNG device whcich effectivly presnets the host /dev/urandom to the guests so perhaps that helps. if you are virtulaising with kvm/libvit i would suggest adding the RNG device and seeing if that has a similar effect.
we had also condidered change openstack default at one point to make it so that RNG devices would be added by default in the futre rather im not sure if we ever did anything in that regard but if it can have that much of an impact on things like osc it proably would be a performace imporvment for many applicaitons that use openssl and need entropy.
Bernd
On 7/28/2019 1:06 PM, Bernd Bausch wrote:
I run Stein Devstack in a virtual machine. Occasionally, openstack client commands take minutes to complete, which didn't happen at earlier releases, and which is rather annoying. The delay seems to take place at an early stage: Even when I forget to set the OS_* environment, it takes minutes until I am told that the request requires authentication.
Interrupting the command, the stack trace confirms this (as a side remark, it also takes a long while until my ^C is processed):
Traceback (most recent call last): File "/usr/local/bin/openstack", line 7, in <module> from openstackclient.shell import main File "/usr/local/lib/python2.7/dist-packages/openstackclient/shell.py", line 22, in <module> from osc_lib.api import auth File "/usr/local/lib/python2.7/dist-packages/osc_lib/api/auth.py", line 19, in <module> from keystoneauth1.loading import base File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/__init__.py", line 13, in <module> from keystoneauth1.loading import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/loading/adapter.py", line 13, in <module> from keystoneauth1 import adapter File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/adapter.py", line 17, in <module> from keystoneauth1 import session File "/usr/local/lib/python2.7/dist-packages/keystoneauth1/session.py", line 25, in <module> import requests File "/usr/local/lib/python2.7/dist-packages/requests/__init__.py", line 95, in <module> from urllib3.contrib import pyopenssl File "/usr/local/lib/python2.7/dist-packages/urllib3/contrib/pyopenssl.py", line 48, in <module> from cryptography.hazmat.backends.openssl import backend as openssl_backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module> from cryptography.hazmat.backends.openssl.backend import backend File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 2419, in <module> backend = Backend() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 119, in __init__ self.activate_osrandom_engine() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 163, in activate_osrandom_engine with self._get_osurandom_engine() as e: File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__ return self.gen.next() File "/usr/local/lib/python2.7/dist-packages/cryptography/hazmat/backends/openssl/backend.py", line 146, in _get_osurandom_engine res = self._lib.ENGINE_init(e) KeyboardInterrupt
My wild speculation is that the delay has something to do with obtaining a random number, which can be a problem on virtual machines due to their lack of entropy. My knowledge in this area is paper-thin, though. And I wonder what happens to all the Devstacks used in CI?
Thus my questions: *Is the delay really caused by my running the cloud on a VM? What can I do to improve this?*
Thanks,
Bernd
On Sat, Jul 27, 2019, at 9:07 PM, Bernd Bausch wrote: snip
My wild speculation is that the delay has something to do with obtaining a random number, which can be a problem on virtual machines due to their lack of entropy. My knowledge in this area is paper-thin, though. And I wonder what happens to all the Devstacks used in CI?
All of the test VMs run haveged. We preinstall that on our images: https://opendev.org/openstack/project-config/src/commit/e02cdfa1f489838aa2bb...
Thus my questions: *Is the delay really caused by my running the cloud on a VM? What can I do to improve this?*
Thanks,
Bernd
participants (4)
-
Bernd Bausch
-
Clark Boylan
-
Mauricio Tavares
-
Sean Mooney