December 2022 - openstack-discuss

[keystone][Meeting] Reminder Keystone meeting is cancelled today
by Dave Wilde 27 Dec '22

27 Dec '22

Just a quick reminder that there won’t be the keystone weekly meeting today. We’ll resume our regulararly scheduled programming on 03-Jan-2023. Please update the agenda if you have anything you’d like to discuess. The reviewathon is also cancelled this week, to be resumed on 06-Jan-2023. /Dave

1 0

[ops][nova] RBD IOPS bottleneck on client-side
by Can Özyurt 27 Dec '22

27 Dec '22

Hi everyone, I hope you are all doing well. We are trying to pinpoint an IOPS problem with RBD and decided to ask you for your take on it. 1 control plane 1 compute node 5 storage with 8 SSD disks each Openstack Stein/Ceph Mimic deployed with kolla-ansible on ubuntu-1804 (kernel 5.4) isolcpus 4-127 on compute vcpu_pin_set 4-127 in nova.conf image_metadatas: hw_scsi_model: virtio-scsi hw_disk_bus: scsi flavor_metadatas: hw:cpu_policy: dedicated What we have tested: fio --directory=. --ioengine=libaio --direct=1 --name=benchmark_random_read_write --filename=test_rand --bs=4k --iodepth=32 --size=1G --readwrite=randrw --rwmixread=50 --time_based --runtime=300s --numjobs=16 1. First we run the fio test above on a guest VM, we see average 5K/5K read/write IOPS consistently. What we realize is that during the test, one single core on compute host is used at max, which is the first of the pinned cpus of the guest. 'top -Hp $qemupid' shows that some threads (notably tp_librbd) share the very same core throughout the test. (also emulatorpin set = vcpupin set as expected) 2. We remove isolcpus and every other configuration stays the same. Now fio tests now show 11K/11K read/write IOPS. No bottlenecked single cpu on the host, observed threads seem to visit all emulatorpins. 3. We bring isolcpus back and redeploy the cluster with Train/Nautilus on ubuntu-1804. Observations are identical to #1. 4. We tried replacing vcpu_pin_set to cpu_shared_set and cpu_dedicated_set to be able to pin emulator cpuset to 0-4 to no avail. Multiple guests on a host can easily deplete resources and IOPS drops. 5. Isolcpus are still in place and we deploy Ussuri with kolla-ansible and Train (to limit the moving parts) with ceph-ansible both on ubuntu-1804. Now we see 7K/7K read/write IOPS. 6. We destroy only the compute node and boot it with ubuntu-2004 with isolcpus set. Add it back to the existing cluster and fio shows slightly above 10K/10K read/write IOPS. What we think happens: 1. Since isolcpus disables scheduling between given cpus, qemu process and its threads are stuck at the same cpu which created the bottleneck. They should be runnable on any given emulatorpin cpus. 2. Ussuri is more performant despite isolcpus, with the improvements made over time. 3. Ubuntu-2004 is more performant despite isolcpus, with the improvements made over time in the kernel. Now the questions are: 1. What else are we missing here? 2. Are any of those assumptions false? 3. If all true, what can we do to solve this issue given that we cannot upgrade openstack nor ceph on production overnight? 4. Has anyone dealt with this issue before? We welcome any opinion and suggestions at this point as we need to make sure that we are on the right path regarding the problem and upgrade is not the only solution. Thanks in advance.

1 0

Container access mounted device
by vincent lee 27 Dec '22

27 Dec '22

Hi, I have deployed a working OpenStack and would like to know if there are any possible options I can enable so that when creating a new container, files can be mounted to it, and a user from the container can access it. For instance, having a mounted device from the compute host and allowing the /dev/(mounted device name) file to be accessible in the container. Best regards, Vincent

1 0

[openstacksdk][cinder] openstacksdk-functional-devstack job is failing
by Rajat Dhasmana 26 Dec '22

26 Dec '22

Hi, Currently the *openstacksdk-functional-devstack* job is failing on the cinder gate[1] due to which gate is blocked. This seems to be due to the new tox 4 enforcement and could be an issue for other projects as well. We've a patch in OpenstackSDK that fixes this issue[2] but that is also failing on the* osc-tox-py38-tips* job. The patch has no update since 21st December which is concerning since it's blocking other projects (at least cinder). I've proposed a patch[3] to make the *openstacksdk-functional-devstack *job non-voting until the situation is resolved on the OpenStackSDK side. [1] https://zuul.opendev.org/t/openstack/build/29c0eb15abd144a8a4f167e70fe61530 [2] https://review.opendev.org/c/openstack/openstacksdk/+/867827 [3] https://review.opendev.org/c/openstack/cinder/+/868596 Thanks Rajat Dhasmana

2 2

[openstack][cinder] Cinder volume id wont change when retype volume
by Nguyễn Hữu Khôi 26 Dec '22

26 Dec '22

Hello guys, I have a problem with cinder: I use cinder volume retype to move my volume to another backend. It worked but when I check on new backend, volume id doest match with id which on Horizon dash board.It expects to have same volume id on Horizon and new backend. I use DellSC and DellPowerstore. Thank you. Nguyen Huu Khoi

2 2

Help overriding Nova policies (Ussuri)
by Jérôme BECOT 26 Dec '22

26 Dec '22

Hello, I should miss something, and I need some help. I've updated some rules that I placed in the /etc/nova/policy.yaml file. I use all defaults for the olso policies (ie scope is not enabled). When I generate the fulle policy with oslopolicy-policy-generator, the rules are applied in the generated output. If I test the policy with oslopolicy-checker under the user's token, the results matches the permissions, but Nova API continue to refuse the operation Policy check for os_compute_api:os-flavor-manage:create failed with credentials I'm using the openstack cli to make the request (openstack flavor create) Thanks for your help J

2 2

[tacker] Cancelling next irc meetings
by Yasufumi Ogawa 25 Dec '22

25 Dec '22

Hi team, Due to the holidays, we are cancelling next two meetings on 27th Dec and 3rd Jun. Cheers, Yasufumi

1 0

[kolla-ansible] telemetry data
by Satish Patel 25 Dec '22

25 Dec '22

Folks, I am deployed kolla-ansible and trying to set up telemetry for billing etc. But not able to get it working may be broken or something. In global.yml I turned on the following options and deployed kolla and it did install all the components but I am not seeing it deploy a ceilometer agent on compute nodes. How does it collect data from compute nodes if the ceilometer agent is not running on compute nodes? Did I miss something here? enable_ceilometer: "yes" enable_gnocchi: "yes" enable_cloudkitty: "yes"

2 2

The CFP for the OpenInfra Summit 2023 is open!
by Erin Disney 25 Dec '22

25 Dec '22

Hi Everyone! The CFP for the 2023 OpenInfra Summit (June 13-15, 2023) is NOW LIVE [1]! Check out the full list of tracks and submit a talk on your topic of expertise [2]. The CFP closes January 10, 2023, at 11:59 p.m. PT We are also now accepting submissions for Forum sessions [3]! What should you submit to the forum vs. the traditional CFP? For the Forum, submit discussion-oriented sessions, including challenges around different software components, working group progress or best practices to tackle common issues. Looking for other resources? Registration [4], sponsorships [5], travel support [6] and visa requests [7] are also all open! Find all the information on the OpenInfra Summit 2023 in one place [8]! Cheers, Erin [1] https://cfp.openinfra.dev/app/vancouver-2023/19/presentations <https://cfp.openinfra.dev/app/vancouver-2023/19/presentationshttps://cfp.op…> [2] https://openinfra.dev/summit/vancouver-2023/summit-tracks/ <https://openinfra.dev/summit/vancouver-2023/summit-tracks/> [3] https://cfp.openinfra.dev/app/vancouver-2023/20/ <https://cfp.openinfra.dev/app/vancouver-2023/20/> [4] http://openinfra.dev/summit/registration <http://openinfra.dev/summit/registration> [5] http://openinfra.dev/summit/vancouver-2023/summit-sponsor/ <http://openinfra.dev/summit/vancouver-2023/summit-sponsor/> [6] https://openinfrafoundation.formstack.com/forms/openinfra_tsp <https://openinfrafoundation.formstack.com/forms/openinfra_tsp> [7] https://openinfrafoundation.formstack.com/forms/visa_yvrsummit2023 <https://openinfrafoundation.formstack.com/forms/visa_yvrsummit2023> [8] https://openinfra.dev/summit/vancouver-2023 <https://openinfra.dev/summit/vancouver-2023>

2 1

the zun service run is OK , but at dashboard web no view zun serivce!
by 无数的星球 24 Dec '22

24 Dec '22

After I successfully installed the zun service, I logged in to the dashboard, but I could not see the zun entry I've rebooted all the machines，but invalid. thanks you !

2 1