November 2023 - openstack-discuss

[ops][cinder][kolla] Cinder + Ceph under high load can lead to timeouts and a failure state.
by Forrest Fuqua 13 Nov '23

13 Nov '23

In my setup of Cinder Zed + Ceph Quincy under very high loads there is a race condition due to a low timeout by default in Openstack Kolla in Cinder.conf of rados_connect_timeout = 5 This can lead to a duplicate volume trying to be made on ceph and cinder erroring out a VM Build in Nova. Nova: nova.exception.BuildAbortException: Build of instance a529f7d5-5a81-44f6-a1fd-93caa39b2439 aborted: Volume b476094b-075e-4dc7-b63e-5bb58a0a9229 did not finish being created even after we waited 30 seconds or 4 attempts. And its status is error. Cinder-Volume: File "/var/lib/kolla/venv/lib/python3.9/site-packages/cinder/volume/drivers/rbd.py", line 230, in __init__\n self.volume = driver.rbd.Image(rados_ioctx,\n', ' File "rbd.pyx", line 2894, in rbd.Image.__init__\n', "rbd.Timeout: [errno 110] RBD operation timeout (error opening image b'volume-b476094b-075e-4dc7-b63e-5bb58a0a9229' at snapshot None)\n"] --- File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 132, in execute\n six.reraise(c, e, tb)\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/six.py", line 719, in reraise\n raise value\n', ' File "/var/lib/kolla/venv/lib/python3.9/site-packages/eventlet/tpool.py", line 86, in tworker\n rv = meth(*args, **kwargs)\n', ' File "rbd.pyx", line 698, in rbd.RBD.clone\n', 'rbd.ImageExists: [errno 17] RBD image already exists (error creating clone)\n'] --- I can confirm that the volume does exist in Ceph now but no longer being tracked in Cinder, leading to a leftover artifact that eats up resources and a failed VM build. This seems like behavior that should be accounted for in some way.

2 2

[swift] Swiftstack Zone/DC decomission
by Muralikrishna Gutha 13 Nov '23

13 Nov '23

Certainly, here's a rewritten version: We currently utilize SwiftStack Object Storage to meet our object storage requirements, and our cluster has a Primary Zone/Secondary Zone topology. Recently, we attempted to remove/disable the Secondary Zone. However, upon doing so, we observed a concerning trend: the storage utilization in the Primary Zone rapidly increased, reaching approximately 90%. This surge was particularly notable due to the use of the 'tworeplicaperregion' policy. In an attempt to address this, we brought the Secondary Zone nodes back online, expecting Swift to redistribute the data back to these nodes and alleviate the storage strain on the Primary Zone. Despite reintegrating the Secondary Zone nodes into the cluster, we continue to witness a persistent rise in storage utilization within the Primary Zone. Interestingly, two out of the three nodes in the Primary Zone seem to bear the brunt of the data and tokens, while the third node is not receiving a proportional distribution of data. At this juncture, we are seeking guidance on potential configurations or adjustments, such as specific knobs, that could expedite the rebalance process. Our objective is to efficiently distribute the data across nodes, particularly in the Primary Zone, and optimize the overall storage utilization in our SwiftStack Object Storage cluster. Any insights or recommendations on tuning parameters for a swifter rebalance would be highly appreciated. Thanks, Murali -- This message may contain confidential and/or privileged information. If you are not the addressee or authorized to receive this on behalf of the addressee you must not use, copy, disclose or take action based on this message or any information herein. If you have received this message in error, please advise the sender immediately by reply email and delete this message. Thank you.

1 0

[tc] Next meeting Tomorrow, Tuesday Nov 14, 1800 UTC
by Jay Faulkner 13 Nov '23

13 Nov '23

Hi all, The next meeting of the OpenStack Technical Committee is tomorrow, Tuesday November 14 at 1800 UTC in #openstack-tc on OFTC. The proposed agenda is: - Roll call - Follow up on tracked action items: https://meetings.opendev.org/meetings/tc/2023/tc.2023-11-07-18.00.html - Jay to sign up to rep TC at PTG OpenInfra Live recap - Gate health check - Leaderless projects (gmann) - Implementation of Unmaintained branch statuses - draft email to openstack-discuss while we figure this out: https://etherpad.opendev.org/p/early-caracal-unmaintained-transition - 2024.1 TC Tracker - https://etherpad.opendev.org/p/tc-2024.1-tracker - Open Discussion and Reviews Thanks all, Jay Faulkner OpenStack TC Chair

1 0

[swift][OpenStackSDK][keystone] Warnings about (my) swift endpoint URL by keystoneauth1 discovery
by Christian Rohmann 13 Nov '23

13 Nov '23

Hey openstack-discuss, we use Ceph RadosGW (Quincy release) to provide Swift as part of an OpenStack cloud. The endpoints are configured as such: > # openstack endpoint list > +----------------------------------+--------+--------------+----------------+---------+-----------+----------------------------------------------------------------------+ > | ID | Region | Service Name | Service > Type | Enabled | Interface | URL | > +----------------------------------+--------+--------------+----------------+---------+-----------+----------------------------------------------------------------------+ > | 1234567890 | region | swift | object-store | True | > public | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > | 2345678901 | region | swift | object-store | True | > internal | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > | 3456789012 | region | swift | object-store | True | > admin | > https://object-store.region.cloud.example.com/swift/v1/AUTH_%(tenant_id)s > | > so according to https://docs.ceph.com/en/latest/radosgw/keystone/#cross-project-tenant-acce…, if I am not mistaken. I can also use the openstack client to access containers: > # openstack container list > +---------------+ > | Name | > +---------------+ > | containera | > | containerb | > +---------------+ and there is no warnings or other apparent issues. But when using "project cleanup" via e.g. "openstack project cleanup --dry-run --project $PROJECT" I see multiple warnings like > Failed to contact the endpoint at > https://object-store.region.cloud.example.com/swift/v1/AUTH_f2bc4bd34567ddc… > for discovery. Fallback to using that endpoint as the base url. > but only mentioning the swift endpoint. Following the warning string this originates from the discovery method of keystoneauth1, see https://github.com/openstack/keystoneauth/blob/28048af9593740315df8d9027c3b…. Could anybody help me understand what is issue might be? Why does this warning not appear for every use of object storage / swift via openstackclient? Is there any way for me to "fix" this? Could this even be a bug? Thanks and with kind regards Christian

4 10

report an error
by 许伟祥 13 Nov '23

13 Nov '23

carry out iaas-pre-host.sh report an error Downloading packages: warning: /var/cache/yum/x86_64/7/iaas/packages/python-fixtures-3.0.0-2.el7.noarch.rpm: Header V4 RSA/SHA1 Signature, key ID 764429e6: NOKEY Public key for python-fixtures-3.0.0-2.el7.noarch.rpm is not installed Please Reboot or Reconnect the terminal How to fix it 从 Windows 版邮件<https://go.microsoft.com/fwlink/?LinkId=550986>发送

1 0

[nova] what nova filter scheduler are you using?
by Satish Patel 13 Nov '23

13 Nov '23

Folks, Just curious about what schedulers people using in their cloud for better distribution or management of VMs? And why? In entire my life I never touch that config also go with default but just curious what people doing and how?

1 0

[kolla-ansible] Galera DR across multiple datacenters
by Jahns, Jay 13 Nov '23

13 Nov '23

We currently run OpenStack on top of two datacenters, and we need to ensure we have datacenter redundancy for the database. With this, we would have 2-3 nodes in each DC running galera. Its been recommended by multiple people that we use an externally managed database server (or cluster) that has its own redundancies built around the two DC layout, and not have kolla manage that. Is there a way we can use kolla to deploy a galera cluster per DC, and use that to provide redundancy in the event that we lose one of the DCs? We were currently looking at proxysql but folks have recommended against it. Thanks.

4 4

[neutron] Bug Deputy Report November 06 - 12
by Lucas Alvares Gomes 13 Nov '23

13 Nov '23

Hi, This is the Neutron bug report from November 6th to 12th. Quieter week than usual. High: * https://bugs.launchpad.net/neutron/+bug/2042938 - "[OVN] Neutron Service Restart Disrupts Octavia OVN Load Balancer Floating IP" Unassigned * https://bugs.launchpad.net/neutron/+bug/2042941 - "neutron-{ovn,ovs}-tempest-with-sqlalchemy-master jobs not installing sqlalchemy/alembic from source" Unassigned * https://bugs.launchpad.net/neutron/+bug/2043141 - " neutron-lib unit tests need update for sqlalchemy 2.0" Assigned to: Brian Haley Medium: * https://bugs.launchpad.net/neutron/+bug/2042925 - "DNS Integration throws expection logs entries were warnings would be sufficient" Assigned to: Jayce Houtman * https://bugs.launchpad.net/neutron/+bug/2042947 - "[stable branches] nftables job inherits from openvswitch-iptables_hybrid master job" Assigned to: Yatin Cheers, Lucas

1 0

[neutron][ovs][kolla-ansible] Does neutron-openvswitch-agent required for sriov only compute node
by Satish Patel 13 Nov '23

13 Nov '23

Folks, I have an openstack cloud just running SRIOV workload, We don't have a requirement to run standard vms on compute nodes. Currently I have noticed neutron-sriov-agent and openvswitch-agent running on all compute nodes. Technically in my case I don't have any use of openvswitch-agent so I am planning to get rid of those agents to reduce pressure on the neutron-server. Just wanted to check with export if that is ok to not run ovs agent. I hope neutron-server won't freak out.

2 2

[release] Release countdown for week R-20, Nov 13-17
by Elõd Illés 10 Nov '23

10 Nov '23

Development Focus ----------------- The Caracal-1 milestone is next week, on November 16th, 2023! Project team plans for the 2024.1 Caracal cycle should now be solidified. General Information ------------------- Libraries need to be released at least once per milestone period. Next week, the release team will propose releases for any library which had changes but has not been otherwise released since the 2023.2 Bobcat release. PTL's or release liaisons, please watch for these and give a +1 to acknowledge them. If there is some reason to hold off on a release, let us know that as well, by posting a -1. If we do not hear anything at all by the end of the week, we will assume things are OK to proceed. NB: If one of your libraries is still releasing 0.x versions, start thinking about when it will be appropriate to do a 1.0 version. The version number does signal the state, real or perceived, of the library, so we strongly encourage going to a full major version once things are in a good and usable state. Upcoming Deadlines & Dates -------------------------- Caracal-1 milestone: November 16th, 2023 Előd Illés irc: elodilles @ #openstack-release

1 0