February 2022 - openstack-discuss

[cinder] Bug deputy report for week of 02-23-2022
by Sofia Enriquez 23 Feb '22

23 Feb '22

This is a bug report from 02-03-2022 to 02-23-2022. Agenda: https://etherpad.opendev.org/p/cinder-bug-squad-meeting The bug report is back! Today's meeting is on video. ----------------------------------------------------------------------------------------- Medium - https://bugs.launchpad.net/cinder/+bug/1960206 "[RBD] Amend the formula of retrieving total_capacity for a rbd pool." Fix proposed to master. Assigned to Gorka. - https://bugs.launchpad.net/os-brick/+bug/1961102 "NVMe-oF doesn't disconnect." Unassigned. Problem with request ID: - https://bugs.launchpad.net/cinder/+bug/1960329 "Wrong request ID on API middleware filters." Fix proposed to master. Assigned to Gorka. - https://bugs.launchpad.net/cinder/+bug/1960021 "Request to list REST API versions doesn't return the request id." Fix proposed to master. Assigned to Gorka. - https://bugs.launchpad.net/cinder/+bug/1960020 "Duplicated request ids in the logs." Fix proposed to master. Assigned to Gorka. - https://bugs.launchpad.net/cinder/+bug/1960019 "REST API responses have wrong request ID when using noauth." Fix proposed to master. Assigned to Gorka. Low - https://bugs.launchpad.net/cinder/+bug/1960315 "[IBM Storwize] Delete volume issue in reverse replication." Unassigned. - https://bugs.launchpad.net/cinder/+bug/1960314 "[IBM Storwize] Resize issue in reverse replication." Unassigned. - https://bugs.launchpad.net/cinder/+bug/1961548 "[Storwize] Fix multiple SVC CLI calls to create volume operation." Fix proposed to master. - https://bugs.launchpad.net/cinder/+bug/1959178 "SolidFire: Error message logged when creating volume from image." Fix proposed to master. Have a +2! Wishlist - https://bugs.launchpad.net/cinder/+bug/1960574 "Provide back-end Qos support for spdk driver." Unassigned. Incomplete - https://bugs.launchpad.net/cinder/+bug/1959350 " [RBD] Volume status becomes available after force delete." Unassigned. Invalid - https://bugs.launchpad.net/cinder/+bug/1961685 "cinder-volume.service crashes: no module named 'wrapt'." If I'm missing bugs please let me know! Cheers! -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[infra] Gerrit email outage?
by Stephen Finucane 23 Feb '22

23 Feb '22

I haven't received any emails from Gerrit since midday (UTC) yesterday. Chatting on IRC, it seems I'm not alone. Has something changed in Gerrit or is this a simple outage? Cheers, Stephen

2 2

[kolla][rabbitmq] Trouble with RMQ 3.8 in Train
by Albert Braden 23 Feb '22

23 Feb '22

We are running kolla-ansible Train and I successfully followed the recommendations in [1] to change our RMQ settings, and our RMQ works a lot better now, but one of the recommendations is to run RMQ 3.8 and so far I haven’t been able to make it work. Our working RMQ container has these settings in the Dockerfile to force 3.7 and the appropriate erlang version: 'erlang-21.3*', 'rabbitmq-server-3.7.23*' I changed those settings to allow 3.8: 'rabbitmq-server-3.*.*el8ost*' Now my RMQ container fails to start with: {"log":"20:29:28.440 [error] Supervisor rabbit_prelaunch_sup had child prelaunch started with rabbit_prelaunch:run_prelaunch_first_phase() at undefined exit with reason no_configuration_schema_found in context start_error\r\n","stream":"stdout","time":"2022-02-22T20:29:28.441064637Z"} What am I missing? [1] https://wiki.openstack.org/wiki/Large_Scale_Configuration_Rabbit

2 1

[qa] Zed PTG planning
by Martin Kopec 23 Feb '22

23 Feb '22

Hi everyone, virtual PTG for the Zed cycle will be held between 4th - 8th April [1]. I've created a Doodle [2] to collect the best time slots for our team. Please fill it in rather sooner than later (the deadline for team signup is March 11th). I've also created an etherpad [3] to collect ideas/topics for the PTG sessions. If you have anything to discuss, please don't hesitate to write it there. [1] http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027051… [2] https://doodle.com/poll/q5yzatzh45brt5v2 [3] https://etherpad.opendev.org/p/qa-zed-ptg Thanks, -- Martin Kopec Senior Software Quality Engineer Red Hat EMEA IM: kopecmartin

1 0

[neutron] Stale Tap Ports
by Ammad Syed 23 Feb '22

23 Feb '22

Hi, I am using neutron 19.1 and ovn 21.09. There is a strange behavior I am seeing on compute nodes that ovs logs shows below logs 2022-02-23T06:14:06.426Z|00226|bridge|WARN|could not open network device tapea0dca9a-32 (No such device) 2022-02-23T06:14:06.428Z|00227|bridge|WARN|could not open network device tapaeaa1749-b5 (No such device) 2022-02-23T06:14:06.429Z|00228|bridge|WARN|could not open network device tap53f06ea0-70 (No such device) 2022-02-23T06:14:06.431Z|00229|bridge|WARN|could not open network device tap724cf0cc-7b (No such device) 2022-02-23T06:14:06.433Z|00230|bridge|WARN|could not open network device tap18864125-6d (No such device) In the ovs-vsctl show I also see alot of ports with these status. Port tapaeaa1749-b5 Interface tapaeaa1749-b5 error: "could not open network device tapaeaa1749-b5 (No such device)" Port tap18864125-6d Interface tap18864125-6d error: "could not open network device tap18864125-6d (No such device)" Port tap54415fc0-dd Interface tap54415fc0-dd error: "could not open network device tap54415fc0-dd (No such device)" I have checked in neutron port list and there is no port found for these tap interfaces. openstack port list --long | grep -i 18864125-6d I have checked in database in ports table, there is also no ports there with these ids. mysql> select * from ports where id like '%18864125-6d%'; These ports even show on compute node even if there is no instance running on it. Are these stale ports ? If yes, is there any tool or utility to run audit and delete these ports ? Ammad

1 0

Re: [Tempest] OpenSack Powered * vs OpeStack Compatible
by yahoo yahho 23 Feb '22

23 Feb '22

Sent from Yahoo Mail on Android

1 0

[all] [requirements] Artificially inflated requirements in os-brick
by Thomas Goirand 22 Feb '22

22 Feb '22

Hi, I've been quite surprised to see $topic. Most requirements for os-brick are now to be found in Yoga only. I've asked why, and I've been told that the reason is because "we've been testing with the new versions only for the number of months". Well, the reason why someone would increase a minimum requirement *must* be only because of the need of a new feature, and should be treated with care. Otherwise, this makes upgrading very dangerous and annoying to do. As much as possible, I would strongly recommend that any dependency in Stable can be found in Stable-1 (only when a new feature is mandatory, then it's ok-ish to require that new version, though I would advocate for a possible fallback if that's not too complex to write). Now, if that's the path we're taking, all is going backward 5 years ago, and then my thinking is: we must reintroduce lower-constraints testing ASAP. Your thoughts everyone? Cheers, Thomas Goirand (zigo) P.S: Please note that this is absolutely not about os-brick, I'm not pointing my finger to any team, it's a global thinking for the whole OpenStack project.

5 10

LetsEncrypt OS Ansible Ussuri
by Marc-Antoine Godde 22 Feb '22

22 Feb '22

Hello, I have a question on how to setup LetsEncrypt with OpenStack Ansible. We are still on OpenStack Ussuri. We added the following variables to user_variables.yml. ================================================================================== haproxy_ssl_letsencrypt_enable: True haproxy_ssl_letsencrypt_install_method: "distro" haproxy_ssl_letsencrypt_setup_extra_params: "--http-01-address {{ ansible_host }} --http-01-port 8888" haproxy_ssl_letsencrypt_email: email(a)example.com haproxy_interval: 2000 user avatar user avatar haproxy_extra_services: # an internal only service for acme-challenge whose backend is certbot on the haproxy host - service: haproxy_service_name: letsencrypt haproxy_backend_nodes: - name: localhost ip_addr: {{ ansible_host }} #certbot binds to the internal IP backend_rise: 1 #quick rise and fall time for multinode deployment to succeed backend_fall: 2 haproxy_bind: - 127.0.0.1 #bind to 127.0.0.1 as the local internal address will be used by certbot haproxy_port: 8888 #certbot is configured with http-01-port to be 8888 haproxy_balance_type: http ================================================================================== Yet, Horizon config for HAproxy is already defined in the default vars (https://github.com/openstack/openstack-ansible/blob/stable/ussuri/inventory… <https://github.com/openstack/openstack-ansible/blob/stable/ussuri/inventory…>) and we don’t know where ta add the required ACL to redirect the traffic from 80 port to 8888: ==================================== haproxy_frontend_acls: #use a frontend ACL specify the backend to use for acme-challenge letsencrypt-acl: rule: "path_beg /.well-known/acme-challenge/" backend_name: letsencrypt ==================================== We know that this is fixed in OpenStack Ansible Victoria. Is it possible with Ussuri tho ? Many thanks, Best, Marc-Antoine Godde

2 5

[cinder] reminder: this week's meeting in video+IRC
by Brian Rosmaita 22 Feb '22

22 Feb '22

Quick reminder that this week's Cinder team meeting on Wednesday 23 February, being the final meeting of the month, will be held in both videoconference and IRC at the regularly scheduled time of 1400 UTC. These are the video meeting rules we've agreed to: * Everyone will keep IRC open during the meeting. * We'll take notes in IRC to leave a record similar to what we have for our regular IRC meetings. * Some people are more comfortable communicating in written English. So at any point, any attendee may request that the discussion of the current topic be conducted entirely in IRC. * The meeting will be recorded. connection info: https://bluejeans.com/3228528973 meeting agenda: https://etherpad.opendev.org/p/cinder-yoga-meetings cheers, brian

1 0

[nova] OOM Killed Processes
by Ammad Syed 22 Feb '22

22 Feb '22

Hi,, I am having trouble with my compute node that nova-compute and ovs process are being killed by OOM. I have alot memory available in system. # free -g total used free shared buff/cache available Mem: 1006 121 881 0 2 879 Swap: 7 0 7 But I am seeing process are being killed in dmesg. [Sat Feb 19 03:46:26 2022] Memory cgroup out of memory: Killed process 2080898 (ovs-vswitchd) total-vm:9474284kB, anon-rss:1076384kB, file-rss:11700kB, shmem-rss:0kB, UID:0 pgtables:2776kB oom_score_adj:0 [Sat Feb 19 03:47:01 2022] Memory cgroup out of memory: Killed process 2081218 (ovs-vswitchd) total-vm:9475332kB, anon-rss:1096988kB, file-rss:11700kB, shmem-rss:0kB, UID:0 pgtables:2780kB oom_score_adj:0 [Sat Feb 19 03:47:06 2022] Memory cgroup out of memory: Killed process 2081616 (ovs-vswitchd) total-vm:9473252kB, anon-rss:1073052kB, file-rss:11700kB, shmem-rss:0kB, UID:0 pgtables:2784kB oom_score_adj:0 [Sat Feb 19 03:47:16 2022] Memory cgroup out of memory: Killed process 2081940 (ovs-vswitchd) total-vm:9471236kB, anon-rss:1070920kB, file-rss:11700kB, shmem-rss:0kB, UID:0 pgtables:2776kB oom_score_adj:0 [Sat Feb 19 03:47:16 2022] Memory cgroup out of memory: Killed process 6098 (nova-compute) total-vm:3428356kB, anon-rss:279920kB, file-rss:9868kB, shmem-rss:0kB, UID:64060 pgtables:1020kB oom_score_adj:0 [Mon Feb 21 11:15:08 2022] Memory cgroup out of memory: Killed process 2082296 (ovs-vswitchd) total-vm:9475372kB, anon-rss:1162636kB, file-rss:11700kB, shmem-rss:0kB, UID:0 pgtables:2864kB oom_score_adj:0 Any advice on how to fix this ? Also any best practices document on configuring memory optimizations in nova compute node. Ammad

2 3

[openstacksdk][magnum] Adding get_coe_cluster_by_id() method to openstack.cloud._coe
by Florian Haas 22 Feb '22

22 Feb '22

Hi everyone, here's something I recently ran into while playing with openstacksdk against Magnum; I could use some help/feedback there if anyone's inclined. :) It looks as though openstacksdk, which presently doesn't include get_coe_cluster_by_id() in cloud._coe[1], will effectively only ever list clusters and then filter client-side, as opposed to looking up individual clusters directly. That means that it only ever hits the Magnum API's /v1/clusters URL[2], which provides only limited properties on each cluster it returns. Were it to implement get_coe_cluster_by_id(), then any Connection object that has use_direct_get set to True could hit /v1/clusters/<uuid> instead[3], which provides a much richer set of properties. I have submitted a patch for this[4], but I have a couple of open questions: (1) I'm not quite sure what's the proper way to test that get_coe_cluster() actually invokes get_coe_cluster_by_id() when passed a UUID. Normally I'd do that with unittest.mock's assert_called() method, but openstacksdk heavily wraps various mock.patch() calls so I'm sure there's a better, preferred way to do it for the openstacksdk codebase. (2) Even if implementing get_coe_cluster_by_id() is the correct approach (which I am entirely not sure of), it still strikes me as a bit unintuitive that there are additional details that could be retrieved only if use_direct_get is set on the Connection object. I think having an explicit "get me more detail" toggle on the call would be preferable, which would hit /v1/clusters first (to retrieve the cluster UUID from the name), and then hit /v1/clusters/<uuid> to retrieve the rich set of properties. Now, the docstring for search_coe_clusters() mentions a "detail" parameter[5], but that appears to be a bit of a red herring as that method doesn't actually use that param. So I'm a bit stuck and unsure what to do there. :) If anyone has thoughts on this they'd like to share, I'd be most grateful! Cheers, Florian References: [1] https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/clou… [2] https://docs.openstack.org/api-ref/container-infrastructure-management/?exp… [3] https://docs.openstack.org/api-ref/container-infrastructure-management/?exp… [4] https://review.opendev.org/c/openstack/openstacksdk/+/828791 [5] https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/clou…

1 1

[CloudKitty] Virtual PTG April 2022
by Rafael Weingärtner 22 Feb '22

22 Feb '22

Hello everyone, As you probably heard our next PTG will be held virtually in April. We've marked April 4, at 13:00-16:00 UTC [1]. We already have a CloudKitty meeting organized for this day. Therefore, it seems the most practical choice. However, if you guys would like some other dates and/or time, just let me know. The room we selected is called "Cactus". I've also created an etherpad [2] to collect ideas/topics for the PTG sessions. If you have anything to discuss, please don't hesitate to write it there. [1] https://ethercalc.openstack.org/7yxdas7suqnd [2] https://etherpad.opendev.org/p/cloudkitty-ptg-zed -- Rafael Weingärtner

1 0

New CI log workflow improvement
by Daniel Pawlik 22 Feb '22

22 Feb '22

Hi, tl;dr Besides sending CI logs, does someone use Logstash on Opendev to push information to the new Elasticsearch (Opensearch) service? Long version: I am currently looking into a new log workflow for OpenDev's OpenSearch that is using the logscraper tool [1], as an improvement over the current workflow [2]. I spotted a few bottlenecks that are complicating pushing logs to the new Elasticsearch service. The most important bottlenecks are: - - gearman worker - this service needs at least 10 instances to keep up with the current influx of build logs. - As a reminder, this service consumes information prepared by gearman client, downloads the logs, adds information and - pushes data to the logstash service, - - logstash service - this service is "freezing" from time to time and it requires many resources such as CPU and RAM. - As a reminder, the service receives data and by using grok filter [3] prepares information that later will be pushed - to Elasticsearch. - The new workflow would eliminate these bottlenecks by pushing data directly into Elasticsearch. This would therefore drop the following services: - gearman client - gearman worker - logstash service Before proceeding further, we need to know if these services, especially logstash, are (apart from sending logs from CI) not used anywhere, so that we don't accidentally pull the plug on any other workflows. If you are aware of these, please let me know. Dan [1] https://opendev.org/openstack/ci-log-processing [2] https://docs.opendev.org/opendev/system-config/latest/logstash.html [3] https://opendev.org/openstack/logstash-filters/src/branch/master/filters/op…

2 1

Call for Outreachy OpenStack mentors - May 2022 round
by Mahati Chamarthy 22 Feb '22

22 Feb '22

Hello! *TL;DR OpenStack is participating in Outreachy for the upcoming round! **Please submit projects.* Outreachy's goal is to support people from groups underrepresented in the technology industry. Interns will work remotely with mentors from our community. We are seeking mentors to propose projects that Outreachy interns can work on during their internship. If you or your colleagues are interested in mentoring in the next round, please submit your project proposal over here by *March 4th, 2022:* https://www.outreachy.org/communities/cfp/openstack/ Mentors should read the mentor FAQ https://www.outreachy.org/mentor/mentor-faq and find more details about the Outreachy program and timeline in https://www.outreachy.org/communities/cfp/ . If you need help crafting your project proposal or have any other queries, please contact me Mahati Chamarthy <mahati.chamarthy(a)gmail.com> or Dmitry Tantsur <divius.inside(a)gmail.com> Best, Mahati

3 3

[kolla][neutron] VM is getting the wrong IP, can't contact metadata service
by Chris DiLorenzo 21 Feb '22

21 Feb '22

Hello, I am having a weird issue. I am running openstack Victoria with OVN. Generally, everything is working. I created a VM using a fixed port with an IP of 10.210.64.6. There are no issues with this VM. When I create a new VM, I get this route info: [ 29.363664] cloud-init[564]: ci-info: ++++++++++++++++++++++++++++++++Route IPv4 info++++++++++++++++++++++++++++++++ [ 29.368568] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+ [ 29.369945] cloud-init[564]: ci-info: | Route | Destination | Gateway | Genmask | Interface | Flags | [ 29.372441] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+ [ 29.376434] cloud-init[564]: ci-info: | 0 | 0.0.0.0 | 10.210.64.1 | 0.0.0.0 | ens3 | UG | [ 29.377820] cloud-init[564]: ci-info: | 1 | 10.210.64.0 | 0.0.0.0 | 255.255.255.192 | ens3 | U | [ 29.380417] cloud-init[564]: ci-info: | 2 | 169.254.169.254 | 10.210.64.6 | 255.255.255.255 | ens3 | UGH | [ 29.384448] cloud-init[564]: ci-info: +-------+-----------------+-------------+-----------------+-----------+-------+ The IP is set correctly, but you can see I am getting a gateway of 10.210.64.6 for the metadata ser. ice. This is preventing the VM from reaching the service so the SSH key is not being added preventing me from logging in. As far as I can tell, there are no error and every VM I build is getting this .6 IP for this route. Thanks, Chris

3 4

[kolla-ansible][manila]Permission denied when using cephfs nfs
by Boxiang Zhu 21 Feb '22

21 Feb '22

Hi all, I have met an issue when I use manila with cephfs nfs backend. Environment Information: - OS: ubuntu 18.04 - OpenStack Version: Train - Kolla-ansible Version: 9.3.2 I used kolla-ansible to deploy my OpenStack environment AIO. Here is the info of my globals.yml as followed: --- kolla_base_distro: "ubuntu" kolla_install_type: "source" openstack_release: "train" kolla_internal_vip_address: "172.16.150.67" network_interface: "ens3" neutron_external_interface: "ens4" enable_haproxy: "no" enable_ceph: "yes" enable_ceph_mds: "yes" enable_ceph_rgw: "yes" enable_ceph_nfs: "yes" enable_cinder: "yes" enable_manila: "yes" enable_manila_backend_cephfs_nfs: "yes" enable_ceph_rgw_keystone: "yes" ceph_osd_store_type: "filestore" I try to use cephfs nfs backend for manila. All containers are running and the services of manila is good. +----+------------------+-------------------+------+---------+-------+----------------------------+ | Id | Binary | Host | Zone | Status | State | Updated_at | +----+------------------+-------------------+------+---------+-------+----------------------------+ | 1 | manila-data | manila | nova | enabled | up | 2022-02-16T03:25:34.000000 | | 2 | manila-scheduler | manila | nova | enabled | up | 2022-02-16T03:25:35.000000 | | 3 | manila-share | manila@cephfsnfs1 | nova | enabled | up | 2022-02-16T03:25:40.000000 | +----+------------------+-------------------+------+---------+-------+----------------------------+ Here is my share type: +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+ | ID | Name | visibility | is_default | required_extra_specs | optional_extra_specs | Description | +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+ | 265a6637-0322-4c4a-9185-f30f01b96d12 | default_share_type | public | - | driver_handles_share_servers : False | | None | +--------------------------------------+--------------------+------------+------------+-------------------------------------+----------------------+-------------+ When I create a share, error information appears in manila-share.log file: 2022-02-16 09:30:04.615 20 INFO manila.share.manager [req-b57991af-5355-4202-8543-dc7ff914d919 - - - - -] Updating share status 2022-02-16 09:30:04.616 20 DEBUG manila.share.driver [req-b57991af-5355-4202-8543-dc7ff914d919 - - - - -] Updating share stats. _update_share_stats /var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/driver.py:1232 2022-02-16 09:30:54.675 20 DEBUG manila.share.drivers.cephfs.driver [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff - - -] create_share CEPHFSNFS1 name=449a52a4-f19c-4d1b-b437-7dc2443e040c size=1 share_group_id=None create_share /var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py:262 2022-02-16 09:30:54.682 20 INFO ceph_volume_client [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff - - -] create_volume: /volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c2022-02-16 09:30:54.927 20 ERROR manila.share.manager [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff - - -] Share instance 449a52a4-f19c-4d1b-b437-7dc2443e040c failed on creation.: cephfs.OSError: error in mkdir volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: Permission denied [Errno 13] 2022-02-16 09:30:54.929 20 WARNING manila.share.manager [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff - - -] Share instance information in exception can not be written to db because it contains {} and it is not a dictionary.: cephfs.OSError: error in mkdir volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: Permission denied [Errno 13] 2022-02-16 09:30:54.955 20 INFO manila.message.api [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff - - -] Creating message record for request_id = req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server [req-d2c8ef2a-d3ce-483b-9e39-e5a3d857d72b eaf0e30a92694889aa46ac5a1d4b7a47 37025e7c9ae447c8975e9ef3a4e5d0ff - - -] Exception during message handling: cephfs.OSError: error in mkdir volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: Permission denied [Errno 13] 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 656, in _mkdir_p 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server self.fs.stat(subpath) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "cephfs.pyx", line 1257, in cephfs.LibCephFS.stat 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server cephfs.ObjectNotFound: error in stat: volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: No such file or directory [Errno 2] 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server During handling of the above exception, another exception occurred: 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server Traceback (most recent call last): 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/server.py", line 165, in _process_incoming 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server res = self.dispatcher.dispatch(message) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 274, in dispatch 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server return self._do_dispatch(endpoint, method, ctxt, args) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_messaging/rpc/dispatcher.py", line 194, in _do_dispatch 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server result = func(ctxt, **new_args) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/manager.py", line 187, in wrapped 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server return f(self, *args, **kwargs) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/utils.py", line 568, in wrapper 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server return func(self, *args, **kwargs) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/manager.py", line 1790, in create_share_instance 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server exception=e) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__ 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server self.force_reraise() 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server six.reraise(self.type_, self.value, self.tb) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/six.py", line 693, in reraise 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server raise value 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/manager.py", line 1753, in create_share_instance 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server context, share_instance, share_server=share_server) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/var/lib/kolla/venv/lib/python3.6/site-packages/manila/share/drivers/cephfs/driver.py", line 272, in create_share 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server mode=self._cephfs_volume_mode) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 677, in create_volume 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server self._mkdir_p(path, mode) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "/usr/lib/python3/dist-packages/ceph_volume_client.py", line 658, in _mkdir_p 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server self.fs.mkdir(subpath, mode) 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server File "cephfs.pyx", line 887, in cephfs.LibCephFS.mkdir 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server cephfs.OSError: error in mkdir volumes/_nogroup/449a52a4-f19c-4d1b-b437-7dc2443e040c: Permission denied [Errno 13] 2022-02-16 09:30:54.971 20 ERROR oslo_messaging.rpc.server

3 4

[neutron] Bug deputy - week of 14.02.2022
by Slawek Kaplonski 21 Feb '22

21 Feb '22

Hi, I was bug deputy last week. It was very quiet week in terms of bugs in Neutron. Here is the summary: ## Critical https://bugs.launchpad.net/neutron/+bug/1961112 - [ovn] overlapping security group rules break neutron-ovn-db-sync-util - fix already proposed long time ago: https://review.opendev.org/c/openstack/neutron/+/801707, ## High https://bugs.launchpad.net/neutron/+bug/1961173 - [fullstack] test_vm_is_accessible_by_local_ip fails sometimes - assigned to Oleg, fix proposed alreadh https://review.opendev.org/c/openstack/neutron/+/829659 ## Medium https://bugs.launchpad.net/neutron/+bug/1961013 - [stable][ovn] frequent OVN DB leader changes increase rate of Neutron API errors - assigned to Frode, it's about backport feature from master to stable branches, ## Low https://bugs.launchpad.net/neutron/+bug/1961184 - [OVN] Virtual ports are always DOWN - assigned to Rodolfo, ## Whishlist (RFEs) https://bugs.launchpad.net/neutron/+bug/1960850 - [RFE][OVN] Support DNS subdomains at a network level - I think it's properly described and needs to be discussed during the drivers meeting now, https://bugs.launchpad.net/neutron/+bug/1961011 - [RFE][L3][OVS] use controller (ovs-agent) send (packet-out) RA to (VMs) ports - I think it's properly described and needs to be discussed during the drivers meeting now, -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[all][tc] What's happening in Technical Committee: summary 18th Feb, 21: Reading: 5 min
by Ghanshyam Mann 19 Feb '22

19 Feb '22

Hello Everyone, Here is this week's summary of the Technical Committee activities. 1. TC Meetings: ============ * We had this week's meeting on 17th. Most of the meeting discussions are summarized below (Completed or in-progress activities section). Meeting full logs are available @https://meetings.opendev.org/meetings/tc/2022/tc.2022-02-17-15.00.log.html * Next week's meeting is on 24th Feb Thursday 15:00 UTC, feel free to add the topic on the agenda[1] by 23rd Feb. 2. What we completed this week: ========================= * Z cycle release name [2] 3. Activities In progress: ================== TC Tracker for Yoga cycle ------------------------------ * This etherpad includes the Yoga cycle targets/working items for TC[3]. 2 out of 9 items are completed. Open Reviews ----------------- * Six open reviews for ongoing activities[4]. Release identification and release name ----------------------------------------------- As Zed is the last release for the current alphabet release name iteration. TC discussed the future release name ideas. Belmiro has proposed the idea of adding "<year>.<release count>" number to each release. The proposal is up for review[5] Switch Keystone back to PTL model ------------------------------------------- Keystone team proposed to move Keystone from DPL to PTL model[6] with Douglas Mendizábal as PTL. Dropping tags framework ------------------------------ yoctozepto has updated part1 of dropping the tag framework[7] which is ready for review. Elod us updating the release scripts to remove the tag ref. Z release cycle Technical Election --------------------------------------- The nomination for PTL and TC elections is closed and there is no voting required. TC Campaigning period is open until 22nd Feb[8]. We will be closing the election on 22nd Feb. There are total 17 projects that missed the nomination, I have created the etherpad to work on the leaderless projects[9]. There is already late candidacy for 8 projects and 9 projects still need leaders. Please add your name if you would like to lead those projects. TC position on release cadence ------------------------------------- No other updates from last week, Dan's proposal is under review[10]. Fixing Zuul config error ---------------------------- Requesting projects with zuul config error to look into those and fix them which should not take much time[11]. Project updates ------------------- * Retire js-openstack-lib (waiting on Adjutant to have new PTL/maintainer) [12] 4. How to contact the TC: ==================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send the email with tag [tc] on openstack-discuss ML[13]. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 15 UTC [14] 3. Ping us using 'tc-members' nickname on #openstack-tc IRC channel. [1] https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting [2] http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027242… [3] https://etherpad.opendev.org/p/tc-yoga-tracker [4] https://review.opendev.org/q/projects:openstack/governance+status:open [5] https://review.opendev.org/c/openstack/governance/+/829563 [6] https://review.opendev.org/c/openstack/governance/+/829037 [7] https://review.opendev.org/c/openstack/governance/+/822900 [8] http://lists.openstack.org/pipermail/openstack-discuss/2022-February/027311… [9] https://etherpad.opendev.org/p/zed-leaderless [10] https://review.opendev.org/c/openstack/governance/+/828777 [11] https://etherpad.opendev.org/p/zuul-config-error-openstack [12] https://review.opendev.org/c/openstack/governance/+/798540 [13] http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-discuss [14] http://eavesdrop.openstack.org/#Technical_Committee_Meeting -gmann

1 0

[release] Release countdown for week R-5, Feb 21-25
by Előd Illés 18 Feb '22

18 Feb '22

Development Focus ----------------- We are getting close to the end of the Yogacycle! Next week on February 24th, 2022is the Yoga-3 milestone, also known as feature freeze. It's time to wrap up feature work in the services and their client libraries, and defer features that won't make it to the Zedcycle. General Information ------------------- This coming week is the deadline for client libraries: their last feature release needs to happen before "Client library freeze" on February 24th, 2022. Only bugfix releases will be allowed beyond this point. When requesting those library releases, you can also include the stable/yogabranching request with the review. As an example, see the "branches" section here: https://opendev.org/openstack/releases/src/branch/master/deliverables/pike/… February 24th, 2022is also the deadline for feature work in all OpenStack deliverables following the cycle-with-rc model. To help those projects produce a first release candidate in time, only bugfixes should be allowed in the master branch beyond this point. Any feature work past that deadline has to be raised as a Feature Freeze Exception (FFE) and approved by the team PTL. Finally, feature freeze is also the deadline for submitting a first version of your cycle-highlights. Cycle highlights are the raw data that helps shape what is communicated in press releases and other release activity at the end of the cycle, avoiding direct contacts from marketing folks. See https://docs.openstack.org/project-team-guide/release-management.html#cycle… for more details. Upcoming Deadlines & Dates -------------------------- Yoga-3 milestone (feature freeze): February 24th, 2022(R-5 week) RC1 deadline: March 10th, 2022(R-3 week) Final RC deadline: March 24th, 2022(R-1 week) Final Yogarelease: March 30th, 2022 Next PTG: April 4 - 8, 2022 (virtual) ElődIllés irc: elodilles

1 0

Masquerading VM works 99%
by Derek O keeffe 18 Feb '22

18 Feb '22

Hi all, We have an openstack cluster with one controller and 4 computes (Victoria) we have set it up using vlan provider networks with linuxbridge agent, distributed routing & ml2 (I am only partly on the networking so there could be more to that which I can find out if needed) So I was tasked with creating two Instances, one (lets call it the external vm) with an external interface 10.55.9.67 and internal interface 192.168.1.2. A second instance (lets call it the internal vm) would then be placed on the 192.168.1.0 network. I configured masquerading on the "external vm" and tried to ping the outside world from the "internal" vm as per something like this https://kifarunix.com/configure-ubuntu-20-04-as-linux-router/?unapproved=49… Both VM's were instantiated on the same compute host (I've tried it with them on separate hosts as well). I can see the ping leave using tcpdumps along the way and it makes it all the way back to the internal interface on the external machine. It just fails on the last hop to the internal machine. I've tried everything in my power to find why this won't work so I would be grateful for any advice at all. I have added the below to show how I followed the ping manually and where it went and when it failed. Thank you in advance. Following the ping from source to destination and back:Generated on the private VMsent to the internal interface on the external vmsent to the external interface on the external vmsent to the tap interface on the computesent to the physical nic on the computesent to the nic on the network device out to the internet received on nic on the network devicefrom the internet received on physical nic on the computereceived on tap interface on compute received on external interface on the external vmreceived on the internal interface on the external vmNEVER gets to last step on the internal vm Regards,Derek

4 6

[cinder] festival of XS reviews 18 February 2022
by Brian Rosmaita 17 Feb '22

17 Feb '22

Hello Argonauts, Apologies for the lateness of this reminder, but hopefully everyone has already used the handy ICS file [0] to have this event on their personal calendars. The most recent edition of the Cinder Festival of XS Reviews will be held tomorrow (Friday 18 February). who: Everyone! what: The Cinder Festival of XS Reviews when: Friday 18 February 2022 from 1400-1600 UTC where: https://meetpad.opendev.org/cinder-festival-of-reviews etherpad: https://etherpad.opendev.org/p/cinder-festival-of-reviews See you there! brian [0] http://eavesdrop.openstack.org/calendars/cinder-festival-of-reviews.ics

1 0

[neutron] Drivers meeting - Friday 18.2.2022 - cancelled
by Lajos Katona 17 Feb '22

17 Feb '22

Hi Neutron Drivers! As Friday is recharge day for RedHat, and most probably we will not have a quorum for the voting, let's cancel tomorrow's meeting. See You at the meeting next week. Lajos Katona (lajoskatona)

1 0

[TripleO] Gate blocker - ansible version
by Rafael Castillo 17 Feb '22

17 Feb '22

Hi All, We have an issue affecting multiple centos stream 8 gate jobs related to an ansible version conflict. Details: https://bugs.launchpad.net/tripleo/+bug/1961035 We are currently deciding on the best course of action for unblocking this issue. We will post an update once the gate is clear. Until then, please hold on rechecking. Thank you!

1 1

[TripleO] Sergii Golovatiuk for TripleO core reviewer (upgrades-related)
by Jose Luis Franco Arza 17 Feb '22

17 Feb '22

Hello folks, I would like to propose Sergii Golovatiuk [1][2][3] as a core reviewer on the TripleO repositories related to the upgrades workflow (openstack/tripleo-heat-templates, openstack/tripleo-ansible, openstack/python-tripleoclient, openstack/tripleo-ci, openstack/tripleo-quickstart, openstack/tripleo-quickstart-extras). Sergii has been working in OpenStack for the last 10 years, he is already core-reviewer of the tripleo-upgrade <https://github.com/openstack/tripleo-upgrade>repository in which he is providing with very valuable feedback, as well as helping getting relevant patches merged. His deep knowledge of the different OpenStack projects, as well as his involvement in the upgrades/updates workflows for the last 9 OpenStack releases makes him a great code reviewer, being able to catch several issues during the code review process. I will monitor the thread for a week and if no objections are exposed I will add him to the tripleo-core group. PD: Here comes the +1 from my side. Thanks. Cheers, José Luis [1]: https://review.opendev.org/q/owner:sgolovat%2540redhat.com [2]: https://www.stackalytics.io/?project_type=all&metric=commits&user_id=sgolov… [3]: Position #25 in the most active TripleO contributors list <https://www.stackalytics.io/report/contribution?module=tripleo-group&projec…>

13 13

[Tripleo] Issue in Baremetal Provisioning from Overcloud
by Anirudh Gupta 17 Feb '22

17 Feb '22

Hi Team I am trying to Provision Bare Metal Node from my tripleo Overcloud. For this, while deploying the overcloud, I have followed the *"default flat" *network approach specified in the below link https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/15… Just to highlight the changes, I have defined the *ironic-config.yaml* parameter_defaults: ... ... IronicIPXEEnabled: true IronicInspectorSubnets: - ip_range: *172.23.3.100,172.23.3.150* IronicInspectorInterface: 'br-baremetal' Also modified the file *~/templates/network-environment.yaml* parameter_defaults: NeutronBridgeMappings: datacentre:br-ex,baremetal:br-baremetal NeutronFlatNetworks: datacentre,baremetal With this I have Followed all the steps of creating br-baremetal bridge on controller, given in the link below: https://access.redhat.com/documentation/en-us/red_hat_openstack_platform/15… - type: ovs_bridge name: br-baremetal use_dhcp: false members: - type: interface name: nic3 Post Deployment, I have also create a flat network on "datacentre" physical network and subnet having the range *172.23.3.200,172.23.3.240 *(as suggested subnet is same as of inspector and range is different) and the router Also created a baremetal node and ran *"openstack baremetal node manage bm1", *the state of which was a success. Observation: On executing "openstack baremetal node *provide* bm1", the machine gets power on and ideally it should take an IP from ironic inspector range and PXE Boot. But nothing of this sort happens and we see an IP from neutron range " *172.23.3.239*" (attached the screenshot) [image: image.png] I have checked overcloud ironic inspector podman logs alongwith the tcpdump. In tcpdump, I can only see dhcp discover request on br-baremetal and nothing happens after that. I have tried to explain my issue in detail, but I would be happy to share more details in case still required. Can someone please help in resolving my issue. Regards Anirudh Gupta

5 20