July 2023 - openstack-discuss - lists.openstack.org

Take the OpenStack User Survey!
by Helena Spease 24 Jul '23

24 Jul '23

Hello everyone, It’s User Survey season! The 2023 OpenStack User Survey is closing August 23, 2023. That’s just under a month to submit your anonymized feedback to the upstream contributors. https://www.openstack.org/user-survey/survey-2023/landing <https://www.openstack.org/user-survey/survey-2023/landing> We have posts from the OpenInfra Foundation social media channels that you can re-share to your networks to encourage them to participate in the User Survey as well. Mastodon: https://social.openinfra.dev/@OpenStack/110768926571772510 <https://social.openinfra.dev/@OpenStack/110768926571772510> Twitter: https://twitter.com/OpenStack/status/1683441577648181248 <https://twitter.com/OpenStack/status/1683441577648181248> LinkedIn: https://www.linkedin.com/posts/openstack_take-the-2023-openstack-user-surve… <https://www.linkedin.com/posts/openstack_take-the-2023-openstack-user-surve…> Evaluating another OpenInfra project? We have User Surveys for them as well! Kata Containers: https://openinfrafoundation.formstack.com/forms/kata_containers_user_survey <https://openinfrafoundation.formstack.com/forms/kata_containers_user_survey> OpenStack: https://www.openstack.org/user-survey/survey-2023/landing <https://www.openstack.org/user-survey/survey-2023/landing> StarlingX: https://openinfrafoundation.formstack.com/forms/starlingx_user_survey <https://openinfrafoundation.formstack.com/forms/starlingx_user_survey> Zuul: https://openinfrafoundation.formstack.com/forms/zuul_user_survey <https://openinfrafoundation.formstack.com/forms/zuul_user_survey> We really appreciate your participation in the User Survey and are really excited to share with you all the cool data we collect! Cheers, Helena helena(a)openinfra.dev <mailto:helena@openinfra.dev> Marketing & Community Coordinator The OpenInfra Foundation

1 0

[zun] container run database insert error
by Jaime Ibar 24 Jul '23

24 Jul '23

Hi all, I have installed and configured zun for container support, all went good and I can list the available services with > openstack appcontainer service list however when I try to launch a container with > openstack appcontainer run --name container --net network=$NET_ID cirros ping 8.8.8.8 as stated in the documentation, I get the following error in zun-api oslo_db.exception.DBReferenceError: (pymysql.err.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`zun`.`container_actions`, CONSTRAINT `container_actions_ibfk_1` FOREIGN KEY (`container_uuid`) REFERENCES `container` (`uuid`))') I have followed this installation guide for yoga release(the one I'm running at the moment) https://docs.openstack.org/zun/yoga/install/index.html. Did anyone see this error before? TIA -- Jaime

1 0

Ussuri: how to delete lbaas loadbalancer left over?
by Michel Jouvin 24 Jul '23

24 Jul '23

Hi, We had a few Magnum (K8s) clusters created a couple of years ago (with Rocky and Stein versions) and forgotten. We started to delete them this spring when we where running Train Neutron service. Basically we managed to do this with the following sequence: - openstack coe cluster delete xxx and waiting for DELETE_FAILED - Use openstack coe cluster show / openstack stack resource list -n2 to identify the neutron entry causing the error and pick the corresponding resource ID - Find the ports associated with the router with openstack port list --router previously_found_id - Use the port subnet to find the port corresponding lbaas load balancer ID, use the neutron CLI to delete the load balancer (deleting one by one all the dependencies preventing the load balancer removal) - Rerun openstack coe cluster delete For some reasons, we didn't cleanup all the abandoned clusters and upgraded Neutron to Ussuri. Unfortunately, since then our previous process is no longer working as it seems that the Neutron server doesn't know anymore anything about the LBAAS load balancers (and "neutron lbaas-loadbalancer list" returns nothing). In the neutron server, any attempt to delete the subnet attached to the load balancer (or to list them with Neutron CLI) results in the following errors in Neutron server.log : ------ 2023-07-07 16:27:31.139 14962 WARNING neutron.pecan_wsgi.controllers.root [req-71e712fc-d8a7-4815-90b3-b406c10e0caa a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa 1367c9a4d5da4b229c35789c271dc7aa] No controller found for: lbaas - returning response code 404: pecan.routing.PecanNotFound 2023-07-07 16:27:31.140 14962 INFO neutron.pecan_wsgi.hooks.translation [req-71e712fc-d8a7-4815-90b3-b406c10e0caa a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa 1367c9a4d5da4b229c35789c271dc7aa] GET failed (client error): The resource could not be found. 2023-07-07 16:27:31.141 14962 INFO neutron.wsgi [req-71e712fc-d8a7-4815-90b3-b406c10e0caa a2b4a88cfee0c18702fe89ccb07ae875de3f34f3f1bb43e505fd83aebcfc094c 245bc968c1b7465dac1b93a30bf67ba9 - 1367c9a4d5da4b229c35789c271dc7aa 1367c9a4d5da4b229c35789c271dc7aa] 157.136.249.153 "GET /v2.0/lbaas/loadbalancers?name=kube_service_964f7e76-d2d5-4126-ab11-cd689f6dd9f9_runnerdeploy-wm9sm-5h52l_hello-node-x-default-x-runnerdeploy-wm9sm-5h52l HTTP/1.1" status: 404 len: 304 time: 0.0052643 ------ Any suggestion to workaround this problem and be able to successfully delete our old Magnum clusters? Thanks in advance for any help. Best regards, Michel

2 8

vPTG October 2023 Team Signup Kickoff
by Kendall Nelson 24 Jul '23

24 Jul '23

Hi Everyone, Hopefully, you have seen by now that the next OpenInfra vPTG[1] which will take place October 23-27, 2023 virtually and registration is open[2]! *To sign up your team, you must complete the survey[3] by August 20th at 7:00 UTC.* We NEED accurate contact information (email and IRC nickname ideally) for the moderator of your team’s sessions. This is because the survey information will be used to organize the schedule sign ups which will be done via the PTGBot. If you are not on IRC, please get setup[4] on the OFTC network and join #openinfra-events. You are also encouraged to familiarize yourself with the PTGBot documentation[5] as well. If you have any questions, please reach out! Information about signing up for time will be sent to moderators late August after the interest survey deadline on August 20. -Kendall (diablo_rojo) [1] PTG Site: https://openinfra.dev/ptg/ [2] PTG Registration: http://ptg2023.openinfra.dev/ [3] Team Survey: https://openinfrafoundation.formstack.com/forms/oct2023_ptg_team_signup [4] Setup IRC: https://docs.openstack.org/contributors/common/irc.html [5] PTGBot README: https://opendev.org/openstack/ptgbot/src/branch/master/README.rst

1 0

[openstack-ansible][wallaby deployment getting failed] error:- Service user token configuration is required for all Nova
by Danish Khan 23 Jul '23

23 Jul '23

Dear Team, I am trying to deploy openstack-ansible wallaby but it is getting failed with error: service user token configuration is required for all Nova services. For more details see the following: https://docs.openstack.org/latest/nova/admin/configuration/service-user-tok… But the mentioned webpage is not available. This is fixed in yoga but wallay is still failing. I tried to copy few variables from Yoga but that is working for me. Can someone please help me on this ? where I need to make some changes to deploy openstack-ansible wallaby? Thanks in advance :) Regards, Danish

2 1

Re: [all] [oslo.messaging] Interest in collaboration on a NATS driver
by Tobias Urdin 22 Jul '23

22 Jul '23

Hello Kai, Great to hear that you’re interested! I’m currently just testing this as a POC directly in devstack so there is real-world testing (not should it yet I think, needs more development first). Best regards Tobias > On 30 Aug 2022, at 18:32, Kai Bojens <kb(a)kbojens.de> wrote: > > Am 29.08.22 um 15:46 schrieb Tobias Urdin: > >> If anybody, or any company, out there would be interested in collaborating in a project to bring this support and maintain it feel free to >> reach out. I’m hoping somebody will bite but atleast I’ve put it out there for all of you. > > Hi, > I am very much interested to take a closer look at your work and maybe contribute to it. Although I'm working with OpenStack for my employer at the moment, I'd do this in my spare time as I'm not sure that I can convince him to add another staging system with a full OpenStack installation just for developing a RabbitMQ replacement. That's not on our agenda as we are mostly users and not developers of OpenStack. > > As I'm pretty new to the messaging topic and had just heard of NATS and your idea, I'll first would have to dive into NATS and then I would take a closer look at your code and maybe try to create some documentation or tests. So, I can't promise anything but as I said I'm very interested in your approach as I also see the massive load from RabbitMQ. > > This brings me to the most important question: Where do you run and test your code? > > Greetings, > Kai

8 30

Superuser / OpenInfra Live Opportunities?
by Kristin Barrientos 21 Jul '23

21 Jul '23

Hi Everyone, OpenInfra Live is looking for ideas and participants to join a panel to discuss those topics. That being said… do you have a topic in mind that you would like to see on OpenInfra Live? We want to hear from you! Some ideas can include: Data Sovereignty Multi-Region Deployment Kolla-Ansible AI/ML And many more! If you would like to participate or have an idea, you can contact me via email (kristin(a)openinfra.dev) or reply here, and we can work on getting something started. If you would rather write a Superuser article, feel free to email me an abstract and we can go from there. I look forward to hearing everyone’s ideas! Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

2 1

[neutron] Next Neutron meetings schedule
by Rodolfo Alonso Hernandez 21 Jul '23

21 Jul '23

Hello Neutrinos: Today we have the Neutron drivers meeting at 1400UTC. This is the agenda: https://wiki.openstack.org/wiki/Meetings/NeutronDrivers. We have one topic: https://bugs.launchpad.net/neutron/+bug/2027742. Next Tuesday we have cancelled the Neutron team meeting (I'll be on PTO). But remember that the Neutron CI meeting will take place at 1500UTC. Regards.

1 0

[OSA][NEUTRON] Failed to bind port XXX on host YYY for vnic_type normal using segments...
by Murilo Morais 21 Jul '23

21 Jul '23

Hey guys. I'm having problems with a new deploy. I'm relatively new to the project, so I still don't know how to guide myself in solving some errors. When I create some Network and try to upload a VM the following errors appear in the Neutron container: ############################### Jul 15 11:31:50 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:31:50.988 5676 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:31:51 dcn2-neutron-server-container-a7122a16 neutron-api[5683]: 2023-07-15 11:31:51.009 5683 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:31:51 dcn2-neutron-server-container-a7122a16 uwsgi[5683]: Sat Jul 15 11:31:51 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/availability_zones (ip 172.29.237.22) !!! Jul 15 11:33:53 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:33:53.860 5676 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:33:58.733 5678 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 uwsgi[5678]: Sat Jul 15 11:33:58 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnetpools (ip 172.29.237.22) !!! Jul 15 11:33:58 dcn2-neutron-server-container-a7122a16 neutron-api[5677]: 2023-07-15 11:33:58.959 5677 INFO neutron.db.segments_db [None req-61542ff4-867d-402f-8239-b45392b2c188 f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Added segment 16ec3850-a4f2-40f3-9dc8-3b4650053935 of type vlan for network 6887bbfb-4680-4ed7-910b-90a65f7731dc Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:33:59.095 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 uwsgi[5680]: Sat Jul 15 11:33:59 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:33:59.584 5680 WARNING neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [None req-5ad409c5-c073-4726-bb26-5024d9c9c65b f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Unable to schedule network 6887bbfb-4680-4ed7-910b-90a65f7731dc: no agents available; will retry on subsequent port and subnet creation events. Jul 15 11:33:59 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:33:59.684 5688 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:41:32 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:41:32.961 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 174, in _new_conn Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn = connection.create_connection( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/connection.py", line 95, in create_connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise err Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/connection.py", line 85, in create_connection Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: sock.connect(sa) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/greenio/base.py", line 256, in connect Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: socket_checkerr(fd) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/greenio/base.py", line 54, in socket_checkerr Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise socket.error(err, errno.errorcode[err]) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: BrokenPipeError: [Errno 32] EPIPE Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 703, in urlopen Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: httplib_response = self._make_request( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 398, in _make_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn.request(method, url, **httplib_request_kw) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 239, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: super(HTTPConnection, self).request(method, url, body=body, headers=headers) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1255, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._send_request(method, url, body, headers, encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1301, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.endheaders(body, encode_chunked=encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1250, in endheaders Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._send_output(message_body, encode_chunked=encode_chunked) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 1010, in _send_output Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.send(msg) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/usr/lib/python3.9/http/client.py", line 950, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.connect() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 205, in connect Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: conn = self._new_conn() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connection.py", line 186, in _new_conn Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise NewConnectionError( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/adapters.py", line 489, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = conn.urlopen( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/connectionpool.py", line 787, in urlopen Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: retries = retries.increment( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/urllib3/util/retry.py", line 592, in increment Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise MaxRetryError(_pool, url, error or ResponseError(cause)) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1022, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = self.session.request(method, url, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/sessions.py", line 587, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = self.send(prep, **send_kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/sessions.py", line 701, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: r = adapter.send(request, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/requests/adapters.py", line 565, in send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise ConnectionError(e, request=request) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: requests.exceptions.ConnectionError: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: During handling of the above exception, another exception occurred: Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Traceback (most recent call last): Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/eventlet/hubs/poll.py", line 111, in wait Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: listener.cb(fileno) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/common/utils.py", line 924, in wrapper Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return func(*args, **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/notifiers/batch_notifier.py", line 58, in synced_send Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self._notify() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/notifiers/batch_notifier.py", line 69, in _notify Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: self.callback(batched_events) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 211, in _send_notifications Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: event.method(event) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 383, in _delete_nova_inventory Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: aggregate_id = self._get_aggregate_id(event.segment_id) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron/services/segments/plugin.py", line 370, in _get_aggregate_id Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: aggregate_uuid = self.p_client.list_aggregates( Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 58, in wrapper Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return f(self, *a, **k) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 554, in list_aggregates Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self._get(url).json() Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/neutron_lib/placement/client.py", line 190, in _get Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self._client.get(url, endpoint_filter=self._ks_filter, Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1141, in get Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: return self.request(url, 'GET', **kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 931, in request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: resp = send(**kwargs) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: File "/openstack/venvs/neutron-26.1.3.dev1/lib/python3.9/site-packages/keystoneauth1/session.py", line 1038, in _send_request Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: raise exceptions.ConnectFailure(msg) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:41:33.144 5688 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] ECONNRESET (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] ECONNRESET Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://172.29.236.2:8780/resource_providers/dd7e74fb-b494-448e-b16a-712a7ad…: HTTPConnectionPool(host='172.29.236.2', port=8780): Max retries exceeded with url: /resource_providers/dd7e74fb-b494-448e-b16a-712a7ad60e6c/aggregates (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7efd5f36ca30>: Failed to establish a new connection: [Errno 32] EPIPE')) Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 uwsgi[5688]: Removing descriptor: 13 Jul 15 11:41:33 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:41:33.542 5689 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:42:36 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:42:36.571 5678 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:40 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:42:40.942 5689 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 1.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:41 dcn2-neutron-server-container-a7122a16 uwsgi[5683]: Sat Jul 15 11:42:41 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:42:42 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:42:42.053 5688 ERROR oslo.messaging._drivers.impl_rabbit [-] Connection failed: [Errno 104] Connection reset by peer (retrying in 3.0 seconds): ConnectionResetError: [Errno 104] Connection reset by peer Jul 15 11:42:45 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:42:45.240 5688 WARNING neutron.api.rpc.agentnotifiers.dhcp_rpc_agent_api [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-0270e149-44c9-49c4-af20-47918e00c4ea f8da6c7abaa64116948db1d2aaa829f1 86e3eede43b940d69d2b43ccba1b256b - - default default] Unable to schedule network 6887bbfb-4680-4ed7-910b-90a65f7731dc: no agents available; will retry on subsequent port and subnet creation events. Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.563 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.564 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 2 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.641 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.641 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 3 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.695 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.695 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 4 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.743 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.743 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 5 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.798 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.799 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 6 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.856 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.856 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 7 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.904 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.904 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 8 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.952 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:46 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:46.952 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 9 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.000 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.001 5682 INFO neutron.plugins.ml2.plugin [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Attempt 10 to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 Jul 15 11:42:47 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:42:47.056 5682 ERROR neutron.plugins.ml2.managers [req-1ded0181-a9ab-446b-b599-e8fbe86ca555 req-80894c6f-8ef3-4df7-86d1-535e20ebc2d2 33ba7b2cad184ddcacff6c1b5017a648 8ee53dea918a4570b4898dc10f542c5c - - default default] Failed to bind port caa01668-d5c0-4b37-a185-0ebc458326f7 on host dcn2 for vnic_type normal using segments [{'id': '16ec3850-a4f2-40f3-9dc8-3b4650053935', 'network_type': 'vlan', 'physical_network': 'vlan', 'segmentation_id': 138, 'network_id': '6887bbfb-4680-4ed7-910b-90a65f7731dc'}] Jul 15 11:42:54 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:42:54.511 5676 INFO neutron.notifiers.nova [-] Nova event matching ['req-0ffbf4aa-051f-4707-afe7-2908726310c5'] response: {'server_uuid': '9f206d9f-6c96-479f-bbaa-116d967129dc', 'name': 'network-vif-deleted', 'tag': 'caa01668-d5c0-4b37-a185-0ebc458326f7', 'status': 'completed', 'code': 200} Jul 15 11:48:24 dcn2-neutron-server-container-a7122a16 neutron-api[5678]: 2023-07-15 11:48:24.062 5678 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 neutron-api[5676]: 2023-07-15 11:48:31.762 5676 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 neutron-api[5677]: 2023-07-15 11:48:31.828 5677 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:31 dcn2-neutron-server-container-a7122a16 uwsgi[5677]: Sat Jul 15 11:48:31 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnets (ip 172.29.237.22) !!! Jul 15 11:48:32 dcn2-neutron-server-container-a7122a16 neutron-api[5686]: 2023-07-15 11:48:32.108 5686 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:32 dcn2-neutron-server-container-a7122a16 uwsgi[5686]: Sat Jul 15 11:48:32 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/networks?limit=21&sort_dir=asc&sort_key=id&router%3Aexternal=True&shared=False (ip 172.29.237.22) !!! Jul 15 11:48:34 dcn2-neutron-server-container-a7122a16 neutron-api[5689]: 2023-07-15 11:48:34.625 5689 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:34 dcn2-neutron-server-container-a7122a16 neutron-api[5688]: 2023-07-15 11:48:34.641 5688 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:35 dcn2-neutron-server-container-a7122a16 neutron-api[5682]: 2023-07-15 11:48:35.978 5682 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 neutron-api[5683]: 2023-07-15 11:48:36.052 5683 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Server unexpectedly closed connection Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 neutron-api[5680]: 2023-07-15 11:48:36.809 5680 INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: [Errno 104] Connection reset by peer Jul 15 11:48:36 dcn2-neutron-server-container-a7122a16 uwsgi[5680]: Sat Jul 15 11:48:36 2023 - SIGPIPE: writing to a closed pipe/socket/fd (probably the client disconnected) on request /v2.0/subnetpools (ip 172.29.237.22) !!! ############################### After the errors appear, the Horizon interface reports the following error: Error: Failed to perform requested operation on instance "test", the instance has an error status: Please try again later [Error: Exceeded maximum number of retries. Exhausted all hosts available for retrying build failures for instance 9f206d9f-6c96-479f-bbaa-116d967129dc.]. What could be happening? I believe I'm missing something to configure, but I don't know where to start.

2 6

[cinder] festival of feature reviews 21st July, 2023
by Rajat Dhasmana 20 Jul '23

20 Jul '23

Hello Argonauts, We will be having our monthly festival of reviews today i.e. 21st July (Friday) from 1400-1600 UTC. Following are some additional details: Date: 21st July, 2023 Time: 1400-1600 UTC Meeting link: will be shared in #openstack-cinder around 1400 UTC etherpad: https://etherpad.opendev.org/p/cinder-festival-of-reviews Attached is the invite if you want to add it to your calendar. Thanks Rajat Dhasmana

1 0

Announcement: OpenStack "C" Release Name
by Kristin Barrientos 19 Jul '23

19 Jul '23

Hi Everyone, Happy 13th birthday to OpenStack! We're proud to announce that the name chosen for the “C” release is… Caracal! Thank you to everyone who voted. Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[neutron] unmanaged router resources - OVN interconnect
by Roberto Bartzen Acosta 19 Jul '23

19 Jul '23

Hello Neutron folks, Regarding the conversation started in March [1] about the use of OVN interconnect with Neutron, I would like to evolve the discussion in relation to resources allocated by OVN-IC and which are not managed by Neutron. In the March PTG [2], the problem related to the db_sync tool was presented, and a proposed solution in which Neutron does not manage these resources. After discussing some architectural designs with Felix/StackIT, it seems to make sense to come up with a proposal to change the mech_driver to validate the external_ids key and not remove resources present in the OVN backend without Neutron "signature". The discussion here is more complex than it seems, because architecturally Neutron does not allow us to interconnect workloads in multiple AZs (with different OpenStacks), but this could be a requirement for a high availability cloud solution (Cloud Region). Additionally, this OVN-IC solution allows interconnecting other cloud backends that use OVN, in the case of kubernetes with ovn-kube. We tested an OVN interconnect integrated with 3 OpenStack installations and it worked very well !!! considering direct L3 traffic at the router level between different network infrastructures. SYNC_REPAIR - problem * Static Routes (learned OVN-IC routes) * Router Port -> Transit Switches Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router Port found in OVN but not in Neutron, port_id=rt2-admin-tenant1 Jul 10 18:34:11 os-infra-1-neutron-server-container-845157ae neutron-server[8632]: 2023-07-10 18:34:11.343 8632 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.ovn_db_sync [req-8d513732-f932-47b8-bc2c-937958c30f47 - - - - -] Router 9823d34b-bb2a-480c-b3f6-cf51fd19db52 static routes [{'destination': ' 10.0.0.1/24', 'nexthop': '169.254.100.1'}, {'destination': '10.0.2.1/24', 'nexthop': '169.254.100.3'}] found in OVN but not in Neutron Any suggestions on how to resolve this db_sync issue? since all other Neutron modules did not present any problem with OVN-IC (as far as I tested). I remember Terry was keen to suggest some things to help. I believe that before proposing some complex mechanism to solve this simple problem, I would like to hear the community opinion. In our use case, a bit change in db_sync with filter by neutron key in external_ids would be enough. Regards, Roberto [1] https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032624.h… [2] https://etherpad.opendev.org/p/neutron-bobcat-ptg Additional logs: OpenStack 1 root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 6b776115-746a-4c59-aa73-6674c70b3498 IPv4 Routes Route Table <main>: 20.0.1.0/24 169.254.200.2 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# ovn-nbctl lr-route-list 23d4552a-62c4-40e1-8bae-d06af3489c07 IPv4 Routes Route Table <main>: 10.0.1.0/24 169.254.100.2 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f931b37c:~# OpenStack 2 root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list dc1e5008-adb9-451e-8b71-09388f3680bc IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.2.0/24 169.254.200.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-30f7e935:~# ovn-nbctl lr-route-list ce45f681-6454-43fe-974f-81344bb8113a IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.2.0/24 169.254.100.3 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip OpenStack 3 root@os-infra-1-neutron-ovn-northd-container-f237db97:~# root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list cfa259d6-311f-4409-bcf2-79a929835cb3 IPv4 Routes Route Table <main>: 20.0.0.0/24 169.254.200.1 dst-ip (learned) 20.0.1.0/24 169.254.200.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip root@os-infra-1-neutron-ovn-northd-container-f237db97:~# ovn-nbctl lr-route-list c5a4dcd8-b9a6-4397-a7cf-88bc1e01b0b0 IPv4 Routes Route Table <main>: 10.0.0.0/24 169.254.100.1 dst-ip (learned) 10.0.1.0/24 169.254.100.2 dst-ip (learned) 0.0.0.0/0 200.200.200.1 dst-ip IPv6 Routes Route Table <main>: ::/0 fc00:ca5a:ca5a:8000:: dst-ip OVN-IC Global database root@ovn-global-db1:~# ovn-ic-sbctl show availability-zone osp1 gateway 832b6c0d-13ce-4600-ab37-78516d8ec4c5 hostname: osp1-gwnode1 type: geneve ip: 192.168.200.28 port admin-rt1-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:01 169.254.100.1/24 fe80::1/64"] port admin-rt1-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:01 169.254.200.1/24"] availability-zone osp2 gateway 17ffabdf-cf47-41ab-9539-d326c13c4ca8 hostname: osp2-gwnode1 type: geneve ip: 192.168.200.128 port admin-rt2-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:bb:02 169.254.100.2/24 fe80::2/64"] port admin-rt2-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:02 169.254.200.2/24"] availability-zone osp3 gateway 97595af9-7896-40d0-a883-beadbff1aa5b hostname: osp3-gwnode1 type: geneve ip: 192.168.200.228 port admin-rt3-tenant1 transit switch: admin-tenant1 address: ["aa:aa:aa:aa:aa:03 169.254.100.3/24 fe80::3/64"] port admin-rt3-tenant1_1 transit switch: admin-tenant1_1 address: ["aa:aa:aa:aa:dd:03 169.254.200.3/24"] -- _‘Esta mensagem é direcionada apenas para os endereços constantes no cabeçalho inicial. Se você não está listado nos endereços constantes no cabeçalho, pedimos-lhe que desconsidere completamente o conteúdo dessa mensagem e cuja cópia, encaminhamento e/ou execução das ações citadas estão imediatamente anuladas e proibidas’._ * **‘Apesar do Magazine Luiza tomar todas as precauções razoáveis para assegurar que nenhum vírus esteja presente nesse e-mail, a empresa não poderá aceitar a responsabilidade por quaisquer perdas ou danos causados por esse e-mail ou por seus anexos’.*

3 19

Cinder Bug Report 2023-07-19
by Sofia Enriquez 19 Jul '23

19 Jul '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> Medium - nvmeof connector _get_host_uuid incompatible when cinder runs in lxd <https://bugs.launchpad.net/cinder/+bug/2027795>. - *Status*: This looks related to bug 2026257 <https://bugs.launchpad.net/bugs/2026257>. Low - Volume to Image upload is very slow due to use of tpool.Proxy() <https://bugs.launchpad.net/cinder/+bug/2027970>. - *Status*: Fix proposed to master <https://review.opendev.org/c/openstack/cinder/+/888676>. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

[all][tc] Proposals for reforming extended maintenance
by Nikolla, Kristi 18 Jul '23

18 Jul '23

Hi all, There have been quite a few discussions related extended maintenance in the last few months[0][1]. I have made a first-go at trying to write a policy for extended maintenance moving forward which tries to preserve the value being brought by extended maintenance while strengthening the requirements for keeping branches under that phase. [2] This was based on multiple discussions that I was present at and feedback that I have received from various teams and community members with which I talked, and is my interpretation of that feedback into something that looks like policy. I have also written two opposing policy proposals about what do to with the current branches under extended maintenance. One which attempts to apply the new requirements defined in the first proposal almost immediately[3], and one that aims for a more gradual EOL[4]. Please provide feedback on the proposals and help shape up the future of extended maintenance. Thank you, Kristi Nikolla [0]. https://lists.openstack.org/pipermail/openstack-discuss/2023-June/033980.ht… [1]. https://etherpad.opendev.org/p/vancouver-2023-em [2]. https://review.opendev.org/c/openstack/governance/+/887966 [3]. https://review.opendev.org/c/openstack/governance/+/887968 [4]. https://review.opendev.org/c/openstack/governance/+/887969

3 3

[tc] Technical Committee next weekly meeting today on July 18, 2023
by Nikolla, Kristi 18 Jul '23

18 Jul '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held Today on Tuesday, July 18, 2023, at 1800 UTC on #openstack-tc on OFTC IRC. Please find below the agenda for today's meeting: * Roll call * Follow up on past action items * knikolla will amend proposal for Unmaintained branches. * Extended Maintenance * https://etherpad.opendev.org/p/openstack-exteneded-maintenance * Release notes guidelines for SLURP/NON-SLURP cadence * Gate health check * Open Discussion and Reviews * https://review.opendev.org/q/projects:openstack/governance+is:open Thank you, Kristi Nikolla

1 0

[openstack][largescale-sig] Openstack multi region deployment
by Nguyễn Hữu Khôi 18 Jul '23

18 Jul '23

Hello guys, I am going to deploy openstack multi regions and I know that keystone replication is the most challenging. I plan to set up 2 regions which use centralize galera cluster(3 nodes). and one standby edge galera cluster(3 nodes) When region 1 is node available, I will map region 2 to use standby edge galera cluster. I hope you give me some experience and advice with multi regions. Thank you very much.

2 2

[ops] Built-in rolling-update/instance refresh capabilities
by Matthew Giassa 18 Jul '23

18 Jul '23

Good day, I was originally discussing this on the IRC channel, #openstack, and was advised to reach out to this mailing list. In short, does OpenStack (i.e. via Heat, Senlin, Masakari, or some other feature/plugin), have the ability to perform a rolling upgrade operation on a group of VMs (i.e. similar to the "rolling update" feature in K8S, or the "instance refresh" feature in AWS auto-scaling groups / ASGs). I've reviewed the available documentation, but all documents regarding "rolling upgrades" are with respect to a rolling upgrade of the OpenStack/IaaS stack itself (i.e. operator-level functionality) as opposed to groups of VMs managed by OpenStack (i.e. user-level functionality). I'd ideally like to use built-in functionality to handle this operation, rather than writing a service from scratch. I've also looked into tools such as Spinnaker.io and Harness.io, but could not find any FOSS or COTS tools for implementing rolling upgrades on OpenStack out-of-the-box (though if there are any options I've overlooked, I'm all ears). Thank you. -- ============================================================ Matthew Giassa e-mail: matthew(a)giassa.net website: www.giassa.net

1 0

CVE-2023-2088 for Charmed Openstack
by Corey Bryant 17 Jul '23

17 Jul '23

Bug LP:#2004555 (CVE-2023-2088) identified that “an unauthorised access to a volume could occur when an iSCSI or FC connection from a host is severed due to a volume being unmapped on the storage system and the device is later reused for another volume on the same host”. While the bug affects iSCSI and FC, the fix introduces a breaking change for configurations upgrading to the fixed packages if they do not have service token support enabled. The new code introduces mandatory use of service tokens to identify requests coming from a trusted control plane service. Service tokens are issued by Keystone for service users. For example, a token can be issued to the Nova service user. They have a particular trust level, defined by the service, that allows them to perform trusted actions. In the case of Nova, the action is detaching volumes from instances. Versions of Ubuntu OpenStack for Yoga and later will have new packages versions released with CVE patches applied that require the use of service tokens to enable the detaching and deletion of volumes on instances. These packages will be released on July 24, 2023. Versions of Ubuntu OpenStack for Xena and earlier will not have package updates released and will still be vulnerable to the “unauthorised detach of volumes” issue until a mitigation policy is applied. The mitigation policy provided requires the use of service tokens. For Charmed OpenStack releases, this means that some charms (identified below) must be upgraded to enable service tokens prior to the packages being upgraded and before the mitigation policy is applied. If not, users will lose the ability to detach volumes from their instances. = Patched packages = The following package versions include patches for CVE-2023-2088. These package versions deliver services that require the use of service tokens. Package versions prior to those listed will not be patched for CVE-2023-2088 and will therefore require policy updates for mitigation. Ubuntu 23.04 (Lunar) * nova >= 3:27.0.0-0ubuntu1.3 * cinder >= 2:22.0.0-0ubuntu1.3 * python-os-brick >= 6.2.0-0ubuntu2.3 * ironic >= 1:21.4.0-0ubuntu1.1 * python-glance-store >= 4.3.0-0ubuntu1.3 Ubuntu Cloud Archive Antelope * nova >= 3:27.0.0-0ubuntu1.3~cloud0 * cinder >= 2:22.0.0-0ubuntu1.3~cloud0 * python-os-brick >= 6.2.0-0ubuntu2.3~cloud0 * ironic >= 1:21.4.0-0ubuntu1.1~cloud0 * python-glance-store >= 4.3.0-0ubuntu1.3~cloud0 Ubuntu 22.10 (Kinetic) * nova >= 3:26.1.1-0ubuntu1.1 * cinder >= 2:21.2.0-0ubuntu1.1 * python-os-brick >= os-brick - 6.1.0-0ubuntu1.3 * ironic >= 1:21.1.0-0ubuntu1.1 * python-glance-store >= 4.1.0-0ubuntu1.3 Ubuntu Cloud Archive Zed * nova >= 3:26.1.1-0ubuntu1.1~cloud0 * cinder >= 2:21.2.0-0ubuntu1.1~cloud0 * python-os-brick >= os-brick - 6.1.0-0ubuntu1.3~cloud0 * ironic >= 1:21.1.0-0ubuntu1.1~cloud0 * python-glance-store >= 4.1.0-0ubuntu1.3~cloud0 Ubuntu 22.04 (Jammy) * nova >= 3:25.1.1-0ubuntu1.1 * cinder >= 2:20.2.0-0ubuntu1.1 * python-os-brick >= 5.2.2-0ubuntu1.2 * ironic >= 1:20.1.0-0ubuntu1.1 * python-glance-store >= 3.0.0-0ubuntu1.3 Ubuntu Cloud Archive Yoga * nova >= 3:25.1.1-0ubuntu1.1~cloud0 * cinder >= 2:20.2.0-0ubuntu1.1~cloud0 * python-os-brick >= 5.2.2-0ubuntu1.2~cloud0 * ironic >= 1:20.1.0-0ubuntu1.1~cloud0 * python-glance-store >= 3.0.0-0ubuntu1.3~cloud0 = Charms with Service Tokens Enabled by Default = Following is the list of charms and earliest revision that supports service tokens: Channel: 2023.1/stable * Keystone revision >= 645 * Cinder revision >= 639 * Nova-compute revision >= 678 * Nova-cloud-controller revision >= 674 * Ironic-api revision >= 54 * Ironic-conductor - s390x revision >= 151 - arm64 revision >= 152 - ppc64el revision >= 153 - amd64 revision >= 149 Channel: zed/stable * Keystone revision >= 646 * Cinder revision >= 647 * Nova-compute revision >= 683 * Nova-cloud-controller revision >= 676 * Ironic-api revision >= 53 * Ironic-conductor - s390x revision >= 148 - arm64 revision >= 154 - ppc64el revision >= 150 - amd64 revision >= 147 Channel: yoga/stable * Keystone revision >= 647 * Cinder revision >= 638 * Nova-compute revision >= 681 * Nova-cloud-controller revision >= 679 * Ironic-api revision >= 48 * Ironic-conductor revision >= 145 Channel: xena/stable * Keystone revision >= 651 * Cinder revision >= 640 * Nova-compute revision >= 680 * Nova-cloud-controller revision >= 677 * Ironic-api revision >= 50 * Ironic-conductor revision >= 143 Channel: wallaby/stable * Keystone revision >= 653 * Cinder revision >= 648 * Nova-compute revision >= 684 * Nova-cloud-controller revision >= 675 * Ironic-api revision >= 52 * Ironic-conductor revision >= 144 Channel: victoria/stable * Keystone revision >= 650 * Cinder revision >= 646 * Nova-compute revision >= 679 * Nova-cloud-controller revision >= 678 * Ironic-api revision >= 49 * Ironic-conductor revision >= 142 Channel: ussuri/stable * Keystone revision >= 652 * Cinder revision >= 645 * Nova-compute revision >= 682 * Nova-cloud-controller revision >= 680 * Ironic-api revision >= 51 * Ironic-conductor revision >= 155 = Using a custom Cinder policy as a mitigation = It’s possible to restrict the access to the operations affected by this CVE by providing a custom Cinder policy file that will prevent users from deleting attachments, detach and force detach. In order to use this mitigation, service token support must be enabled. The exact method for customising this depends on the deployment tooling being used. For Charmed OpenStack users it is the following: 1) Identify the nova service user id running the following command openstack user show --domain service_domain -f value -c id nova The command output looks like this ae034831d6bf41099327bf9d34988fbc 2) Create a policy file named cve-2023-2088-cinder.yaml with the following content: "is_service": "role:service or service_user_id:<nova_service_uuid>" "legacy_system_admin_or_project_member": "(role:admin) or (role:member and project_id:%(project_id)s)" "volume:attachment_delete": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:terminate_connection": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:detach": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_admin_actions:force_detach": "!" Replacing “<nova_service_uuid>” would result in a snippet that looks like this: "is_service": "role:service or service_user_id:ae034831d6bf41099327bf9d34988fbc" "legacy_system_admin_or_project_member": "(role:admin) or (role:member and project_id:%(project_id)s)" "volume:attachment_delete": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:terminate_connection": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:detach": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_admin_actions:force_detach": "!" 3) Create zip files from the yaml file: zip cve-2023-2088-cinder.zip cve-2023-2088-cinder.yaml 4) Attach the zip file as a resource to the cinder applications: juju attach-resource cinder policyd-override=cve-2023-2088-cinder.zip 5) Enable the overrides via the use-policyd-override charm option: juju config cinder use-policyd-override=true For more details on using policy overrides see the OpenStack Charm Guide. = Frequently Asked Questions = Q: What can I do if my deployed cloud doesn’t have service tokens enabled and my deployment tooling doesn’t have the option to enable them? A: Pin the version of the affected packages to block the upgrades to the versions listed in “Affected packages” section or newer, until service tokens are enabled. Q: For Charmed OpenStack, what’s the upgrade order for charms and deb packages? A: Upgrade the charms to the latest versions available in the charm channel tracked by your environment or at the least to the revisions listed in section “Charms with Service Token Enabled By Default”. After that, upgrade the deb packages with your preferred method (e.g. Landscape, apt-get, etc). = References = * CVE-2023-2088: https://ubuntu.com/security/CVE-2023-2088 * OSSA-2023-003 Unauthorized volume access through deleted volume attachments: https://security.openstack.org/ossa/OSSA-2023-003.html * OSSN-0092 Using Configuration as a Short-Term Mitigation for OSSA-2023-003: https://wiki.openstack.org/wiki/OSSN/OSSN-0092 * Bug 2004555: https://bugs.launchpad.net/nova/+bug/2004555 * OpenStack Charm Guide Policy overrides: https://docs.openstack.org/charm-guide/latest/admin/policy-overrides.html

1 0

CVE-2023-2088 for Ubuntu Openstack
by Corey Bryant 17 Jul '23

17 Jul '23

Bug LP:#2004555 (CVE-2023-2088) identified that “an unauthorised access to a volume could occur when an iSCSI or FC connection from a host is severed due to a volume being unmapped on the storage system and the device is later reused for another volume on the same host”. While the bug affects iSCSI and FC, the fix introduces a breaking change for configurations upgrading to the fixed packages if they do not have service token support enabled. The new code introduces mandatory use of service tokens to identify requests coming from a trusted control plane service. Service tokens are issued by Keystone for service users. For example, a token can be issued to the Nova service user. They have a particular trust level, defined by the service, that allows them to perform trusted actions. In the case of Nova, the action is detaching volumes from instances. Versions of Ubuntu OpenStack for Yoga and later will have new packages versions released with CVE patches applied that require the use of service tokens to enable the detaching and deletion of volumes on instances. These packages will be released on July 24, 2023. Versions of Ubuntu OpenStack for Xena and earlier will not have package updates released and will still be vulnerable to the “unauthorised detach of volumes” issue until a mitigation policy is applied. The mitigation policy provided requires the use of service tokens. = Patched packages = The following package versions include patches for CVE-2023-2088. These package versions deliver services that require the use of service tokens. Package versions prior to those listed will not be patched for CVE-2023-2088 and will therefore require policy updates for mitigation. Ubuntu 23.04 (Lunar) * nova >= 3:27.0.0-0ubuntu1.3 * cinder >= 2:22.0.0-0ubuntu1.3 * python-os-brick >= 6.2.0-0ubuntu2.3 * ironic >= 1:21.4.0-0ubuntu1.1 * python-glance-store >= 4.3.0-0ubuntu1.3 Ubuntu Cloud Archive Antelope * nova >= 3:27.0.0-0ubuntu1.3~cloud0 * cinder >= 2:22.0.0-0ubuntu1.3~cloud0 * python-os-brick >= 6.2.0-0ubuntu2.3~cloud0 * ironic >= 1:21.4.0-0ubuntu1.1~cloud0 * python-glance-store >= 4.3.0-0ubuntu1.3~cloud0 Ubuntu 22.10 (Kinetic) * nova >= 3:26.1.1-0ubuntu1.1 * cinder >= 2:21.2.0-0ubuntu1.1 * python-os-brick >= os-brick - 6.1.0-0ubuntu1.3 * ironic >= 1:21.1.0-0ubuntu1.1 * python-glance-store >= 4.1.0-0ubuntu1.3 Ubuntu Cloud Archive Zed * nova >= 3:26.1.1-0ubuntu1.1~cloud0 * cinder >= 2:21.2.0-0ubuntu1.1~cloud0 * python-os-brick >= os-brick - 6.1.0-0ubuntu1.3~cloud0 * ironic >= 1:21.1.0-0ubuntu1.1~cloud0 * python-glance-store >= 4.1.0-0ubuntu1.3~cloud0 Ubuntu 22.04 (Jammy) * nova >= 3:25.1.1-0ubuntu1.1 * cinder >= 2:20.2.0-0ubuntu1.1 * python-os-brick >= 5.2.2-0ubuntu1.2 * ironic >= 1:20.1.0-0ubuntu1.1 * python-glance-store >= 3.0.0-0ubuntu1.3 Ubuntu Cloud Archive Yoga * nova >= 3:25.1.1-0ubuntu1.1~cloud0 * cinder >= 2:20.2.0-0ubuntu1.1~cloud0 * python-os-brick >= 5.2.2-0ubuntu1.2~cloud0 * ironic >= 1:20.1.0-0ubuntu1.1~cloud0 * python-glance-store >= 3.0.0-0ubuntu1.3~cloud0 = Using a custom Cinder policy as a mitigation = It’s possible to restrict the access to the operations affected by this CVE by providing a custom Cinder policy file that will prevent users from deleting attachments, detach and force detach. In order to use this mitigation, service token support must be enabled. The exact method for customising this depends on the deployment tooling being used. The following steps can be used as a guildeline: 1) Identify the nova service user id running the following command openstack user show --domain service_domain -f value -c id nova The command output looks like this ae034831d6bf41099327bf9d34988fbc 2) Create and apply the following policy to cinder, using the output of the previous command: "is_service": "role:service or service_user_id:ae034831d6bf41099327bf9d34988fbc" "legacy_system_admin_or_project_member": "(role:admin) or (role:member and project_id:%(project_id)s)" "volume:attachment_delete": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:terminate_connection": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_actions:detach": "rule:legacy_system_admin_or_project_member and rule:is_service" "volume_extension:volume_admin_actions:force_detach": "!" = Frequently Asked Questions = Q: What can I do if my deployed cloud doesn’t have service tokens enabled and my deployment tooling doesn’t have the option to enable them? A: Pin the version of the affected packages to block the upgrades to the versions listed in “Affected packages” section or newer, until service tokens are enabled. = References = * CVE-2023-2088: https://ubuntu.com/security/CVE-2023-2088 * OSSA-2023-003 Unauthorized volume access through deleted volume attachments: https://security.openstack.org/ossa/OSSA-2023-003.html * OSSN-0092 Using Configuration as a Short-Term Mitigation for OSSA-2023-003: https://wiki.openstack.org/wiki/OSSN/OSSN-0092 * Bug 2004555: https://bugs.launchpad.net/nova/+bug/2004555

1 0

[designate] About delegating PTR using shared zones on a shared internal network
by Thomas Goirand 17 Jul '23

17 Jul '23

Hi there! I have just watched Michael Johnson's video of the presentation at the last Vancouver summit: https://www.youtube.com/watch?v=zdhvNezU1w4 This was very interesting, and thanks Michael, for that awesome feature. In our use case, we created an OpenStack internal network, that is shared for all of our customers, so that they can allocate public IPs assigned directly to their instances, without the need to create a router and use NAT. For the moment (since we don't have the shared zones feature yet), we manage the PTR record through support tickets. We'd like to move forward, and delegate the PTR handling to our customer, whenever they allocate a port on that specific shared network. How can this be done? Should we create a new sink of our own, so that we would automate the the shared zone creation, so that it would delegate the in-addr.arpa. range, when a port on that network is created, and the same way, remove the delegation (and records) whenever the port is deleted? Would this be something similar, probably, we the sink we implemented to automatically delete PTR records in floating IPs, available here [1]? [1] https://salsa.debian.org/openstack-team/services/designate/-/blob/debian/an… I'd love to have some pointers / examples, so we could implement this directly in Designate, so that the feature could be shared in the OpenStack community. For sure, we aren't the only public cloud provider with the use case... Cheers, Thomas Goirand (zigo)

2 3

UX
by A.R. Rumman 17 Jul '23

17 Jul '23

Hi, I am a UX Designer. I would love to contribute to OpenStack. So what's the next step or process? Thank you.

2 1

OpenStack "C" Release Voting Reminder
by Kristin Barrientos 17 Jul '23

17 Jul '23

Hi everyone, Just a reminder to get your votes in for the OpenStack “C” Release. Voting closes Tuesday, July 18! Vote here: https://civs1.civs.us/cgi-bin/vote.pl?id=E_852cf3e53467d72e&akey=1504cb6ad9… Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[publiccloud-sig] Reminder - next meeting July 19th - 0700 UTC
by Felix Kronlage-Dammers 17 Jul '23

17 Jul '23

Hi everyone, this Wednesday the next meeting of the public cloud sig is going to happen. We meet on IRC in #openstack-operators at 0700 UTC. This time, we will switch to a jitsi-based video-call directly after start. The url will be posted on #openstack-operators at the time of the meeting. A preliminary agenda can be found in the pad: https://etherpad.opendev.org/p/publiccloud-sig-meeting See also here for all other details: https://wiki.openstack.org/wiki/PublicCloudSIG read^see you on Wednesday! felix -- Felix Kronlage-Dammers Product Owner IaaS & Operations Sovereign Cloud Stack Sovereign Cloud Stack — standardized, built and operated by many Ein Projekt der Open Source Business Alliance - Bundesverband für digitale Souveränität e.V. Tel.: +49-30-206539-205 | Matrix: @fkronlage:matrix.org | fkr(a)osb-alliance.com

1 0

[neutron] Bugs deputy report July 10th - 16th
by Miguel Lavalle 17 Jul '23

17 Jul '23

Hi, Here's last week's bugs deputy report: Critical ====== https://bugs.launchpad.net/neutron/+bug/2027817 neutron slow jobs before xena broken with ERROR: unknown environment 'slow'. Assigned to Ghanshyam Mann. Proposed fix: https://review.opendev.org/c/openstack/tempest/+/888590. In progress High ==== https://bugs.launchpad.net/neutron/+bug/2027605 [OVN][Trunk] Sometimes subport doesn't reach status ACTIVE after live-migration. Requires owner Medium ====== https://bugs.launchpad.net/neutron/+bug/2027595 [sqlalchemy-20] Check represent "network" model regardless of the elements order. Assigned to Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/888286. Merged. https://bugs.launchpad.net/neutron/+bug/2027602 [sqlalchemy-20] "FromClause.join" accepts only one model and an optional expression to define the join. Assigned to Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/888295. Merged. https://bugs.launchpad.net/neutron/+bug/2027604 [sqlalchemy-20] The Query.get() method is considered legacy as of the 1.x series of SQLAlchemy and becomes a legacy construct in 2.0. Assigned to Rodolfo Alonso https://bugs.launchpad.net/neutron/+bug/2027610 [sqlalchemy-20] Error in "test_notify_port_status_all_values" using an wrong OVO parameter Assigned to Rodolfo Alonso. Proposed fix: https://review.opendev.org/c/openstack/neutron/+/888305. Merged RFEs ==== https://bugs.launchpad.net/neutron/+bug/2027742 [RFE] unmanaged dynamic router resources - OVN Heads up from the Ironic team: ======================== https://bugs.launchpad.net/neutron/+bug/2026757. dnsmasq on Ubuntu Jammy/Lunar crashes on neutron-dhcp-agent updates. Julia Kreger indicates that most likely this is not a neutron bug. However, she believes we might encounter it at some point. Yatin Karel is watching it. Bringing it up just for visibility to the Neutron team

1 0

[kolla-ansible][octavia]
by Nguyễn Hữu Khôi 15 Jul '23

15 Jul '23

Hello guys. I am using openstack yoga by kolla-ansible but when i reconfigure octavia then it stucked at TASK [octavia : Create loadbalancer management network] \"Unable to create the network. The VLAN xxx on physical network physnet1 is in use.\ I see that it cannot create lb network because someone used it. Will we need to change some kolla-ansible tasks to fix it? Thank you very much. Nguyen Huu Khoi

2 3

[neutron] Neutron drivers meeting
by Rodolfo Alonso Hernandez 14 Jul '23

14 Jul '23

Hello Neutrinos: Today's meeting will be cancelled due to the lack of quorum. At least three of our drivers members [1] are in PTO today, thus we'll postpone the requested RFE discussion [2] to next week. Sorry for the inconveniences. Regards. [1]https://launchpad.net/~neutron-drivers/+members [2]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0

[cinder] I/O throttling using blkio cgroups
by Jon Bernard 14 Jul '23

14 Jul '23

I started to look at a cinder bug [1] to migrate our cgroup usage from v1 to v2 and I'm wondering: is anyone using this feature to throttle cinder's local I/O per cinder-volume instance? Docker since 20.10 and kubernetes since version 1.19 support cgroup v2. Since cinder services are deployed within containers, I think it makes sense to consider implementing I/O throttling at the container-level as opposed to application-specific code within a particular service (cinder in this case, but I believe glance also leverages cgroups). That would mean instead of updating our implementation to use the v2 interface, we could drop our cgroup code entirely in favor of a more generic solution that could be applied to any container that performs local I/O. Would anyone be opposed to this idea? Thoughts? 1: https://bugs.launchpad.net/cinder/+bug/1942203 -- Jon

1 0

SNAT failure with OVN under Antelope
by Gary Molenkamp 13 Jul '23

13 Jul '23

Good morning, I'm having a problem with snat routing under OVN but I'm not sure if something is mis-configured or just my understanding of how OVN is architected is wrong. I've built a Zed cloud, since upgraded to Antelope, using the Neutron Manual install method here: https://docs.openstack.org/neutron/latest/install/ovn/manual_install.html I'm using a multi-tenent configuration using geneve and the flat provider network is present on each hypervisor. Each hypervisor is connected to the physical provider network, along with the tenent network and is tagged as an external chassis under OVN. br-int exists, as does br-provider ovs-vsctl set open . external-ids:ovn-cms-options=enable-chassis-as-gw For most cases, distributed FIP based connectivity is working without issue, but I'm having an issue where VMs without a FIP are not always able to use the SNAT services of the tenent network router. Scenario: Internal network named cs3319: with subnet 172.31.100.0/23 Has a router named cs3319_router with external gateway set (snat enabled) This network has 3 vms: - #1 has a FIP and can be accessed externally - #2 has no FIP, can be accessed via VM1 and can access external resources via SNAT (ie OS repos, DNS, etc) - #3 has no FIP, can be accessed via VM1 but has no external SNAT connectivity From what I can tell, the chassis config is correct, compute05 is the hypervisor and the faulty VM has a port binding on this hypervisor: ovn-sbctl show ... Chassis "8e0fa17c-e480-4b60-9015-bd8833412561" hostname: compute05.cloud.sci.uwo.ca Encap geneve ip: "192.168.0.105" options: {csum="true"} Port_Binding "7a5257eb-caea-45bf-b48c-620c5dff4b39" Port_Binding "50e16602-78e6-429b-8c2f-e7e838ece1b4" Port_Binding "f121c9f4-c3fe-4ea9-b754-a809be95a3fd" The router has the candidate gateways, and the snat set: ovn-nbctl show 92df19a7-4ebe-43ea-b233-f4e9f5a46e7c router 92df19a7-4ebe-43ea-b233-f4e9f5a46e7c (neutron-389439b5-07f8-44b6-a35b-c76651b48be5) (aka cs3319_public_router) port lrp-44ae1753-845e-4822-9e3d-a41e0469e257 mac: "fa:16:3e:9a:db:d8" networks: ["129.100.21.94/22"] gateway chassis: [5c039d38-70b2-4ee6-9df1-596f82c68106 99facd23-ad17-4b68-a8c2-1ff6da15ac5f 1694116c-6d30-4c31-b5ea-0f411878316e 2a4bbaf9-228a-462e-8970-0cdbf59086e6 9332c61b-93e1-4a70-9547-701a014bfd98] port lrp-509bba37-fa06-42d6-9210-2342045490db mac: "fa:16:3e:ff:0f:3b" networks: ["172.31.100.1/23"] nat 11e0565a-4695-4f67-b4ee-101f1b1b9a4f external ip: "129.100.21.94" logical ip: "172.31.100.0/23" type: "snat" nat 21e4be02-d81c-46e8-8fa8-3f94edb4aed1 external ip: "129.100.21.87" logical ip: "172.31.100.49" type: "dnat_and_snat" Each network agent on the hypervisors shows the ovn controller up : OVN Controller Gateway agent | compute05.cloud.sci.uwo.ca | | :-) | UP | ovn-controller The ovs vswitch on the hypervisor looks correct afaict and ovn ports bfd status are all forwarding to other hypervisors. ie: Port ovn-2a4bba-0 Interface ovn-2a4bba-0 type: geneve options: {csum="true", key=flow, remote_ip="192.168.0.106"} bfd_status: {diagnostic="No Diagnostic", flap_count="1", forwarding="true", remote_diagnostic="No Diagnostic", remote_state=up, state=up} Any advice on where to look would be appreciated. PS. Version info: Neutron 22.0.0-1 OVN 22.12 neutron options: enable_distributed_floating_ip = true ovn_l3_scheduler = leastloaded Thanks Gary -- Gary Molenkamp Science Technology Services Systems/Cloud Administrator University of Western Ontario molenkam(a)uwo.ca http://sts.sci.uwo.ca (519) 661-2111 x86882 (519) 661-3566

5 14

OpenStack "C" Release Voting
by Kristin Barrientos 13 Jul '23

13 Jul '23

Hi everyone, It’s time to vote for the OpenStack “C” Release! The nominees are: - Celine - Caracal - Cashew Get your votes in! Vote here: https://civs1.civs.us/cgi-bin/vote.pl?id=E_852cf3e53467d72e&akey=1504cb6ad9… Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation

1 0

[cinder]cinder qos question.
by Nguyễn Hữu Khôi 13 Jul '23

13 Jul '23

Hello guys. I am trying to use cinder qos. But i have question. - I added qos after a volume was create and I see that qos did not apply to this volume. - i create new volume with assosiate qos then it is ok but when i changed extra specs in qos, volume keep old specs. Is there any way to make qos apply with two above cases. Many thanks.

2 5

Re: [kuryr][tacker] fails to bump the k8s version to v1.26
by Michał Dulko 13 Jul '23

13 Jul '23

On Thu, 2023-07-13 at 10:16 +0000, Ayumu Ueha (Fujitsu) wrote: > Hi Michał, > > Thanks for your reply, > > I trying it in my local machine and check logs, but I am not familiar > with troubleshooting with kubelet/kubeadm, so I do not know why it > errored as for now. > We will continue to investigate, but it would be helpful if you could > share any small findings. > > For your reference, I will attach a log of the kubelet.service when > it failed in my local environment. Looks like kube-apiserver and kube-scheduler are failing to start. Checking logs of them would be my next idea. You might want to use `crictl` to find them. > Thank you. > > Best Regards, > Ueha > > -----Original Message----- > From: Michał Dulko <mdulko(a)redhat.com> > Sent: Thursday, July 13, 2023 3:48 PM > To: openstack-discuss(a)lists.openstack.org > Subject: Re: [kuryr][tacker] fails to bump the k8s version to v1.26 > > On Tue, 2023-07-04 at 07:53 +0000, Ayumu Ueha (Fujitsu) wrote: > > > > > > > > Hi kuryr-kubetenetes team, > > > > Tacker uses the kuryr-kubernetes as the setup for the kubernetes > > Environment. > > In the Bobcat version, we will bump the supported version of > > kubenetes > > to 1.26.6, and when I tried to bump the version in devstack's > > local.conf by the patch [1], the following error occurred. (please > > refer the details to Zuul log > > [2]) > > ============================== > > [wait-control-plane] Waiting for the kubelet to boot up the control > > plane as static Pods from directory "/etc/kubernetes/manifests". > > This > > can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. > > > > Unfortunately, an error has occurred: > > timed out waiting for the condition > > > > This error is likely caused by: > > - The kubelet is not running > > - The kubelet is unhealthy due to a misconfiguration > > of > > the node in some way (required cgroups disabled) > > > > If you are on a systemd-powered system, you can try to troubleshoot > > the error with the following commands: > > - 'systemctl status kubelet' > > - 'journalctl -xeu kubelet' > > > > Additionally, a control plane component may have crashed or exited > > when started by the container runtime. > > To troubleshoot, list all containers using your preferred container > > runtimes CLI. > > Here is one example how you may list all running Kubernetes > > containers > > by using crictl: > > - 'crictl --runtime-endpoint > > unix:///var/run/crio/crio.sock ps -a | grep kube | grep -v pause' > > Once you have found the failing container, you can > > inspect its logs with: > > - 'crictl --runtime-endpoint > > unix:///var/run/crio/crio.sock logs CONTAINERID' > > error execution phase wait-control-plane: couldn't initialize a > > Kubernetes cluster ============================== > > > > I know it is not yet supporting version 1.26 of the K8S at the > > kuryr- > > kubernetes as for now, but do you know any way to avoid the above > > error? > > I suspect this is due to a change from version 1.25, but I'm not > > sure > > which one is affecting... > > > > Also, when will you support the K8S v 1.26 at the kuryr-kubernetes? > > Will the Bobcat release support it? > > Please kindly let me know if you know anything. Thank you. > > Kuryr-kubernetes itself supports K8s 1.26 and 1.27 as we test it with > these versions as part of OpenShift. What doesn't seem to work is the > DevStack plugin, I guess due to some breaking changes in kubelet or > kubeadm. > > I can advise to try to deploy DevStack with these settings on your > local machine and then you should have access to all the logs > required to figure out what is causing the problem with the version > bump. > > > [1] https://review.opendev.org/c/openstack/tacker/+/886935 > > Change the parameters: > > KURYR_KUBERNETES_VERSION: 1.25.6 to 1.26.6 > > CRIO_VERSION: 1.25 to 1.26 > > [2] > > https://zuul.opendev.org/t/openstack/build/1a4061da74e640368da133ba219 > > b54d9/log/controller-k8s/logs/devstacklog.txt#9761-9816 > > > > Best Regards, > > Ueha > >

1 0

[cinder-backup][kolla] mix cinder-volume and cinder-backup types (nfs, ceph)
by garcetto 13 Jul '23

13 Jul '23

good evening, it is possible to mix cinder-volume backend (say ceph) and cinder-backup backend (say nfs)? and also do incrementals backup? thank you

3 3

[kuryr][tacker] fails to bump the k8s version to v1.26
by Ayumu Ueha (Fujitsu) 13 Jul '23

13 Jul '23

Hi kuryr-kubetenetes team, Tacker uses the kuryr-kubernetes as the setup for the kubernetes Environment. In the Bobcat version, we will bump the supported version of kubenetes to 1.26.6, and when I tried to bump the version in devstack's local.conf by the patch [1], the following error occurred. (please refer the details to Zuul log [2]) ============================== [wait-control-plane] Waiting for the kubelet to boot up the control plane as static Pods from directory "/etc/kubernetes/manifests". This can take up to 4m0s [kubelet-check] Initial timeout of 40s passed. Unfortunately, an error has occurred: timed out waiting for the condition This error is likely caused by: - The kubelet is not running - The kubelet is unhealthy due to a misconfiguration of the node in some way (required cgroups disabled) If you are on a systemd-powered system, you can try to troubleshoot the error with the following commands: - 'systemctl status kubelet' - 'journalctl -xeu kubelet' Additionally, a control plane component may have crashed or exited when started by the container runtime. To troubleshoot, list all containers using your preferred container runtimes CLI. Here is one example how you may list all running Kubernetes containers by using crictl: - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock ps -a | grep kube | grep -v pause' Once you have found the failing container, you can inspect its logs with: - 'crictl --runtime-endpoint unix:///var/run/crio/crio.sock logs CONTAINERID' error execution phase wait-control-plane: couldn't initialize a Kubernetes cluster ============================== I know it is not yet supporting version 1.26 of the K8S at the kuryr-kubernetes as for now, but do you know any way to avoid the above error? I suspect this is due to a change from version 1.25, but I'm not sure which one is affecting... Also, when will you support the K8S v 1.26 at the kuryr-kubernetes? Will the Bobcat release support it? Please kindly let me know if you know anything. Thank you. [1] https://review.opendev.org/c/openstack/tacker/+/886935 Change the parameters: KURYR_KUBERNETES_VERSION: 1.25.6 to 1.26.6 CRIO_VERSION: 1.25 to 1.26 [2] https://zuul.opendev.org/t/openstack/build/1a4061da74e640368da133ba219b54d9… Best Regards, Ueha

2 2

[horizon][keystone] Adding different rules in the same protocol for federated logon
by James Leong 13 Jul '23

13 Jul '23

Hi all, I have yoga version openstack with the deployment tool of kolla-ansible. I am trying to combine different mapping rules such as allowing user to login to different domain. However, I am not able to do that in a single JSON file. When I try to include different rule in the same JSON file, only the first rule is being considered. Is there a way to allow multiple rule to redirect user to their account in a different domain. Best, James

2 4

[Glance] Cancelling Weekly Meeting 13th July
by Pranali Deore 12 Jul '23

12 Jul '23

Hello, I will not be around during the meeting time due to some personal reasons and there is nothing much on agenda, so cancelling the meeting for this week. Let's meet next week on 20th July !! Thanks, Pranali D

1 0

Need help
by Gk Gk 12 Jul '23

12 Jul '23

Hi All, If I use the openstacksdk to connect to an openstack cloud, I have to use clouds.yaml file for specifying the cloud configuration which includes username and password as well. Since its a plain text file, how can I mask the password mentioned in clouds.yaml file for security purposes? Thanks Kumar

5 5

Cinder Bug Report 2023-07-12
by Sofia Enriquez 12 Jul '23

12 Jul '23

Hello Argonauts, Cinder Bug Meeting Etherpad <https://etherpad.opendev.org/p/cinder-bug-squad-meeting> High - Cannot create a volume from a backup <https://bugs.launchpad.net/cinder/+bug/2025277>. - *Status*: Fix proposed to master <https://review.opendev.org/c/openstack/cinder/+/887443>, but it needs a release note. - Cinder backup appears as down <https://bugs.launchpad.net/cinder/+bug/2026877>. - *Status*: Unassigned. Medium - [api] Sort by boot and pass in the marker Exception <https://bugs.launchpad.net/cinder/+bug/2027532>. - *Status*: Unassigned. - [os-brick] _get_host_uuid incompatible with btrfs snapshots <https://bugs.launchpad.net/os-brick/+bug/2026257>. - *Status*: Unassigned. Workaround on the launchpad bug report. Low - HPE 3par: Issue observed during retype/migrate. <https://bugs.launchpad.net/cinder/+bug/2026718> - *Status*: Unassigned. Cheers, -- Sofía Enriquez she/her Software Engineer Red Hat PnT <https://www.redhat.com> IRC: @enriquetaso @RedHat <https://twitter.com/redhat> Red Hat <https://www.linkedin.com/company/red-hat> Red Hat <https://www.facebook.com/RedHatInc> <https://www.redhat.com>

1 0

instance console something went wrong, connection is closed | Wallaby DCN
by Swogat Pradhan 12 Jul '23

12 Jul '23

Hi, I am creating instances in my DCN site and i am unable to get the console sometimes, error: something went wrong, connection is closed I have 3 instances now running on my hci02 node and there is console access on 1 of the vm's and the rest two i am not getting the console, i have used the same flavor, same image same security group for the VM's. Please suggest what can be done. With regards, Swogat Pradhan

3 13

Need help
by Gk Gk 11 Jul '23

11 Jul '23

Hi Folks, I work on openstack platforms. Can anyone suggest some good books or resources which will help me in becoming a cloud solutions architect ? What would this path look like ? What would be my responsibilities ? Thanks Kumar

2 3

[neutron] Bug deputy 3 July - 9 July
by Rodolfo Alonso Hernandez 11 Jul '23

11 Jul '23

Hello Neutrinos: This is the list of patches opened last week: Critical: * https://bugs.launchpad.net/neutron/+bug/2025753. "Job openstack-tox-py310-with-neutron-lib-master is failing every day 02.07.2023". Patch proposed: https://review.opendev.org/c/openstack/neutron/+/887942 High: * https://bugs.launchpad.net/neutron/+bug/2025618. "[UT] "test_negative_update_floatingip_port_forwarding" does not create the ports in the correct subnet". Patch merged: https://review.opendev.org/c/openstack/neutron/+/887495 * https://bugs.launchpad.net/neutron/+bug/2025946. "Neutron 504 Gateway Timeout Openstack Kolla-Ansible : Ussuri". *Not assigned*. Medium: * https://bugs.launchpad.net/neutron/+bug/2025637. "[ovn-octavia-provider] FIP traffic not distributed for members' FIP after lb cascade deletion". Patch merged: https://review.opendev.org/c/openstack/ovn-octavia-provider/+/886611 * https://bugs.launchpad.net/neutron/+bug/2026122. "neutron-l3-agent permission denied when execute (list_network_namespaces)". Assigned, under investigation right now, there is also a configuration fix tested. * https://bugs.launchpad.net/neutron/+bug/2026182. "Add support for the service role in neutron API policies". Assigned, under development right now. * https://bugs.launchpad.net/neutron/+bug/2026264. " "--target" parameter in "network log object create" is only expected for ML2/OVS". Assigned, under development right now. * https://bugs.launchpad.net/neutron/+bug/2026775. "Metadata agents do not parse X-Forwarded-For headers properly". Patch proposed: https://review.opendev.org/c/openstack/neutron/+/888067. * https://bugs.launchpad.net/neutron/+bug/2026825. " [OVN] Disable LSP_OPTIONS_MCAST_FLOOD_REPORTS for LSPs". Patch proposed: https://review.opendev.org/c/openstack/neutron/+/888127. Low: * https://bugs.launchpad.net/neutron/+bug/2026547. "[RBAC] Remove "AddressScope" shared field". Assigned. Wishlist: * https://bugs.launchpad.net/neutron/+bug/2025969. "bgp os_ken ssh_console support". *Not assigned*. * https://bugs.launchpad.net/neutron/+bug/2026286. "Missing documentation for notification events". *Not assigned.* * https://bugs.launchpad.net/neutron/+bug/2026489. " [RFE][quota] Implement "router_route" quota using the Neutron quota engine, replacing "max_routes" config option". *Not assigned.* Incomplete: * https://bugs.launchpad.net/neutron/+bug/2025587. "neutron request body or parameters cause "Internal Server Error"" Regards.

1 0

[ptl] Need PTL volunteer for OpenStack Vitrage
by Ghanshyam Mann 11 Jul '23

11 Jul '23

Hi Eyal, I am reaching out to you as you were PTL for OpenStack Vitrage project in the last cycle. There is no PTL candidate for the next cycle (2023.2), and it is on the leaderless project list. Please check if you or anyone you know would like to lead this project. - https://etherpad.opendev.org/p/2023.2-leaderless Also, if anyone else would like to help leading this project, this is time to let TC knows. -gmann

3 3

[oslo] HTTP base direct RPC oslo.messaging driver contribution
by 室井雅仁 11 Jul '23

11 Jul '23

Hi oslo team, We'd like to contribute HTTP base direct RPC driver to the oslo.messaging community. We have developed the HTTP base driver internally. We have been using the driver in the production with over 10K hypervisors now. I checked the IRC meeting log of the oslo team[1], but there is no regluar meeting in 2023. Is it okay to submit oslo-spec[2] to propose the driver directly, or is there another good place to discuss the feature before submitting a spec? 1. https://meetings.opendev.org/#Oslo_Team_Meeting 2. https://opendev.org/openstack/oslo-specs best regards, Masahito

8 13

vPTG October 2023 Dates & Registration Kickoff
by Kendall Nelson 11 Jul '23

11 Jul '23

Hello Everyone! The next PTG[1] will take place October 23-27, 2023 virtually! Registration is now open[2] and free for all to attend. Keep an eye out in the coming weeks because I will send along the team sign up to collect groups interested in participating. Thanks! -Kendall Nelson(diablo_rojo) [1] https://openinfra.dev/ptg/ [2] http://ptg2023.openinfra.dev/

1 0

[tc] Technical Committee next weekly meeting on July 11, 2023
by Nikolla, Kristi 10 Jul '23

10 Jul '23

Hi all, This is a reminder that the next weekly Technical Committee meeting is to be held on Tuesday, July 11, 2023 at 1800 UTC on Zoom. Meeting link and agenda can be found at https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee#Next_Meeting Please find below the current agenda for the meeting. • Roll call • Follow up on past action items • knikolla will write alternative proposals for Extended Maintenance. • Extended Maintenance • https://etherpad.opendev.org/p/openstack-exteneded-maintenance • Gate health check • Open Discussion and Reviews • https://review.opendev.org/q/projects:openstack/governance+is:open Thank you, Kristi Nikolla

1 0

[cinder] Extending driver merge deadline to 14th July
by Rajat Dhasmana 10 Jul '23

10 Jul '23

Hi, As discussed in the last cinder meeting[1], due to less review bandwidth, Cinder drivers proposed for 2023.2 Bobcat didn't get much reviews. As a result, we will be extending the deadline to this week and the new date for driver merge deadline will be 14 July, 2023. Here is a list of all the drivers proposed for 2023.1 Bobcat cycle[2], feel free to review and add your comment even if you are not a core reviewer, all reviews are appreciated. [1] https://meetings.opendev.org/irclogs/%23openstack-meeting-alt/%23openstack-… [2] https://etherpad.opendev.org/p/cinder-2023-2-bobcat-drivers Thanks Rajat Dhasmana

1 0

[keystone] Weekly Meeting Move Proposal
by Dave Wilde 10 Jul '23

10 Jul '23

Hello, I'd like to propose moving the Keystone weekly meeting from Tuesdays at 15:00 UTC to Wednesdays at 15:00 UTC. Please let me know any feedback on this proposal, without objection our first Wednesday meeting will be 19-Jul-2023 at 15:00 UTC. Thank you! /Dave

1 0

[tacker][oslo][kolla-ansible] Issue reports for session management in oslo when using kolla-ansible
by Yuta Kazato 10 Jul '23

10 Jul '23

Hi, OpenStack Community. Our Tacker team found an issue with session management in oslo and sqlalchemy when using kolla-ansible. The following report details the issue. [Issue details] - In the case that we build Tacker and other components by using kolla-ansible, an issue with session management in oslo and sqlalchemy occurs when Tacker update the DB with `with context.session.begin(subtransactions=True):` - code: https://github.com/openstack/tacker/blob/7eaf841525d7ead30d7883abbde6c4fb86… - error details === * cli logs tacker@tacker:~/tacker-nic_internal/samples/takt-docker-sample/deployments/tacker$ curl -s -X PATCH http://"10.244.121.249:9890"/vnffm/v1/alarms/f23f39cc-f9a4-45bf-9a87-2f8148541b0a \ -H "Content-type: application/json" \ -H "Version: 1.3.0" \ -d "{\"ackState\": \"ACKNOWLEDGED\"}" \ -w "\n%{http_code}\n" {"status": 500, "detail": "this TransactionFactory is already started"} * tacker logs 2023-06-12 05:28:56.028 1 INFO tacker.service [-] Tacker service started, listening on 0.0.0.0:9890 2023-06-12 05:28:56.028 1 INFO tacker.wsgi [-] (1) wsgi starting up on http://0.0.0.0:9890 2023-06-12 05:30:07.021 1 DEBUG tacker.wsgi [-] (1) accepted ('10.100.2.3', 52508) server /usr/local/lib/python3.10/dist-packages/eventlet/wsgi.py:1004 2023-06-12 05:30:07.027 1 INFO tacker.sol_refactored.api.wsgi [-] PATCH http://10.244.121.249:9890/vnffm/v1/alarms/f23f39cc-f9a4-45bf-9a87-2f814854… 2023-06-12 05:30:07.052 1 DEBUG tacker.sol_refactored.common.coordinate [None req-a19d9e85-ba9f-4f6b-938a-3fde04bce62b - - - - - -] resources f23f39cc-f9a4-45bf-9a87-2f8148541b0a locked. wrapper /usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/common/coordinate.py:95 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi [None req-a19d9e85-ba9f-4f6b-938a-3fde04bce62b - - - - - -] Unknown error: oslo_db.sqlalchemy.enginefacade.AlreadyStartedError: this TransactionFactory is already started 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi Traceback (most recent call last): 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/api/wsgi.py", line 134, in __call__ 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi result = self._dispatch(request, action, args) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/api/wsgi.py", line 170, in _dispatch 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi return controller_method(request=request, **action_args) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/api/validator.py", line 48, in wrapper 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi return func(*args, **kwargs) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/common/coordinate.py", line 96, in wrapper 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi return func(*args, **kwargs) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/sol_refactored/controller/vnffm_v1.py", line 101, in update 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi with context.session.begin(subtransactions=True): 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/context.py", line 183, in session 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi self._session = db_api.get_session() 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/db/api.py", line 46, in get_session 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi facade = _create_facade_lazily() 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/tacker/db/api.py", line 32, in _create_facade_lazily 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi context_manager.configure(sqlite_fk=True, **cfg.CONF.database) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 812, in configure 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi self._factory.configure(**kw) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 319, in configure 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi self._configure(False, kw) 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi File "/usr/local/lib/python3.10/dist-packages/oslo_db/sqlalchemy/enginefacade.py", line 332, in _configure 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi raise AlreadyStartedError( 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi oslo_db.sqlalchemy.enginefacade.AlreadyStartedError: this TransactionFactory is already started 2023-06-12 05:30:07.157 1 ERROR tacker.sol_refactored.api.wsgi 2023-06-12 05:30:07.159 1 INFO tacker.sol_refactored.api.wsgi [None req-a19d9e85-ba9f-4f6b-938a-3fde04bce62b - - - - - -] http://10.244.121.249:9890/vnffm/v1/alarms/f23f39cc-f9a4-45bf-9a87-2f814854… returned with HTTP 500 2023-06-12 05:30:07.159 1 INFO tacker.wsgi [None req-a19d9e85-ba9f-4f6b-938a-3fde04bce62b - - - - - -] 10.100.2.3 - - [12/Jun/2023 05:30:07] "PATCH /vnffm/v1/alarms/f23f39cc-f9a4-45bf-9a87-2f8148541b0a HTTP/1.1" 500 309 0.135927 === - This issue does not occur with a regular Tacker Deploy (manual deployment or devstack), but kola-ansible or container envs only. [Temporary solution] - Remove the codes, but we investigate to check if there are any other impacts. To solve these problems will require the cooperation of not only Tacker but also oslo and kolla-ansible projects. We invite your feedback on these problems and especially welcome your share of similar situation info and solutions. Best Regards, Yuta -- Yuta Kazato NTT Network Innovation Center. tel: +81-422-59-6754 mail: yuta.kazato(a)ntt.com

1 0

TripleO | OS-net config logs | openstack overcloud node provision
by Lokendra Rathour 09 Jul '23

09 Jul '23

Hi Team, Scenario: We were performing overcloud deployment. During this, we were continuously facing the following errors on two of the nodes out of the four nodes: PLAY [Overcloud Node Grow Volumes] ********************************************* 2023-06-02 19:07:33.483504 | 52540049-4965-0177-e681-00000000000a | TASK | Wait for provisioned nodes to boot [WARNING]: Unhandled error in Python interpreter discovery for host overcloud- controller-0: Failed to connect to the host via ssh: ssh: connect to host 30.30.30.219 port 22: No route to host [WARNING]: Unhandled error in Python interpreter discovery for host overcloud- controller-0: Failed to connect to the host via ssh: ssh: connect to host 30.30.30.219 port 22: Connection refused We faced this error when we were performing overcloud node provisioning. To resolve this issue, we had to update the image and create a user to access the nodes. We were able to access the nodes via this user through the BMC console. This implied that OS was getting correctly installed on the node. Once we accessed the logs on the failed node we came across the following logs: 2023-06-13 09:02:16.120 INFO os_net_config.utils._ordered_nics Finding active nics 2023-06-13 09:02:16.120 INFO os_net_config.utils._ordered_nics eno1 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens2f0 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno4 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno2 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens1f0 is not an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics lo is not an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens2f1 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics eno3 is an embedded active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics ens1f1 is an active nic 2023-06-13 09:02:16.121 INFO os_net_config.utils._ordered_nics No DPDK mapping available in path (/var/lib/os-net-config/dpdk_mapping.yaml) 2023-06-13 09:02:16.122 INFO os_net_config.utils._ordered_nics Active nics are ['eno1', 'eno2', 'eno3', 'eno4', 'ens1f1', 'ens2f0', 'ens2f1'] 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic2 mapped to: eno2 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic6 mapped to: ens2f0 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic5 mapped to: ens1f1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic3 mapped to: eno3 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic7 mapped to: ens2f1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic1 mapped to: eno1 2023-06-13 09:02:16.122 INFO os_net_config.objects.mapped_nics nic4 mapped to: eno4 As we can see in the above logs, only 7 nics are active. But as per our bond config file, we required 8 interfaces: a. nic-1 for management IP b. nic-3 and nic-4 for the br-tenant-2 network. c. nic-5 and nic-6 for the br-tenant network. d. nic-7 and nic 8 for the br-ex network. The wire was faulty in this case and hence only 7 interfaces were available Query: The logs which are visible while we run "openstack overcloud node provision" are not very informative. Is there any way where we could get these logs on the undercloud node itself? This could save some time for future references -- ~ Lokendra skype: lokendrarathour

2 1

[release] Release countdown for week R-12, Jul 10-14
by Herve Beraud 07 Jul '23

07 Jul '23

Development Focus ----------------- We are now past the Bobcat-2 milestone, and entering the last development phase of the cycle. Teams should be focused on implementing planned work for the cycle. Now is a good time to review those plans and reprioritize anything if needed based on the what progress has been made and what looks realistic to complete in the next few weeks. General Information ------------------- Looking ahead to the end of the release cycle, please be aware of the feature freeze dates. Those vary depending on deliverable type: * General libraries (except client libraries) need to have their last feature release before Non-client library freeze (August 24). Their stable branches are cut early. * Client libraries (think python-*client libraries) need to have their last feature release before Client library freeze (August 31) * Deliverables following a cycle-with-rc model (that would be most services) observe a Feature freeze on that same date, August 31. Any feature addition beyond that date should be discussed on the mailing-list and get PTL approval. After feature freeze, cycle-with-rc deliverables need to produce a first release candidate (and a stable branch) before RC1 deadline (September 14) * Deliverables following cycle-with-intermediary model can release as necessary, but in all cases before Final RC deadline (September 28) Upcoming Deadlines & Dates -------------------------- Non-client library freeze: August 24 (R-6 week) Client library freeze: August 31 (R-5 week) Bobcat-3 milestone: August 31 (R-5 week) Final 2023.2 Bobcat release: October 4th, 2023 -- Hervé Beraud Senior Software Engineer at Red Hat irc: hberaud https://github.com/4383/

1 0

[neutron] Neutron drivers meeting cancelled
by Rodolfo Alonso Hernandez 07 Jul '23

07 Jul '23

Hello Neutrinos: Due to the lack of agenda [1], today's meeting is cancelled. Have a nice weekend! [1]https://wiki.openstack.org/wiki/Meetings/NeutronDrivers

1 0