March 2023 - openstack-discuss - lists.openstack.org

Wallaby on Ubuntu 20.04, Neutron 18.6.0 neutron-dhcp-agent RPC unusually slow
by Zakhar Kirpichenko 14 Mar '23

14 Mar '23

Hi! We're running Openstack Wallaby on Ubuntu 20.04, 3 high-performance infra nodes with a RabbitMQ cluster. I updated Neutron components to version 18.6.0, which recently became available in the cloud repository ( http://ubuntu-cloud.archive.canonical.com/ubuntu focal-updates/wallaby main). The exact package versions updated are as follows: Install: libunbound8:amd64 (1.9.4-2ubuntu1.4, automatic), openvswitch-common:amd64 (2.15.2-0ubuntu1~cloud0, automatic) Upgrade: neutron-common:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), python3-werkzeug:amd64 (0.16.1+dfsg1-2, 0.16.1+dfsg1-2ubuntu0.1), neutron-dhcp-agent:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), neutron-l3-agent:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), python3-neutron:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), neutron-server:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), neutron-plugin-ml2:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), neutron-metadata-agent:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1), neutron-linuxbridge-agent:amd64 (2:18.5.0-0ubuntu1~cloud0, 2:18.6.0-0ubuntu1~cloud1) Installed Neutron packages: ii neutron-common 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - common ii neutron-dhcp-agent 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - DHCP agent Firewall-as-a-Service driver for OpenStack Neutron ii neutron-l3-agent 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - l3 agent ii neutron-linuxbridge-agent 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - linuxbridge agent ii neutron-metadata-agent 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - metadata agent ii neutron-plugin-ml2 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - ML2 plugin ii neutron-server 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - server ii python3-neutron 2:18.6.0-0ubuntu1~cloud1 all Neutron is a virtual network service for Openstack - Python library ii python3-neutron-lib 2.10.1-0ubuntu1~cloud0 all Neutron shared routines and utilities - Python 3.x ii python3-neutronclient 1:7.2.1-0ubuntu1~cloud0 all client API library for Neutron - Python 3.x Normally this would be an easy update, but this time neutron-dhcp-agent doesn't work properly: 2023-03-14 05:44:27.572 2534501 INFO neutron.agent.dhcp.agent [req-4a362701-cc1f-4b9d-87e6-045b6a388709 - - - - -] Synchronizing state complete 2023-03-14 05:44:38.868 2534501 ERROR neutron_lib.rpc [req-cb1dc604-1372-44cd-bc06-09496ed5f68f - - - - -] Timeout in RPC method dhcp_ready_on_ports. Waiting for 55 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID bd97110b004e413cb2d6b05d9fb3b57c 2023-03-14 05:44:38.871 2534501 WARNING neutron_lib.rpc [req-cb1dc604-1372-44cd-bc06-09496ed5f68f - - - - -] Increasing timeout for dhcp_ready_on_ports calls to 120 seconds. Restart the agent to restore it to the default value.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID bd97110b004e413cb2d6b05d9fb3b57c 2023-03-14 05:45:34.244 2534501 ERROR neutron.agent.dhcp.agent [req-cb1dc604-1372-44cd-bc06-09496ed5f68f - - - - -] Timeout notifying server of ports ready. Retrying...: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID bd97110b004e413cb2d6b05d9fb3b57c 2023-03-14 05:47:10.876 2534501 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : bd97110b004e413cb2d6b05d9fb3b57c 2023-03-14 05:47:34.353 2534501 ERROR neutron_lib.rpc [req-607a9252-49b1-4043-aa0d-2457b78dc99e - - - - -] Timeout in RPC method dhcp_ready_on_ports. Waiting for 27 seconds before next attempt. If the server is not down, consider increasing the rpc_response_timeout option as Neutron server(s) may be overloaded and unable to respond quickly enough.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID f254f735998243c4b0a58ce95c974534 2023-03-14 05:47:34.354 2534501 WARNING neutron_lib.rpc [req-607a9252-49b1-4043-aa0d-2457b78dc99e - - - - -] Increasing timeout for dhcp_ready_on_ports calls to 240 seconds. Restart the agent to restore it to the default value.: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID f254f735998243c4b0a58ce95c974534 2023-03-14 05:47:46.681 2534501 INFO oslo_messaging._drivers.amqpdriver [-] No calling threads waiting for msg_id : f254f735998243c4b0a58ce95c974534 2023-03-14 05:48:01.086 2534501 ERROR neutron.agent.dhcp.agent [req-607a9252-49b1-4043-aa0d-2457b78dc99e - - - - -] Timeout notifying server of ports ready. Retrying...: oslo_messaging.exceptions.MessagingTimeout: Timed out waiting for a reply to message ID f254f735998243c4b0a58ce95c974534 2023-03-14 05:49:45.035 2534501 INFO neutron.agent.dhcp.agent [req-5935a0d0-a981-463c-a4ea-23ccbb54c896 - - - - -] DHCP configuration for ports ... (A successful configuration here). While neutron-dhcp-agent is waiting, neutron-server log gets filled up with: neutron-server.log:2023-03-14 05:47:05.761 4171971 INFO neutron.plugins.ml2.plugin [req-cb1dc604-1372-44cd-bc06-09496ed5f68f - - - - -] Attempt 1 to provision port 18cddbb8-f3ed-4b49-9c6f-c0c67b4f7c76 ... neutron-server.log:2023-03-14 05:47:10.727 4171971 INFO neutron.plugins.ml2.plugin [req-cb1dc604-1372-44cd-bc06-09496ed5f68f - - - - -] Attempt 10 to provision port 18cddbb8-f3ed-4b49-9c6f-c0c67b4f7c76 This repeats for each port of each network neutron-dhcp-agent needs to configure. Each subsequent configuration for each network takes about 1-2 minutes, depending on the network size. With earlier Neutron versions the whole process of configuring all networks would finish in under a minute, i.e. DHCP configuration per port (and network) is several orders of magnitude slower than it should be. Once neutron-dhcp-agent finishes synchronization, it seems to work without issues although there aren't that many changes in our cloud to tell whether it's fast or slow, individual port updates seem to happen quickly. All other services are working well, RabbitMQ cluster is working well, infra nodes are not overloaded and there are no apparent issues other than this one with Neutron, thus I am inclined to think that the issue is specific to version 18.6.0 of neutron-dhcp-agent or neutron-server. I would appreciate any advice! Best regards, Zakhar

1 1

[openstack][backup] Experience for instance backup
by Nguyễn Hữu Khôi 14 Mar '23

14 Mar '23

Hello guys. I am looking for instance backup solution. I am using Cinder backup with nfs backup but it looks not too fast. I am using a 10Gbps network. I would like to know experience for best practice for instance backup solutions on Openstack. Thank you. Nguyen Huu Khoi

4 4

[neutron] Bug deputy - report from week of 6th March
by Slawek Kaplonski 14 Mar '23

14 Mar '23

Hi, Sorry for sending it so late but I was off Monday and Tuesday and I forgot about it. Here's list of new bugs reported last week: ## High * https://bugs.launchpad.net/neutron/+bug/2009678 - [OVN] The OVN agent config entry point is incorrect in setup.cfg - assigned to Rodolfo, fix proposed * https://bugs.launchpad.net/neutron/+bug/2009632 - ARP requests from ovnmeta namespaces are sent to physical interfaces of computing nodes - **not assigned yet** * https://bugs.launchpad.net/neutron/+bug/2009703 - [OVN] HW offload event "QoSMinimumBandwidthEvent" fails if the min-bw rule is removed - assigned to Rodolfo, fix proposed ## Medium * https://bugs.launchpad.net/neutron/+bug/2009509 - Large number of FIPs and subnets causes slow sync_routers response - assigned to Adam Oswick * https://bugs.launchpad.net/neutron/+bug/2009804 - [OVN] Method ``get_port_qos`` should always return 2 values - **not assigned yet, low hanging fruit bug** ## Low * https://bugs.launchpad.net/neutron/+bug/2009728 - [OVS] "permitted_ethertypes" should be validated and filtered during the OVS agent initialization - **not assigned yet, low hanging fruit bug** * https://bugs.launchpad.net/neutron/+bug/2009832 - FWaaS docs lack required packages - **not assigned yet, low hanging fruit bug** * https://bugs.launchpad.net/neutron/+bug/2009831 - VPNaaS docs lack packages to install - **not assigned yet, low hanging fruit bug** ## Incomlete * https://bugs.launchpad.net/neutron/+bug/2009807 - Not able to create external physical network - needs some more info but for now it looks more like user's issue rather than bug ## Others * https://bugs.launchpad.net/neutron/+bug/2009705 - [FWaaS ]Openstack Zed - firewall group status doesn't change to ACTIVE. - **unassigned**, it needs someone from FWaaS to look at it and ZhaoHeng is looking into it already. -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[tc][all] What's happening in the Technical Committee - 2023 March 14
by Nikolla, Kristi 14 Mar '23

14 Mar '23

Hi all, Please find below a summary for the last 2 weeks of the TC. Meetings ======= - March 1, 2023 A video recoding of the meeting is available https://www.youtube.com/watch?v=HA1owc9qGiE - March 8, 2023 Meeting notes are available https://meetings.opendev.org/meetings/tc/2023/tc.2023-03-08-15.59.html - The next Technical Committee meeting will be tomorrow, March 14, 2023 at 16:00 UTC, chaired by Jay Faulkner. Agenda is available at the link below. Please contact me or Jay if you notice missing items that should be discussed. https://wiki.openstack.org/wiki/Meetings/TechnicalCommittee Merged Changes ============== - Appoint Felipe Reyes as OpenStack_Charms PTL (formal-vote) https://review.opendev.org/c/openstack/governance/+/874971 - Appoint Liu as Senlin PTL (formal-vote) https://review.opendev.org/c/openstack/governance/+/874969 - Adding mailto link in upstream opportunities doc (documentation-change) https://review.opendev.org/c/openstack/governance/+/874968 - Move heat-translator and tosca-parser to tacker's governance (project-update) https://review.opendev.org/c/openstack/governance/+/876012 - Fix doc referencing 'admin_api' rule (typo-fix) https://review.opendev.org/c/openstack/governance/+/875860 - Correct CHAIR.rst: reflect vice-chair nom practice https://review.opendev.org/c/openstack/governance/+/875788 - Ironic program adopting x/virtualpdu https://review.opendev.org/c/openstack/governance/+/876208 - Volunteer to serve at 2023.2 TC vice chair https://review.opendev.org/c/openstack/governance/+/875787 - Add chair role to knikolla for 2023.2 https://review.opendev.org/c/openstack/governance/+/875742 Open Changes ============ There are 15 open changes to the Governance repo. https://review.opendev.org/q/project:openstack/governance+status:open Happenings ========== New TC Chair and Vice Chair ---------------------------------------- The TC has a new Chair (Kristi Nikolla) and Vice Chair (Jay Faulkner). Thank you Ghanshyam Mann for your 2 years of service as the TC chair. https://lists.openstack.org/pipermail/openstack-discuss/2023-March/032506.h… TripleO is being deprecated -------------------------------------- The TC voted to continue supporting the Zed release of TripleO, however without a project team. https://review.opendev.org/c/openstack/governance/+/877132 Leaderless projects -------------------------- We're now down to 7 projects without any PTL candidacies (Monasca, Rally, Sahara, Swift, TripleO, Vitrage, and Winstackers). The TC has started reaching out to the teams and former PTLs. If you have any information or interest, please reach out to the TC via the etherpad linked below. https://etherpad.opendev.org/p/2023.2-leaderless Release naming for projects -------------------------------------- The TC continued discussing changes related to version naming for projects. Specifically, there are two proposals about allowing projects to omit the OpenStack version when referring to their project, or flipping the OpenStack and project version order. https://review.opendev.org/c/openstack/governance/+/874484 https://review.opendev.org/c/openstack/governance/+/875942 Virtual PTG ---------------- The TC is considering scheduling the same PTG time slots for itself as the previous PTG. Namely, Thursday and Friday 15:00 - 19:00 UTC. I will be booking the slots later today. https://etherpad.opendev.org/p/tc-2023-2-ptg How to contact the TC: ================== If you would like to discuss or give feedback to TC, you can reach out to us in multiple ways: 1. Email: you can send an email with the tag [tc] on the openstack-discuss mailing list. 2. Weekly meeting: The Technical Committee conduct a weekly meeting every Thursday 16:00 UTC 3. IRC: Ping us using the 'tc-members' keyword on the #openstack-tc IRC channel on OFTC.

1 0

[ptls][Antelope] OpenInfra Live: OpenStack Antelope
by Kristin Barrientos 14 Mar '23

14 Mar '23

Hi everyone, As we get closer to the OpenStack release, I wanted to reach out to see if any PTL’s were interested in providing their Antelope cycle highlights in an OpenInfra Live[1] episode on Thursday, March 23 at 1500 UTC. Ideally, we would get 4-6 projects represented. Previous examples of OpenStack release episodes can be found here[2] <http://www.youtube.com/watch?v=hwPfjvshxOM>and here[3] <https://www.youtube.com/watch?v=MSbB3L9_MeY>. Please let me know if you’re interested and I can provide next steps. If you would like to provide a project update but that time doesn’t work for you, please share a recording with me and I can get it added to the project navigator. Thanks, Kristin Barrientos Marketing Coordinator OpenInfra Foundation [1] https://openinfra.dev/live/ [2] https://www.youtube.com/watch?v=hwPfjvshxOM [3] https://www.youtube.com/watch?v=MSbB3L9_MeY

5 4

Re: [manila] support for encryption
by Alvaro Soto 13 Mar '23

13 Mar '23

I don't think so, and it also doesn't sound like a feature that Manila should implement; try to do that in your backend storage. Cheers! On Sat, Mar 11, 2023 at 7:10 AM garcetto <garcetto(a)gmail.com> wrote: > good afternoon, > data encryption. > thank you > > Il Ven 10 Mar 2023, 21:48 Alvaro Soto <alsotoes(a)gmail.com> ha scritto: > >> You mean data or token encryption? >> >> Cheers! >> >> On Fri, Mar 10, 2023 at 9:08 AM garcetto <garcetto(a)gmail.com> wrote: >> >>> good afternoon, >>> does manila support encryption in some sort ? >>> >>> thank you >>> >> >> >> -- >> >> Alvaro Soto >> >> *Note: My work hours may not be your work hours. Please do not feel the >> need to respond during a time that is not convenient for you.* >> ---------------------------------------------------------- >> Great people talk about ideas, >> ordinary people talk about things, >> small people talk... about other people. >> > -- Alvaro Soto *Note: My work hours may not be your work hours. Please do not feel the need to respond during a time that is not convenient for you.* ---------------------------------------------------------- Great people talk about ideas, ordinary people talk about things, small people talk... about other people.

1 0

[manila] create snapshot from share not permitted
by garcetto 13 Mar '23

13 Mar '23

good morning, i am using manila and generic driver with dhss true, but cannot create snapshot from shares, any help? where can i look at? (cinder backend is a linux nfs server) thank you $ manila snapshot-create share-01 --name Snapshot1 ERROR: Snapshots cannot be created for share '2c8b1b3d-ef82-4372-94df-678539f0d843' since it does not have that capability. (HTTP 422) (Request-ID: req-cab23a46-37dc-4f2b-b26c-d6b21b7453ba)

2 1

[Diversity] Diversity and Inclusion WG Meeting reminder
by Amy Marrich 13 Mar '23

13 Mar '23

This is a reminder that the Diversity and Inclusion WG will be meeting tomorrow at 14:00 UTC in the #openinfra-diversity channel on OFTC. We hope members of all OpenInfra projects join us as we look at the Code of Conduct, and continue working on planning for the OpenInfra Summit as well as Foundation-wide diversity surveys. Thanks, Amy (spotz) 0 - https://etherpad.opendev.org/p/diversity-wg-agenda

1 0

*Migration)Red Hat Open Stack Migration Lift and Shift
by Adivya Singh 13 Mar '23

13 Mar '23

Hi Team, I am making a plan for Red hat OpenStack migration(Lift and Shift), The Director is in VM sourced in VMware, which we are migrated in other DC, But the IP and VLAN will change Will it be advisable to change the Provisioning IP and other IP , according to the New VLAN designed in the new DC, Configured in the template , Do a introspection of the new design and Configured the Red hat OpenStack, Any addition on this or guidance would be highly usefull Regards Adivya Singh

2 1

[cinder] proposing Jon Bernard for cinder core
by Rajat Dhasmana 13 Mar '23

13 Mar '23

Hello everyone, I would like to propose Jon Bernard as cinder core. Looking at the review stats for the past 60[1], 90[2], 120[3] days, he has been consistently in the top 5 reviewers with a good +/- ratio and leaving helpful comments indicating good quality of reviews. He has been managing the stable branch releases for the past 2 cycles (Zed and 2023.1) and has helped in releasing security issues as well. Jon has been part of the cinder and OpenStack community for a long time and has shown very active interest in upstream activities, be it release liaison, review contribution, attending cinder meetings and also involving in outreachy activities. He will be a very good addition to our team helping out with the review bandwidth and adding valuable input in our discussions. I will leave this thread open for a week and if there are no objections, I will add Jon Bernard to the cinder core team. [1] https://www.stackalytics.io/report/contribution?module=cinder-group&project… [2] https://www.stackalytics.io/report/contribution?module=cinder-group&project… [3] https://www.stackalytics.io/report/contribution?module=cinder-group&project… Thanks Rajat Dhasmana

5 5