April 2022 - openstack-discuss - lists.openstack.org

[neutron][ops] OVN scale issues and reviving midonet plugin
by Sam Morrison 12 Apr '22

12 Apr '22

Hi, We recently tried to migrate our install from ML2 midonet -> OVN driver on our Victoria Openstack install with ~1000 hypervisors. Victoria was the last release where midonet plugin was supported so was a good motivation to move. Unfortunately when we changed the neutron-server config over to use OVN in our production install everything went very bad. We got lots of things like: ovsdbapp.exceptions.TimeoutException: Commands [<neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.commands.CheckRevisionNumberCommand object at 0x7fa3b84bd8b0>, <neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.commands.SetLSwitchPortCommand object at 0x7fa37b04cf40>, <ovsdbapp.schema.ovn_northbound.commands.PgDelPortCommand object at 0x7fa375ae0fd0>] exceeded timeout 180 seconds, cause: TXN queue is ful ovsdbapp.exceptions.TimeoutException: Commands [<neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.commands.CheckRevisionNumberCommand object at 0x7f0ee5197dc0>, <neutron.plugins.ml2.drivers.ovn.mech_driver.ovsdb.commands.SetLSwitchPortCommand object at 0x7f0ee50a92e0>, <ovsdbapp.schema.ovn_northbound.commands.PgAddPortCommand object at 0x7f0ee5181760>] exceeded timeout 180 seconds, cause: Result queue is empty ovsdbapp.exceptions.TimeoutException: Commands [<ovsdbapp.backend.ovs_idl.command.DbDestroyCommand object at 0x7fcd99f9ea90>] exceeded timeout 180 seconds, cause: Result queue is empt among other things, which forced us to roll back. Our next approach is to get everything up to yoga and try again, (with some better live testing before we make the switch somehow) In the mean time we have revived the networking-midonet plugin in our own branch but just wanted to check to see if anyone else is in this situation and are or have looked into running midonet on wallaby and beyond? Cheers, Sam

1 0

[manila][ussuri] Manila return 404 resource not found after simple install
by CHANU ROMAIN 11 Apr '22

11 Apr '22

Hello, I tried to install manila on Ubuntu servers and I'm stuck during the first steps. I followed this documentation: https://docs.openstack.org/manila/ussuri/install/install-controller-ubuntu.… When I tried to verify the operation I always get the same message: # manila service-list ERROR: Not Found (HTTP 404) In manila-api.log 2022-04-11 12:12:56.533 125 INFO eventlet.wsgi.server [req-c03111de- 46bb-40e1-9440-2f62a94a61f0 ee3ae5e43b4c6246c4d95264b2978bf05dce57920b6b852d62b450ad6e7fb392 59bc7ad25c364c809a97c1a55caec161 - - -] IP_CLIENT,IP_LB "GET /v2/services HTTP/1.1" status: 404 len: 228 time: 0.1233838 If I use -d #manila -d service-list DEBUG (connectionpool:208) Starting new HTTP connection (1): FQDN DEBUG (connectionpool:396) http://FQDN:8786 "GET /v2/services HTTP/1.1" 404 112 RESP: [404] {'Content-Length': '112', 'Content-Type': 'application/json', 'Date': 'Mon, 11 Apr 2022 12:14:51 GMT'} RESP BODY: {"message": "The resource could not be found.<br /><br />\n\n\n", "code": "404 Not Found", "title": "Not Found"} Services are running, database is populated, rabbitmq has some exchanges... I currently run out of ideas... Did someone already face this issue? Best regards, Romain

2 1

Re: [neutron] exposing ip address of external Network from
by A Monster 11 Apr '22

11 Apr '22

Hi, indeed I have dhcp disabled for this network, and that's because I have an external dhcp server which attributes ip addresses to hosts on this network, so I cannot check the dhcp option when creating subnets for this public network.

1 0

[neutron] Bug Deputy Report January 03 - 10
by Lucas Alvares Gomes 11 Apr '22

11 Apr '22

Hi, This is the Neutron bug report from January 3rd to 10th. Critical: * https://bugs.launchpad.net/neutron/+bug/1956344 - "Functional test test_gateway_chassis_rebalance is failing intermittently" - Assigned to: Rodolfo Alonso High: * https://bugs.launchpad.net/neutron/+bug/1956034 - "ovn load balancer health monitor cause mac address conflict" - Unassigned * https://bugs.launchpad.net/neutron/+bug/1956745 - " [ovn-octavia-provider] Load Balancer remained with ACTIVE state even with PENDING_UPDATE listener" - Assigned to: Luis Tomas Bolivar * https://bugs.launchpad.net/neutron/+bug/1956763 - "[OVN] Overlapping configuration for security group logging is not applied correctly" - Assigned to: Elvira Garcia Ruiz Medium: * https://bugs.launchpad.net/neutron/+bug/1956035 - "ovn load balancer member failover not working when accessed from floating ip" - Unassigned * https://bugs.launchpad.net/neutron/+bug/1956632 - "DNS integration documentation and scenario tests need update after dns_assignment calculation was changed" - Assigned to: Miguel Lavalle * https://bugs.launchpad.net/neutron/+bug/1956785 - "Duplicate validator TypeError for dict values Edit" - Assigned to: Harald Jensås Low: * https://bugs.launchpad.net/neutron/+bug/1956476 - "[OVN] Disallow multiple physnets per bridge" - Assigned to: Rodolfo Alonso * https://bugs.launchpad.net/neutron/+bug/1956770 - " keepalived no_track implemented in >= 2.0.3" - Assigned to: Rodolfo Alonso Needs further triage: * https://bugs.launchpad.net/neutron/+bug/1956435 - "OVS: support multiple segments per host" - Unassigned * https://bugs.launchpad.net/neutron/+bug/1956460 - "The VM can't get metadata from dhcp-agent" - Unassigned * https://bugs.launchpad.net/neutron/+bug/1956846 - "ha router duplicated routes" - Unassigned Cheers, Lucas

4 7

[zuul][neutron][neutron-fwaas][tap-as-a-service] Errors in the jobs definitions and EOL of some old neutron-fwaas and tap-as-a-service branches
by Lajos Katona 11 Apr '22

11 Apr '22

Hi, During the PTG we got questions about projects which cause zuul config errors (see [1]). Slawek started a thread regarding networking-midonet (see [2]), I join to that with neutron-fwaas and tap-as-a-servie. - neutron-fwaas' stable/rocky and stable/queens branches - tap-as-a-service stable/ocata, stable/pike, stable/queens and stable/rocky branches are affected - as tap-as-a-service was under x/ namespace previously I am not sure what can happen to those branches. If there is nobody to fix the config and gate errors on these old branches, then the best would be to EOL and later delete these branches. [1]: https://zuul.opendev.org/t/openstack/config-errors [2]: http://lists.openstack.org/pipermail/openstack-discuss/2022-April/028120.ht…

1 0

[requirements][kolla][infra] Python versions in u-c
by Marcin Juszkiewicz 11 Apr '22

11 Apr '22

OpenStack Zed lists Python 3.8 as minimal version. Debian 'bullseye' (current stable) as Python 3.9, Ubuntu 20.04 'focal' has 3.8. Ubuntu 22.04 'jammy' has 3.10, CentOS Stream 9 (so RHEL 9, Rocky Linux 9) has 3.9 version. We ignore anything RHEL 8 based as they have Python 3.6 only and there is no work on getting OpenStack Zed supported there. So there will be 3.8, 3.9, 3.10 used with Zed. Two question to requirements team: 1. Are there plans to get rid of 3.6 entries from openstack/requirements/upper-constraints.txt in near future? 2. Can we get "python_version=='3.8'" lines be changed to ">=" ones? Or once 3.6 entries drop just assume that we have 3.8+ and drop 'python_version' entries? websocket-client===1.3.1;python_version=='3.6' websocket-client===1.3.2;python_version=='3.8' Asking due to my recent Kolla/infra work on getting CentOS Stream 9 supported in both places. Infra team has CI job which builds Python wheel cache to make sure that other CI jobs do not have to do it. Times are nicely cut due to this stuff. There are 55 entries with "python_version=='3.8'". Probably none of them are time critical anymore as upstream projects provide aarch64 wheels most of times too.

2 1

[Zuul] [neutron] Errors in the jobs definitions and EOL of some old networking-midonet branches
by Slawek Kaplonski 11 Apr '22

11 Apr '22

Hi, Today I got back to checking my old patches which are going to fix some zuul jobs' definitions. And I found that CI for networking-midonet in some stable branches is totally broken: * stable/ussuri: https://review.opendev.org/c/openstack/networking-midonet/+/823273[1] - everything is red here * stable/train: https://review.opendev.org/c/openstack/networking-midonet/+/823275[2] - most of the jobs are red * stable/stein: https://review.opendev.org/c/openstack/networking-midonet/+/823276[3] - here there were errors in the definition of the jobs, I changed it now, maybe it will run, So my question is if we want, and have resources, to fix it somehow or should we maybe set those branches as EOL now? Also, another question to the Zuul team - will it be ok if such branch will be EOL to get rid of the Zuul configuration errors warnings? Or should we somehow force to merge those patches first to get rid of the errors in https://zuul.opendev.org/t/openstack/config-errors[4] ? -- Slawek Kaplonski Principal Software Engineer Red Hat -------- [1] https://review.opendev.org/c/openstack/networking-midonet/+/823273 [2] https://review.opendev.org/c/openstack/networking-midonet/+/823275 [3] https://review.opendev.org/c/openstack/networking-midonet/+/823276 [4] https://zuul.opendev.org/t/openstack/config-errors

2 2

[all][neutron] Pike - End of Life
by Lajos Katona 11 Apr '22

11 Apr '22

Hi, As time and cycles are passing it's time to bring up the topic of EOLing an old branch again. In Neutron (and its satellite projects, aka stadium) stable/pike has had low activity in the last cycle and recently even the jobs for it started to fail, and we cause some zuul config errors which are hard to fix as the gate is broken. Based on the Ocata EOL mail (see [1]), Neutron team decided to EOL Pike, see the discussions during the Zed PTG (see [2]) [1]: http://lists.openstack.org/pipermail/openstack-discuss/2021-April/021949.ht… [2]: https://etherpad.opendev.org/p/neutron-zed-ptg#L254 Lajos Katona (lajoskatona)

1 0

[vmware-nsx] EOL old branches
by Slawek Kaplonski 11 Apr '22

11 Apr '22

Hi Salvatore, According to Your comments in my patches to x/vmware-nsx project: * stable/stein: https://review.opendev.org/c/x/vmware-nsx/+/823277[1] * stable/rocky: https://review.opendev.org/c/x/vmware-nsx/+/823291[2] Can You maybe make them EOL finally? -- Slawek Kaplonski Principal Software Engineer Red Hat -------- [1] https://review.opendev.org/c/x/vmware-nsx/+/823277 [2] https://review.opendev.org/c/x/vmware-nsx/+/823291

1 0

[neutron] Bug Deputy Report April 04 - 11
by Lucas Alvares Gomes 11 Apr '22

11 Apr '22

Hi, This is the Neutron bug report from April 4th to 11th. Critical: * https://bugs.launchpad.net/neutron/+bug/1967893 - " [stable/yoga] tempest.scenario.test_network_qos_placement.MinBwAllocationPlacementTest fails in neutron-ovs-tempest-multinode-full job Edit" - Assigned to: Lajos Katona High: * https://bugs.launchpad.net/neutron/+bug/1967839 - " [L3] NDP extension not handing "ha_state_change" correctly" - Assigned to: Rodolfo Alonso * https://bugs.launchpad.net/neutron/+bug/1967996 - "[OVN] External subscribers to feed generated in Openstack fails if no internal VM is subscribed to that feed first" - Unassigned Medium: * https://bugs.launchpad.net/neutron/+bug/1967742 - "ML2 - Network Context, not possible to see original/current segments" - Unassigned * https://bugs.launchpad.net/neutron/+bug/1968343 - "Security Group Rule create with forged integer security_group_id causes exceptions" - Assigned to: Andrew Karpow Needs further triage: * https://bugs.launchpad.net/neutron/+bug/1968057 - "Doc needed for configuring neutron AZs for edge" - Unassigned * https://bugs.launchpad.net/neutron/+bug/1968269 - " [router qos] qos binding is not clear after remove gateway" - Unassigned Cheers, Lucas

1 0