<div dir="ltr"><div dir="ltr">Hello,<br><br>I appreciate the prompt feedback. Unfortunately, after making multiple changes, we cannot make external networks to connect via gateway-hosts. Our follow up investigation has shown the following:<br><br>1. Removed flat and vlan provider_networks from /etc/openstack_deploy/openstack_user_config.yml. Only management 

provider_networks  was defined here :<br><br>provider_networks:<br>    - network:<br>        container_bridge: "br-mgmt"<br>        container_type: "veth"<br>        container_interface: "ens4"<br>        ip_from_q: "management"<br>        type: "raw"<br>        group_binds:<br>          - all_containers<br>          - hosts<br>        is_management_address: true<br><br> <br>2. Defined ML2 information and network types in /etc/openstack_deploy/user_variables.yml:<br><br>neutron_ml2_conf_ini_overrides:<br>  ml2:<br>    tenant_network_types: geneve<br>  ml2_type_flat:<br>    flat_networks: flat<br>  ml2_type_geneve:<br>    vni_ranges: 1:1000<br>    max_header_size: 38<br><br> <br>3. Moved neutron_provider_networks configuration on a per-host basis and removed network_mappings and network_interface_mappings for compute hosts in /etc/openstack_deploy/host_vars/<br><br>compute node<br>/etc/openstack_deploy/host_vars/cmp3:<br><br>neutron_provider_networks:<br>  network_types: "geneve"<br>  network_geneve_ranges: "1:1000"<br><br><br>gateway node<br>/etc/openstack_deploy/host_vars/net1:<br><br>neutron_provider_networks:<br>  network_types: "geneve"<br>  network_geneve_ranges: "1:1000"<br>  network_mappings: "flat:br-flat"<br>  network_interface_mappings: "br-flat:ens2"<br> <br><br>4. Upon checking the new recreated inventory targets the correct neutron_ovn_gateway hosts<br>/etc/openstack_deploy/openstack_inventory.json<br><br>…<br>"component": "neutron_ovn_gateway",<br>                "container_name": "net1",<br>                "container_networks": {<br>                    "management_address": {<br>                        "address": "172.16.0.31",<br>                        "bridge": "br-mgmt",<br><br>--<br><br>                "component": "neutron_ovn_gateway",<br>                "container_name": "net2",<br>                "container_networks": {<br>                    "management_address": {<br>                        "address": "172.16.0.32",<br>                        "bridge": "br-mgmt",<br><br><div>--<br>"neutron_ovn_gateway": {<br>        "children": [],<br>        "hosts": [<br>            "net1",<br>            "net2"<br>…<br><br> <br><br> <br>5. The correct ovn-cms-options=enable-chassis-as-gw is set on gateway nodes only:<br><br>ovn-sbctl list chassis | grep 'hostname\|ovn-cms-options'<br><br>hostname            : net2<br>other_config        : {ct-no-masked-label="true", datapath-type=system, iface-types="afxdp,afxdp-nonpmd,bareudp,erspan,geneve,gre,gtpu,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", mac-binding-timestamp="true", ovn-bridge-mappings="flat:br-flat", ovn-chassis-mac-mappings="", ovn-cms-options=enable-chassis-as-gw, ovn-ct-lb-related="true", ovn-enable-lflow-cache="true", ovn-limit-lflow-cache="", ovn-memlimit-lflow-cache-kb="", ovn-monitor-all="false", ovn-trim-limit-lflow-cache="", ovn-trim-timeout-ms="", ovn-trim-wmark-perc-lflow-cache="", port-up-notif="true"}<br><br>hostname            : net1<br>other_config        : {ct-no-masked-label="true", datapath-type=system, iface-types="afxdp,afxdp-nonpmd,bareudp,erspan,geneve,gre,gtpu,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", mac-binding-timestamp="true", ovn-bridge-mappings="flat:br-flat", ovn-chassis-mac-mappings="", ovn-cms-options=enable-chassis-as-gw, ovn-ct-lb-related="true", ovn-enable-lflow-cache="true", ovn-limit-lflow-cache="", ovn-memlimit-lflow-cache-kb="", ovn-monitor-all="false", ovn-trim-limit-lflow-cache="", ovn-trim-timeout-ms="", ovn-trim-wmark-perc-lflow-cache="", port-up-notif="true"}<br><br>hostname            : cmp3<br>other_config        : {ct-no-masked-label="true", datapath-type=system, iface-types="afxdp,afxdp-nonpmd,bareudp,erspan,geneve,gre,gtpu,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", mac-binding-timestamp="true", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options="", ovn-ct-lb-related="true", ovn-enable-lflow-cache="true", ovn-limit-lflow-cache="", ovn-memlimit-lflow-cache-kb="", ovn-monitor-all="false", ovn-trim-limit-lflow-cache="", ovn-trim-timeout-ms="", ovn-trim-wmark-perc-lflow-cache="", port-up-notif="true"}<br><br>hostname            : cmp4<br>other_config        : {ct-no-masked-label="true", datapath-type=system, iface-types="afxdp,afxdp-nonpmd,bareudp,erspan,geneve,gre,gtpu,internal,ip6erspan,ip6gre,lisp,patch,stt,system,tap,vxlan", is-interconn="false", mac-binding-timestamp="true", ovn-bridge-mappings="", ovn-chassis-mac-mappings="", ovn-cms-options="", ovn-ct-lb-related="true", ovn-enable-lflow-cache="true", ovn-limit-lflow-cache="", ovn-memlimit-lflow-cache-kb="", ovn-monitor-all="false", ovn-trim-limit-lflow-cache="", ovn-trim-timeout-ms="", ovn-trim-wmark-perc-lflow-cache="", port-up-notif="true"}<br><br> <br>RESULT: VMs fail to launch with external network (flat). Error logs "Binding failed for port":<br> <br><br>Sep  6 19:42:37 net1 nova-conductor[4270]: 2023-09-06 19:42:37.599 4270 ERROR nova.scheduler.utils [None req-25a6a8d6-8122-4621-a2c2-8ca0be5e594c 52059c7247434072b6823d1701fec23e 116579f970b242b996ac717fa7580311 - - default default] [instance: 8760706e-d38f-454d-b90f-b9d5d322ba99] Error from last host: dev-usc1-ost-cmp4 (node dev-usc1-ost-cmp4.openstack.local): ['Traceback (most recent call last):\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/compute/manager.py", line 2607, in _build_and_run_instance\n    self.driver.spawn(context, instance, image_meta,\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 4383, in spawn\n    xml = self._get_guest_xml(context, instance, network_info,\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/virt/libvirt/driver.py", line 7516, in _get_guest_xml\n    network_info_str = str(network_info)\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/network/model.py", line 620, in __str__\n    return self._sync_wrapper(fn, *args, **kwargs)\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/network/model.py", line 603, in _sync_wrapper\n    self.wait()\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/network/model.py", line 635, in wait\n    self[:] = self._gt.wait()\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/eventlet/greenthread.py", line 181, in wait\n    return self._exit_event.wait()\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/eventlet/event.py", line 132, in wait\n    current.throw(*self._exc)\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/eventlet/greenthread.py", line 221, in main\n    result = function(*args, **kwargs)\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/utils.py", line 654, in context_wrapper\n    return func(*args, **kwargs)\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/compute/manager.py", line 1987, in _allocate_network_async\n    raise e\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/compute/manager.py", line 1965, in _allocate_network_async\n    nwinfo = self.network_api.allocate_for_instance(\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/network/neutron.py", line 1216, in allocate_for_instance\n    created_port_ids = self._update_ports_for_instance(\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/network/neutron.py", line 1352, in _update_ports_for_instance\n    with excutils.save_and_reraise_exception():\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/oslo_utils/excutils.py", line 227, in __exit__\n    self.force_reraise()\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/oslo_utils/excutils.py", line 200, in force_reraise\n    raise self.value\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/network/neutron.py", line 1327, in _update_ports_for_instance\n    updated_port = self._update_port(\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/network/neutron.py", line 585, in _update_port\n    _ensure_no_port_binding_failure(port)\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/network/neutron.py", line 294, in _ensure_no_port_binding_failure\n    raise exception.PortBindingFailed(port_id=port[\'id\'])\n', 'nova.exception.PortBindingFailed: Binding failed for port b82f4518-ecba-49d9-a21d-2646d3f33efd, please check neutron logs for more information.\n', '\nDuring handling of the above exception, another exception occurred:\n\n', 'Traceback (most recent call last):\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/compute/manager.py", line 2428, in _do_build_and_run_instance\n    self._build_and_run_instance(context, instance, image,\n', '  File "/openstack/venvs/nova-0.1.0.dev8112/lib/python3.10/site-packages/nova/compute/manager.py", line 2703, in _build_and_run_instance\n    raise exception.RescheduledException(\n', 'nova.exception.RescheduledException: Build of instance 8760706e-d38f-454d-b90f-b9d5d322ba99 was re-scheduled: Binding failed for port b82f4518-ecba-49d9-a21d-2646d3f33efd, please check neutron logs for more information.\n']<br><br><br>All we need is to make sure external networks are routed via gateway-hosts and not via compute nodes. In our case, compute nodes have only one physical interface with an IP address and no connectivity to the flat network. No layer 2 connectivity is available on compute nodes either. That's the reason why we must traverse external traffic via gateway nodes only.<br><br>It is worth noting that tenant/internal networks work fine. <br><br>What are we doing wrong?<br><br>Thank you.</div></div></div>