Thanks for your help and advice. It doesn't appear to be causing any serious problems. The customer who complained about it was worried that it could interfere with traffic routing for hosts with dual interfaces, but he worked around that by deleting the routes from route-<interface> on his VMs and that seems to have worked. For future cluster builds we will prevent the routes from being created by not specifying --gateway and letting it default to the 1st IP in the subnet.

Now that I understand what is happening, I don't think that this is necessarily a bad change; we just need to change our config to match the new code.
On Friday, April 1, 2022, 12:38:07 PM EDT, Brian Haley <haleyb.dev@gmail.com> wrote:


Hi Albert,

Thanks for the command line, it helped me track down the code in neutron
that changed, and it was really the --network-segment arg that is
triggering this along with --gateway (and I haven't defined any segments
so don't see it in my setup).

Anyways, there are a few changes that added the update of host routes in
the segment plugin code to support routed networks better. Looking at
https://bugs.launchpad.net/neutron/+bug/1766380 shows them all, but
https://review.opendev.org/c/openstack/neutron/+/570405/ and
https://review.opendev.org/c/openstack/neutron/+/573897 where the two
main ones.

It doesn't look like there's a way to disable it, but I cc'd Harald to
get his thoughts on it.

My only follow-on question would be are these host routes causing an
issue or just something that was noticed in your upgrade?

Thanks,

-Brian


On 3/31/22 16:06, Albert Braden wrote:
> Here's what I get when I create a 4th subnet:
>
> $ openstack network segment create --physical-network physnet_bo-az3
> --network-type vlan --segment 1115 --network trust trust-az4
> +------------------+--------------------------------------+
> | Field | Value |
> +------------------+--------------------------------------+
> | description | |
> | id | 92355e6d-3406-4b29-a956-1b05c4c9a33e |
> | name | private-provider-trust-az4 |
> | network_id | ac30a487-bccc-c3de-93eb-c422ad9f3ce5 |
> | network_type | vlan |
> | physical_network | physnet_bo-az3 |
> | segmentation_id | 1115 |
> +------------------+--------------------------------------+
>
> $ openstack subnet create --no-dhcp --network private-provider-trust
> --network-segment private-provider-trust-az4 --ip-version 4
> --allocation-pool start=10.52.172.14,end=10.52.172.235 --subnet-range
> 10.52.172.0/22 --dns-nameserver 10.10.10.10 --gateway 10.52.172.1
> private-provider-trust-az4-subnet
> +----------------------+------------------------------------------------------+
> | Field | Value |
> +----------------------+------------------------------------------------------+
> | allocation_pools | 10.52.172.10-10.52.172.245 |
> | cidr | 10.52.172.0/22 |
> | created_at | 2022-03-31T19:26:48Z |
> | description | |
> | dns_nameservers | 10.10.10.10 |
> | dns_publish_fixed_ip | None |
> | enable_dhcp | False |
> | gateway_ip | 10.52.172.1 |
> | host_routes | destination='10.52.160.0/22', gateway='10.52.172.1' |
> | | destination='10.52.164.0/22', gateway='10.52.172.1' |
> | | destination='10.52.168.0/22', gateway='10.52.172.1' |
> | id | 04a15cdd-d22b-4e58-8bbd-8b956d8c10ba |
> | ip_version | 4 |
> | ipv6_address_mode | None |
> | ipv6_ra_mode | None |
> | name | private-provider-trust-az4-subnet |
> | network_id | ac30a487-bccc-4ac5-93eb-c422ad9f3ce5 |
> | prefix_length | None |
> | project_id | 561e8d2236634ece81ffa22203e80dc7 |
> | revision_number | 0 |
> | segment_id | 92355e6d-a5de-4b29-a956-1b05c4c9a33e |
> | service_types | |
> | subnetpool_id | None |
> | tags | |
> | updated_at | 2022-03-31T19:26:48Z |
> +----------------------+------------------------------------------------------+
>
> If I create the 4th subnet without specifying a gateway, then the routes
> are not created. It looks like this may be what changed from Queens to
> Train:
>
> $ openstack subnet create --no-dhcp --network private-provider-trust
> --network-segment private-provider-trust-az4 --ip-version 4
> --allocation-pool start=10.52.172.10,end=10.52.172.245 --subnet-range
> 10.52.172.0/22 --dns-nameserver 10.10.10.10
> private-provider-trust-az4-subnet
> +----------------------+--------------------------------------+
> | Field | Value |
> +----------------------+--------------------------------------+
> | allocation_pools | 10.52.172.10-10.52.172.245 |
> | cidr | 10.52.172.0/22 |
> | created_at | 2022-03-31T20:00:44Z |
> | description | |
> | dns_nameservers | 10.10.10.10 |
> | dns_publish_fixed_ip | None |
> | enable_dhcp | False |
> | gateway_ip | 10.52.172.1 |
> | host_routes | |
> | id | 11757c89-2057-4c7c-9730-9b7d976e361e |
> | ip_version | 4 |
> | ipv6_address_mode | None |
> | ipv6_ra_mode | None |
> | name | private-provider-trust-az4-subnet |
> | network_id | ac30a487-bccc-4ac5-93eb-c422ad9f3ce5 |
> | prefix_length | None |
> | project_id | 561e8d2236634ece81ffa22203e80dc7 |
> | revision_number | 0 |
> | segment_id | 92355e6d-a5de-4b29-a956-1b05c4c9a33e |
> | service_types | |
> | subnetpool_id | None |
> | tags | |
> | updated_at | 2022-03-31T20:00:44Z |
> +----------------------+--------------------------------------+
> On Wednesday, March 30, 2022, 09:01:23 PM EDT, Brian Haley
> <haleyb.dev@gmail.com> wrote:
>
>
> Hi,
>
> On 3/30/22 15:27, Albert Braden wrote:
>  > The command that we use to create subnets looks like this:
>  >
>  > openstack subnet create --no-dhcp --network trust --network-segment
>  > trust-az1-seg --ip-version 4 --allocation-pool
>  > start=10.52.160.14,end=10.52.160.235 --subnet-range 10.52.160.0/24
>  > --dns-nameserver 10.10.10.10 --gateway 10.52.160.1 trust-az1
>
> Since you're not specifying --host-route there should be none, can you
> paste the created object returned from this call since for me
> host_routes is blank (see below).
>
>  > My co-workers tell me that we also specified "--gateway" when we created
>  > our Queens subnets, but this did not cause static routes to be created.
>  > Did the handling of "--gateway" change from Queens to Train?
>
> I don't believe so, and --gateway will default to the first IP in the
> subnet if not given so isn't required.
>
> -Brian
>
>
> $ openstack subnet create --subnet-pool
> f5e3f133-a932-4adc-9592-0b525aec278f --network private private-subnet-2
> +----------------------+---------------------------+
> | Field                | Value                    |
> +----------------------+---------------------------+
> | allocation_pools    | 10.0.0.66-10.0.0.126      |
> | cidr                | 10.0.0.64/26              |
> | created_at          | 2022-03-30T17:38:40Z      |
> | description          |                          |
> | dns_nameservers      |                          |
> | dns_publish_fixed_ip | None                      |
> | enable_dhcp          | True                      |
> | gateway_ip          | 10.0.0.65                |
> | host_routes          |                          |
> | id                  | ce09a038-b918-4208-9a3d-c8c259ae7433 |
> | ip_version          | 4                        |
> | ipv6_address_mode    | None                      |
> | ipv6_ra_mode        | None                      |
> | name                | private-subnet-2          |
> | network_id          | baf6c62d-4cec-464e-a768-253074df8879 |
> | project_id          | 657e6d647c0446438c1f06da70d79bed |
> | revision_number      | 0                        |
>          | segment_id          | None                      |
>
> | service_types        |                          |
> | subnetpool_id        | f5e3f133-a932-4adc-9592-0b525aec278f |
> | tags                |                          |
> | updated_at          | 2022-03-30T17:38:40Z      |
>
> +----------------------+---------------------------+
>
>  > On Wednesday, March 30, 2022, 01:45:52 PM EDT, Brian Haley
>  > <haleyb.dev@gmail.com <mailto:haleyb.dev@gmail.com>> wrote:

>  >
>  >
>  > Hi Albert,
>  >
>  > On 3/29/22 17:04, Albert Braden wrote:
>  >  > After upgrading our kolla-ansible clusters from Queens to Train, we
>  > are seeing static routes when we create subnets. We didn’t see this in
>  > Queens. For example, in our de6 region we have a network called “trust”
>  > with 3 subnets:
>  >  >
>  >  > Subnet                CIDR                                  Gateway
>  >  > trust-az1:            10.52.160.0/22  10.52.160.1
>  >  > trust-az2:            10.52.164.0/22  10.52.164.1
>  >  > trust-az3:            10.52.168.0/22  10.52.168.1
>  >  >
>  >  > Each of these subnets has 2 entries under “host_routes:” that point
>  > to the other two subnets. For example, subnet trust-az1 has these two
>  > routes:
>  >  >
>  >  > host_routes          | destination='10.52.164.0/22',
>  > gateway='10.52.160.1' |
>  >  > |                      | destination='10.52.168.0/22',
>  > gateway='10.52.160.1' |
>  >  >
>  >  > How can we prevent these host routes from being created in Train? Do
>  > we need to change something in our config?
>  >
>  >
>  >  From the neutron side of things, host_routes of a subnet is not
>  > automatically calculated and filled-in, they have to be manually added.
>  > So perhaps this is something kolla is doing? At least on my Yoga setup
>  > it is completely blank using 'openstack subnet create ...' even with
>  > multiple subnets on a network.
>  >
>  > How exactly are the subnets getting created?
>  >
>  > -Brian
>  >
>