Open Stack

Hi all,

We've caused ourselves some bother with our neutron database, and I 
found the following thread instructive, so I hope we can get some help 
to resolve the bother.
https://lists.openstack.org/pipermail/openstack-discuss/2022-November/031312.html

We're running kolla-ansible train (yes, upgrades planned). We wanted to 
test something with a wallaby container, and while mainly just pulling 
train versions of containers on a single compute node, we also changed 
openstack_release variable to wallaby in globals.yml - this was the mistake.

Later that day, we ran kolla-ansible deploy --limit $HOST for a new 
compute node. We stopped at the following error, realising the above 
mistake.
---
2023-04-05 12:25:50.487 7 ERROR nova 
nova.exception.TooOldComputeService: Current Nova version does not 
support computes older than Victoria but the minimum compute service 
level in your cell is 40 and the oldest supported service level is 52.
2023-04-05 12:25:50.487 7 ERROR nova
---

Now when we deploy a node it gets to task "Running Neutron bootstrap 
container" and gives error:
----
"alembic.script.revision.ResolutionError: No such revision or branch 
'6135a7bd4425'"
----

Using insight from the above list thread, I fetched DB entries:
MariaDB [neutron]> select * from alembic_version;
+--------------+
| version_num  |
+--------------+
| 5c85685d616d |
| 6135a7bd4425 |
+--------------+

Running the same from the test cluster which was deployed in the same way:
MariaDB [neutron]> select * from alembic_version;
+--------------+
| version_num  |
+--------------+
| 5c85685d616d |
| c613d0b82681 |
+--------------+

Also, comparing `neutron-db-manage history --verbose` across the two 
clusters, output is the same in both test and prod, except for the order 
of 2 lines:
----
-Branches into: 56893333aa52, 30018084ed99
-Branches into: c40fbb377ad, 67c8e8d61d5
+Branches into: 30018084ed99, 56893333aa52
+Branches into: 67c8e8d61d5, c40fbb377ad
----

Finally, I looked at the schema of the neutron database across the two. 
There are significant differences.
for i in `mysql neutron -bNe "show tables;"`; do echo "TABLE: $i"; mysql 
neutron -e "desc $i;"; done > prod-neutron-descs
Repeat for test.
$ cat prod-neutron-descs | nc termbin.com 9999
https://termbin.com/8ppi
$ cat test-neutron-descs | nc termbin.com 9999
https://termbin.com/cfqz

Our cluster and Neutron in particular is operating fine, but we can't 
deploy new nodes to it.

May I have some help on how to resolve this?

Thank you,
Greg.