[neutron] [kolla-ansible] alembic mismatch, schema differences

Eugen Block eblock at nde.ag
Tue Apr 11 09:16:38 UTC 2023


Hi,

I don't really have more input than what I reported in the mentioned  
thread. I'm also not familiar with kolla-ansible, unfortunately. But I  
would check for files with the non-existing revision number on the  
control node(s)? Something like this depending on your distro could  
reveal something:

# find  
/usr/lib/python3/dist-packages/neutron/db/migration/alembic_migrations/versions/ -name  
*6135a7bd4425*

If there are files that don't belong there remove them and retry the  
neutron-db-manage commands with the correct revisions in the table.  
But if you're using containers I'd be curious how those files would  
get there... Does a release change in the globals.yaml also redeploy  
control containers? But on the other hand, you seem to have deployed  
only a compute node with wallaby? I hope someone with kolla-ansible  
experience can chime in here.

Sorry if I can't be more helpful...

Eugen

Zitat von Gregory Orange <gregory.orange at pawsey.org.au>:

> Hi all,
>
> We've caused ourselves some bother with our neutron database, and I  
> found the following thread instructive, so I hope we can get some  
> help to resolve the bother.
> https://lists.openstack.org/pipermail/openstack-discuss/2022-November/031312.html
>
> We're running kolla-ansible train (yes, upgrades planned). We wanted  
> to test something with a wallaby container, and while mainly just  
> pulling train versions of containers on a single compute node, we  
> also changed openstack_release variable to wallaby in globals.yml -  
> this was the mistake.
>
> Later that day, we ran kolla-ansible deploy --limit $HOST for a new  
> compute node. We stopped at the following error, realising the above  
> mistake.
> ---
> 2023-04-05 12:25:50.487 7 ERROR nova  
> nova.exception.TooOldComputeService: Current Nova version does not  
> support computes older than Victoria but the minimum compute service  
> level in your cell is 40 and the oldest supported service level is 52.
> 2023-04-05 12:25:50.487 7 ERROR nova
> ---
>
> Now when we deploy a node it gets to task "Running Neutron bootstrap  
> container" and gives error:
> ----
> "alembic.script.revision.ResolutionError: No such revision or branch  
> '6135a7bd4425'"
> ----
>
> Using insight from the above list thread, I fetched DB entries:
> MariaDB [neutron]> select * from alembic_version;
> +--------------+
> | version_num  |
> +--------------+
> | 5c85685d616d |
> | 6135a7bd4425 |
> +--------------+
>
> Running the same from the test cluster which was deployed in the same way:
> MariaDB [neutron]> select * from alembic_version;
> +--------------+
> | version_num  |
> +--------------+
> | 5c85685d616d |
> | c613d0b82681 |
> +--------------+
>
> Also, comparing `neutron-db-manage history --verbose` across the two  
> clusters, output is the same in both test and prod, except for the  
> order of 2 lines:
> ----
> -Branches into: 56893333aa52, 30018084ed99
> -Branches into: c40fbb377ad, 67c8e8d61d5
> +Branches into: 30018084ed99, 56893333aa52
> +Branches into: 67c8e8d61d5, c40fbb377ad
> ----
>
> Finally, I looked at the schema of the neutron database across the  
> two. There are significant differences.
> for i in `mysql neutron -bNe "show tables;"`; do echo "TABLE: $i";  
> mysql neutron -e "desc $i;"; done > prod-neutron-descs
> Repeat for test.
> $ cat prod-neutron-descs | nc termbin.com 9999
> https://termbin.com/8ppi
> $ cat test-neutron-descs | nc termbin.com 9999
> https://termbin.com/cfqz
>
> Our cluster and Neutron in particular is operating fine, but we  
> can't deploy new nodes to it.
>
> May I have some help on how to resolve this?
>
> Thank you,
> Greg.






More information about the openstack-discuss mailing list