[neutron] neutron-db-manage multiple heads
Hi *, I'd like to ask you for advice on how to clean up my neutron db. At some point (which I don't know exactly, probably train) my neutron database got inconsistent, apparently one of the upgrades did not go as planned. The interesting thing is that the database still works, I just upgraded from ussuri to victoria where that issue popped up again during 'neutron-db-manage upgrade --expand', I'll add the information at the end of this email. Apparently, I have multiple heads, and one of them is from train, it seems as if I never ran --contract (or it failed and I didn't notice). Just some additional information what I did with this database: this cloud started out as a test environment with a single control node and then became a production environment. About two and a half years ago we decided to reinstall this cloud with version ussuri and import the databases. I had a virtual machine in which I upgraded the database dump from production to the latest versions at that time. That all worked quite well, I only didn't notice that something was missing. Now that I finished the U --> V upgrade I want to fix this inconsistency, I just have no idea how to do it. As I'm not sure how all the neutron-db-manage commands work exactly I'd like to ask for some guidance. For example, could the "stamp" command possibly help? Or how else can I get rid of the train head and/or how to get the train revision to "contract" so I can finish the upgrade and contract the victoria revision? I can paste the whole neutron-db history if necessary (neutron-db-manage history), please let me know what information would be required to get to the bottom of this. Any help is greatly appreciated! Thanks! Eugen ---snip--- controller01:~ # neutron-db-manage upgrade --expand [...] alembic.script.revision.MultipleHeads: Multiple heads are present for given argument 'expand@head'; 633d74ebbc4b, I38991de2b4 controller01:~ # neutron-db-manage current --verbose Running current for neutron ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn/neutron: Rev: bebe95aae4d4 (head) Parent: b5344a66e818 Branch names: contract Path: /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py Rev: 633d74ebbc4b (head) Parent: 6c9eb0469914 Branch names: expand Path: /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py Rev: I38991de2b4 (head) Parent: 49d8622c5221 Branch names: expand Path: /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py OK ---snip---
Hi Eugen: In Neutron we don't support contract operations since Newton. If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule) Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided. Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version. Regards. On Fri, Nov 25, 2022 at 1:58 PM Eugen Block <eblock@nde.ag> wrote:
Hi *,
I'd like to ask you for advice on how to clean up my neutron db. At some point (which I don't know exactly, probably train) my neutron database got inconsistent, apparently one of the upgrades did not go as planned. The interesting thing is that the database still works, I just upgraded from ussuri to victoria where that issue popped up again during 'neutron-db-manage upgrade --expand', I'll add the information at the end of this email. Apparently, I have multiple heads, and one of them is from train, it seems as if I never ran --contract (or it failed and I didn't notice). Just some additional information what I did with this database: this cloud started out as a test environment with a single control node and then became a production environment. About two and a half years ago we decided to reinstall this cloud with version ussuri and import the databases. I had a virtual machine in which I upgraded the database dump from production to the latest versions at that time. That all worked quite well, I only didn't notice that something was missing. Now that I finished the U --> V upgrade I want to fix this inconsistency, I just have no idea how to do it. As I'm not sure how all the neutron-db-manage commands work exactly I'd like to ask for some guidance. For example, could the "stamp" command possibly help? Or how else can I get rid of the train head and/or how to get the train revision to "contract" so I can finish the upgrade and contract the victoria revision? I can paste the whole neutron-db history if necessary (neutron-db-manage history), please let me know what information would be required to get to the bottom of this. Any help is greatly appreciated!
Thanks! Eugen
---snip--- controller01:~ # neutron-db-manage upgrade --expand [...] alembic.script.revision.MultipleHeads: Multiple heads are present for given argument 'expand@head'; 633d74ebbc4b, I38991de2b4
controller01:~ # neutron-db-manage current --verbose Running current for neutron ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn /neutron: Rev: bebe95aae4d4 (head) Parent: b5344a66e818 Branch names: contract Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py
Rev: 633d74ebbc4b (head) Parent: 6c9eb0469914 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
Rev: I38991de2b4 (head) Parent: 49d8622c5221 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py
OK ---snip---
Hi, thanks for your quick response.
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
That explains why I saw the newton revision in a new victoria cluster :-)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided.
The table alembic_versions contains the three versions I provided at the end of my email: MariaDB [neutron]> select * from alembic_version; +--------------+ | version_num | +--------------+ | 633d74ebbc4b | | bebe95aae4d4 | | I38991de2b4 | +--------------+ I already tried to manipulate the table so I would only have those two versions you already mentioned, but then the upgrade --expand command alternates the database again with the mentioned error message ("Multiple heads are present").
Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version.
That's what I wanted to try next, export only the db schema (no data) from a working victoria neutron database, then export only data from our production db and merge those, then import that into the production and try to run upgrade --expand and --contract again. But I didn't want to fiddle around too much in the production, that's why I wanted to ask for your guidance first. But IIUC even if I changed the table alembic_versions again and import the merged db, wouldn't upgrade --expand somehow try to alternate the table again? I don't see where the train revision comes from exactly, could you clarify, please? It seems like I always get back to square one when running the --expand command. Thanks! Eugen Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided. Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version.
Regards.
On Fri, Nov 25, 2022 at 1:58 PM Eugen Block <eblock@nde.ag> wrote:
Hi *,
I'd like to ask you for advice on how to clean up my neutron db. At some point (which I don't know exactly, probably train) my neutron database got inconsistent, apparently one of the upgrades did not go as planned. The interesting thing is that the database still works, I just upgraded from ussuri to victoria where that issue popped up again during 'neutron-db-manage upgrade --expand', I'll add the information at the end of this email. Apparently, I have multiple heads, and one of them is from train, it seems as if I never ran --contract (or it failed and I didn't notice). Just some additional information what I did with this database: this cloud started out as a test environment with a single control node and then became a production environment. About two and a half years ago we decided to reinstall this cloud with version ussuri and import the databases. I had a virtual machine in which I upgraded the database dump from production to the latest versions at that time. That all worked quite well, I only didn't notice that something was missing. Now that I finished the U --> V upgrade I want to fix this inconsistency, I just have no idea how to do it. As I'm not sure how all the neutron-db-manage commands work exactly I'd like to ask for some guidance. For example, could the "stamp" command possibly help? Or how else can I get rid of the train head and/or how to get the train revision to "contract" so I can finish the upgrade and contract the victoria revision? I can paste the whole neutron-db history if necessary (neutron-db-manage history), please let me know what information would be required to get to the bottom of this. Any help is greatly appreciated!
Thanks! Eugen
---snip--- controller01:~ # neutron-db-manage upgrade --expand [...] alembic.script.revision.MultipleHeads: Multiple heads are present for given argument 'expand@head'; 633d74ebbc4b, I38991de2b4
controller01:~ # neutron-db-manage current --verbose Running current for neutron ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn /neutron: Rev: bebe95aae4d4 (head) Parent: b5344a66e818 Branch names: contract Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py
Rev: 633d74ebbc4b (head) Parent: 6c9eb0469914 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
Rev: I38991de2b4 (head) Parent: 49d8622c5221 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py
OK ---snip---
Hi Eugen: I don't know how it is possible that you have 3 registers in this table. And the first two are not IDs of any Neutron revision. I would suggest you to (1) check the DB schema deployed against a fresh deployed system (in Victoria version) and (2) fix this table to point to the correct revision numbers. Regards. On Fri, Nov 25, 2022 at 3:51 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
thanks for your quick response.
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
That explains why I saw the newton revision in a new victoria cluster :-)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided.
The table alembic_versions contains the three versions I provided at the end of my email:
MariaDB [neutron]> select * from alembic_version; +--------------+ | version_num | +--------------+ | 633d74ebbc4b | | bebe95aae4d4 | | I38991de2b4 | +--------------+
I already tried to manipulate the table so I would only have those two versions you already mentioned, but then the upgrade --expand command alternates the database again with the mentioned error message ("Multiple heads are present").
Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version.
That's what I wanted to try next, export only the db schema (no data) from a working victoria neutron database, then export only data from our production db and merge those, then import that into the production and try to run upgrade --expand and --contract again. But I didn't want to fiddle around too much in the production, that's why I wanted to ask for your guidance first. But IIUC even if I changed the table alembic_versions again and import the merged db, wouldn't upgrade --expand somehow try to alternate the table again? I don't see where the train revision comes from exactly, could you clarify, please? It seems like I always get back to square one when running the --expand command.
Thanks! Eugen
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided. Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version.
Regards.
On Fri, Nov 25, 2022 at 1:58 PM Eugen Block <eblock@nde.ag> wrote:
Hi *,
I'd like to ask you for advice on how to clean up my neutron db. At some point (which I don't know exactly, probably train) my neutron database got inconsistent, apparently one of the upgrades did not go as planned. The interesting thing is that the database still works, I just upgraded from ussuri to victoria where that issue popped up again during 'neutron-db-manage upgrade --expand', I'll add the information at the end of this email. Apparently, I have multiple heads, and one of them is from train, it seems as if I never ran --contract (or it failed and I didn't notice). Just some additional information what I did with this database: this cloud started out as a test environment with a single control node and then became a production environment. About two and a half years ago we decided to reinstall this cloud with version ussuri and import the databases. I had a virtual machine in which I upgraded the database dump from production to the latest versions at that time. That all worked quite well, I only didn't notice that something was missing. Now that I finished the U --> V upgrade I want to fix this inconsistency, I just have no idea how to do it. As I'm not sure how all the neutron-db-manage commands work exactly I'd like to ask for some guidance. For example, could the "stamp" command possibly help? Or how else can I get rid of the train head and/or how to get the train revision to "contract" so I can finish the upgrade and contract the victoria revision? I can paste the whole neutron-db history if necessary (neutron-db-manage history), please let me know what information would be required to get to the bottom of this. Any help is greatly appreciated!
Thanks! Eugen
---snip--- controller01:~ # neutron-db-manage upgrade --expand [...] alembic.script.revision.MultipleHeads: Multiple heads are present for given argument 'expand@head'; 633d74ebbc4b, I38991de2b4
controller01:~ # neutron-db-manage current --verbose Running current for neutron ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn /neutron: Rev: bebe95aae4d4 (head) Parent: b5344a66e818 Branch names: contract Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py
Rev: 633d74ebbc4b (head) Parent: 6c9eb0469914 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
Rev: I38991de2b4 (head) Parent: 49d8622c5221 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py
OK ---snip---
Hi, I believe they are neutron revisions, here's the output from yesterday's neutron-db-manage attempt: ---snip--- Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 5c85685d616d -> c43a0ddb6a03 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c43a0ddb6a03 -> b5344a66e818 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade b5344a66e818 -> bebe95aae4d4 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c613d0b82681 -> 6c9eb0469914 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 6c9eb0469914 -> 633d74ebbc4b Nov 23 12:51:52 controller01 neutron-db-manage[25913]: Running upgrade for neutron ... Nov 23 12:51:52 controller01 neutron-db-manage[25913]: OK ---snip--- And here's where they are located, apparently from train version: ---snip--- controller01:~ # grep -r 633d74ebbc4b /usr/lib/python3.6/site-packages/ Übereinstimmungen in Binärdatei /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/__pycache__/633d74ebbc4b_.cpython-36.pyc /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:Revision ID: 633d74ebbc4b /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:revision = '633d74ebbc4b' ---snip--- Next week we'll try it with merging fresh victoria schema with our production data, then run the upgrade command again. Thanks, Eugen Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
I don't know how it is possible that you have 3 registers in this table. And the first two are not IDs of any Neutron revision. I would suggest you to (1) check the DB schema deployed against a fresh deployed system (in Victoria version) and (2) fix this table to point to the correct revision numbers.
Regards.
On Fri, Nov 25, 2022 at 3:51 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
thanks for your quick response.
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
That explains why I saw the newton revision in a new victoria cluster :-)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided.
The table alembic_versions contains the three versions I provided at the end of my email:
MariaDB [neutron]> select * from alembic_version; +--------------+ | version_num | +--------------+ | 633d74ebbc4b | | bebe95aae4d4 | | I38991de2b4 | +--------------+
I already tried to manipulate the table so I would only have those two versions you already mentioned, but then the upgrade --expand command alternates the database again with the mentioned error message ("Multiple heads are present").
Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version.
That's what I wanted to try next, export only the db schema (no data) from a working victoria neutron database, then export only data from our production db and merge those, then import that into the production and try to run upgrade --expand and --contract again. But I didn't want to fiddle around too much in the production, that's why I wanted to ask for your guidance first. But IIUC even if I changed the table alembic_versions again and import the merged db, wouldn't upgrade --expand somehow try to alternate the table again? I don't see where the train revision comes from exactly, could you clarify, please? It seems like I always get back to square one when running the --expand command.
Thanks! Eugen
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided. Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version.
Regards.
On Fri, Nov 25, 2022 at 1:58 PM Eugen Block <eblock@nde.ag> wrote:
Hi *,
I'd like to ask you for advice on how to clean up my neutron db. At some point (which I don't know exactly, probably train) my neutron database got inconsistent, apparently one of the upgrades did not go as planned. The interesting thing is that the database still works, I just upgraded from ussuri to victoria where that issue popped up again during 'neutron-db-manage upgrade --expand', I'll add the information at the end of this email. Apparently, I have multiple heads, and one of them is from train, it seems as if I never ran --contract (or it failed and I didn't notice). Just some additional information what I did with this database: this cloud started out as a test environment with a single control node and then became a production environment. About two and a half years ago we decided to reinstall this cloud with version ussuri and import the databases. I had a virtual machine in which I upgraded the database dump from production to the latest versions at that time. That all worked quite well, I only didn't notice that something was missing. Now that I finished the U --> V upgrade I want to fix this inconsistency, I just have no idea how to do it. As I'm not sure how all the neutron-db-manage commands work exactly I'd like to ask for some guidance. For example, could the "stamp" command possibly help? Or how else can I get rid of the train head and/or how to get the train revision to "contract" so I can finish the upgrade and contract the victoria revision? I can paste the whole neutron-db history if necessary (neutron-db-manage history), please let me know what information would be required to get to the bottom of this. Any help is greatly appreciated!
Thanks! Eugen
---snip--- controller01:~ # neutron-db-manage upgrade --expand [...] alembic.script.revision.MultipleHeads: Multiple heads are present for given argument 'expand@head'; 633d74ebbc4b, I38991de2b4
controller01:~ # neutron-db-manage current --verbose Running current for neutron ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn /neutron: Rev: bebe95aae4d4 (head) Parent: b5344a66e818 Branch names: contract Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py
Rev: 633d74ebbc4b (head) Parent: 6c9eb0469914 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
Rev: I38991de2b4 (head) Parent: 49d8622c5221 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py
OK ---snip---
Hi Eugen: Please check the code you have. Those revisions (633d74ebbc4b, bebe95aae4d4) do not exist in the Neutron repository. File [1] (or something similar with the same prefix) does not exist. Are you using a customized Neutron repository? Regards. [1]/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py On Fri, Nov 25, 2022 at 8:57 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
I believe they are neutron revisions, here's the output from yesterday's neutron-db-manage attempt:
---snip--- Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 5c85685d616d -> c43a0ddb6a03 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c43a0ddb6a03 -> b5344a66e818 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade b5344a66e818 -> bebe95aae4d4 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c613d0b82681 -> 6c9eb0469914 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 6c9eb0469914 -> 633d74ebbc4b Nov 23 12:51:52 controller01 neutron-db-manage[25913]: Running upgrade for neutron ... Nov 23 12:51:52 controller01 neutron-db-manage[25913]: OK ---snip---
And here's where they are located, apparently from train version:
---snip--- controller01:~ # grep -r 633d74ebbc4b /usr/lib/python3.6/site-packages/ Übereinstimmungen in Binärdatei
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/__pycache__/633d74ebbc4b_.cpython-36.pyc /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:Revision ID: 633d74ebbc4b /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:revision = '633d74ebbc4b' ---snip---
Next week we'll try it with merging fresh victoria schema with our production data, then run the upgrade command again.
Thanks, Eugen Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
I don't know how it is possible that you have 3 registers in this table. And the first two are not IDs of any Neutron revision. I would suggest you to (1) check the DB schema deployed against a fresh deployed system (in Victoria version) and (2) fix this table to point to the correct revision numbers.
Regards.
On Fri, Nov 25, 2022 at 3:51 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
thanks for your quick response.
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
That explains why I saw the newton revision in a new victoria cluster :-)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided.
The table alembic_versions contains the three versions I provided at the end of my email:
MariaDB [neutron]> select * from alembic_version; +--------------+ | version_num | +--------------+ | 633d74ebbc4b | | bebe95aae4d4 | | I38991de2b4 | +--------------+
I already tried to manipulate the table so I would only have those two versions you already mentioned, but then the upgrade --expand command alternates the database again with the mentioned error message ("Multiple heads are present").
Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version.
That's what I wanted to try next, export only the db schema (no data) from a working victoria neutron database, then export only data from our production db and merge those, then import that into the production and try to run upgrade --expand and --contract again. But I didn't want to fiddle around too much in the production, that's why I wanted to ask for your guidance first. But IIUC even if I changed the table alembic_versions again and import the merged db, wouldn't upgrade --expand somehow try to alternate the table again? I don't see where the train revision comes from exactly, could you clarify, please? It seems like I always get back to square one when running the --expand command.
Thanks! Eugen
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided. Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version.
Regards.
On Fri, Nov 25, 2022 at 1:58 PM Eugen Block <eblock@nde.ag> wrote:
Hi *,
I'd like to ask you for advice on how to clean up my neutron db. At some point (which I don't know exactly, probably train) my neutron database got inconsistent, apparently one of the upgrades did not go as planned. The interesting thing is that the database still works, I just upgraded from ussuri to victoria where that issue popped up again during 'neutron-db-manage upgrade --expand', I'll add the information at the end of this email. Apparently, I have multiple heads, and one of them is from train, it seems as if I never ran --contract (or it failed and I didn't notice). Just some additional information what I did with this database: this cloud started out as a test environment with a single control node and then became a production environment. About two and a half years ago we decided to reinstall this cloud with version ussuri and import the databases. I had a virtual machine in which I upgraded the database dump from production to the latest versions at that time. That all worked quite well, I only didn't notice that something was missing. Now that I finished the U --> V upgrade I want to fix this inconsistency, I just have no idea how to do it. As I'm not sure how all the neutron-db-manage commands work exactly I'd like to ask for some guidance. For example, could the "stamp" command possibly help? Or how else can I get rid of the train head and/or how to get the train revision to "contract" so I can finish the upgrade and contract the victoria revision? I can paste the whole neutron-db history if necessary (neutron-db-manage history), please let me know what information would be required to get to the bottom of this. Any help is greatly appreciated!
Thanks! Eugen
---snip--- controller01:~ # neutron-db-manage upgrade --expand [...] alembic.script.revision.MultipleHeads: Multiple heads are present for given argument 'expand@head'; 633d74ebbc4b, I38991de2b4
controller01:~ # neutron-db-manage current --verbose Running current for neutron ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn /neutron: Rev: bebe95aae4d4 (head) Parent: b5344a66e818 Branch names: contract Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py
Rev: 633d74ebbc4b (head) Parent: 6c9eb0469914 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
Rev: I38991de2b4 (head) Parent: 49d8622c5221 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py
OK ---snip---
Hi, not really, no. I have no explanation how those files got there, to be honest. We're using openSUSE Leap (currently 15.2) and the respective repos from openSUSE. By the way, I only see those files on one of the control nodes, that's irritating me even more. But if those files are not known, maybe I should just delete them and the contract directories as well? Because during the next upgrade I'll probably have the same issue again. So if I see it correctly the two "contract" directories should be removed /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/contract /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract as well as this revision file: /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py Comparing with the "native" V installation (and the other control node) I should only keep two of these files: controller01:~ # ll /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/ insgesamt 16 -rw-r--r-- 1 root root 900 30. Mär 2021 633d74ebbc4b_.py <-- delete -rw-r--r-- 1 root root 1694 14. Nov 16:07 63fd95af7dcd_conntrack_helper.py -rw-r--r-- 1 root root 900 30. Mär 2021 6c9eb0469914_.py <-- delete -rw-r--r-- 1 root root 1134 14. Nov 16:07 c613d0b82681_subnet_force_network_id.py drwxr-xr-x 2 root root 312 23. Nov 11:09 __pycache__ I believe that should clean it up. Then I'll import the merged neutron database and run the upgrade commands again. Does that make sense? Thanks! Eugen Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
Please check the code you have. Those revisions (633d74ebbc4b, bebe95aae4d4) do not exist in the Neutron repository. File [1] (or something similar with the same prefix) does not exist. Are you using a customized Neutron repository?
Regards.
[1]/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
On Fri, Nov 25, 2022 at 8:57 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
I believe they are neutron revisions, here's the output from yesterday's neutron-db-manage attempt:
---snip--- Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 5c85685d616d -> c43a0ddb6a03 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c43a0ddb6a03 -> b5344a66e818 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade b5344a66e818 -> bebe95aae4d4 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c613d0b82681 -> 6c9eb0469914 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 6c9eb0469914 -> 633d74ebbc4b Nov 23 12:51:52 controller01 neutron-db-manage[25913]: Running upgrade for neutron ... Nov 23 12:51:52 controller01 neutron-db-manage[25913]: OK ---snip---
And here's where they are located, apparently from train version:
---snip--- controller01:~ # grep -r 633d74ebbc4b /usr/lib/python3.6/site-packages/ Übereinstimmungen in Binärdatei
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/__pycache__/633d74ebbc4b_.cpython-36.pyc /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:Revision ID: 633d74ebbc4b /usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:revision = '633d74ebbc4b' ---snip---
Next week we'll try it with merging fresh victoria schema with our production data, then run the upgrade command again.
Thanks, Eugen Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
I don't know how it is possible that you have 3 registers in this table. And the first two are not IDs of any Neutron revision. I would suggest you to (1) check the DB schema deployed against a fresh deployed system (in Victoria version) and (2) fix this table to point to the correct revision numbers.
Regards.
On Fri, Nov 25, 2022 at 3:51 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
thanks for your quick response.
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
That explains why I saw the newton revision in a new victoria cluster :-)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided.
The table alembic_versions contains the three versions I provided at the end of my email:
MariaDB [neutron]> select * from alembic_version; +--------------+ | version_num | +--------------+ | 633d74ebbc4b | | bebe95aae4d4 | | I38991de2b4 | +--------------+
I already tried to manipulate the table so I would only have those two versions you already mentioned, but then the upgrade --expand command alternates the database again with the mentioned error message ("Multiple heads are present").
Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version.
That's what I wanted to try next, export only the db schema (no data) from a working victoria neutron database, then export only data from our production db and merge those, then import that into the production and try to run upgrade --expand and --contract again. But I didn't want to fiddle around too much in the production, that's why I wanted to ask for your guidance first. But IIUC even if I changed the table alembic_versions again and import the merged db, wouldn't upgrade --expand somehow try to alternate the table again? I don't see where the train revision comes from exactly, could you clarify, please? It seems like I always get back to square one when running the --expand command.
Thanks! Eugen
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided. Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run this version.
Regards.
On Fri, Nov 25, 2022 at 1:58 PM Eugen Block <eblock@nde.ag> wrote:
Hi *,
I'd like to ask you for advice on how to clean up my neutron db. At some point (which I don't know exactly, probably train) my neutron database got inconsistent, apparently one of the upgrades did not go as planned. The interesting thing is that the database still works, I just upgraded from ussuri to victoria where that issue popped up again during 'neutron-db-manage upgrade --expand', I'll add the information at the end of this email. Apparently, I have multiple heads, and one of them is from train, it seems as if I never ran --contract (or it failed and I didn't notice). Just some additional information what I did with this database: this cloud started out as a test environment with a single control node and then became a production environment. About two and a half years ago we decided to reinstall this cloud with version ussuri and import the databases. I had a virtual machine in which I upgraded the database dump from production to the latest versions at that time. That all worked quite well, I only didn't notice that something was missing. Now that I finished the U --> V upgrade I want to fix this inconsistency, I just have no idea how to do it. As I'm not sure how all the neutron-db-manage commands work exactly I'd like to ask for some guidance. For example, could the "stamp" command possibly help? Or how else can I get rid of the train head and/or how to get the train revision to "contract" so I can finish the upgrade and contract the victoria revision? I can paste the whole neutron-db history if necessary (neutron-db-manage history), please let me know what information would be required to get to the bottom of this. Any help is greatly appreciated!
Thanks! Eugen
---snip--- controller01:~ # neutron-db-manage upgrade --expand [...] alembic.script.revision.MultipleHeads: Multiple heads are present for given argument 'expand@head'; 633d74ebbc4b, I38991de2b4
controller01:~ # neutron-db-manage current --verbose Running current for neutron ... INFO [alembic.runtime.migration] Context impl MySQLImpl. INFO [alembic.runtime.migration] Will assume non-transactional DDL. Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn /neutron: Rev: bebe95aae4d4 (head) Parent: b5344a66e818 Branch names: contract Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py
Rev: 633d74ebbc4b (head) Parent: 6c9eb0469914 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
Rev: I38991de2b4 (head) Parent: 49d8622c5221 Branch names: expand Path:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py
OK ---snip---
Yes, but you should also be sure what is the status of the DB schema. That means to check what is the latest migration file applied and set that revision ID on the "neutron.alembic_version" table. On Mon, Nov 28, 2022 at 11:31 AM Eugen Block <eblock@nde.ag> wrote:
Hi,
not really, no. I have no explanation how those files got there, to be honest. We're using openSUSE Leap (currently 15.2) and the respective repos from openSUSE. By the way, I only see those files on one of the control nodes, that's irritating me even more. But if those files are not known, maybe I should just delete them and the contract directories as well? Because during the next upgrade I'll probably have the same issue again. So if I see it correctly the two "contract" directories should be removed
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/contract
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract
as well as this revision file:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
Comparing with the "native" V installation (and the other control node) I should only keep two of these files:
controller01:~ # ll
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/ insgesamt 16 -rw-r--r-- 1 root root 900 30. Mär 2021 633d74ebbc4b_.py <-- delete -rw-r--r-- 1 root root 1694 14. Nov 16:07 63fd95af7dcd_conntrack_helper.py -rw-r--r-- 1 root root 900 30. Mär 2021 6c9eb0469914_.py <-- delete -rw-r--r-- 1 root root 1134 14. Nov 16:07 c613d0b82681_subnet_force_network_id.py drwxr-xr-x 2 root root 312 23. Nov 11:09 __pycache__
I believe that should clean it up. Then I'll import the merged neutron database and run the upgrade commands again. Does that make sense?
Thanks! Eugen
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
Please check the code you have. Those revisions (633d74ebbc4b, bebe95aae4d4) do not exist in the Neutron repository. File [1] (or something similar with the same prefix) does not exist. Are you using a customized Neutron repository?
Regards.
[1]/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
On Fri, Nov 25, 2022 at 8:57 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
I believe they are neutron revisions, here's the output from yesterday's neutron-db-manage attempt:
---snip--- Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 5c85685d616d -> c43a0ddb6a03 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c43a0ddb6a03 -> b5344a66e818 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade b5344a66e818 -> bebe95aae4d4 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c613d0b82681 -> 6c9eb0469914 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 6c9eb0469914 -> 633d74ebbc4b Nov 23 12:51:52 controller01 neutron-db-manage[25913]: Running upgrade for neutron ... Nov 23 12:51:52 controller01 neutron-db-manage[25913]: OK ---snip---
And here's where they are located, apparently from train version:
---snip--- controller01:~ # grep -r 633d74ebbc4b /usr/lib/python3.6/site-packages/ Übereinstimmungen in Binärdatei
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:Revision
ID: 633d74ebbc4b
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:revision
= '633d74ebbc4b' ---snip---
Next week we'll try it with merging fresh victoria schema with our production data, then run the upgrade command again.
Thanks, Eugen Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
I don't know how it is possible that you have 3 registers in this
And the first two are not IDs of any Neutron revision. I would suggest you to (1) check the DB schema deployed against a fresh deployed system (in Victoria version) and (2) fix this table to point to the correct revision numbers.
Regards.
On Fri, Nov 25, 2022 at 3:51 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
thanks for your quick response.
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
That explains why I saw the newton revision in a new victoria cluster :-)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided.
The table alembic_versions contains the three versions I provided at the end of my email:
MariaDB [neutron]> select * from alembic_version; +--------------+ | version_num | +--------------+ | 633d74ebbc4b | | bebe95aae4d4 | | I38991de2b4 | +--------------+
I already tried to manipulate the table so I would only have those two versions you already mentioned, but then the upgrade --expand command alternates the database again with the mentioned error message ("Multiple heads are present").
Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run
version.
That's what I wanted to try next, export only the db schema (no data) from a working victoria neutron database, then export only data from our production db and merge those, then import that into the production and try to run upgrade --expand and --contract again. But I didn't want to fiddle around too much in the production, that's why I wanted to ask for your guidance first. But IIUC even if I changed the table alembic_versions again and import the merged db, wouldn't upgrade --expand somehow try to alternate the table again? I don't see where the train revision comes from exactly, could you clarify, please? It seems like I always get back to square one when running the --expand command.
Thanks! Eugen
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
In Neutron we don't support contract operations since Newton.
If you are in Victoria and you correctly finished the DB migration, your HEADs should be: * contract: 5c85685d616d (from Newton) * expand: I38991de2b4 (from the last DB change in Victoria, source_and_destination_ip_prefix_neutron_metering_rule)
Please check what you have in the DB table neutron.alembic_version. The first register should be the expand number, the second the contract one. If not, update them with the ones I've provided. Before executing the migration tool again, be sure the DB schema matches the latest migration patch for your version. You can deploy a VM with devstack and run
version.
Regards.
On Fri, Nov 25, 2022 at 1:58 PM Eugen Block <eblock@nde.ag> wrote:
> Hi *, > > I'd like to ask you for advice on how to clean up my neutron db. At > some point (which I don't know exactly, probably train) my neutron > database got inconsistent, apparently one of the upgrades did not go > as planned. The interesting thing is that the database still works, I > just upgraded from ussuri to victoria where that issue popped up again > during 'neutron-db-manage upgrade --expand', I'll add the information > at the end of this email. Apparently, I have multiple heads, and one > of them is from train, it seems as if I never ran --contract (or it > failed and I didn't notice). > Just some additional information what I did with this database:
> cloud started out as a test environment with a single control node and > then became a production environment. About two and a half years ago > we decided to reinstall this cloud with version ussuri and import
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/__pycache__/633d74ebbc4b_.cpython-36.pyc table. this this this the
> databases. I had a virtual machine in which I upgraded the database > dump from production to the latest versions at that time. That all > worked quite well, I only didn't notice that something was missing. > Now that I finished the U --> V upgrade I want to fix this > inconsistency, I just have no idea how to do it. As I'm not sure how > all the neutron-db-manage commands work exactly I'd like to ask for > some guidance. For example, could the "stamp" command possibly help? > Or how else can I get rid of the train head and/or how to get the > train revision to "contract" so I can finish the upgrade and contract > the victoria revision? I can paste the whole neutron-db history if > necessary (neutron-db-manage history), please let me know what > information would be required to get to the bottom of this. > Any help is greatly appreciated! > > Thanks! > Eugen > > > ---snip--- > controller01:~ # neutron-db-manage upgrade --expand > [...] > alembic.script.revision.MultipleHeads: Multiple heads are present for > given argument 'expand@head'; 633d74ebbc4b, I38991de2b4 > > controller01:~ # neutron-db-manage current --verbose > Running current for neutron ... > INFO [alembic.runtime.migration] Context impl MySQLImpl. > INFO [alembic.runtime.migration] Will assume non-transactional DDL. > Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn > /neutron: > Rev: bebe95aae4d4 (head) > Parent: b5344a66e818 > Branch names: contract > Path: > >
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py
> > Rev: 633d74ebbc4b (head) > Parent: 6c9eb0469914 > Branch names: expand > Path: > >
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
> > Rev: I38991de2b4 (head) > Parent: 49d8622c5221 > Branch names: expand > Path: > >
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py
> > OK > ---snip--- > > >
How do I check what the latest applied migration file was? Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Yes, but you should also be sure what is the status of the DB schema. That means to check what is the latest migration file applied and set that revision ID on the "neutron.alembic_version" table.
On Mon, Nov 28, 2022 at 11:31 AM Eugen Block <eblock@nde.ag> wrote:
Hi,
not really, no. I have no explanation how those files got there, to be honest. We're using openSUSE Leap (currently 15.2) and the respective repos from openSUSE. By the way, I only see those files on one of the control nodes, that's irritating me even more. But if those files are not known, maybe I should just delete them and the contract directories as well? Because during the next upgrade I'll probably have the same issue again. So if I see it correctly the two "contract" directories should be removed
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/contract
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract
as well as this revision file:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
Comparing with the "native" V installation (and the other control node) I should only keep two of these files:
controller01:~ # ll
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/ insgesamt 16 -rw-r--r-- 1 root root 900 30. Mär 2021 633d74ebbc4b_.py <-- delete -rw-r--r-- 1 root root 1694 14. Nov 16:07 63fd95af7dcd_conntrack_helper.py -rw-r--r-- 1 root root 900 30. Mär 2021 6c9eb0469914_.py <-- delete -rw-r--r-- 1 root root 1134 14. Nov 16:07 c613d0b82681_subnet_force_network_id.py drwxr-xr-x 2 root root 312 23. Nov 11:09 __pycache__
I believe that should clean it up. Then I'll import the merged neutron database and run the upgrade commands again. Does that make sense?
Thanks! Eugen
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
Please check the code you have. Those revisions (633d74ebbc4b, bebe95aae4d4) do not exist in the Neutron repository. File [1] (or something similar with the same prefix) does not exist. Are you using a customized Neutron repository?
Regards.
[1]/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
On Fri, Nov 25, 2022 at 8:57 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
I believe they are neutron revisions, here's the output from yesterday's neutron-db-manage attempt:
---snip--- Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 5c85685d616d -> c43a0ddb6a03 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c43a0ddb6a03 -> b5344a66e818 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade b5344a66e818 -> bebe95aae4d4 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c613d0b82681 -> 6c9eb0469914 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 6c9eb0469914 -> 633d74ebbc4b Nov 23 12:51:52 controller01 neutron-db-manage[25913]: Running upgrade for neutron ... Nov 23 12:51:52 controller01 neutron-db-manage[25913]: OK ---snip---
And here's where they are located, apparently from train version:
---snip--- controller01:~ # grep -r 633d74ebbc4b /usr/lib/python3.6/site-packages/ Übereinstimmungen in Binärdatei
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:Revision
ID: 633d74ebbc4b
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:revision
= '633d74ebbc4b' ---snip---
Next week we'll try it with merging fresh victoria schema with our production data, then run the upgrade command again.
Thanks, Eugen Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
I don't know how it is possible that you have 3 registers in this
And the first two are not IDs of any Neutron revision. I would suggest you to (1) check the DB schema deployed against a fresh deployed system (in Victoria version) and (2) fix this table to point to the correct revision numbers.
Regards.
On Fri, Nov 25, 2022 at 3:51 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
thanks for your quick response.
> In Neutron we don't support contract operations since Newton. > > If you are in Victoria and you correctly finished the DB migration, your > HEADs should be: > * contract: 5c85685d616d (from Newton) > * expand: I38991de2b4 (from the last DB change in Victoria, > source_and_destination_ip_prefix_neutron_metering_rule)
That explains why I saw the newton revision in a new victoria cluster :-)
> Please check what you have in the DB table neutron.alembic_version. The > first register should be the expand number, the second the contract one. If > not, update them with the ones I've provided.
The table alembic_versions contains the three versions I provided at the end of my email:
MariaDB [neutron]> select * from alembic_version; +--------------+ | version_num | +--------------+ | 633d74ebbc4b | | bebe95aae4d4 | | I38991de2b4 | +--------------+
I already tried to manipulate the table so I would only have those two versions you already mentioned, but then the upgrade --expand command alternates the database again with the mentioned error message ("Multiple heads are present").
> Before executing the > migration tool again, be sure the DB schema matches the latest migration > patch for your version. You can deploy a VM with devstack and run
> version.
That's what I wanted to try next, export only the db schema (no data) from a working victoria neutron database, then export only data from our production db and merge those, then import that into the production and try to run upgrade --expand and --contract again. But I didn't want to fiddle around too much in the production, that's why I wanted to ask for your guidance first. But IIUC even if I changed the table alembic_versions again and import the merged db, wouldn't upgrade --expand somehow try to alternate the table again? I don't see where the train revision comes from exactly, could you clarify, please? It seems like I always get back to square one when running the --expand command.
Thanks! Eugen
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
> Hi Eugen: > > In Neutron we don't support contract operations since Newton. > > If you are in Victoria and you correctly finished the DB migration, your > HEADs should be: > * contract: 5c85685d616d (from Newton) > * expand: I38991de2b4 (from the last DB change in Victoria, > source_and_destination_ip_prefix_neutron_metering_rule) > > Please check what you have in the DB table neutron.alembic_version. The > first register should be the expand number, the second the contract one. If > not, update them with the ones I've provided. Before executing the > migration tool again, be sure the DB schema matches the latest migration > patch for your version. You can deploy a VM with devstack and run
> version. > > Regards. > > > On Fri, Nov 25, 2022 at 1:58 PM Eugen Block <eblock@nde.ag> wrote: > >> Hi *, >> >> I'd like to ask you for advice on how to clean up my neutron db. At >> some point (which I don't know exactly, probably train) my neutron >> database got inconsistent, apparently one of the upgrades did not go >> as planned. The interesting thing is that the database still works, I >> just upgraded from ussuri to victoria where that issue popped up again >> during 'neutron-db-manage upgrade --expand', I'll add the information >> at the end of this email. Apparently, I have multiple heads, and one >> of them is from train, it seems as if I never ran --contract (or it >> failed and I didn't notice). >> Just some additional information what I did with this database:
>> cloud started out as a test environment with a single control node and >> then became a production environment. About two and a half years ago >> we decided to reinstall this cloud with version ussuri and import
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/__pycache__/633d74ebbc4b_.cpython-36.pyc table. this this this the
>> databases. I had a virtual machine in which I upgraded the database >> dump from production to the latest versions at that time. That all >> worked quite well, I only didn't notice that something was missing. >> Now that I finished the U --> V upgrade I want to fix this >> inconsistency, I just have no idea how to do it. As I'm not sure how >> all the neutron-db-manage commands work exactly I'd like to ask for >> some guidance. For example, could the "stamp" command possibly help? >> Or how else can I get rid of the train head and/or how to get the >> train revision to "contract" so I can finish the upgrade and contract >> the victoria revision? I can paste the whole neutron-db history if >> necessary (neutron-db-manage history), please let me know what >> information would be required to get to the bottom of this. >> Any help is greatly appreciated! >> >> Thanks! >> Eugen >> >> >> ---snip--- >> controller01:~ # neutron-db-manage upgrade --expand >> [...] >> alembic.script.revision.MultipleHeads: Multiple heads are present for >> given argument 'expand@head'; 633d74ebbc4b, I38991de2b4 >> >> controller01:~ # neutron-db-manage current --verbose >> Running current for neutron ... >> INFO [alembic.runtime.migration] Context impl MySQLImpl. >> INFO [alembic.runtime.migration] Will assume non-transactional DDL. >> Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn >> /neutron: >> Rev: bebe95aae4d4 (head) >> Parent: b5344a66e818 >> Branch names: contract >> Path: >> >>
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py
>> >> Rev: 633d74ebbc4b (head) >> Parent: 6c9eb0469914 >> Branch names: expand >> Path: >> >>
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
>> >> Rev: I38991de2b4 (head) >> Parent: 49d8622c5221 >> Branch names: expand >> Path: >> >>
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py
>> >> OK >> ---snip--- >> >> >>
Use "neutron-db-manage history" to check what your alembic migration current status is. On Mon, Nov 28, 2022 at 12:03 PM Eugen Block <eblock@nde.ag> wrote:
How do I check what the latest applied migration file was?
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Yes, but you should also be sure what is the status of the DB schema. That means to check what is the latest migration file applied and set that revision ID on the "neutron.alembic_version" table.
On Mon, Nov 28, 2022 at 11:31 AM Eugen Block <eblock@nde.ag> wrote:
Hi,
not really, no. I have no explanation how those files got there, to be honest. We're using openSUSE Leap (currently 15.2) and the respective repos from openSUSE. By the way, I only see those files on one of the control nodes, that's irritating me even more. But if those files are not known, maybe I should just delete them and the contract directories as well? Because during the next upgrade I'll probably have the same issue again. So if I see it correctly the two "contract" directories should be removed
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/contract
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract
as well as this revision file:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
Comparing with the "native" V installation (and the other control node) I should only keep two of these files:
controller01:~ # ll
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/
insgesamt 16 -rw-r--r-- 1 root root 900 30. Mär 2021 633d74ebbc4b_.py <-- delete -rw-r--r-- 1 root root 1694 14. Nov 16:07 63fd95af7dcd_conntrack_helper.py -rw-r--r-- 1 root root 900 30. Mär 2021 6c9eb0469914_.py <-- delete -rw-r--r-- 1 root root 1134 14. Nov 16:07 c613d0b82681_subnet_force_network_id.py drwxr-xr-x 2 root root 312 23. Nov 11:09 __pycache__
I believe that should clean it up. Then I'll import the merged neutron database and run the upgrade commands again. Does that make sense?
Thanks! Eugen
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
Please check the code you have. Those revisions (633d74ebbc4b, bebe95aae4d4) do not exist in the Neutron repository. File [1] (or something similar with the same prefix) does not exist. Are you using a customized Neutron repository?
Regards.
[1]/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
On Fri, Nov 25, 2022 at 8:57 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
I believe they are neutron revisions, here's the output from yesterday's neutron-db-manage attempt:
---snip--- Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 5c85685d616d ->
c43a0ddb6a03
Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c43a0ddb6a03 -> b5344a66e818 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade b5344a66e818 -> bebe95aae4d4 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c613d0b82681 -> 6c9eb0469914 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 6c9eb0469914 -> 633d74ebbc4b Nov 23 12:51:52 controller01 neutron-db-manage[25913]: Running upgrade for neutron ... Nov 23 12:51:52 controller01 neutron-db-manage[25913]: OK ---snip---
And here's where they are located, apparently from train version:
---snip--- controller01:~ # grep -r 633d74ebbc4b /usr/lib/python3.6/site-packages/ Übereinstimmungen in Binärdatei
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/__pycache__/633d74ebbc4b_.cpython-36.pyc
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:Revision
ID: 633d74ebbc4b
= '633d74ebbc4b' ---snip---
Next week we'll try it with merging fresh victoria schema with our production data, then run the upgrade command again.
Thanks, Eugen Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
I don't know how it is possible that you have 3 registers in this table. And the first two are not IDs of any Neutron revision. I would suggest you to (1) check the DB schema deployed against a fresh deployed system (in Victoria version) and (2) fix this table to point to the correct revision numbers.
Regards.
On Fri, Nov 25, 2022 at 3:51 PM Eugen Block <eblock@nde.ag> wrote:
> Hi, > > thanks for your quick response. > > > In Neutron we don't support contract operations since Newton. > > > > If you are in Victoria and you correctly finished the DB migration, your > > HEADs should be: > > * contract: 5c85685d616d (from Newton) > > * expand: I38991de2b4 (from the last DB change in Victoria, > > source_and_destination_ip_prefix_neutron_metering_rule) > > That explains why I saw the newton revision in a new victoria cluster :-) > > > Please check what you have in the DB table neutron.alembic_version. The > > first register should be the expand number, the second the contract one. > If > > not, update them with the ones I've provided. > > The table alembic_versions contains the three versions I provided at > the end of my email: > > MariaDB [neutron]> select * from alembic_version; > +--------------+ > | version_num | > +--------------+ > | 633d74ebbc4b | > | bebe95aae4d4 | > | I38991de2b4 | > +--------------+ > > I already tried to manipulate the table so I would only have those two > versions you already mentioned, but then the upgrade --expand command > alternates the database again with the mentioned error message > ("Multiple heads are present"). > > > Before executing the > > migration tool again, be sure the DB schema matches the latest migration > > patch for your version. You can deploy a VM with devstack and run this > > version. > > That's what I wanted to try next, export only the db schema (no data) > from a working victoria neutron database, then export only data from > our production db and merge those, then import that into the > production and try to run upgrade --expand and --contract again. But I > didn't want to fiddle around too much in the production, that's why I > wanted to ask for your guidance first. > But IIUC even if I changed the table alembic_versions again and import > the merged db, wouldn't upgrade --expand somehow try to alternate
> table again? I don't see where the train revision comes from exactly, > could you clarify, please? It seems like I always get back to square > one when running the --expand command. > > Thanks! > Eugen > > Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>: > > > Hi Eugen: > > > > In Neutron we don't support contract operations since Newton. > > > > If you are in Victoria and you correctly finished the DB migration, your > > HEADs should be: > > * contract: 5c85685d616d (from Newton) > > * expand: I38991de2b4 (from the last DB change in Victoria, > > source_and_destination_ip_prefix_neutron_metering_rule) > > > > Please check what you have in the DB table neutron.alembic_version. The > > first register should be the expand number, the second the contract one. > If > > not, update them with the ones I've provided. Before executing
> > migration tool again, be sure the DB schema matches the latest migration > > patch for your version. You can deploy a VM with devstack and run this > > version. > > > > Regards. > > > > > > On Fri, Nov 25, 2022 at 1:58 PM Eugen Block <eblock@nde.ag> wrote: > > > >> Hi *, > >> > >> I'd like to ask you for advice on how to clean up my neutron db. At > >> some point (which I don't know exactly, probably train) my neutron > >> database got inconsistent, apparently one of the upgrades did not go > >> as planned. The interesting thing is that the database still works, I > >> just upgraded from ussuri to victoria where that issue popped up again > >> during 'neutron-db-manage upgrade --expand', I'll add the information > >> at the end of this email. Apparently, I have multiple heads, and one > >> of them is from train, it seems as if I never ran --contract (or it > >> failed and I didn't notice). > >> Just some additional information what I did with this database: this > >> cloud started out as a test environment with a single control node and > >> then became a production environment. About two and a half years ago > >> we decided to reinstall this cloud with version ussuri and import the > >> databases. I had a virtual machine in which I upgraded the database > >> dump from production to the latest versions at that time. That all > >> worked quite well, I only didn't notice that something was missing. > >> Now that I finished the U --> V upgrade I want to fix this > >> inconsistency, I just have no idea how to do it. As I'm not sure how > >> all the neutron-db-manage commands work exactly I'd like to ask for > >> some guidance. For example, could the "stamp" command possibly help? > >> Or how else can I get rid of the train head and/or how to get
> >> train revision to "contract" so I can finish the upgrade and contract > >> the victoria revision? I can paste the whole neutron-db history if > >> necessary (neutron-db-manage history), please let me know what > >> information would be required to get to the bottom of this. > >> Any help is greatly appreciated! > >> > >> Thanks! > >> Eugen > >> > >> > >> ---snip--- > >> controller01:~ # neutron-db-manage upgrade --expand > >> [...] > >> alembic.script.revision.MultipleHeads: Multiple heads are
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:revision the the the present
for
> >> given argument 'expand@head'; 633d74ebbc4b, I38991de2b4 > >> > >> controller01:~ # neutron-db-manage current --verbose > >> Running current for neutron ... > >> INFO [alembic.runtime.migration] Context impl MySQLImpl. > >> INFO [alembic.runtime.migration] Will assume non-transactional DDL. > >> Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn > >> /neutron: > >> Rev: bebe95aae4d4 (head) > >> Parent: b5344a66e818 > >> Branch names: contract > >> Path: > >> > >> >
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py
> >> > >> Rev: 633d74ebbc4b (head) > >> Parent: 6c9eb0469914 > >> Branch names: expand > >> Path: > >> > >> >
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
> >> > >> Rev: I38991de2b4 (head) > >> Parent: 49d8622c5221 > >> Branch names: expand > >> Path: > >> > >> >
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py
> >> > >> OK > >> ---snip--- > >> > >> > >> > > > >
Hi Rodolfo, thanks again for your assistance, I appreciate it! I managed to recreate the neutron database in our production cloud by merging the schema from a "native" victoria cloud and our data dump. The services came up successfully and some test networks + instances started successfully, dhcp is working, so all seems good now. Hopefully, this won't be such an issue during the next upgrade. Thanks! Eugen Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Use "neutron-db-manage history" to check what your alembic migration current status is.
On Mon, Nov 28, 2022 at 12:03 PM Eugen Block <eblock@nde.ag> wrote:
How do I check what the latest applied migration file was?
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Yes, but you should also be sure what is the status of the DB schema. That means to check what is the latest migration file applied and set that revision ID on the "neutron.alembic_version" table.
On Mon, Nov 28, 2022 at 11:31 AM Eugen Block <eblock@nde.ag> wrote:
Hi,
not really, no. I have no explanation how those files got there, to be honest. We're using openSUSE Leap (currently 15.2) and the respective repos from openSUSE. By the way, I only see those files on one of the control nodes, that's irritating me even more. But if those files are not known, maybe I should just delete them and the contract directories as well? Because during the next upgrade I'll probably have the same issue again. So if I see it correctly the two "contract" directories should be removed
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/contract
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract
as well as this revision file:
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
Comparing with the "native" V installation (and the other control node) I should only keep two of these files:
controller01:~ # ll
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/
insgesamt 16 -rw-r--r-- 1 root root 900 30. Mär 2021 633d74ebbc4b_.py <-- delete -rw-r--r-- 1 root root 1694 14. Nov 16:07 63fd95af7dcd_conntrack_helper.py -rw-r--r-- 1 root root 900 30. Mär 2021 6c9eb0469914_.py <-- delete -rw-r--r-- 1 root root 1134 14. Nov 16:07 c613d0b82681_subnet_force_network_id.py drwxr-xr-x 2 root root 312 23. Nov 11:09 __pycache__
I believe that should clean it up. Then I'll import the merged neutron database and run the upgrade commands again. Does that make sense?
Thanks! Eugen
Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
Hi Eugen:
Please check the code you have. Those revisions (633d74ebbc4b, bebe95aae4d4) do not exist in the Neutron repository. File [1] (or something similar with the same prefix) does not exist. Are you using a customized Neutron repository?
Regards.
[1]/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
On Fri, Nov 25, 2022 at 8:57 PM Eugen Block <eblock@nde.ag> wrote:
Hi,
I believe they are neutron revisions, here's the output from yesterday's neutron-db-manage attempt:
---snip--- Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Context impl MySQLImpl. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Will assume non-transactional DDL. Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 5c85685d616d ->
c43a0ddb6a03
Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c43a0ddb6a03 -> b5344a66e818 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade b5344a66e818 -> bebe95aae4d4 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade c613d0b82681 -> 6c9eb0469914 Nov 23 12:51:51 controller01 neutron-db-manage[25913]: INFO [alembic.runtime.migration] Running upgrade 6c9eb0469914 -> 633d74ebbc4b Nov 23 12:51:52 controller01 neutron-db-manage[25913]: Running upgrade for neutron ... Nov 23 12:51:52 controller01 neutron-db-manage[25913]: OK ---snip---
And here's where they are located, apparently from train version:
---snip--- controller01:~ # grep -r 633d74ebbc4b /usr/lib/python3.6/site-packages/ Übereinstimmungen in Binärdatei
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/__pycache__/633d74ebbc4b_.cpython-36.pyc
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:Revision
ID: 633d74ebbc4b
= '633d74ebbc4b' ---snip---
Next week we'll try it with merging fresh victoria schema with our production data, then run the upgrade command again.
Thanks, Eugen Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>:
> Hi Eugen: > > I don't know how it is possible that you have 3 registers in this table. > And the first two are not IDs of any Neutron revision. I would suggest you > to (1) check the DB schema deployed against a fresh deployed system (in > Victoria version) and (2) fix this table to point to the correct revision > numbers. > > Regards. > > > On Fri, Nov 25, 2022 at 3:51 PM Eugen Block <eblock@nde.ag> wrote: > >> Hi, >> >> thanks for your quick response. >> >> > In Neutron we don't support contract operations since Newton. >> > >> > If you are in Victoria and you correctly finished the DB migration, your >> > HEADs should be: >> > * contract: 5c85685d616d (from Newton) >> > * expand: I38991de2b4 (from the last DB change in Victoria, >> > source_and_destination_ip_prefix_neutron_metering_rule) >> >> That explains why I saw the newton revision in a new victoria cluster :-) >> >> > Please check what you have in the DB table neutron.alembic_version. The >> > first register should be the expand number, the second the contract one. >> If >> > not, update them with the ones I've provided. >> >> The table alembic_versions contains the three versions I provided at >> the end of my email: >> >> MariaDB [neutron]> select * from alembic_version; >> +--------------+ >> | version_num | >> +--------------+ >> | 633d74ebbc4b | >> | bebe95aae4d4 | >> | I38991de2b4 | >> +--------------+ >> >> I already tried to manipulate the table so I would only have those two >> versions you already mentioned, but then the upgrade --expand command >> alternates the database again with the mentioned error message >> ("Multiple heads are present"). >> >> > Before executing the >> > migration tool again, be sure the DB schema matches the latest migration >> > patch for your version. You can deploy a VM with devstack and run this >> > version. >> >> That's what I wanted to try next, export only the db schema (no data) >> from a working victoria neutron database, then export only data from >> our production db and merge those, then import that into the >> production and try to run upgrade --expand and --contract again. But I >> didn't want to fiddle around too much in the production, that's why I >> wanted to ask for your guidance first. >> But IIUC even if I changed the table alembic_versions again and import >> the merged db, wouldn't upgrade --expand somehow try to alternate
>> table again? I don't see where the train revision comes from exactly, >> could you clarify, please? It seems like I always get back to square >> one when running the --expand command. >> >> Thanks! >> Eugen >> >> Zitat von Rodolfo Alonso Hernandez <ralonsoh@redhat.com>: >> >> > Hi Eugen: >> > >> > In Neutron we don't support contract operations since Newton. >> > >> > If you are in Victoria and you correctly finished the DB migration, your >> > HEADs should be: >> > * contract: 5c85685d616d (from Newton) >> > * expand: I38991de2b4 (from the last DB change in Victoria, >> > source_and_destination_ip_prefix_neutron_metering_rule) >> > >> > Please check what you have in the DB table neutron.alembic_version. The >> > first register should be the expand number, the second the contract one. >> If >> > not, update them with the ones I've provided. Before executing
>> > migration tool again, be sure the DB schema matches the latest migration >> > patch for your version. You can deploy a VM with devstack and run this >> > version. >> > >> > Regards. >> > >> > >> > On Fri, Nov 25, 2022 at 1:58 PM Eugen Block <eblock@nde.ag> wrote: >> > >> >> Hi *, >> >> >> >> I'd like to ask you for advice on how to clean up my neutron db. At >> >> some point (which I don't know exactly, probably train) my neutron >> >> database got inconsistent, apparently one of the upgrades did not go >> >> as planned. The interesting thing is that the database still works, I >> >> just upgraded from ussuri to victoria where that issue popped up again >> >> during 'neutron-db-manage upgrade --expand', I'll add the information >> >> at the end of this email. Apparently, I have multiple heads, and one >> >> of them is from train, it seems as if I never ran --contract (or it >> >> failed and I didn't notice). >> >> Just some additional information what I did with this database: this >> >> cloud started out as a test environment with a single control node and >> >> then became a production environment. About two and a half years ago >> >> we decided to reinstall this cloud with version ussuri and import the >> >> databases. I had a virtual machine in which I upgraded the database >> >> dump from production to the latest versions at that time. That all >> >> worked quite well, I only didn't notice that something was missing. >> >> Now that I finished the U --> V upgrade I want to fix this >> >> inconsistency, I just have no idea how to do it. As I'm not sure how >> >> all the neutron-db-manage commands work exactly I'd like to ask for >> >> some guidance. For example, could the "stamp" command possibly help? >> >> Or how else can I get rid of the train head and/or how to get
>> >> train revision to "contract" so I can finish the upgrade and contract >> >> the victoria revision? I can paste the whole neutron-db history if >> >> necessary (neutron-db-manage history), please let me know what >> >> information would be required to get to the bottom of this. >> >> Any help is greatly appreciated! >> >> >> >> Thanks! >> >> Eugen >> >> >> >> >> >> ---snip--- >> >> controller01:~ # neutron-db-manage upgrade --expand >> >> [...] >> >> alembic.script.revision.MultipleHeads: Multiple heads are
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py:revision the the the present
for
>> >> given argument 'expand@head'; 633d74ebbc4b, I38991de2b4 >> >> >> >> controller01:~ # neutron-db-manage current --verbose >> >> Running current for neutron ... >> >> INFO [alembic.runtime.migration] Context impl MySQLImpl. >> >> INFO [alembic.runtime.migration] Will assume non-transactional DDL. >> >> Current revision(s) for mysql+pymysql://neutron:XXXXX@controller.fqdn >> >> /neutron: >> >> Rev: bebe95aae4d4 (head) >> >> Parent: b5344a66e818 >> >> Branch names: contract >> >> Path: >> >> >> >> >>
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/ussuri/contract/bebe95aae4d4_.py
>> >> >> >> Rev: 633d74ebbc4b (head) >> >> Parent: 6c9eb0469914 >> >> Branch names: expand >> >> Path: >> >> >> >> >>
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/train/expand/633d74ebbc4b_.py
>> >> >> >> Rev: I38991de2b4 (head) >> >> Parent: 49d8622c5221 >> >> Branch names: expand >> >> Path: >> >> >> >> >>
/usr/lib/python3.6/site-packages/neutron/db/migration/alembic_migrations/versions/victoria/expand/I38991de2b4_source_and_destination_ip_prefix_neutron_metering_rule.py
>> >> >> >> OK >> >> ---snip--- >> >> >> >> >> >> >> >> >> >>
participants (2)
-
Eugen Block
-
Rodolfo Alonso Hernandez