[openstack-dev] [nova] unit test migration failure specific to MySQL/MariaDB - 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey'

Matt Riedemann mriedem at linux.vnet.ibm.com
Wed Jan 7 18:27:39 UTC 2015



On 1/6/2015 5:40 PM, Mike Bayer wrote:
> Hello -
>
> Victor Sergeyev and I are both observing the following test failure which occurs with all the tests underneath nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.    This is against master with a brand new tox environment and everything at the default.
>
> It does not seem to be occurring on gates that run these tests and interestingly the tests seem to complete very quickly (under seven seconds) on the gate as well; the failures here take between 50-100 seconds to occur, not fully deterministically, and only on the MySQL backend; the Postgresql and SQLite versions of these tests pass.  I’m running against MariaDB server 10.0.14 with Python 2.7.8 on Fedora 21.
>
> Below is the test just for test_walk_versions, but the warnings (not necessarily the failures themselves) here also occur for test_migration_267 as well as test_innodb_tables.
>
> I’m still looking into what the cause of this is, I’d imagine it’s something related to newer MySQL versions or perhaps MariaDB vs. MySQL, I’m just putting it up here in case someone already knows what this is or has some clue to save me some time figuring it out.  I apologize if I’m just doing something dumb, I’ve only recently begun to run Nova’s test suite in full against all backends, so I haven’t yet put intelligent thought into this nor have I tried to yet look at the migration in question causing the problem.  Will do that next.
>
>
> [mbayer at thinkpad nova]$ tox -e py27 -- nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
> py27 develop-inst-noop: /home/mbayer/dev/openstack/nova
> py27 runtests: PYTHONHASHSEED='0'
> py27 runtests: commands[0] | find . -type f -name *.pyc -delete
> py27 runtests: commands[1] | bash tools/pretty_tox.sh nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
> running testr
> running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
> ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --list
> running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
> OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
> OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
> ${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests}  --load-list /tmp/tmpw7zqhE
>
> 2015-01-06 18:28:12.913 32435 WARNING oslo.db.sqlalchemy.session [req-5cc6731f-00ef-43df-8aec-4914a44d12c5 ] MySQL SQL mode is '', consider enabling TRADITIONAL or STRICT_ALL_TABLES
> {0} nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions [51.553131s] ... FAILED
>
> Captured traceback:
> ~~~~~~~~~~~~~~~~~~~
>      Traceback (most recent call last):
>        File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
>          self.walk_versions(self.snake_walk, self.downgrade)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
>          self.migrate_up(version, with_data=True)
>        File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
>          super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
>          self.REPOSITORY, version)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
>          return _migrate(url, repository, version, upgrade=True, err=err, **opts)
>        File "<string>", line 2, in _migrate
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
>          return f(*a, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
>          schema.runchange(ver, change, changeset.step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
>          change.run(self.engine, step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
>          script_func(engine)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
>          process_null_records(meta, scan=False)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
>          table.columns.uuid.alter(nullable=False)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
>          return alter_column(self, *p, **k)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
>          engine._run_visitor(visitorcallable, delta)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
>          conn._run_visitor(visitorcallable, element, **kwargs)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
>          **kwargs).traverse_single(element)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
>          ret = super(AlterTableVisitor, self).traverse_single(elem)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
>          return meth(obj, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
>          self.execute()
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
>          return self.connection.execute(self.buffer.getvalue())
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
>          return self._execute_text(object, multiparams, params)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
>          statement, parameters
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
>          util.raise_from_cause(newraise, exc_info)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
>          reraise(type(exception), exception, tb=exc_tb)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
>          cursor.execute(statement, parameters)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
>          self.errorhandler(self, exc, value)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
>          raise errorclass, errorvalue
>      OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
>      Traceback (most recent call last):
>      _StringException: Empty attachments:
>        stdout
>
>      stderr: {{{
>      /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
>        cursor.execute(statement, parameters)
>      }}}
>
>      Traceback (most recent call last):
>        File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
>          self.walk_versions(self.snake_walk, self.downgrade)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
>          self.migrate_up(version, with_data=True)
>        File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
>          super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
>          self.REPOSITORY, version)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
>          return _migrate(url, repository, version, upgrade=True, err=err, **opts)
>        File "<string>", line 2, in _migrate
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
>          return f(*a, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
>          schema.runchange(ver, change, changeset.step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
>          change.run(self.engine, step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
>          script_func(engine)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
>          process_null_records(meta, scan=False)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
>          table.columns.uuid.alter(nullable=False)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
>          return alter_column(self, *p, **k)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
>          engine._run_visitor(visitorcallable, delta)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
>          conn._run_visitor(visitorcallable, element, **kwargs)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
>          **kwargs).traverse_single(element)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
>          ret = super(AlterTableVisitor, self).traverse_single(elem)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
>          return meth(obj, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
>          self.execute()
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
>          return self.connection.execute(self.buffer.getvalue())
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
>          return self._execute_text(object, multiparams, params)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
>          statement, parameters
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
>          util.raise_from_cause(newraise, exc_info)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
>          reraise(type(exception), exception, tb=exc_tb)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
>          cursor.execute(statement, parameters)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
>          self.errorhandler(self, exc, value)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
>          raise errorclass, errorvalue
>      OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
>
>      Traceback (most recent call last):
>      _StringException: Empty attachments:
>        stdout
>
>      stderr: {{{
>      /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
>        cursor.execute(statement, parameters)
>      }}}
>
>      Traceback (most recent call last):
>        File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
>          self.walk_versions(self.snake_walk, self.downgrade)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
>          self.migrate_up(version, with_data=True)
>        File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
>          super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
>          self.REPOSITORY, version)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
>          return _migrate(url, repository, version, upgrade=True, err=err, **opts)
>        File "<string>", line 2, in _migrate
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
>          return f(*a, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
>          schema.runchange(ver, change, changeset.step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
>          change.run(self.engine, step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
>          script_func(engine)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
>          process_null_records(meta, scan=False)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
>          table.columns.uuid.alter(nullable=False)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
>          return alter_column(self, *p, **k)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
>          engine._run_visitor(visitorcallable, delta)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
>          conn._run_visitor(visitorcallable, element, **kwargs)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
>          **kwargs).traverse_single(element)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
>          ret = super(AlterTableVisitor, self).traverse_single(elem)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
>          return meth(obj, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
>          self.execute()
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
>          return self.connection.execute(self.buffer.getvalue())
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
>          return self._execute_text(object, multiparams, params)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
>          statement, parameters
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
>          util.raise_from_cause(newraise, exc_info)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
>          reraise(type(exception), exception, tb=exc_tb)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
>          cursor.execute(statement, parameters)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
>          self.errorhandler(self, exc, value)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
>          raise errorclass, errorvalue
>      OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
>
>
>
> Captured stderr:
> ~~~~~~~~~~~~~~~~
>      /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
>        cursor.execute(statement, parameters)
>
> Slowest Tests
> Test id                                                                        Runtime (s)
> -----------------------------------------------------------------------------  -----------
> nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions  51.553
>
> ==============================
> Failed 1 tests - output below:
> ==============================
>
> nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
> -----------------------------------------------------------------------------
>
> Captured traceback:
> ~~~~~~~~~~~~~~~~~~~
>      Traceback (most recent call last):
>        File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
>          self.walk_versions(self.snake_walk, self.downgrade)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
>          self.migrate_up(version, with_data=True)
>        File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
>          super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
>          self.REPOSITORY, version)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
>          return _migrate(url, repository, version, upgrade=True, err=err, **opts)
>        File "<string>", line 2, in _migrate
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
>          return f(*a, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
>          schema.runchange(ver, change, changeset.step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
>          change.run(self.engine, step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
>          script_func(engine)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
>          process_null_records(meta, scan=False)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
>          table.columns.uuid.alter(nullable=False)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
>          return alter_column(self, *p, **k)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
>          engine._run_visitor(visitorcallable, delta)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
>          conn._run_visitor(visitorcallable, element, **kwargs)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
>          **kwargs).traverse_single(element)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
>          ret = super(AlterTableVisitor, self).traverse_single(elem)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
>          return meth(obj, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
>          self.execute()
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
>          return self.connection.execute(self.buffer.getvalue())
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
>          return self._execute_text(object, multiparams, params)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
>          statement, parameters
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
>          util.raise_from_cause(newraise, exc_info)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
>          reraise(type(exception), exception, tb=exc_tb)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
>          cursor.execute(statement, parameters)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
>          self.errorhandler(self, exc, value)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
>          raise errorclass, errorvalue
>      OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
>      Traceback (most recent call last):
>      _StringException: Empty attachments:
>        stdout
>
>      stderr: {{{
>      /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
>        cursor.execute(statement, parameters)
>      }}}
>
>      Traceback (most recent call last):
>        File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
>          self.walk_versions(self.snake_walk, self.downgrade)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
>          self.migrate_up(version, with_data=True)
>        File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
>          super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
>          self.REPOSITORY, version)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
>          return _migrate(url, repository, version, upgrade=True, err=err, **opts)
>        File "<string>", line 2, in _migrate
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
>          return f(*a, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
>          schema.runchange(ver, change, changeset.step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
>          change.run(self.engine, step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
>          script_func(engine)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
>          process_null_records(meta, scan=False)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
>          table.columns.uuid.alter(nullable=False)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
>          return alter_column(self, *p, **k)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
>          engine._run_visitor(visitorcallable, delta)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
>          conn._run_visitor(visitorcallable, element, **kwargs)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
>          **kwargs).traverse_single(element)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
>          ret = super(AlterTableVisitor, self).traverse_single(elem)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
>          return meth(obj, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
>          self.execute()
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
>          return self.connection.execute(self.buffer.getvalue())
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
>          return self._execute_text(object, multiparams, params)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
>          statement, parameters
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
>          util.raise_from_cause(newraise, exc_info)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
>          reraise(type(exception), exception, tb=exc_tb)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
>          cursor.execute(statement, parameters)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
>          self.errorhandler(self, exc, value)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
>          raise errorclass, errorvalue
>      OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
>
>      Traceback (most recent call last):
>      _StringException: Empty attachments:
>        stdout
>
>      stderr: {{{
>      /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
>        cursor.execute(statement, parameters)
>      }}}
>
>      Traceback (most recent call last):
>        File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
>          self.walk_versions(self.snake_walk, self.downgrade)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
>          self.migrate_up(version, with_data=True)
>        File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
>          super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
>          self.REPOSITORY, version)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
>          return _migrate(url, repository, version, upgrade=True, err=err, **opts)
>        File "<string>", line 2, in _migrate
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
>          return f(*a, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
>          schema.runchange(ver, change, changeset.step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
>          change.run(self.engine, step)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
>          script_func(engine)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
>          process_null_records(meta, scan=False)
>        File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
>          table.columns.uuid.alter(nullable=False)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
>          return alter_column(self, *p, **k)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
>          engine._run_visitor(visitorcallable, delta)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
>          conn._run_visitor(visitorcallable, element, **kwargs)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
>          **kwargs).traverse_single(element)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
>          ret = super(AlterTableVisitor, self).traverse_single(elem)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
>          return meth(obj, **kw)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
>          self.execute()
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
>          return self.connection.execute(self.buffer.getvalue())
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
>          return self._execute_text(object, multiparams, params)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
>          statement, parameters
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
>          util.raise_from_cause(newraise, exc_info)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
>          reraise(type(exception), exception, tb=exc_tb)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
>          context)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
>          cursor.execute(statement, parameters)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
>          self.errorhandler(self, exc, value)
>        File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
>          raise errorclass, errorvalue
>      OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
>
>
>
> Captured stderr:
> ~~~~~~~~~~~~~~~~
>      /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
>        cursor.execute(statement, parameters)
>
>
>
> ======
> Totals
> ======
> Run: 1 in 51.553131 sec.
>   - Passed: 0
>   - Skipped: 0
>   - Failed: 1
>
> ==============
> Worker Balance
> ==============
>   - Worker 0 (1 tests) => 0:00:51.553131s
> ERROR: InvocationError: '/usr/bin/bash tools/pretty_tox.sh nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions'
> ____________________________________________________________________________________ summary ____________________________________________________________________________________
> ERROR:   py27: commands failed
> [mbayer at thinkpad nova]$
>
>
>
>
> _______________________________________________
> OpenStack-dev mailing list
> OpenStack-dev at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

We're running the migrations with Tempest in an internal CI with MariaDB 
and RHEL 7 and haven't had migration issues like this.

The deprecation warning is odd, unless that's coming from the 216 
migration where the duplicate index is defined, but it's removed in the 
249 migration so that shouldn't be a problem.

As for the 267 migration, it's failing to set the instances.uuid column 
to nullable=False.  Before it does this, it scans the tables looking for 
tables with an instance_uuid column that has a foreign key back to the 
instances table and where the instance_uuid value is NULL, it will fail 
it if finds one of those (on purpose, you have to manually purge those 
rows).

Once it's checked that there aren't any null instance_uuid entries with 
fkeys back to instances, it alters the instances.uuid column to be 
non-nullable so the UniqueConstraint on instances.uuid can be created.

There is a known issue with a unique index being created on 
instances.uuid in the 216 migration and the 267 migration adds the 
unique constraint, so there is some duplication there, but I don't think 
that's what's causing your failure.

-- 

Thanks,

Matt Riedemann




More information about the OpenStack-dev mailing list