Open Stack

Hello -

Victor Sergeyev and I are both observing the following test failure which occurs with all the tests underneath nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.    This is against master with a brand new tox environment and everything at the default.

It does not seem to be occurring on gates that run these tests and interestingly the tests seem to complete very quickly (under seven seconds) on the gate as well; the failures here take between 50-100 seconds to occur, not fully deterministically, and only on the MySQL backend; the Postgresql and SQLite versions of these tests pass.  I’m running against MariaDB server 10.0.14 with Python 2.7.8 on Fedora 21.   

Below is the test just for test_walk_versions, but the warnings (not necessarily the failures themselves) here also occur for test_migration_267 as well as test_innodb_tables.

I’m still looking into what the cause of this is, I’d imagine it’s something related to newer MySQL versions or perhaps MariaDB vs. MySQL, I’m just putting it up here in case someone already knows what this is or has some clue to save me some time figuring it out.  I apologize if I’m just doing something dumb, I’ve only recently begun to run Nova’s test suite in full against all backends, so I haven’t yet put intelligent thought into this nor have I tried to yet look at the migration in question causing the problem.  Will do that next.


[mbayer at thinkpad nova]$ tox -e py27 -- nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
py27 develop-inst-noop: /home/mbayer/dev/openstack/nova
py27 runtests: PYTHONHASHSEED='0'
py27 runtests: commands[0] | find . -type f -name *.pyc -delete
py27 runtests: commands[1] | bash tools/pretty_tox.sh nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
running testr
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests} --list 
running=OS_STDOUT_CAPTURE=${OS_STDOUT_CAPTURE:-1} \
OS_STDERR_CAPTURE=${OS_STDERR_CAPTURE:-1} \
OS_TEST_TIMEOUT=${OS_TEST_TIMEOUT:-160} \
${PYTHON:-python} -m subunit.run discover -t ./ ${OS_TEST_PATH:-./nova/tests}  --load-list /tmp/tmpw7zqhE

2015-01-06 18:28:12.913 32435 WARNING oslo.db.sqlalchemy.session [req-5cc6731f-00ef-43df-8aec-4914a44d12c5 ] MySQL SQL mode is '', consider enabling TRADITIONAL or STRICT_ALL_TABLES
{0} nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions [51.553131s] ... FAILED

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
        self.walk_versions(self.snake_walk, self.downgrade)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
        self.migrate_up(version, with_data=True)
      File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
        super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
        self.REPOSITORY, version)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
        return _migrate(url, repository, version, upgrade=True, err=err, **opts)
      File "<string>", line 2, in _migrate
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
        return f(*a, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
        schema.runchange(ver, change, changeset.step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
        change.run(self.engine, step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
        script_func(engine)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
        process_null_records(meta, scan=False)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
        table.columns.uuid.alter(nullable=False)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
        return alter_column(self, *p, **k)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
        engine._run_visitor(visitorcallable, delta)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
        conn._run_visitor(visitorcallable, element, **kwargs)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
        **kwargs).traverse_single(element)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
        ret = super(AlterTableVisitor, self).traverse_single(elem)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
        return meth(obj, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
        self.execute()
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
        return self.connection.execute(self.buffer.getvalue())
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
        return self._execute_text(object, multiparams, params)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
        statement, parameters
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
        util.raise_from_cause(newraise, exc_info)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
        cursor.execute(statement, parameters)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
        self.errorhandler(self, exc, value)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
        raise errorclass, errorvalue
    OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
    Traceback (most recent call last):
    _StringException: Empty attachments:
      stdout
    
    stderr: {{{
    /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
      cursor.execute(statement, parameters)
    }}}
    
    Traceback (most recent call last):
      File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
        self.walk_versions(self.snake_walk, self.downgrade)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
        self.migrate_up(version, with_data=True)
      File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
        super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
        self.REPOSITORY, version)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
        return _migrate(url, repository, version, upgrade=True, err=err, **opts)
      File "<string>", line 2, in _migrate
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
        return f(*a, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
        schema.runchange(ver, change, changeset.step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
        change.run(self.engine, step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
        script_func(engine)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
        process_null_records(meta, scan=False)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
        table.columns.uuid.alter(nullable=False)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
        return alter_column(self, *p, **k)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
        engine._run_visitor(visitorcallable, delta)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
        conn._run_visitor(visitorcallable, element, **kwargs)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
        **kwargs).traverse_single(element)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
        ret = super(AlterTableVisitor, self).traverse_single(elem)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
        return meth(obj, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
        self.execute()
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
        return self.connection.execute(self.buffer.getvalue())
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
        return self._execute_text(object, multiparams, params)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
        statement, parameters
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
        util.raise_from_cause(newraise, exc_info)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
        cursor.execute(statement, parameters)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
        self.errorhandler(self, exc, value)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
        raise errorclass, errorvalue
    OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
    
    Traceback (most recent call last):
    _StringException: Empty attachments:
      stdout
    
    stderr: {{{
    /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
      cursor.execute(statement, parameters)
    }}}
    
    Traceback (most recent call last):
      File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
        self.walk_versions(self.snake_walk, self.downgrade)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
        self.migrate_up(version, with_data=True)
      File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
        super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
        self.REPOSITORY, version)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
        return _migrate(url, repository, version, upgrade=True, err=err, **opts)
      File "<string>", line 2, in _migrate
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
        return f(*a, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
        schema.runchange(ver, change, changeset.step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
        change.run(self.engine, step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
        script_func(engine)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
        process_null_records(meta, scan=False)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
        table.columns.uuid.alter(nullable=False)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
        return alter_column(self, *p, **k)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
        engine._run_visitor(visitorcallable, delta)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
        conn._run_visitor(visitorcallable, element, **kwargs)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
        **kwargs).traverse_single(element)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
        ret = super(AlterTableVisitor, self).traverse_single(elem)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
        return meth(obj, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
        self.execute()
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
        return self.connection.execute(self.buffer.getvalue())
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
        return self._execute_text(object, multiparams, params)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
        statement, parameters
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
        util.raise_from_cause(newraise, exc_info)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
        cursor.execute(statement, parameters)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
        self.errorhandler(self, exc, value)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
        raise errorclass, errorvalue
    OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
    
    

Captured stderr:
~~~~~~~~~~~~~~~~
    /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
      cursor.execute(statement, parameters)
    
Slowest Tests
Test id                                                                        Runtime (s)
-----------------------------------------------------------------------------  -----------
nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions  51.553

==============================
Failed 1 tests - output below:
==============================

nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions
-----------------------------------------------------------------------------

Captured traceback:
~~~~~~~~~~~~~~~~~~~
    Traceback (most recent call last):
      File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
        self.walk_versions(self.snake_walk, self.downgrade)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
        self.migrate_up(version, with_data=True)
      File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
        super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
        self.REPOSITORY, version)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
        return _migrate(url, repository, version, upgrade=True, err=err, **opts)
      File "<string>", line 2, in _migrate
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
        return f(*a, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
        schema.runchange(ver, change, changeset.step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
        change.run(self.engine, step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
        script_func(engine)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
        process_null_records(meta, scan=False)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
        table.columns.uuid.alter(nullable=False)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
        return alter_column(self, *p, **k)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
        engine._run_visitor(visitorcallable, delta)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
        conn._run_visitor(visitorcallable, element, **kwargs)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
        **kwargs).traverse_single(element)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
        ret = super(AlterTableVisitor, self).traverse_single(elem)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
        return meth(obj, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
        self.execute()
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
        return self.connection.execute(self.buffer.getvalue())
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
        return self._execute_text(object, multiparams, params)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
        statement, parameters
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
        util.raise_from_cause(newraise, exc_info)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
        cursor.execute(statement, parameters)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
        self.errorhandler(self, exc, value)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
        raise errorclass, errorvalue
    OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
    Traceback (most recent call last):
    _StringException: Empty attachments:
      stdout
    
    stderr: {{{
    /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
      cursor.execute(statement, parameters)
    }}}
    
    Traceback (most recent call last):
      File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
        self.walk_versions(self.snake_walk, self.downgrade)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
        self.migrate_up(version, with_data=True)
      File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
        super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
        self.REPOSITORY, version)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
        return _migrate(url, repository, version, upgrade=True, err=err, **opts)
      File "<string>", line 2, in _migrate
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
        return f(*a, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
        schema.runchange(ver, change, changeset.step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
        change.run(self.engine, step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
        script_func(engine)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
        process_null_records(meta, scan=False)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
        table.columns.uuid.alter(nullable=False)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
        return alter_column(self, *p, **k)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
        engine._run_visitor(visitorcallable, delta)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
        conn._run_visitor(visitorcallable, element, **kwargs)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
        **kwargs).traverse_single(element)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
        ret = super(AlterTableVisitor, self).traverse_single(elem)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
        return meth(obj, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
        self.execute()
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
        return self.connection.execute(self.buffer.getvalue())
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
        return self._execute_text(object, multiparams, params)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
        statement, parameters
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
        util.raise_from_cause(newraise, exc_info)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
        cursor.execute(statement, parameters)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
        self.errorhandler(self, exc, value)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
        raise errorclass, errorvalue
    OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
    
    Traceback (most recent call last):
    _StringException: Empty attachments:
      stdout
    
    stderr: {{{
    /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
      cursor.execute(statement, parameters)
    }}}
    
    Traceback (most recent call last):
      File "nova/tests/unit/db/test_migrations.py", line 151, in test_walk_versions
        self.walk_versions(self.snake_walk, self.downgrade)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 193, in walk_versions
        self.migrate_up(version, with_data=True)
      File "nova/tests/unit/db/test_migrations.py", line 148, in migrate_up
        super(NovaMigrationsCheckers, self).migrate_up(version, with_data)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/oslo/db/sqlalchemy/test_migrations.py", line 263, in migrate_up
        self.REPOSITORY, version)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 186, in upgrade
        return _migrate(url, repository, version, upgrade=True, err=err, **opts)
      File "<string>", line 2, in _migrate
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/util/__init__.py", line 160, in with_engine
        return f(*a, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/api.py", line 366, in _migrate
        schema.runchange(ver, change, changeset.step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/schema.py", line 93, in runchange
        change.run(self.engine, step)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/versioning/script/py.py", line 148, in run
        script_func(engine)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 103, in upgrade
        process_null_records(meta, scan=False)
      File "/home/mbayer/dev/openstack/nova/nova/db/sqlalchemy/migrate_repo/versions/267_instance_uuid_non_nullable.py", line 89, in process_null_records
        table.columns.uuid.alter(nullable=False)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 534, in alter
        return alter_column(self, *p, **k)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/schema.py", line 141, in alter_column
        engine._run_visitor(visitorcallable, delta)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1616, in _run_visitor
        conn._run_visitor(visitorcallable, element, **kwargs)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1245, in _run_visitor
        **kwargs).traverse_single(element)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 56, in traverse_single
        ret = super(AlterTableVisitor, self).traverse_single(elem)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/sql/visitors.py", line 120, in traverse_single
        return meth(obj, **kw)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/databases/mysql.py", line 47, in visit_column
        self.execute()
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/migrate/changeset/ansisql.py", line 44, in execute
        return self.connection.execute(self.buffer.getvalue())
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 721, in execute
        return self._execute_text(object, multiparams, params)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 870, in _execute_text
        statement, parameters
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 958, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1155, in _handle_dbapi_exception
        util.raise_from_cause(newraise, exc_info)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause
        reraise(type(exception), exception, tb=exc_tb)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 951, in _execute_context
        context)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 436, in do_execute
        cursor.execute(statement, parameters)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/cursors.py", line 205, in execute
        self.errorhandler(self, exc, value)
      File "/home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler
        raise errorclass, errorvalue
    OperationalError: (OperationalError) (1833, "Cannot change column 'uuid': used in a foreign key constraint 'block_device_mapping_instance_uuid_fkey' of table 'qzclautyzr.block_device_mapping'") '\nALTER TABLE instances CHANGE COLUMN uuid uuid VARCHAR(36) NOT NULL' ()
    
    

Captured stderr:
~~~~~~~~~~~~~~~~
    /home/mbayer/dev/openstack/nova/.tox/py27/lib/python2.7/site-packages/sqlalchemy/engine/default.py:436: Warning: Duplicate index 'block_device_mapping_instance_uuid_virtual_name_device_name_idx' defined on the table 'qzclautyzr.block_device_mapping'. This is deprecated and will be disallowed in a future release.
      cursor.execute(statement, parameters)
    


======
Totals
======
Run: 1 in 51.553131 sec.
 - Passed: 0
 - Skipped: 0
 - Failed: 1

==============
Worker Balance
==============
 - Worker 0 (1 tests) => 0:00:51.553131s
ERROR: InvocationError: '/usr/bin/bash tools/pretty_tox.sh nova.tests.unit.db.test_migrations.TestNovaMigrationsMySQL.test_walk_versions'
____________________________________________________________________________________ summary ____________________________________________________________________________________
ERROR:   py27: commands failed
[mbayer at thinkpad nova]$