<div dir="ltr">Hello,<div><br></div><div>I encountered an issue on a clean openstack havana install (one controller node, two compute nodes, and one network node)  when creating multiple instances. Some instances were created but other entered in the error state. Looking in the logs I noticed that the openstack-nova-scheduler had some database related errors (logs can be found bellow). Has anyone encountered this issue? What is causing this?</div>
<div><br></div><div>Regards,</div><div>Stefan</div><div><br></div><div>The error in the nova-scheduler log:</div><div><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">Error during SchedulerManager._expire_reservations: (OperationalError) (1054, "Unknown column 'reservations.user_id' in 'field list'") 'SELECT reservations.created_at AS reservations_created_at, reservations.updated_at AS reservations_updated_at, reservations.deleted_at AS reservations_deleted_at, reservations.deleted AS reservations_deleted, <a href="http://reservations.id">reservations.id</a> AS reservations_id, reservations.uuid AS reservations_uuid, reservations.usage_id AS reservations_usage_id, reservations.project_id AS reservations_project_id, reservations.user_id AS reservations_user_id, reservations.resource AS reservations_resource, reservations.delta AS reservations_delta, reservations.expire AS reservations_expire \nFROM reservations INNER JOIN quota_usages ON <a href="http://quota_usages.id">quota_usages.id</a> = reservations.usage_id \nWHERE reservations.deleted = %s AND reservations.expire < %s' (0, datetime.datetime(2014, 3, 18, 12, 58, 8, 784970))</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task Traceback (most recent call last):</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/openstack/common/periodic_task.py", line 180, in run_periodic_tasks</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task task(self, context)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/scheduler/manager.py", line 273, in _expire_reservations</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task QUOTAS.expire(context)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/quota.py", line 1368, in expire</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task self._driver.expire(context)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/quota.py", line 609, in expire</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task db.reservation_expire(context)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/db/api.py", line 1112, in reservation_expire</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task return IMPL.reservation_expire(context)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 108, in wrapper</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task return f(*args, **kwargs)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib/python2.6/site-packages/nova/db/sqlalchemy/api.py", line 3337, in reservation_expire</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task for reservation in reservation_query.join(models.QuotaUsage).all():</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2115, in all</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task return list(self)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2227, in _</span><em style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">iter</em><span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">_</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task return self._execute_and_instances(context)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/query.py", line 2242, in _execute_and_instances</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task result = conn.execute(querycontext.statement, self._params)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1449, in execute</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task params)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task compiled_sql, distilled_params</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1698, in _execute_context</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task context)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1691, in _execute_context</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task context)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/engine/default.py", line 331, in do_execute</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task cursor.execute(statement, parameters)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/site-packages/MySQLdb/cursors.py", line 173, in execute</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task self.errorhandler(self, exc, value)</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task File "/usr/lib64/python2.6/site-packages/MySQLdb/connections.py", line 36, in defaulterrorhandler</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task raise errorclass, errorvalue</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task OperationalError: (OperationalError) (1054, "Unknown column 'reservations.user_id' in 'field list'") 'SELECT reservations.created_at AS reservations_created_at, reservations.updated_at AS reservations_updated_at, reservations.deleted_at AS reservations_deleted_at, reservations.deleted AS reservations_deleted, <a href="http://reservations.id">reservations.id</a> AS reservations_id, reservations.uuid AS reservations_uuid, reservations.usage_id AS reservations_usage_id, reservations.project_id AS reservations_project_id, reservations.user_id AS reservations_user_id, reservations.resource AS reservations_resource, reservations.delta AS reservations_delta, reservations.expire AS reservations_expire \nFROM reservations INNER JOIN quota_usages ON <a href="http://quota_usages.id">quota_usages.id</a> = reservations.usage_id \nWHERE reservations.deleted = %s AND reservations.expire < %s' (0, datetime.datetime(2014, 3, 18, 12, 58, 8, 784970))</span><br style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">
<span style="color:rgb(51,51,51);font-family:Arial,sans-serif;font-size:14px;line-height:20px">2014-03-18 05:58:08.827 24056 TRACE nova.openstack.common.periodic_task</span><br></div></div>