[openstack-dev] [keystone] Field 'domain_id' doesn't have a default value
Lance Bragstad
lbragstad at gmail.com
Thu Jan 26 17:37:56 UTC 2017
We were able to resolve the issue in IRC [0]. Posting the outcome here so
that others following the thread stay updated.
The issue was a service running Newton code after the --contract phase was
executed. This broke because the old code didn't understand the new schema.
Instead, the services should be upgraded individually to the new code
in-between the --migrate and --contract phases. This ensures all services
are running the new code before the --contract phase changes the schema.
Thanks for asking Eduardo, and let us know if you find anything else
related to rolling upgrades.
[0]
http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-01-26.log.html#t2017-01-26T17:16:25
On Thu, Jan 26, 2017 at 10:47 AM, Lance Bragstad <lbragstad at gmail.com>
wrote:
> Hi Eduardo,
>
> Master should populate the domain_id for a user before it gets to the sql
> layer [0] [1]. Do you have `[identity] default_domain_id` specified in your
> keystone.conf?
>
> Can you give some specifics on the upgrade scenario? Number of nodes?
> Specific request you're making to create users?
>
>
> [0] https://github.com/openstack/keystone/blob/
> 169e66ab8800148c4052a46d2cb321af33e44f77/keystone/identity/
> controllers.py#L218
> [1] https://github.com/openstack/keystone/blob/
> 169e66ab8800148c4052a46d2cb321af33e44f77/keystone/common/
> controller.py#L737-L741
>
> On Thu, Jan 26, 2017 at 10:15 AM, Eduardo Gonzalez <dabarren at gmail.com>
> wrote:
>
>> Hi.
>> I’m testing upgrades from Newton to master branch using keystone’s
>> zero-downtime upgrade method:
>>
>> keystone-manage db_sync --expand
>> keystone-manage db_sync --migrate
>> keystone-manage db_sync --contract
>>
>> After upgrade is made with no errors in logs, I cannot create users.
>> Other keystone commands works fine.
>>
>> Error message: “Field ‘domain_id’ doesn’t have a default value”
>>
>> Full trace:
>>
>> 2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters [req-576e5683-6631-49b9-b532-65f06a23bbeb 4b3ff53812734e72b5aea42103349571 04fe68cf58724855a2ee67781a14b446 - default default] DBAPIError exception wrapped from (pymysql.err.InternalError) (1364, u"Field 'domain_id' doesn't have a default value") [SQL: u'INSERT INTO user (id, enabled, extra, default_project_id, created_at, last_active_at) VALUES (%(id)s, %(enabled)s, %(extra)s, %(default_project_id)s, %(created_at)s, %(last_active_at)s)'] [parameters: {'last_active_at': None, 'extra': '{}', 'created_at': datetime.datetime(2017, 1, 26, 15, 32, 10, 974903), 'enabled': 1, 'default_project_id': None, 'id': '8989be945c954f14a1c9ebaf45988fad'}]2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters Traceback (most recent call last):2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1139, in _execute_context2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters context)2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 450, in do_execute2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters cursor.execute(statement, parameters)2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/cursors.py", line 167, in execute2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters result = self._query(query)2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/cursors.py", line 323, in _query2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters conn.query(q)2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 836, in query2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters self._affected_rows = self._read_query_result(unbuffered=unbuffered)2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1020, in _read_query_result2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters result.read()2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 1303, in read2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters first_packet = self.connection._read_packet()2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 982, in _read_packet2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters packet.check_error()2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/connections.py", line 394, in check_error2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters err.raise_mysql_exception(self._data)2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/err.py", line 120, in raise_mysql_exception2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters _check_mysql_exception(errinfo)2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters File "/var/lib/kolla/venv/lib/python2.7/site-packages/pymysql/err.py", line 115, in _check_mysql_exception2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters raise InternalError(errno, errorvalue)2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters InternalError: (1364, u"Field 'domain_id' doesn't have a default value")2017-01-26 15:32:10.978 17 ERROR oslo_db.sqlalchemy.exc_filters
>>
>> Database migration logs:
>>
>> 2017-01-26 15:27:11.425 18 INFO migrate.versioning.api [-] 4 -> 5... 2017-01-26 15:27:11.540 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:11.540 18 INFO migrate.versioning.api [-] 5 -> 6... 2017-01-26 15:27:11.643 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:11.645 18 INFO migrate.versioning.api [-] 6 -> 7... 2017-01-26 15:27:11.758 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:11.759 18 INFO migrate.versioning.api [-] 7 -> 8... 2017-01-26 15:27:11.872 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:11.879 18 INFO migrate.versioning.api [-] 8 -> 9... 2017-01-26 15:27:12.082 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:12.083 18 INFO migrate.versioning.api [-] 9 -> 10... 2017-01-26 15:27:13.327 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:13.329 18 INFO migrate.versioning.api [-] 10 -> 11... 2017-01-26 15:27:13.430 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:13.431 18 INFO migrate.versioning.api [-] 11 -> 12... 2017-01-26 15:27:14.958 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:14.958 18 INFO migrate.versioning.api [-] 12 -> 13... 2017-01-26 15:27:15.054 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:15.055 18 INFO migrate.versioning.api [-] 13 -> 14... 2017-01-26 15:27:17.623 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:17.625 18 INFO migrate.versioning.api [-] 14 -> 15... 2017-01-26 15:27:17.871 18 INFO migrate.versioning.api [-] done2017-01-26 15:27:19.000 26 INFO migrate.versioning.api [-] 4 -> 5... 2017-01-26 15:27:19.106 26 INFO migrate.versioning.api [-] done2017-01-26 15:27:19.108 26 INFO migrate.versioning.api [-] 5 -> 6... 2017-01-26 15:27:19.229 26 INFO migrate.versioning.api [-] done2017-01-26 15:27:19.229 26 INFO migrate.versioning.api [-] 6 -> 7... 2017-01-26 15:27:19.327 26 INFO migrate.versioning.api [-] done2017-01-26 15:27:19.328 26 INFO migrate.versioning.api [-] 7 -> 8... 2017-01-26 15:27:19.435 26 INFO migrate.versioning.api [-] done2017-01-26 15:27:19.436 26 INFO migrate.versioning.api [-] 8 -> 9... 2017-01-26 15:27:19.542 26 INFO migrate.versioning.api [-] done2017-01-26 15:27:19.543 26 INFO migrate.versioning.api [-] 9 -> 10... 2017-01-26 15:27:19.640 26 INFO migrate.versioning.api [-] done2017-01-26 15:27:19.641 26 INFO migrate.versioning.api [-] 10 -> 11... 2017-01-26 15:27:19.901 26 INFO migrate.versioning.api [-] done2017-01-26 15:27:19.903 26 INFO migrate.versioning.api [-] 11 -> 12... 2017-01-26 15:27:20.070 26 INFO migrate.versioning.api [-] done2017-01-26 15:27:20.072 26 INFO migrate.versioning.api [-] 12 -> 13... 2017-01-26 15:27:20.192 26 INFO migrate.versioning.api [-] done2017-01-26 15:27:20.193 26 INFO migrate.versioning.api [-] 13 -> 14... 2017-01-26 15:27:20.471 26 INFO migrate.versioning.api [-] done2017-01-26 15:27:20.471 26 INFO migrate.versioning.api [-] 14 -> 15... 2017-01-26 15:27:20.609 26 INFO migrate.versioning.api [-] done
>>
>> keystone DB version:
>>
>> keystone-manage db_version109
>>
>> User table:
>>
>> MariaDB [keystone]> select * from user;
>> +----------------------------------+-------+---------+--------------------+---------------------+----------------+-----------+
>> | id | extra | enabled | default_project_id | created_at | last_active_at | domain_id |
>> +----------------------------------+-------+---------+--------------------+---------------------+----------------+-----------+
>> | 0343be260fbb4d69810ed75e43aa2962 | {} | 1 | NULL | 2017-01-26 15:25:43 | NULL | default |
>> | 08db2d8915e44ef68bfc5c5023fd38ee | {} | 1 | NULL | 2017-01-26 15:25:45 | NULL | default |
>> | 0ef1ada772ab4c0b8b4b6b3c75a05589 | {} | 1 | NULL | 2017-01-26 15:25:14 | NULL | default |
>> | 13b61e1aacc2485588899c92547e6982 | {} | 1 | NULL | 2017-01-26 15:25:11 | NULL | default |
>> | 190a2fec80d74bde9ac2b81590d22f30 | {} | 1 | NULL | 2017-01-26 15:25:01 | NULL | default |
>> | 1e4bc38a544242a9b93773280409e230 | {} | 1 | NULL | 2017-01-26 15:25:06 | NULL | default |
>> | 24e0229506db45e882ebd5d535ad84d7 | {} | 1 | NULL | 2017-01-26 15:25:29 | NULL | default |
>> | 2532002aea1c47c5a9d40a87ffcaeb93 | {} | 1 | NULL | 2017-01-26 15:25:37 | NULL | default |
>> | 2a6f627730b0472bbb07d961c15a6fdf | {} | 1 | NULL | 2017-01-26 15:25:08 | NULL | default |
>> | 3508815fea414038886fd693d4f8c6d9 | {} | 1 | NULL | 2017-01-26 15:25:13 | NULL | default |
>> | 491846111aef4942b3849fcff758bd68 | {} | 1 | NULL | 2017-01-26 15:25:16 | NULL | default |
>> | 4b3ff53812734e72b5aea42103349571 | {} | 1 | NULL | 2017-01-26 15:23:55 | NULL | default |
>> | 516f3bab0cc14e068417763df5396720 | {} | 1 | NULL | 2017-01-26 15:25:41 | NULL | default |
>> | 541ca0cc0bcf4840aea941ca6befee5c | {} | 1 | NULL | 2017-01-26 15:25:35 | NULL | default |
>> | 5accc9b70a2d47dc996b65e472131f1d | {} | 1 | NULL | 2017-01-26 15:25:20 | NULL | default |
>> | 68b8e0ca81864401925623ccdd77d687 | {} | 1 | NULL | 2017-01-26 15:25:33 | NULL | default |
>> | 6a2df372d7fe44f785d15411955396a0 | {} | 1 | NULL | 2017-01-26 15:25:22 | NULL | default |
>> | 96f5580c12714469aa13575c8f28e244 | {} | 1 | NULL | 2017-01-26 15:25:31 | NULL | default |
>> | a12df0e1cb534a0f901b7be8b2275549 | {} | 1 | NULL | 2017-01-26 15:25:18 | NULL | default |
>> | a7c560692f204c44841b5e5b1d9a42c6 | {} | 1 | NULL | 2017-01-26 15:25:09 | NULL | default |
>> | aab83cbea56c4b52962eeaa57f574333 | {} | 1 | NULL | 2017-01-26 15:25:50 | NULL | default |
>> | ae35973c8f734b83bad4215783eb35f1 | {} | 1 | NULL | 2017-01-26 15:25:27 | NULL | default |
>> | d6f622ea204d4e1da47a503d662db885 | {} | 1 | NULL | 2017-01-26 15:25:39 | NULL | default |
>> | fb14a9aefecd476595bc24070ce72d2d | {} | 1 | NULL | 2017-01-26 15:25:25 | NULL | default |
>> | fc5b8e8f93274507b8c34a01a2039a64 | {} | 1 | NULL | 2017-01-26 15:25:47 | NULL | default |
>> | ff738be529a94095a9c9474a698b6b72 | {} | 1 | NULL | 2017-01-26 15:25:23 | NULL | default |
>> +----------------------------------+-------+---------+--------------------+---------------------+----------------+-----------+
>>
>> Any help will be appreciated
>>
>> Regards, Eduardo
>>
>>
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170126/459c419e/attachment-0001.html>
More information about the OpenStack-dev
mailing list