I am working on making Kolla images Python 3 only. So far images are py3 but then there are issues during deployment phase which I do not know how to solve. https://review.opendev.org/#/c/642375/ is a patch. 'kolla-ansible-ubuntu-source' CI job deploys using Ubuntu 18.04 based images. And fails. Log [1] shows something which looks like 'works in py2, not tested with py3' code: 1. http://logs.openstack.org/75/642375/19/check/kolla-ansible-ubuntu-source/408... "+++ neutron-db-manage --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/plugins/ml2/ml2_conf.ini upgrade head", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "INFO [alembic.runtime.migration] Running upgrade -> kilo", "INFO [alembic.runtime.migration] Running upgrade kilo -> 354db87e3225", "INFO [alembic.runtime.migration] Running upgrade 354db87e3225 -> 599c6a226151", "INFO [alembic.runtime.migration] Running upgrade 599c6a226151 -> 52c5312f6baf", "INFO [alembic.runtime.migration] Running upgrade 52c5312f6baf -> 313373c0ffee", "INFO [alembic.runtime.migration] Running upgrade 313373c0ffee -> 8675309a5c4f", "INFO [alembic.runtime.migration] Running upgrade 8675309a5c4f -> 45f955889773", "INFO [alembic.runtime.migration] Running upgrade 45f955889773 -> 26c371498592", "INFO [alembic.runtime.migration] Running upgrade 26c371498592 -> 1c844d1677f7", "INFO [alembic.runtime.migration] Running upgrade 1c844d1677f7 -> 1b4c6e320f79", "INFO [alembic.runtime.migration] Running upgrade 1b4c6e320f79 -> 48153cb5f051", "INFO [alembic.runtime.migration] Running upgrade 48153cb5f051 -> 9859ac9c136", "INFO [alembic.runtime.migration] Running upgrade 9859ac9c136 -> 34af2b5c5a59", "INFO [alembic.runtime.migration] Running upgrade 34af2b5c5a59 -> 59cb5b6cf4d", "INFO [alembic.runtime.migration] Running upgrade 59cb5b6cf4d -> 13cfb89f881a", "INFO [alembic.runtime.migration] Running upgrade 13cfb89f881a -> 32e5974ada25", "INFO [alembic.runtime.migration] Running upgrade 32e5974ada25 -> ec7fcfbf72ee", "INFO [alembic.runtime.migration] Running upgrade ec7fcfbf72ee -> dce3ec7a25c9", "INFO [alembic.runtime.migration] Running upgrade dce3ec7a25c9 -> c3a73f615e4", "INFO [alembic.runtime.migration] Running upgrade c3a73f615e4 -> 659bf3d90664", "INFO [alembic.runtime.migration] Running upgrade 659bf3d90664 -> 1df244e556f5", "INFO [alembic.runtime.migration] Running upgrade 1df244e556f5 -> 19f26505c74f", "INFO [alembic.runtime.migration] Running upgrade 19f26505c74f -> 15be73214821", "INFO [alembic.runtime.migration] Running upgrade 15be73214821 -> b4caf27aae4", "INFO [alembic.runtime.migration] Running upgrade b4caf27aae4 -> 15e43b934f81", "INFO [alembic.runtime.migration] Running upgrade 15e43b934f81 -> 31ed664953e6", "INFO [alembic.runtime.migration] Running upgrade 31ed664953e6 -> 2f9e956e7532", "INFO [alembic.runtime.migration] Running upgrade 2f9e956e7532 -> 3894bccad37f", "INFO [alembic.runtime.migration] Running upgrade 3894bccad37f -> 0e66c5227a8a", "INFO [alembic.runtime.migration] Running upgrade 0e66c5227a8a -> 45f8dd33480b", "INFO [alembic.runtime.migration] Running upgrade 45f8dd33480b -> 5abc0278ca73", "INFO [alembic.runtime.migration] Running upgrade 5abc0278ca73 -> d3435b514502", "INFO [alembic.runtime.migration] Running upgrade d3435b514502 -> 30107ab6a3ee", "INFO [alembic.runtime.migration] Running upgrade 30107ab6a3ee -> c415aab1c048", "INFO [alembic.runtime.migration] Running upgrade c415aab1c048 -> a963b38d82f4", "INFO [alembic.runtime.migration] Running upgrade kilo -> 30018084ec99", "INFO [alembic.runtime.migration] Running upgrade 30018084ec99 -> 4ffceebfada", "INFO [alembic.runtime.migration] Running upgrade 4ffceebfada -> 5498d17be016", "INFO [alembic.runtime.migration] Running upgrade 5498d17be016 -> 2a16083502f3", "INFO [alembic.runtime.migration] Running upgrade 2a16083502f3 -> 2e5352a0ad4d", "INFO [alembic.runtime.migration] Running upgrade 2e5352a0ad4d -> 11926bcfe72d", "INFO [alembic.runtime.migration] Running upgrade 11926bcfe72d -> 4af11ca47297", "INFO [alembic.runtime.migration] Running upgrade 4af11ca47297 -> 1b294093239c", "INFO [alembic.runtime.migration] Running upgrade 1b294093239c -> 8a6d8bdae39", "INFO [alembic.runtime.migration] Running upgrade 8a6d8bdae39 -> 2b4c2465d44b", "INFO [alembic.runtime.migration] Running upgrade 2b4c2465d44b -> e3278ee65050", "INFO [alembic.runtime.migration] Running upgrade e3278ee65050 -> c6c112992c9", "INFO [alembic.runtime.migration] Running upgrade c6c112992c9 -> 5ffceebfada", "INFO [alembic.runtime.migration] Running upgrade 5ffceebfada -> 4ffceebfcdc", "INFO [alembic.runtime.migration] Running upgrade 4ffceebfcdc -> 7bbb25278f53", "INFO [alembic.runtime.migration] Running upgrade 7bbb25278f53 -> 89ab9a816d70", "INFO [alembic.runtime.migration] Running upgrade 89ab9a816d70 -> c879c5e1ee90", "INFO [alembic.runtime.migration] Running upgrade c879c5e1ee90 -> 8fd3918ef6f4", "INFO [alembic.runtime.migration] Running upgrade 8fd3918ef6f4 -> 4bcd4df1f426", "INFO [alembic.runtime.migration] Running upgrade 4bcd4df1f426 -> b67e765a3524", "INFO [alembic.runtime.migration] Running upgrade a963b38d82f4 -> 3d0e74aa7d37", "INFO [alembic.runtime.migration] Running upgrade 3d0e74aa7d37 -> 030a959ceafa", "INFO [alembic.runtime.migration] Running upgrade 030a959ceafa -> a5648cfeeadf", "INFO [alembic.runtime.migration] Running upgrade a5648cfeeadf -> 0f5bef0f87d4", "INFO [alembic.runtime.migration] Running upgrade 0f5bef0f87d4 -> 67daae611b6e", "INFO [alembic.runtime.migration] Running upgrade 67daae611b6e -> 6b461a21bcfc", "INFO [alembic.runtime.migration] Running upgrade 6b461a21bcfc -> 5cd92597d11d", "INFO [alembic.runtime.migration] Running upgrade 5cd92597d11d -> 929c968efe70", "INFO [alembic.runtime.migration] Running upgrade 929c968efe70 -> a9c43481023c", "INFO [alembic.runtime.migration] Running upgrade a9c43481023c -> 804a3c76314c", "INFO [alembic.runtime.migration] Running upgrade 804a3c76314c -> 2b42d90729da", "INFO [alembic.runtime.migration] Running upgrade 2b42d90729da -> 62c781cb6192", "INFO [alembic.runtime.migration] Running upgrade 62c781cb6192 -> c8c222d42aa9", "INFO [alembic.runtime.migration] Running upgrade c8c222d42aa9 -> 349b6fd605a6", "INFO [alembic.runtime.migration] Running upgrade 349b6fd605a6 -> 7d32f979895f", "INFO [alembic.runtime.migration] Running upgrade 7d32f979895f -> 594422d373ee", "INFO [alembic.runtime.migration] Running upgrade 594422d373ee -> 61663558142c", "INFO [alembic.runtime.migration] Running upgrade 61663558142c -> 867d39095bf4, port forwarding", "INFO [alembic.runtime.migration] Running upgrade 867d39095bf4 -> d72db3e25539, modify uniq port forwarding", "INFO [alembic.runtime.migration] Running upgrade d72db3e25539 -> cada2437bf41", "INFO [alembic.runtime.migration] Running upgrade cada2437bf41 -> 195176fb410d, router gateway IP QoS", "INFO [alembic.runtime.migration] Running upgrade 195176fb410d -> fb0167bd9639", "INFO [alembic.runtime.migration] Running upgrade fb0167bd9639 -> 0ff9e3881597", "INFO [alembic.runtime.migration] Running upgrade 0ff9e3881597 -> 9bfad3f1e780", "INFO [alembic.runtime.migration] Running upgrade b67e765a3524 -> a84ccf28f06a", "INFO [alembic.runtime.migration] Running upgrade a84ccf28f06a -> 7d9d8eeec6ad", "INFO [alembic.runtime.migration] Running upgrade 7d9d8eeec6ad -> a8b517cff8ab", "INFO [alembic.runtime.migration] Running upgrade a8b517cff8ab -> 3b935b28e7a0", "INFO [alembic.runtime.migration] Running upgrade 3b935b28e7a0 -> b12a3ef66e62", "INFO [alembic.runtime.migration] Running upgrade b12a3ef66e62 -> 97c25b0d2353", "INFO [alembic.runtime.migration] Running upgrade 97c25b0d2353 -> 2e0d7a8a1586", "INFO [alembic.runtime.migration] Running upgrade 2e0d7a8a1586 -> 5c85685d616d", "INFO [alembic.runtime.migration] Context impl MySQLImpl.", "INFO [alembic.runtime.migration] Will assume non-transactional DDL.", "Traceback (most recent call last):", " File \"/var/lib/kolla/venv/bin/neutron-db-manage\", line 10, in <module>", " sys.exit(main())", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/db/migration/cli.py\", line 657, in main", " return_val |= bool(CONF.command.func(config, CONF.command.name))", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/db/migration/cli.py\", line 179, in do_upgrade", " run_sanity_checks(config, revision)", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/db/migration/cli.py\", line 641, in run_sanity_checks", " script_dir.run_env()", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/script/base.py\", line 475, in run_env", " util.load_python_file(self.dir, \"env.py\")", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/util/pyfiles.py\", line 90, in load_python_file", " module = load_module_py(module_id, path)", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/util/compat.py\", line 156, in load_module_py", " spec.loader.exec_module(module)", " File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module", " File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/networking_infoblox/neutron/db/migration/alembic_migrations/env.py\", line 88, in <module>", " run_migrations_online()", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/networking_infoblox/neutron/db/migration/alembic_migrations/env.py\", line 79, in run_migrations_online", " context.run_migrations()", " File \"<string>\", line 8, in run_migrations", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/runtime/environment.py\", line 839, in run_migrations", " self.get_context().run_migrations(**kw)", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/runtime/migration.py\", line 350, in run_migrations", " for step in self._migrations_fn(heads, self):", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/neutron/db/migration/cli.py\", line 632, in check_sanity", " revision, rev, implicit_base=True):", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/script/revision.py\", line 767, in _iterate_revisions", " uppers = util.dedupe_tuple(self.get_revisions(upper))", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/script/revision.py\", line 321, in get_revisions", " resolved_id, branch_label = self._resolve_revision_number(id_)", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/script/revision.py\", line 491, in _resolve_revision_number", " self._revision_map", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/util/langhelpers.py\", line 230, in __get__", " obj.__dict__[self.__name__] = result = self.fget(obj)", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/script/revision.py\", line 123, in _revision_map", " for revision in self._generator():", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/script/base.py\", line 109, in _load_revisions", " script = Script._from_filename(self, vers, file_)", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/script/base.py\", line 887, in _from_filename", " module = util.load_python_file(dir_, filename)", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/util/pyfiles.py\", line 90, in load_python_file", " module = load_module_py(module_id, path)", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/alembic/util/compat.py\", line 156, in load_module_py", " spec.loader.exec_module(module)", " File \"<frozen importlib._bootstrap_external>\", line 678, in exec_module", " File \"<frozen importlib._bootstrap>\", line 219, in _call_with_frames_removed", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/networking_infoblox/neutron/db/migration/alembic_migrations/versions/4d0bb1d080f8_member_sync_improvement.py\", line 43, in <module>", " default=utils.get_hash()))", " File \"/var/lib/kolla/venv/lib/python3.6/site-packages/networking_infoblox/neutron/common/utils.py\", line 374, in get_hash", " return hashlib.md5(str(time.time())).hexdigest()", "TypeError: Unicode-objects must be encoded before hashing" ], Any ideas which project goes wrong? And how/where to fix it?