Open Stack

Hello,

This makes me recall a commit I saw in oslo.config when troubleshooting another issue.
https://github.com/openstack/oslo.config/commit/e3e2ba55eeeb86a9bc0624bb2592e46583e839e7

Best regards
________________________________________
From: Mark Goddard <mark at stackhpc.com>
Sent: Tuesday, April 21, 2020 4:28 PM
To: openstack-discuss
Subject: [kolla][tripleo][nova] nova scheduler startup errors in Train on CentOS 8

Hi,

Our (kolla-ansible) CentOS 8 binary (RDO) deploy jobs have been
failing on stable/train since added. Because the source jobs were
passing, I was kind of hoping it was a packaging issue that would
right itself, but no such luck yet.

The majority of the job succeeds, but we grep our logs for CRITICAL
messages, and nova-scheduler has a few when it starts up that cause us
to fail:

2020-04-21 08:59:30.125 24 CRITICAL nova
[req-86f6dad1-25f2-4c77-9d7d-ac7422801b4e - - - - -] Unhandled error:
RuntimeError: dictionary changed size during iteration
2020-04-21 08:59:30.125 24 ERROR nova Traceback (most recent call last):
2020-04-21 08:59:30.125 24 ERROR nova   File
"/usr/bin/nova-scheduler", line 10, in <module>
2020-04-21 08:59:30.125 24 ERROR nova     sys.exit(main())
2020-04-21 08:59:30.125 24 ERROR nova   File
"/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 53, in
main
2020-04-21 08:59:30.125 24 ERROR nova     service.serve(server, workers=workers)
2020-04-21 08:59:30.125 24 ERROR nova   File
"/usr/lib/python3.6/site-packages/nova/service.py", line 454, in serve
2020-04-21 08:59:30.125 24 ERROR nova     restart_method='mutate')
2020-04-21 08:59:30.125 24 ERROR nova   File
"/usr/lib/python3.6/site-packages/oslo_service/service.py", line 842,
in launch
2020-04-21 08:59:30.125 24 ERROR nova
launcher.launch_service(service, workers=workers)
2020-04-21 08:59:30.125 24 ERROR nova   File
"/usr/lib/python3.6/site-packages/oslo_service/service.py", line 606,
in launch_service
2020-04-21 08:59:30.125 24 ERROR nova     self._start_child(wrap)
2020-04-21 08:59:30.125 24 ERROR nova   File
"/usr/lib/python3.6/site-packages/oslo_service/service.py", line 575,
in _start_child
2020-04-21 08:59:30.125 24 ERROR nova     self.launcher.restart()
2020-04-21 08:59:30.125 24 ERROR nova   File
"/usr/lib/python3.6/site-packages/oslo_service/service.py", line 311,
in restart
2020-04-21 08:59:30.125 24 ERROR nova     self.conf.mutate_config_files()
2020-04-21 08:59:30.125 24 ERROR nova   File
"/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3013, in
mutate_config_files
2020-04-21 08:59:30.125 24 ERROR nova     self._warn_immutability()
2020-04-21 08:59:30.125 24 ERROR nova   File
"/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3040, in
_warn_immutability
2020-04-21 08:59:30.125 24 ERROR nova     for info, group in
self._all_opt_infos():
2020-04-21 08:59:30.125 24 ERROR nova   File
"/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2502, in
_all_opt_infos
2020-04-21 08:59:30.125 24 ERROR nova     for info in self._opts.values():
2020-04-21 08:59:30.125 24 ERROR nova RuntimeError: dictionary changed
size during iteration

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_339/721329/1/check/kolla-ansible-centos8-binary/339767b/primary/logs/kolla/nova/nova-scheduler.txt

The service does appear to start up successfully. It doesn't happen on master.

Has anyone seen this before?

Cheers,
Mark