On Tue, 21 Apr 2020 at 16:05, Tobias Urdin tobias.urdin@binero.com wrote:
Hello,
This makes me recall a commit I saw in oslo.config when troubleshooting another issue. https://github.com/openstack/oslo.config/commit/e3e2ba55eeeb86a9bc0624bb2592...
Thanks Tobias, that does look relevant. It was cherry-picked to stable/train, and available in oslo.config 6.11.2. However, both source and binary images include that version of the package. The code path looks a little different in this case also.
Best regards ________________________________________ From: Mark Goddard mark@stackhpc.com Sent: Tuesday, April 21, 2020 4:28 PM To: openstack-discuss Subject: [kolla][tripleo][nova] nova scheduler startup errors in Train on CentOS 8
Hi,
Our (kolla-ansible) CentOS 8 binary (RDO) deploy jobs have been failing on stable/train since added. Because the source jobs were passing, I was kind of hoping it was a packaging issue that would right itself, but no such luck yet.
The majority of the job succeeds, but we grep our logs for CRITICAL messages, and nova-scheduler has a few when it starts up that cause us to fail:
2020-04-21 08:59:30.125 24 CRITICAL nova [req-86f6dad1-25f2-4c77-9d7d-ac7422801b4e - - - - -] Unhandled error: RuntimeError: dictionary changed size during iteration 2020-04-21 08:59:30.125 24 ERROR nova Traceback (most recent call last): 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/bin/nova-scheduler", line 10, in <module> 2020-04-21 08:59:30.125 24 ERROR nova sys.exit(main()) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 53, in main 2020-04-21 08:59:30.125 24 ERROR nova service.serve(server, workers=workers) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 454, in serve 2020-04-21 08:59:30.125 24 ERROR nova restart_method='mutate') 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 842, in launch 2020-04-21 08:59:30.125 24 ERROR nova launcher.launch_service(service, workers=workers) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 606, in launch_service 2020-04-21 08:59:30.125 24 ERROR nova self._start_child(wrap) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 575, in _start_child 2020-04-21 08:59:30.125 24 ERROR nova self.launcher.restart() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 311, in restart 2020-04-21 08:59:30.125 24 ERROR nova self.conf.mutate_config_files() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3013, in mutate_config_files 2020-04-21 08:59:30.125 24 ERROR nova self._warn_immutability() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3040, in _warn_immutability 2020-04-21 08:59:30.125 24 ERROR nova for info, group in self._all_opt_infos(): 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2502, in _all_opt_infos 2020-04-21 08:59:30.125 24 ERROR nova for info in self._opts.values(): 2020-04-21 08:59:30.125 24 ERROR nova RuntimeError: dictionary changed size during iteration
https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z...
The service does appear to start up successfully. It doesn't happen on master.
Has anyone seen this before?
Cheers, Mark