On Tue, 21 Apr 2020 at 16:05, Tobias Urdin tobias.urdin@binero.com wrote:

Hello,

This makes me recall a commit I saw in oslo.config when troubleshooting another issue. https://github.com/openstack/oslo.config/commit/e3e2ba55eeeb86a9bc0624bb2592...

Thanks Tobias, that does look relevant. It was cherry-picked to stable/train, and available in oslo.config 6.11.2. However, both source and binary images include that version of the package. The code path looks a little different in this case also.

Best regards ________________________________________ From: Mark Goddard mark@stackhpc.com Sent: Tuesday, April 21, 2020 4:28 PM To: openstack-discuss Subject: [kolla][tripleo][nova] nova scheduler startup errors in Train on CentOS 8

Hi,

Our (kolla-ansible) CentOS 8 binary (RDO) deploy jobs have been failing on stable/train since added. Because the source jobs were passing, I was kind of hoping it was a packaging issue that would right itself, but no such luck yet.

The majority of the job succeeds, but we grep our logs for CRITICAL messages, and nova-scheduler has a few when it starts up that cause us to fail:

2020-04-21 08:59:30.125 24 CRITICAL nova [req-86f6dad1-25f2-4c77-9d7d-ac7422801b4e - - - - -] Unhandled error: RuntimeError: dictionary changed size during iteration 2020-04-21 08:59:30.125 24 ERROR nova Traceback (most recent call last): 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/bin/nova-scheduler", line 10, in <module> 2020-04-21 08:59:30.125 24 ERROR nova sys.exit(main()) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 53, in main 2020-04-21 08:59:30.125 24 ERROR nova service.serve(server, workers=workers) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 454, in serve 2020-04-21 08:59:30.125 24 ERROR nova restart_method='mutate') 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 842, in launch 2020-04-21 08:59:30.125 24 ERROR nova launcher.launch_service(service, workers=workers) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 606, in launch_service 2020-04-21 08:59:30.125 24 ERROR nova self._start_child(wrap) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 575, in _start_child 2020-04-21 08:59:30.125 24 ERROR nova self.launcher.restart() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 311, in restart 2020-04-21 08:59:30.125 24 ERROR nova self.conf.mutate_config_files() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3013, in mutate_config_files 2020-04-21 08:59:30.125 24 ERROR nova self._warn_immutability() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3040, in _warn_immutability 2020-04-21 08:59:30.125 24 ERROR nova for info, group in self._all_opt_infos(): 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2502, in _all_opt_infos 2020-04-21 08:59:30.125 24 ERROR nova for info in self._opts.values(): 2020-04-21 08:59:30.125 24 ERROR nova RuntimeError: dictionary changed size during iteration

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z...

The service does appear to start up successfully. It doesn't happen on master.

Has anyone seen this before?

Cheers, Mark

On Tue, 21 Apr 2020 at 18:12, Radosław Piliszek radoslaw.piliszek@gmail.com wrote:

This is definitely the same issue (looping over mutated sequence), but it is present somewhere in the current packaging of RDO for Train on CentOS 8. Only Nova is hit so far.

It seems to be a different call trace to the referenced bug. I wonder if it's some distro-packaged configuration that triggers the immutability warnings, or a difference in timing.

-yoctozepto

On Tue, Apr 21, 2020 at 5:33 PM Mark Goddard mark@stackhpc.com wrote:

...

On Tue, 21 Apr 2020 at 16:05, Tobias Urdin tobias.urdin@binero.com wrote:

...

Hello,

This makes me recall a commit I saw in oslo.config when troubleshooting another issue. https://github.com/openstack/oslo.config/commit/e3e2ba55eeeb86a9bc0624bb2592...

Thanks Tobias, that does look relevant. It was cherry-picked to stable/train, and available in oslo.config 6.11.2. However, both source and binary images include that version of the package. The code path looks a little different in this case also.

...

Best regards ________________________________________ From: Mark Goddard mark@stackhpc.com Sent: Tuesday, April 21, 2020 4:28 PM To: openstack-discuss Subject: [kolla][tripleo][nova] nova scheduler startup errors in Train on CentOS 8

Hi,

Our (kolla-ansible) CentOS 8 binary (RDO) deploy jobs have been failing on stable/train since added. Because the source jobs were passing, I was kind of hoping it was a packaging issue that would right itself, but no such luck yet.

The majority of the job succeeds, but we grep our logs for CRITICAL messages, and nova-scheduler has a few when it starts up that cause us to fail:

2020-04-21 08:59:30.125 24 CRITICAL nova [req-86f6dad1-25f2-4c77-9d7d-ac7422801b4e - - - - -] Unhandled error: RuntimeError: dictionary changed size during iteration 2020-04-21 08:59:30.125 24 ERROR nova Traceback (most recent call last): 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/bin/nova-scheduler", line 10, in <module> 2020-04-21 08:59:30.125 24 ERROR nova sys.exit(main()) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 53, in main 2020-04-21 08:59:30.125 24 ERROR nova service.serve(server, workers=workers) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/nova/service.py", line 454, in serve 2020-04-21 08:59:30.125 24 ERROR nova restart_method='mutate') 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 842, in launch 2020-04-21 08:59:30.125 24 ERROR nova launcher.launch_service(service, workers=workers) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 606, in launch_service 2020-04-21 08:59:30.125 24 ERROR nova self._start_child(wrap) 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 575, in _start_child 2020-04-21 08:59:30.125 24 ERROR nova self.launcher.restart() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 311, in restart 2020-04-21 08:59:30.125 24 ERROR nova self.conf.mutate_config_files() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3013, in mutate_config_files 2020-04-21 08:59:30.125 24 ERROR nova self._warn_immutability() 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3040, in _warn_immutability 2020-04-21 08:59:30.125 24 ERROR nova for info, group in self._all_opt_infos(): 2020-04-21 08:59:30.125 24 ERROR nova File "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2502, in _all_opt_infos 2020-04-21 08:59:30.125 24 ERROR nova for info in self._opts.values(): 2020-04-21 08:59:30.125 24 ERROR nova RuntimeError: dictionary changed size during iteration

https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/z...

The service does appear to start up successfully. It doesn't happen on master.

Has anyone seen this before?

Cheers, Mark