Open Stack

On Tue, 21 Apr 2020 at 16:05, Tobias Urdin <tobias.urdin at binero.com> wrote:
>
> Hello,
>
> This makes me recall a commit I saw in oslo.config when troubleshooting another issue.
> https://github.com/openstack/oslo.config/commit/e3e2ba55eeeb86a9bc0624bb2592e46583e839e7

Thanks Tobias, that does look relevant. It was cherry-picked to
stable/train, and available in oslo.config 6.11.2. However, both
source and binary images include that version of the package. The code
path looks a little different in this case also.

>
> Best regards
> ________________________________________
> From: Mark Goddard <mark at stackhpc.com>
> Sent: Tuesday, April 21, 2020 4:28 PM
> To: openstack-discuss
> Subject: [kolla][tripleo][nova] nova scheduler startup errors in Train on CentOS 8
>
> Hi,
>
> Our (kolla-ansible) CentOS 8 binary (RDO) deploy jobs have been
> failing on stable/train since added. Because the source jobs were
> passing, I was kind of hoping it was a packaging issue that would
> right itself, but no such luck yet.
>
> The majority of the job succeeds, but we grep our logs for CRITICAL
> messages, and nova-scheduler has a few when it starts up that cause us
> to fail:
>
> 2020-04-21 08:59:30.125 24 CRITICAL nova
> [req-86f6dad1-25f2-4c77-9d7d-ac7422801b4e - - - - -] Unhandled error:
> RuntimeError: dictionary changed size during iteration
> 2020-04-21 08:59:30.125 24 ERROR nova Traceback (most recent call last):
> 2020-04-21 08:59:30.125 24 ERROR nova   File
> "/usr/bin/nova-scheduler", line 10, in <module>
> 2020-04-21 08:59:30.125 24 ERROR nova     sys.exit(main())
> 2020-04-21 08:59:30.125 24 ERROR nova   File
> "/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 53, in
> main
> 2020-04-21 08:59:30.125 24 ERROR nova     service.serve(server, workers=workers)
> 2020-04-21 08:59:30.125 24 ERROR nova   File
> "/usr/lib/python3.6/site-packages/nova/service.py", line 454, in serve
> 2020-04-21 08:59:30.125 24 ERROR nova     restart_method='mutate')
> 2020-04-21 08:59:30.125 24 ERROR nova   File
> "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 842,
> in launch
> 2020-04-21 08:59:30.125 24 ERROR nova
> launcher.launch_service(service, workers=workers)
> 2020-04-21 08:59:30.125 24 ERROR nova   File
> "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 606,
> in launch_service
> 2020-04-21 08:59:30.125 24 ERROR nova     self._start_child(wrap)
> 2020-04-21 08:59:30.125 24 ERROR nova   File
> "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 575,
> in _start_child
> 2020-04-21 08:59:30.125 24 ERROR nova     self.launcher.restart()
> 2020-04-21 08:59:30.125 24 ERROR nova   File
> "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 311,
> in restart
> 2020-04-21 08:59:30.125 24 ERROR nova     self.conf.mutate_config_files()
> 2020-04-21 08:59:30.125 24 ERROR nova   File
> "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3013, in
> mutate_config_files
> 2020-04-21 08:59:30.125 24 ERROR nova     self._warn_immutability()
> 2020-04-21 08:59:30.125 24 ERROR nova   File
> "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3040, in
> _warn_immutability
> 2020-04-21 08:59:30.125 24 ERROR nova     for info, group in
> self._all_opt_infos():
> 2020-04-21 08:59:30.125 24 ERROR nova   File
> "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2502, in
> _all_opt_infos
> 2020-04-21 08:59:30.125 24 ERROR nova     for info in self._opts.values():
> 2020-04-21 08:59:30.125 24 ERROR nova RuntimeError: dictionary changed
> size during iteration
>
> https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_339/721329/1/check/kolla-ansible-centos8-binary/339767b/primary/logs/kolla/nova/nova-scheduler.txt
>
> The service does appear to start up successfully. It doesn't happen on master.
>
> Has anyone seen this before?
>
> Cheers,
> Mark
>
>