Open Stack

This is definitely the same issue (looping over mutated sequence), but
it is present somewhere in the current packaging of RDO for Train on
CentOS 8.
Only Nova is hit so far.

-yoctozepto

On Tue, Apr 21, 2020 at 5:33 PM Mark Goddard <mark at stackhpc.com> wrote:
>
> On Tue, 21 Apr 2020 at 16:05, Tobias Urdin <tobias.urdin at binero.com> wrote:
> >
> > Hello,
> >
> > This makes me recall a commit I saw in oslo.config when troubleshooting another issue.
> > https://github.com/openstack/oslo.config/commit/e3e2ba55eeeb86a9bc0624bb2592e46583e839e7
>
> Thanks Tobias, that does look relevant. It was cherry-picked to
> stable/train, and available in oslo.config 6.11.2. However, both
> source and binary images include that version of the package. The code
> path looks a little different in this case also.
>
> >
> > Best regards
> > ________________________________________
> > From: Mark Goddard <mark at stackhpc.com>
> > Sent: Tuesday, April 21, 2020 4:28 PM
> > To: openstack-discuss
> > Subject: [kolla][tripleo][nova] nova scheduler startup errors in Train on CentOS 8
> >
> > Hi,
> >
> > Our (kolla-ansible) CentOS 8 binary (RDO) deploy jobs have been
> > failing on stable/train since added. Because the source jobs were
> > passing, I was kind of hoping it was a packaging issue that would
> > right itself, but no such luck yet.
> >
> > The majority of the job succeeds, but we grep our logs for CRITICAL
> > messages, and nova-scheduler has a few when it starts up that cause us
> > to fail:
> >
> > 2020-04-21 08:59:30.125 24 CRITICAL nova
> > [req-86f6dad1-25f2-4c77-9d7d-ac7422801b4e - - - - -] Unhandled error:
> > RuntimeError: dictionary changed size during iteration
> > 2020-04-21 08:59:30.125 24 ERROR nova Traceback (most recent call last):
> > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > "/usr/bin/nova-scheduler", line 10, in <module>
> > 2020-04-21 08:59:30.125 24 ERROR nova     sys.exit(main())
> > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > "/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 53, in
> > main
> > 2020-04-21 08:59:30.125 24 ERROR nova     service.serve(server, workers=workers)
> > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > "/usr/lib/python3.6/site-packages/nova/service.py", line 454, in serve
> > 2020-04-21 08:59:30.125 24 ERROR nova     restart_method='mutate')
> > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 842,
> > in launch
> > 2020-04-21 08:59:30.125 24 ERROR nova
> > launcher.launch_service(service, workers=workers)
> > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 606,
> > in launch_service
> > 2020-04-21 08:59:30.125 24 ERROR nova     self._start_child(wrap)
> > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 575,
> > in _start_child
> > 2020-04-21 08:59:30.125 24 ERROR nova     self.launcher.restart()
> > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 311,
> > in restart
> > 2020-04-21 08:59:30.125 24 ERROR nova     self.conf.mutate_config_files()
> > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3013, in
> > mutate_config_files
> > 2020-04-21 08:59:30.125 24 ERROR nova     self._warn_immutability()
> > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3040, in
> > _warn_immutability
> > 2020-04-21 08:59:30.125 24 ERROR nova     for info, group in
> > self._all_opt_infos():
> > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2502, in
> > _all_opt_infos
> > 2020-04-21 08:59:30.125 24 ERROR nova     for info in self._opts.values():
> > 2020-04-21 08:59:30.125 24 ERROR nova RuntimeError: dictionary changed
> > size during iteration
> >
> > https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_339/721329/1/check/kolla-ansible-centos8-binary/339767b/primary/logs/kolla/nova/nova-scheduler.txt
> >
> > The service does appear to start up successfully. It doesn't happen on master.
> >
> > Has anyone seen this before?
> >
> > Cheers,
> > Mark
> >
> >
>