[kolla][tripleo][nova] nova scheduler startup errors in Train on CentOS 8

Mark Goddard mark at stackhpc.com
Wed Apr 22 13:41:55 UTC 2020


On Tue, 21 Apr 2020 at 18:12, Radosław Piliszek
<radoslaw.piliszek at gmail.com> wrote:
>
> This is definitely the same issue (looping over mutated sequence), but
> it is present somewhere in the current packaging of RDO for Train on
> CentOS 8.
> Only Nova is hit so far.

It seems to be a different call trace to the referenced bug. I wonder
if it's some distro-packaged configuration that triggers the
immutability warnings, or a difference in timing.

>
> -yoctozepto
>
> On Tue, Apr 21, 2020 at 5:33 PM Mark Goddard <mark at stackhpc.com> wrote:
> >
> > On Tue, 21 Apr 2020 at 16:05, Tobias Urdin <tobias.urdin at binero.com> wrote:
> > >
> > > Hello,
> > >
> > > This makes me recall a commit I saw in oslo.config when troubleshooting another issue.
> > > https://github.com/openstack/oslo.config/commit/e3e2ba55eeeb86a9bc0624bb2592e46583e839e7
> >
> > Thanks Tobias, that does look relevant. It was cherry-picked to
> > stable/train, and available in oslo.config 6.11.2. However, both
> > source and binary images include that version of the package. The code
> > path looks a little different in this case also.
> >
> > >
> > > Best regards
> > > ________________________________________
> > > From: Mark Goddard <mark at stackhpc.com>
> > > Sent: Tuesday, April 21, 2020 4:28 PM
> > > To: openstack-discuss
> > > Subject: [kolla][tripleo][nova] nova scheduler startup errors in Train on CentOS 8
> > >
> > > Hi,
> > >
> > > Our (kolla-ansible) CentOS 8 binary (RDO) deploy jobs have been
> > > failing on stable/train since added. Because the source jobs were
> > > passing, I was kind of hoping it was a packaging issue that would
> > > right itself, but no such luck yet.
> > >
> > > The majority of the job succeeds, but we grep our logs for CRITICAL
> > > messages, and nova-scheduler has a few when it starts up that cause us
> > > to fail:
> > >
> > > 2020-04-21 08:59:30.125 24 CRITICAL nova
> > > [req-86f6dad1-25f2-4c77-9d7d-ac7422801b4e - - - - -] Unhandled error:
> > > RuntimeError: dictionary changed size during iteration
> > > 2020-04-21 08:59:30.125 24 ERROR nova Traceback (most recent call last):
> > > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > > "/usr/bin/nova-scheduler", line 10, in <module>
> > > 2020-04-21 08:59:30.125 24 ERROR nova     sys.exit(main())
> > > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > > "/usr/lib/python3.6/site-packages/nova/cmd/scheduler.py", line 53, in
> > > main
> > > 2020-04-21 08:59:30.125 24 ERROR nova     service.serve(server, workers=workers)
> > > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > > "/usr/lib/python3.6/site-packages/nova/service.py", line 454, in serve
> > > 2020-04-21 08:59:30.125 24 ERROR nova     restart_method='mutate')
> > > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > > "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 842,
> > > in launch
> > > 2020-04-21 08:59:30.125 24 ERROR nova
> > > launcher.launch_service(service, workers=workers)
> > > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > > "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 606,
> > > in launch_service
> > > 2020-04-21 08:59:30.125 24 ERROR nova     self._start_child(wrap)
> > > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > > "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 575,
> > > in _start_child
> > > 2020-04-21 08:59:30.125 24 ERROR nova     self.launcher.restart()
> > > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > > "/usr/lib/python3.6/site-packages/oslo_service/service.py", line 311,
> > > in restart
> > > 2020-04-21 08:59:30.125 24 ERROR nova     self.conf.mutate_config_files()
> > > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > > "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3013, in
> > > mutate_config_files
> > > 2020-04-21 08:59:30.125 24 ERROR nova     self._warn_immutability()
> > > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > > "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 3040, in
> > > _warn_immutability
> > > 2020-04-21 08:59:30.125 24 ERROR nova     for info, group in
> > > self._all_opt_infos():
> > > 2020-04-21 08:59:30.125 24 ERROR nova   File
> > > "/usr/lib/python3.6/site-packages/oslo_config/cfg.py", line 2502, in
> > > _all_opt_infos
> > > 2020-04-21 08:59:30.125 24 ERROR nova     for info in self._opts.values():
> > > 2020-04-21 08:59:30.125 24 ERROR nova RuntimeError: dictionary changed
> > > size during iteration
> > >
> > > https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_339/721329/1/check/kolla-ansible-centos8-binary/339767b/primary/logs/kolla/nova/nova-scheduler.txt
> > >
> > > The service does appear to start up successfully. It doesn't happen on master.
> > >
> > > Has anyone seen this before?
> > >
> > > Cheers,
> > > Mark
> > >
> > >
> >



More information about the openstack-discuss mailing list