Hi James, Thanks, that explains a lot. Nova-compute was in enforce mode on the testbed deployment, while it was on complain mode on the other environment. Setting it to complain makes life a lot easier. Cheers, Eyle

On 15 Feb 2021, at 11:06, James Page <james.page@canonical.com> wrote:

Hi Eyle

On Mon, Feb 15, 2021 at 9:59 AM Eyle Brinkhuis <eyle.brinkhuis@surf.nl <mailto:eyle.brinkhuis@surf.nl>> wrote: Hi,

Deploying a new testbed, we experience an issue with nova-compute (using canonical charmed openstack Train). We use networking-vpp, that uses python-etcd. After installation of openstack, and configuration of the networking-vpp setup, we cannot start nova-compute anymore:

Feb 15 09:35:19 sr635-1 nova-compute[131674]: Traceback (most recent call last): Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/bin/nova-compute", line 6, in <module> Feb 15 09:35:19 sr635-1 nova-compute[131674]: from nova.cmd.compute import main Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/nova/cmd/__init__.py", line 16, in <module> Feb 15 09:35:19 sr635-1 nova-compute[131674]: import nova.monkey_patch # noqa Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/nova/monkey_patch.py", line 98, in <module> Feb 15 09:35:19 sr635-1 nova-compute[131674]: _monkey_patch() Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/nova/monkey_patch.py", line 67, in _monkey_patch Feb 15 09:35:19 sr635-1 nova-compute[131674]: import oslo_service # noqa Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/oslo_service/__init__.py", line 16, in <module> Feb 15 09:35:19 sr635-1 nova-compute[131674]: from oslo_log import log as logging Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/oslo_log/log.py", line 41, in <module> Feb 15 09:35:19 sr635-1 nova-compute[131674]: from oslo_config import cfg Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/oslo_config/cfg.py", line 53, in <module> Feb 15 09:35:19 sr635-1 nova-compute[131674]: from oslo_config import types Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/oslo_config/types.py", line 28, in <module> Feb 15 09:35:19 sr635-1 nova-compute[131674]: from debtcollector import removals Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/debtcollector/__init__.py", line 20, in <module> Feb 15 09:35:19 sr635-1 nova-compute[131674]: 'debtcollector').version_string() Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pbr/version.py", line 467, in version_string Feb 15 09:35:19 sr635-1 nova-compute[131674]: return self.semantic_version().brief_string() Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pbr/version.py", line 462, in semantic_version Feb 15 09:35:19 sr635-1 nova-compute[131674]: self._semantic = self._get_version_from_pkg_resources() Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pbr/version.py", line 439, in _get_version_from_pkg_resources Feb 15 09:35:19 sr635-1 nova-compute[131674]: import pkg_resources Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3250, in <module> Feb 15 09:35:19 sr635-1 nova-compute[131674]: @_call_aside Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3234, in _call_aside Feb 15 09:35:19 sr635-1 nova-compute[131674]: f(*args, **kwargs) Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 3263, in _initialize_master_working_set Feb 15 09:35:19 sr635-1 nova-compute[131674]: working_set = WorkingSet._build_master() Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 574, in _build_master Feb 15 09:35:19 sr635-1 nova-compute[131674]: ws = cls() Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 567, in __init__ Feb 15 09:35:19 sr635-1 nova-compute[131674]: self.add_entry(entry) Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 623, in add_entry Feb 15 09:35:19 sr635-1 nova-compute[131674]: for dist in find_distributions(entry, True): Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2065, in find_on_path Feb 15 09:35:19 sr635-1 nova-compute[131674]: for dist in factory(fullpath): Feb 15 09:35:19 sr635-1 nova-compute[131674]: File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2127, in distributions_from_metadata Feb 15 09:35:19 sr635-1 nova-compute[131674]: if len(os.listdir(path)) == 0: Feb 15 09:35:19 sr635-1 nova-compute[131674]: PermissionError: [Errno 13] Permission denied: '/usr/local/lib/python3.6/dist-packages/python_etcd-0.4.5.dist-info' Feb 15 09:35:19 sr635-1 systemd[1]: nova-compute.service: Main process exited, code=exited, status=1/FAILURE

Meanwhile, in a different deployment, which runs nova-compute 20.3.0-ubuntu1~cloud0, we do not experience this issue. Permissions of all the packages in /usr/local/lib/python3.6/dist-packages/ are all identical. When we remove all packages once again (python-etcd etc), nova-compute is able to start without any issue.

Anyone experiencing the same, or has a workaround?

I've not experienced this directly - have you got the apparmor enabled on one of the deployments but not the other? That's a charm configuration option. You would also see a DENIED message in the kern.log file if this is enabled (I think).

Regards

James