Open Stack

Wracking my brains over this one, would appreciate any pointers...

Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu
Bionic. The first compute node is an NFS server for
/var/lib/nova/instances, and the other compute nodes mount that as NFS
clients.

Problem: Sometimes, when launching an instance which is scheduled to one of
the client nodes, nova-compute (in imagebackend.py) gets Permission Denied
(errno 13) when calling utime to touch the timestamp on the instance file.

Through various bits of debugging and hackery, I've established that:

- it looks like the problem never occurs when this is the call that
bootstraps the privsep setup; but it does occur quite frequently on later
calls

- when the problem occurs, retrying doesn't help (5 times, with 0.5s in
between)

- the instance file does exist, and is owned by root with read/write
permission for root

- the privsep helper is running as root

- the privsep helper receives and executes the request - so it's not a
problem with communication between nova-compute and the helper

- root is uid 0 on both NFS server and client

- NFS setup does not have the root_squash option

- there is some AppArmor setup, on both client and server, and I haven't
yet worked out whether that might be relevant.

Any ideas?

Many thanks,
      Neil
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20181019/b244a2ad/attachment.html>