[openstack-dev] [nova][NFS] Inexplicable utime permission denied when launching instance

Neil Jerram neil at tigera.io
Wed Oct 24 13:41:07 UTC 2018


Thanks so much for these hints, Erlon.  I will look closer at AppArmor.

    Neil

On Wed, Oct 24, 2018 at 1:41 PM Erlon Cruz <sombrafam at gmail.com> wrote:
>
> PS. Don't forget that if you change or disable AppArmor you will have to reboot the host so the kernel gets reloaded.
>
> Em qua, 24 de out de 2018 às 09:40, Erlon Cruz <sombrafam at gmail.com> escreveu:
>>
>> I think that there's a change that AppArmor is blocking the access. Have you checked the dmesg messages related with apparmor?
>>
>> Em sex, 19 de out de 2018 às 09:38, Neil Jerram <neil at tigera.io> escreveu:
>>>
>>> Wracking my brains over this one, would appreciate any pointers...
>>>
>>> Setup: Small test deployment with just 3 compute nodes, Queens on Ubuntu Bionic. The first compute node is an NFS server for /var/lib/nova/instances, and the other compute nodes mount that as NFS clients.
>>>
>>> Problem: Sometimes, when launching an instance which is scheduled to one of the client nodes, nova-compute (in imagebackend.py) gets Permission Denied (errno 13) when calling utime to touch the timestamp on the instance file.
>>>
>>> Through various bits of debugging and hackery, I've established that:
>>>
>>> - it looks like the problem never occurs when this is the call that bootstraps the privsep setup; but it does occur quite frequently on later calls
>>>
>>> - when the problem occurs, retrying doesn't help (5 times, with 0.5s in between)
>>>
>>> - the instance file does exist, and is owned by root with read/write permission for root
>>>
>>> - the privsep helper is running as root
>>>
>>> - the privsep helper receives and executes the request - so it's not a problem with communication between nova-compute and the helper
>>>
>>> - root is uid 0 on both NFS server and client
>>>
>>> - NFS setup does not have the root_squash option
>>>
>>> - there is some AppArmor setup, on both client and server, and I haven't yet worked out whether that might be relevant.
>>>
>>> Any ideas?
>>>
>>> Many thanks,
>>>       Neil
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev



More information about the OpenStack-dev mailing list