On Tue, Apr 8, 2025 at 5:41 PM Nell Jerram <nell@tigera.io> wrote:
On Tue, Apr 8, 2025 at 10:22 AM Thomas Goirand <zigo@debian.org> wrote:
On 4/8/25 11:15, Nell Jerram wrote:
> Many thanks for this Thomas.  I am using OpenStack Caracal packages on
> Ubuntu Jammy (using cloud-archive:caracal), so I'm surprised I'm not
> seeing those numbers.
>
> Aha, my setup script creates the nova user with
>
>      useradd -m -p password -s /bin/bash nova
>
> _before_ installing the nova packages.  I wonder if this is indeed the
> problem...
>
> Best wishes - Nell

Well, I just had a quick look at the Ubuntu packages, and they are
creating the UID / GID for you (see below). I'd suggest either doing
what the package does in your script, or just let the package do it.

Cheers,

Thomas Goirand (zigo)

#!/bin/sh -e

NOVA_UID=64060
NOVA_GID=64060

if [ "$1" = "configure" ]; then
     if ! getent group nova > /dev/null 2>&1; then
         addgroup --quiet --system \
             --gid $NOVA_GID nova 2>/dev/null
     fi

     if ! getent passwd nova > /dev/null 2>&1; then
         adduser --quiet --system \
             --home /var/lib/nova \
             --no-create-home \
             --uid $NOVA_UID \
             --gid $NOVA_GID \
             --shell /usr/sbin/nologin nova 2>/dev/null
     fi

     if [ -z "$2" ]; then
         # New install - blanket permissions
         chown -R nova:nova /var/lib/nova/
     fi

     chown nova:adm /var/log/nova
     chmod 0750 /var/log/nova


Unfortunately the problem is still happening in the same way with the Ubuntu packaging UIDs and GIDs.

I will keep digging and report back!

Best wishes - Nell

auditctl is a nice tool!  (Thank you https://serverfault.com/questions/619722/how-do-i-detect-what-is-changing-file-ownership-on-linux )

This is the audit entry for the operation that converts it to root ownership: 

time->Tue Apr  8 16:49:46 2025
type=PROCTITLE msg=audit(1744130986.261:218): proctitle="/usr/sbin/libvirtd"
type=PATH msg=audit(1744130986.261:218): item=0 name="/var/lib/nova/instances/c63af63f-a5b4-44b6-af9d-b26c85f091b6/disk" inode=802497 dev=00:30 mode=0100640 ouid=64055 ogid=109 rdev=00:00 nametype=NORMAL cap_fp=0 cap_fi=0 cap_fe=0 cap_fver=0 cap_frootid=0
type=CWD msg=audit(1744130986.261:218): cwd="/"
type=SYSCALL msg=audit(1744130986.261:218): arch=c000003e syscall=92 success=yes exit=0 a0=7556ac0bafb0 a1=0 a2=0 a3=0 items=1 ppid=46198 pid=49872 auid=4294967295 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=(none) ses=4294967295 comm="rpc-libvirtd" exe="/usr/sbin/libvirtd" subj=libvirtd key="njdisk"

syscall 92 is chown, so that's libvirtd running as root and chowning, presumably to itself.

That then led me to https://bugs.launchpad.net/ubuntu/+source/libvirt/+bug/1784001 and https://netapp-openstack-dev.github.io/openstack-docs/ussuri/cinder/deployment_choice/section_nfs-security.html, and it appears that a solution is to add

dynamic_ownership = 0
user = "nova"

to /etc/libvirt/qemu.conf

I don't yet feel confident that that is _the_ right solution, or know if it might cause other regressions in my testing, but this feels like progress.

Best wishes - Nell