It appears that the extra * was the issue. After removing it I can run the rootwrap daemon without errors. I'm not 100% sure because the issue took 2 weeks to show up after the initial config change, but this seems to have fixed the problem. From: Erik Olof Gunnar Andersson <eandersson@blizzard.com> Sent: Thursday, October 10, 2019 6:21 PM To: Albert Braden <albertb@synopsys.com>; Chris Apsey <bitskrieg@bitskrieg.net> Cc: openstack-discuss@lists.openstack.org Subject: Re: Port creation times out for some VMs in large group Btw I still think your suders is slightly incorrect. I feel like that is significant, but not a hundred. Drop the star at the end of the last line. root@us01odc-qa-ctrl3:/var/log/neutron# cat /etc/sudoers.d/neutron_sudoers Defaults:neutron !requiretty neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf * neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf ________________________________ From: Erik Olof Gunnar Andersson <eandersson@blizzard.com<mailto:eandersson@blizzard.com>> Sent: Thursday, October 10, 2019 6:18 PM To: Albert Braden <Albert.Braden@synopsys.com<mailto:Albert.Braden@synopsys.com>>; Chris Apsey <bitskrieg@bitskrieg.net<mailto:bitskrieg@bitskrieg.net>> Cc: openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: Re: Port creation times out for some VMs in large group Maybe double check that your rootwrap config is up to date? /etc/neutron/rootwrap .conf and /etc/neutron/rootwrap.d (Make sure to pick the appropriate branch in github) https://github.com/openstack/neutron/blob/master/etc/rootwrap.conf<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_neutron_blob_master_etc_rootwrap.conf&d=DwMFIw&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=29xksG5hik0V-K7r47f3d2k4DKuxpLDhvkdGmQ2rX5o&s=gAk-Efh-K5lP8Zm2rRhHbkXiZvUUFvZJtNKuqbmMukM&e=> https://github.com/openstack/neutron/tree/master/etc/neutron/rootwrap.d<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_openstack_neutron_tree_master_etc_neutron_rootwrap.d&d=DwMFIw&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=29xksG5hik0V-K7r47f3d2k4DKuxpLDhvkdGmQ2rX5o&s=asd9vqO5bTt0d0a1Y2ckk_5HyxY4criuTMgVu-MQlDU&e=> ________________________________ From: Albert Braden <Albert.Braden@synopsys.com<mailto:Albert.Braden@synopsys.com>> Sent: Thursday, October 10, 2019 1:45 PM To: Erik Olof Gunnar Andersson <eandersson@blizzard.com<mailto:eandersson@blizzard.com>>; Chris Apsey <bitskrieg@bitskrieg.net<mailto:bitskrieg@bitskrieg.net>> Cc: openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> <openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org>> Subject: RE: Port creation times out for some VMs in large group The errors appear to start with this line: 2019-10-10 13:42:48.261 1211336 ERROR neutron.agent.linux.utils [req-42c530f6-6e08-47c1-8ed4-dcb31c9cd972 - - - - -] Rootwrap error running command: ['iptables-save', '-t', 'raw']: Exception: Failed to spawn rootwrap process. We're not running iptables. Do we need it, to use the rootwrap daemon? From: Albert Braden <Albert.Braden@synopsys.com<mailto:Albert.Braden@synopsys.com>> Sent: Thursday, October 10, 2019 12:13 PM To: Erik Olof Gunnar Andersson <eandersson@blizzard.com<mailto:eandersson@blizzard.com>>; Chris Apsey <bitskrieg@bitskrieg.net<mailto:bitskrieg@bitskrieg.net>> Cc: openstack-discuss@lists.openstack.org<mailto:openstack-discuss@lists.openstack.org> Subject: RE: Port creation times out for some VMs in large group It looks like something is still missing. I added the line to /etc/sudoers.d/neutron_sudoers: root@us01odc-qa-ctrl3:/var/log/neutron# cat /etc/sudoers.d/neutron_sudoers Defaults:neutron !requiretty neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf * neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf * Then I restarted neutron services and the error was gone... for a few minutes, and then it came back on ctrl3. Ctrl1/2 aren't erroring at this time. I changed neutron's shell and tested the daemon command and it seems to work: root@us01odc-qa-ctrl3:~# su - neutron neutron@us01odc-qa-ctrl3:~$ /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf /tmp/rootwrap-5b1QoP/rootwrap.sock Z%▒"▒▒▒Vs▒▒5-▒,a▒▒▒▒G▒▒▒▒v▒▒ But neutron-linuxbridge-agent.log still scrolls errors: http://paste.openstack.org/show/782740/<https://urldefense.com/v3/__http:/paste.openstack.org/show/782740/__;!2E0gRdhhnqPNNL0!z5cwPxQ1y_zz0MvtFzMZSCIh7-3d80kxciHbPtkj4LbHCzSkzNpf36RwLi8kWGm1Ew$> It appears that there is another factor besides the config, because even when the sudoers line was missing, it would work for hours or days before the error started. It has been working in our prod cluster for about a week now, without the sudoers line. It seems like it should not work that way. What am I missing?