Open Stack

The errors appear to start with this line:

2019-10-10 13:42:48.261 1211336 ERROR neutron.agent.linux.utils [req-42c530f6-6e08-47c1-8ed4-dcb31c9cd972 - - - - -] Rootwrap error running command: ['iptables-save', '-t', 'raw']: Exception: Failed to spawn rootwrap process.

We’re not running iptables. Do we need it, to use the rootwrap daemon?

From: Albert Braden <Albert.Braden at synopsys.com>
Sent: Thursday, October 10, 2019 12:13 PM
To: Erik Olof Gunnar Andersson <eandersson at blizzard.com>; Chris Apsey <bitskrieg at bitskrieg.net>
Cc: openstack-discuss at lists.openstack.org
Subject: RE: Port creation times out for some VMs in large group

It looks like something is still missing. I added the line to /etc/sudoers.d/neutron_sudoers:

root at us01odc-qa-ctrl3:/var/log/neutron# cat /etc/sudoers.d/neutron_sudoers
Defaults:neutron !requiretty

neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf *

Then I restarted neutron services and the error was gone… for a few minutes, and then it came back on ctrl3. Ctrl1/2 aren’t erroring at this time. I changed neutron’s shell and tested the daemon command and it seems to work:

root at us01odc-qa-ctrl3:~# su - neutron
neutron at us01odc-qa-ctrl3:~$ /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf
/tmp/rootwrap-5b1QoP/rootwrap.sock
Z%▒"▒▒▒Vs▒▒5-▒,a▒▒▒▒G▒▒▒▒v▒▒

But neutron-linuxbridge-agent.log still scrolls errors:

http://paste.openstack.org/show/782740/

It appears that there is another factor besides the config, because even when the sudoers line was missing, it would work for hours or days before the error started. It has been working in our prod cluster for about a week now, without the sudoers line. It seems like it should not work that way. What am I missing?


From: Erik Olof Gunnar Andersson <eandersson at blizzard.com<mailto:eandersson at blizzard.com>>
Sent: Thursday, October 10, 2019 11:08 AM
To: Albert Braden <albertb at synopsys.com<mailto:albertb at synopsys.com>>; Chris Apsey <bitskrieg at bitskrieg.net<mailto:bitskrieg at bitskrieg.net>>
Cc: openstack-discuss at lists.openstack.org<mailto:openstack-discuss at lists.openstack.org>
Subject: RE: Port creation times out for some VMs in large group

Yea – if you look at your sudoers its only allowing the old traditional rootwrap, and not the new daemon. You need both.

Defaults:neutron !requiretty

neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *
neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf

Best Regards, Erik Olof Gunnar Andersson

From: Albert Braden <Albert.Braden at synopsys.com<mailto:Albert.Braden at synopsys.com>>
Sent: Thursday, October 10, 2019 11:05 AM
To: Erik Olof Gunnar Andersson <eandersson at blizzard.com<mailto:eandersson at blizzard.com>>; Chris Apsey <bitskrieg at bitskrieg.net<mailto:bitskrieg at bitskrieg.net>>
Cc: openstack-discuss at lists.openstack.org<mailto:openstack-discuss at lists.openstack.org>
Subject: RE: Port creation times out for some VMs in large group

I have the neutron sudoers line under sudoers.d:

root at us01odc-qa-ctrl1:/etc/sudoers.d#<mailto:root at us01odc-qa-ctrl1:/etc/sudoers.d> cat neutron_sudoers
Defaults:neutron !requiretty

neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap /etc/neutron/rootwrap.conf *

Whatever is causing this didn’t start until I had been running the rootwrap daemon for 2 weeks, and it has not started in our prod cluster.

From: Erik Olof Gunnar Andersson <eandersson at blizzard.com<mailto:eandersson at blizzard.com>>
Sent: Wednesday, October 9, 2019 6:40 PM
To: Albert Braden <albertb at synopsys.com<mailto:albertb at synopsys.com>>; Chris Apsey <bitskrieg at bitskrieg.net<mailto:bitskrieg at bitskrieg.net>>
Cc: openstack-discuss at lists.openstack.org<mailto:openstack-discuss at lists.openstack.org>
Subject: Re: Port creation times out for some VMs in large group

You are probably missing an entry in your sudoers file.

You need something like


neutron ALL = (root) NOPASSWD: /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf

________________________________
From: Albert Braden <Albert.Braden at synopsys.com<mailto:Albert.Braden at synopsys.com>>
Sent: Wednesday, October 9, 2019 5:20 PM
To: Chris Apsey <bitskrieg at bitskrieg.net<mailto:bitskrieg at bitskrieg.net>>
Cc: openstack-discuss at lists.openstack.org<mailto:openstack-discuss at lists.openstack.org> <openstack-discuss at lists.openstack.org<mailto:openstack-discuss at lists.openstack.org>>
Subject: RE: Port creation times out for some VMs in large group


We tested this in dev and qa and then implemented in production and it did make a difference, but 2 weeks later we started seeing an issue, first in dev, and then in qa. In syslog we see neutron-linuxbridge-agent.service stopping and starting[1]. In neutron-linuxbridge-agent.log we see a rootwrap error[2]: “Exception: Failed to spawn rootwrap process.”



If I comment out ‘root_helper_daemon = "sudo /usr/bin/neutron-rootwrap-daemon /etc/neutron/rootwrap.conf"’ and restart neutron services then the error goes away.



How can I use the root_helper_daemon setting without creating this new error?



http://paste.openstack.org/show/782622/<https://urldefense.proofpoint.com/v2/url?u=http-3A__paste.openstack.org_show_782622_&d=DwMFAg&c=DPL6_X_6JkXFx7AXWqB0tg&r=XrJBXYlVPpvOXkMqGPz6KucRW_ils95ZMrEmlTflPm8&m=hT1YhRjyM0zYEXl5feVL1lmrbHaM7sytttrPvi1aZzg&s=mxxRA-SpuIF9xc1Pgx9RrbC3UjGdAFrXm4X6lH6UbR8&e=>


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-discuss/attachments/20191010/af80d115/attachment-0001.html>