<div dir="ltr">Thanks for the update, Matt.<div><br></div><div>I will join our meeting next week.</div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>Regards,<br>Ivan Kolodyazhny,<br><a href="http://blog.e0ne.info/" target="_blank">http://blog.e0ne.info/</a></div></div></div></div>
<br><div class="gmail_quote">On Tue, Jul 12, 2016 at 4:25 PM, Matt Riedemann <span dir="ltr"><<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 7/12/2016 6:29 AM, Ivan Kolodyazhny wrote:<br>
</span><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
Hi team,<br>
<br>
Do we have any decision on this issue? I've found few patches but both<br>
of them are -1'ed.<br>
<br>
>From Cinder perspective, it blocks us to release new os-brick with<br>
features, which are needed for other projects like Cinder and<br>
python-brick-cinderclient-ext.<br>
<br>
Regards,<br>
Ivan Kolodyazhny,<br>
<a href="http://blog.e0ne.info/" rel="noreferrer" target="_blank">http://blog.e0ne.info/</a><br>
<br>
On Wed, Jun 22, 2016 at 5:47 PM, Matt Riedemann<br></span><span class="">
<<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a> <mailto:<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a>>> wrote:<br>
<br>
On 6/21/2016 10:12 PM, Angus Lees wrote:<br>
<br>
On Wed, 22 Jun 2016 at 05:59 Matt Riedemann<br>
<<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a> <mailto:<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a>><br></span>
<mailto:<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a><div><div class="h5"><br>
<mailto:<a href="mailto:mriedem@linux.vnet.ibm.com" target="_blank">mriedem@linux.vnet.ibm.com</a>>>> wrote:<br>
<br>
Angus, what should we be looking at from the privsep side<br>
for debugging<br>
this?<br>
<br>
<br>
The line above the screen-n-cpu.txt.gz failure you linked to is:<br>
2016-06-21 16:21:30.994<br>
<<a href="http://logs.openstack.org/85/331885/2/check/gate-grenade-dsvm-multinode/415e1bc/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2016-06-21_16_21_30_994" rel="noreferrer" target="_blank">http://logs.openstack.org/85/331885/2/check/gate-grenade-dsvm-multinode/415e1bc/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2016-06-21_16_21_30_994</a>>1840<br>
WARNING oslo.privsep.daemon [-] privsep log:<br>
/usr/local/bin/nova-rootwrap: Unauthorized command: privsep-helper<br>
--config-file /etc/nova/nova.conf --privsep_context<br>
os_brick.privileged.default --privsep_sock_path<br>
/tmp/tmpV5w2VC/privsep.sock (no filter matched)<br>
<br>
.. so nova-rootwrap is rejecting the privsep-helper command line<br>
because no filter matched. This indicates the nova<br>
compute.filters file<br>
has not been updated, or is incorrect.<br>
<br>
<br>
As was later pointed out by mtreinish, grenade is attempting to<br>
run the<br>
newton code against mitaka configs, and this includes using mitaka<br>
rootwrap filters. Unfortunately, the change to add privsep to<br>
nova's<br>
rootwrap filters wasn't approved until the newton cycle (so that<br>
all the<br>
os-brick privsep-related changes could be approved together), and so<br>
this doesn't Just Work.<br>
<br>
Digging in further, it appears that there *is* a mechanism in<br>
grenade to<br>
upgrade rootwrap filters between major releases, but this needs<br>
to be<br>
explicitly updated for each project+release and hasn't been for<br>
nova+mitaka->newton. I'm not sure how this is *meant* to work,<br>
since<br>
the grenade "theory of upgrade" doesn't mention when configs<br>
should be<br>
updated - the only mechanism provided is an "exception ... used<br>
sparingly."<br>
<br>
<br>
As noted in the review, my understanding of the config changes is<br>
deprecation of options across release boundaries so that you can't<br>
drop a config option that would break someone from release to<br>
release without it being deprecated first. So deprecate option foo<br>
in mitaka, people upgrading from liberty to mitaka aren't broken,<br>
but they get warnings in mitaka so that when you drop the option in<br>
newton it's not a surprise and consumers should have adjusted during<br>
mitaka.<br>
<br>
For rootwrap filters I agree this is more complicated.<br>
<br>
<br>
Anyway, I added an upgrade step for nova mitaka->newton that updates<br>
rootwrap filters appropriately(*). Again, I'm not sure what this<br>
communicates to deployers compared to cinder (which *did* have the<br>
updated rootwrap filter merged in mitaka, but of course that update<br>
still needs to be installed at some point).<br>
(*) <a href="https://review.openstack.org/#/c/332610" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/332610</a><br>
<br>
- Gus<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br></div></div>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><span class=""><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
<br>
Alternatively Walter had a potential workaround to fallback to<br>
rootwrap for os-brick:<br>
<br>
<a href="https://review.openstack.org/#/c/329586/" rel="noreferrer" target="_blank">https://review.openstack.org/#/c/329586/</a><br>
<br>
So we could maybe use that for newton. But os-vif doesn't have<br>
anything like that, so we'd have to see what kind of (immediately<br>
deprecated) workaround could happen for os-vif in newton and then<br>
drop that in ocata.<br>
<br>
I'm told danpb is out until tomorrow though so we'll probably need<br>
to wait to talk to him about options there.<br>
<br>
<br>
--<br>
<br>
Thanks,<br>
<br>
Matt Riedemann<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br></span>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><span class=""><br>
<br>
<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
<br>
</span></blockquote>
<br>
We probably aren't doing anything while Sean Dague is on vacation. He's back next week and we have the nova/cinder meetups, so I'm planning on talking about the grenade issue in person and hopefully we'll have a plan by the end of next week to move forward.<div class="HOEnZb"><div class="h5"><br>
<br>
-- <br>
<br>
Thanks,<br>
<br>
Matt Riedemann<br>
<br>
<br>
__________________________________________________________________________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.openstack.org?subject:unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev</a><br>
</div></div></blockquote></div><br></div>