[openstack-dev] [cinder] [nova] os-brick privsep failures and an upgrade strategy?
Ivan Kolodyazhny
e0ne at e0ne.info
Wed Jul 13 13:54:50 UTC 2016
Thanks for the update, Matt.
I will join our meeting next week.
Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/
On Tue, Jul 12, 2016 at 4:25 PM, Matt Riedemann <mriedem at linux.vnet.ibm.com>
wrote:
> On 7/12/2016 6:29 AM, Ivan Kolodyazhny wrote:
>
>> Hi team,
>>
>> Do we have any decision on this issue? I've found few patches but both
>> of them are -1'ed.
>>
>> From Cinder perspective, it blocks us to release new os-brick with
>> features, which are needed for other projects like Cinder and
>> python-brick-cinderclient-ext.
>>
>> Regards,
>> Ivan Kolodyazhny,
>> http://blog.e0ne.info/
>>
>> On Wed, Jun 22, 2016 at 5:47 PM, Matt Riedemann
>> <mriedem at linux.vnet.ibm.com <mailto:mriedem at linux.vnet.ibm.com>> wrote:
>>
>> On 6/21/2016 10:12 PM, Angus Lees wrote:
>>
>> On Wed, 22 Jun 2016 at 05:59 Matt Riedemann
>> <mriedem at linux.vnet.ibm.com <mailto:mriedem at linux.vnet.ibm.com>
>> <mailto:mriedem at linux.vnet.ibm.com
>>
>> <mailto:mriedem at linux.vnet.ibm.com>>> wrote:
>>
>> Angus, what should we be looking at from the privsep side
>> for debugging
>> this?
>>
>>
>> The line above the screen-n-cpu.txt.gz failure you linked to is:
>> 2016-06-21 16:21:30.994
>> <
>> http://logs.openstack.org/85/331885/2/check/gate-grenade-dsvm-multinode/415e1bc/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2016-06-21_16_21_30_994
>> >1840
>> WARNING oslo.privsep.daemon [-] privsep log:
>> /usr/local/bin/nova-rootwrap: Unauthorized command: privsep-helper
>> --config-file /etc/nova/nova.conf --privsep_context
>> os_brick.privileged.default --privsep_sock_path
>> /tmp/tmpV5w2VC/privsep.sock (no filter matched)
>>
>> .. so nova-rootwrap is rejecting the privsep-helper command line
>> because no filter matched. This indicates the nova
>> compute.filters file
>> has not been updated, or is incorrect.
>>
>>
>> As was later pointed out by mtreinish, grenade is attempting to
>> run the
>> newton code against mitaka configs, and this includes using mitaka
>> rootwrap filters. Unfortunately, the change to add privsep to
>> nova's
>> rootwrap filters wasn't approved until the newton cycle (so that
>> all the
>> os-brick privsep-related changes could be approved together), and
>> so
>> this doesn't Just Work.
>>
>> Digging in further, it appears that there *is* a mechanism in
>> grenade to
>> upgrade rootwrap filters between major releases, but this needs
>> to be
>> explicitly updated for each project+release and hasn't been for
>> nova+mitaka->newton. I'm not sure how this is *meant* to work,
>> since
>> the grenade "theory of upgrade" doesn't mention when configs
>> should be
>> updated - the only mechanism provided is an "exception ... used
>> sparingly."
>>
>>
>> As noted in the review, my understanding of the config changes is
>> deprecation of options across release boundaries so that you can't
>> drop a config option that would break someone from release to
>> release without it being deprecated first. So deprecate option foo
>> in mitaka, people upgrading from liberty to mitaka aren't broken,
>> but they get warnings in mitaka so that when you drop the option in
>> newton it's not a surprise and consumers should have adjusted during
>> mitaka.
>>
>> For rootwrap filters I agree this is more complicated.
>>
>>
>> Anyway, I added an upgrade step for nova mitaka->newton that
>> updates
>> rootwrap filters appropriately(*). Again, I'm not sure what this
>> communicates to deployers compared to cinder (which *did* have the
>> updated rootwrap filter merged in mitaka, but of course that
>> update
>> still needs to be installed at some point).
>> (*) https://review.openstack.org/#/c/332610
>>
>> - Gus
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>> Alternatively Walter had a potential workaround to fallback to
>> rootwrap for os-brick:
>>
>> https://review.openstack.org/#/c/329586/
>>
>> So we could maybe use that for newton. But os-vif doesn't have
>> anything like that, so we'd have to see what kind of (immediately
>> deprecated) workaround could happen for os-vif in newton and then
>> drop that in ocata.
>>
>> I'm told danpb is out until tomorrow though so we'll probably need
>> to wait to talk to him about options there.
>>
>>
>> --
>>
>> Thanks,
>>
>> Matt Riedemann
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>> >
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> We probably aren't doing anything while Sean Dague is on vacation. He's
> back next week and we have the nova/cinder meetups, so I'm planning on
> talking about the grenade issue in person and hopefully we'll have a plan
> by the end of next week to move forward.
>
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160713/bdab20d1/attachment.html>
More information about the OpenStack-dev
mailing list