[openstack-dev] [cinder] [nova] os-brick privsep failures and an upgrade strategy?

Ivan Kolodyazhny e0ne at e0ne.info
Wed Jul 13 13:54:50 UTC 2016


Thanks for the update, Matt.

I will join our meeting next week.

Regards,
Ivan Kolodyazhny,
http://blog.e0ne.info/

On Tue, Jul 12, 2016 at 4:25 PM, Matt Riedemann <mriedem at linux.vnet.ibm.com>
wrote:

> On 7/12/2016 6:29 AM, Ivan Kolodyazhny wrote:
>
>> Hi team,
>>
>> Do we have any decision on this issue? I've found few patches but both
>> of them are -1'ed.
>>
>> From Cinder perspective, it blocks us to release new os-brick with
>> features, which are needed for other projects like Cinder and
>> python-brick-cinderclient-ext.
>>
>> Regards,
>> Ivan Kolodyazhny,
>> http://blog.e0ne.info/
>>
>> On Wed, Jun 22, 2016 at 5:47 PM, Matt Riedemann
>> <mriedem at linux.vnet.ibm.com <mailto:mriedem at linux.vnet.ibm.com>> wrote:
>>
>>     On 6/21/2016 10:12 PM, Angus Lees wrote:
>>
>>         On Wed, 22 Jun 2016 at 05:59 Matt Riedemann
>>         <mriedem at linux.vnet.ibm.com <mailto:mriedem at linux.vnet.ibm.com>
>>         <mailto:mriedem at linux.vnet.ibm.com
>>
>>         <mailto:mriedem at linux.vnet.ibm.com>>> wrote:
>>
>>             Angus, what should we be looking at from the privsep side
>>         for debugging
>>             this?
>>
>>
>>         The line above the screen-n-cpu.txt.gz failure you linked to is:
>>         2016-06-21 16:21:30.994
>>         <
>> http://logs.openstack.org/85/331885/2/check/gate-grenade-dsvm-multinode/415e1bc/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2016-06-21_16_21_30_994
>> >1840
>>         WARNING oslo.privsep.daemon [-] privsep log:
>>         /usr/local/bin/nova-rootwrap: Unauthorized command: privsep-helper
>>         --config-file /etc/nova/nova.conf --privsep_context
>>         os_brick.privileged.default --privsep_sock_path
>>         /tmp/tmpV5w2VC/privsep.sock (no filter matched)
>>
>>          .. so nova-rootwrap is rejecting the privsep-helper command line
>>         because no filter matched.  This indicates the nova
>>         compute.filters file
>>         has not been updated, or is incorrect.
>>
>>
>>         As was later pointed out by mtreinish, grenade is attempting to
>>         run the
>>         newton code against mitaka configs, and this includes using mitaka
>>         rootwrap filters.   Unfortunately, the change to add privsep to
>>         nova's
>>         rootwrap filters wasn't approved until the newton cycle (so that
>>         all the
>>         os-brick privsep-related changes could be approved together), and
>> so
>>         this doesn't Just Work.
>>
>>         Digging in further, it appears that there *is* a mechanism in
>>         grenade to
>>         upgrade rootwrap filters between major releases, but this needs
>>         to be
>>         explicitly updated for each project+release and hasn't been for
>>         nova+mitaka->newton.  I'm not sure how this is *meant* to work,
>>         since
>>         the grenade "theory of upgrade" doesn't mention when configs
>>         should be
>>         updated - the only mechanism provided is an "exception ... used
>>         sparingly."
>>
>>
>>     As noted in the review, my understanding of the config changes is
>>     deprecation of options across release boundaries so that you can't
>>     drop a config option that would break someone from release to
>>     release without it being deprecated first. So deprecate option foo
>>     in mitaka, people upgrading from liberty to mitaka aren't broken,
>>     but they get warnings in mitaka so that when you drop the option in
>>     newton it's not a surprise and consumers should have adjusted during
>>     mitaka.
>>
>>     For rootwrap filters I agree this is more complicated.
>>
>>
>>         Anyway, I added an upgrade step for nova mitaka->newton that
>> updates
>>         rootwrap filters appropriately(*).  Again, I'm not sure what this
>>         communicates to deployers compared to cinder (which *did* have the
>>         updated rootwrap filter merged in mitaka, but of course that
>> update
>>         still needs to be installed at some point).
>>         (*) https://review.openstack.org/#/c/332610
>>
>>          - Gus
>>
>>
>>
>> __________________________________________________________________________
>>         OpenStack Development Mailing List (not for usage questions)
>>         Unsubscribe:
>>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>         <
>> http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>     Alternatively Walter had a potential workaround to fallback to
>>     rootwrap for os-brick:
>>
>>     https://review.openstack.org/#/c/329586/
>>
>>     So we could maybe use that for newton. But os-vif doesn't have
>>     anything like that, so we'd have to see what kind of (immediately
>>     deprecated) workaround could happen for os-vif in newton and then
>>     drop that in ocata.
>>
>>     I'm told danpb is out until tomorrow though so we'll probably need
>>     to wait to talk to him about options there.
>>
>>
>>     --
>>
>>     Thanks,
>>
>>     Matt Riedemann
>>
>>
>>
>> __________________________________________________________________________
>>     OpenStack Development Mailing List (not for usage questions)
>>     Unsubscribe:
>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>> >
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>>
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> We probably aren't doing anything while Sean Dague is on vacation. He's
> back next week and we have the nova/cinder meetups, so I'm planning on
> talking about the grenade issue in person and hopefully we'll have a plan
> by the end of next week to move forward.
>
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20160713/bdab20d1/attachment.html>


More information about the OpenStack-dev mailing list