[openstack-dev] [cinder] [nova] os-brick privsep failures and an upgrade strategy?

Matt Riedemann mriedem at linux.vnet.ibm.com
Tue Jul 12 13:25:06 UTC 2016


On 7/12/2016 6:29 AM, Ivan Kolodyazhny wrote:
> Hi team,
>
> Do we have any decision on this issue? I've found few patches but both
> of them are -1'ed.
>
> From Cinder perspective, it blocks us to release new os-brick with
> features, which are needed for other projects like Cinder and
> python-brick-cinderclient-ext.
>
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
>
> On Wed, Jun 22, 2016 at 5:47 PM, Matt Riedemann
> <mriedem at linux.vnet.ibm.com <mailto:mriedem at linux.vnet.ibm.com>> wrote:
>
>     On 6/21/2016 10:12 PM, Angus Lees wrote:
>
>         On Wed, 22 Jun 2016 at 05:59 Matt Riedemann
>         <mriedem at linux.vnet.ibm.com <mailto:mriedem at linux.vnet.ibm.com>
>         <mailto:mriedem at linux.vnet.ibm.com
>         <mailto:mriedem at linux.vnet.ibm.com>>> wrote:
>
>             Angus, what should we be looking at from the privsep side
>         for debugging
>             this?
>
>
>         The line above the screen-n-cpu.txt.gz failure you linked to is:
>         2016-06-21 16:21:30.994
>         <http://logs.openstack.org/85/331885/2/check/gate-grenade-dsvm-multinode/415e1bc/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2016-06-21_16_21_30_994>1840
>         WARNING oslo.privsep.daemon [-] privsep log:
>         /usr/local/bin/nova-rootwrap: Unauthorized command: privsep-helper
>         --config-file /etc/nova/nova.conf --privsep_context
>         os_brick.privileged.default --privsep_sock_path
>         /tmp/tmpV5w2VC/privsep.sock (no filter matched)
>
>          .. so nova-rootwrap is rejecting the privsep-helper command line
>         because no filter matched.  This indicates the nova
>         compute.filters file
>         has not been updated, or is incorrect.
>
>
>         As was later pointed out by mtreinish, grenade is attempting to
>         run the
>         newton code against mitaka configs, and this includes using mitaka
>         rootwrap filters.   Unfortunately, the change to add privsep to
>         nova's
>         rootwrap filters wasn't approved until the newton cycle (so that
>         all the
>         os-brick privsep-related changes could be approved together), and so
>         this doesn't Just Work.
>
>         Digging in further, it appears that there *is* a mechanism in
>         grenade to
>         upgrade rootwrap filters between major releases, but this needs
>         to be
>         explicitly updated for each project+release and hasn't been for
>         nova+mitaka->newton.  I'm not sure how this is *meant* to work,
>         since
>         the grenade "theory of upgrade" doesn't mention when configs
>         should be
>         updated - the only mechanism provided is an "exception ... used
>         sparingly."
>
>
>     As noted in the review, my understanding of the config changes is
>     deprecation of options across release boundaries so that you can't
>     drop a config option that would break someone from release to
>     release without it being deprecated first. So deprecate option foo
>     in mitaka, people upgrading from liberty to mitaka aren't broken,
>     but they get warnings in mitaka so that when you drop the option in
>     newton it's not a surprise and consumers should have adjusted during
>     mitaka.
>
>     For rootwrap filters I agree this is more complicated.
>
>
>         Anyway, I added an upgrade step for nova mitaka->newton that updates
>         rootwrap filters appropriately(*).  Again, I'm not sure what this
>         communicates to deployers compared to cinder (which *did* have the
>         updated rootwrap filter merged in mitaka, but of course that update
>         still needs to be installed at some point).
>         (*) https://review.openstack.org/#/c/332610
>
>          - Gus
>
>
>         __________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>     Alternatively Walter had a potential workaround to fallback to
>     rootwrap for os-brick:
>
>     https://review.openstack.org/#/c/329586/
>
>     So we could maybe use that for newton. But os-vif doesn't have
>     anything like that, so we'd have to see what kind of (immediately
>     deprecated) workaround could happen for os-vif in newton and then
>     drop that in ocata.
>
>     I'm told danpb is out until tomorrow though so we'll probably need
>     to wait to talk to him about options there.
>
>
>     --
>
>     Thanks,
>
>     Matt Riedemann
>
>
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>

We probably aren't doing anything while Sean Dague is on vacation. He's 
back next week and we have the nova/cinder meetups, so I'm planning on 
talking about the grenade issue in person and hopefully we'll have a 
plan by the end of next week to move forward.

-- 

Thanks,

Matt Riedemann




More information about the OpenStack-dev mailing list