[openstack-dev] [cinder] [nova] os-brick privsep failures and an upgrade strategy?
Matt Riedemann
mriedem at linux.vnet.ibm.com
Tue Jul 12 13:25:06 UTC 2016
On 7/12/2016 6:29 AM, Ivan Kolodyazhny wrote:
> Hi team,
>
> Do we have any decision on this issue? I've found few patches but both
> of them are -1'ed.
>
> From Cinder perspective, it blocks us to release new os-brick with
> features, which are needed for other projects like Cinder and
> python-brick-cinderclient-ext.
>
> Regards,
> Ivan Kolodyazhny,
> http://blog.e0ne.info/
>
> On Wed, Jun 22, 2016 at 5:47 PM, Matt Riedemann
> <mriedem at linux.vnet.ibm.com <mailto:mriedem at linux.vnet.ibm.com>> wrote:
>
> On 6/21/2016 10:12 PM, Angus Lees wrote:
>
> On Wed, 22 Jun 2016 at 05:59 Matt Riedemann
> <mriedem at linux.vnet.ibm.com <mailto:mriedem at linux.vnet.ibm.com>
> <mailto:mriedem at linux.vnet.ibm.com
> <mailto:mriedem at linux.vnet.ibm.com>>> wrote:
>
> Angus, what should we be looking at from the privsep side
> for debugging
> this?
>
>
> The line above the screen-n-cpu.txt.gz failure you linked to is:
> 2016-06-21 16:21:30.994
> <http://logs.openstack.org/85/331885/2/check/gate-grenade-dsvm-multinode/415e1bc/logs/new/screen-n-cpu.txt.gz?level=TRACE#_2016-06-21_16_21_30_994>1840
> WARNING oslo.privsep.daemon [-] privsep log:
> /usr/local/bin/nova-rootwrap: Unauthorized command: privsep-helper
> --config-file /etc/nova/nova.conf --privsep_context
> os_brick.privileged.default --privsep_sock_path
> /tmp/tmpV5w2VC/privsep.sock (no filter matched)
>
> .. so nova-rootwrap is rejecting the privsep-helper command line
> because no filter matched. This indicates the nova
> compute.filters file
> has not been updated, or is incorrect.
>
>
> As was later pointed out by mtreinish, grenade is attempting to
> run the
> newton code against mitaka configs, and this includes using mitaka
> rootwrap filters. Unfortunately, the change to add privsep to
> nova's
> rootwrap filters wasn't approved until the newton cycle (so that
> all the
> os-brick privsep-related changes could be approved together), and so
> this doesn't Just Work.
>
> Digging in further, it appears that there *is* a mechanism in
> grenade to
> upgrade rootwrap filters between major releases, but this needs
> to be
> explicitly updated for each project+release and hasn't been for
> nova+mitaka->newton. I'm not sure how this is *meant* to work,
> since
> the grenade "theory of upgrade" doesn't mention when configs
> should be
> updated - the only mechanism provided is an "exception ... used
> sparingly."
>
>
> As noted in the review, my understanding of the config changes is
> deprecation of options across release boundaries so that you can't
> drop a config option that would break someone from release to
> release without it being deprecated first. So deprecate option foo
> in mitaka, people upgrading from liberty to mitaka aren't broken,
> but they get warnings in mitaka so that when you drop the option in
> newton it's not a surprise and consumers should have adjusted during
> mitaka.
>
> For rootwrap filters I agree this is more complicated.
>
>
> Anyway, I added an upgrade step for nova mitaka->newton that updates
> rootwrap filters appropriately(*). Again, I'm not sure what this
> communicates to deployers compared to cinder (which *did* have the
> updated rootwrap filter merged in mitaka, but of course that update
> still needs to be installed at some point).
> (*) https://review.openstack.org/#/c/332610
>
> - Gus
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
> Alternatively Walter had a potential workaround to fallback to
> rootwrap for os-brick:
>
> https://review.openstack.org/#/c/329586/
>
> So we could maybe use that for newton. But os-vif doesn't have
> anything like that, so we'd have to see what kind of (immediately
> deprecated) workaround could happen for os-vif in newton and then
> drop that in ocata.
>
> I'm told danpb is out until tomorrow though so we'll probably need
> to wait to talk to him about options there.
>
>
> --
>
> Thanks,
>
> Matt Riedemann
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe:
> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
We probably aren't doing anything while Sean Dague is on vacation. He's
back next week and we have the nova/cinder meetups, so I'm planning on
talking about the grenade issue in person and hopefully we'll have a
plan by the end of next week to move forward.
--
Thanks,
Matt Riedemann
More information about the OpenStack-dev
mailing list