Open Stack

On 06/08/2017 01:34 PM, Lance Bragstad wrote:
> After digging into etcd a bit, one place this might be help deployer 
> experience would be the handling of fernet keys for token encryption in 
> keystone. Currently, all keys used to encrypt and decrypt tokens are 
> kept on disk for each keystone node in the deployment. While simple, it 
> requires operators to perform rotation on a single node and then push, 
> or sync, the new key set to the rest of the nodes. This must be done in 
> lock step in order to prevent early token invalidation and inconsistent 
> token responses.
> 
> An alternative would be to keep the keys in etcd and make the fernet 
> bits pluggable so that it's possible to read keys from disk or etcd 
> (pending configuration). The advantage would be that operators could 
> initiate key rotations from any keystone node in the deployment (or 
> using etcd directly) and not have to worry about distributing the new 
> key set. Since etcd associates metadata to the key-value pairs, we might 
> be able to simplify the rotation strategy as well.

Interesting, I had the mis-conception that "fernet" keys no longer 
required any server-side storage (how is "kept-on-disk" now 
implemented?) .  We've had continuous issues with the pre-fernet 
Keystone tokens filling up databases, even when operators were correctly 
expunging old tokens; some environments just did so many requests that 
the keystone-token table still blew up to where MySQL can no longer 
delete from it without producing a too-large transaction for Galera.

So after all the "finally fernet solves this problem" we propose, hey 
lets put them *back* in the database :).  That's great.  But, lets 
please not leave "cleaning out old tokens" as some kind of 
cron/worry-about-it-later thing.  that was a terrible architectural 
decision, with apologies to whoever made it.    if you're putting some 
kind of "we create an infinite, rapidly growing, 
turns-to-garbage-in-30-seconds" kind of data in a database, removing 
that data robustly and ASAP needs to be part of the process.





> 
> On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <mbayer at redhat.com 
> <mailto:mbayer at redhat.com>> wrote:
> 
> 
> 
>     On 06/08/2017 12:47 AM, Joshua Harlow wrote:
> 
>         So just out of curiosity, but do people really even know what
>         etcd is good for? I am thinking that there should be some
>         guidance from folks in the community as to where etcd should be
>         used and where it shouldn't (otherwise we just all end up in a
>         mess).
> 
> 
>     So far I've seen a proposal of etcd3 as a replacement for memcached
>     in keystone, and a new dogpile connector was added to oslo.cache to
>     handle referring to etcd3 as a cache backend.  This is a really
>     simplistic / minimal kind of use case for a key-store.
> 
>     But, keeping in mind I don't know anything about etcd3 other than
>     "it's another key-store", it's the only database used by Kubernetes
>     as a whole, which suggests it's doing a better job than Redis in
>     terms of "durable".   So I wouldn't be surprised if new / existing
>     openstack applications express some gravitational pull towards using
>     it as their own datastore as well.    I'll be trying to hang onto
>     the etcd3 track as much as possible so that if/when that happens I
>     still have a job :).
> 
> 
> 
> 
> 
>         Perhaps a good idea to actually give examples of how it should
>         be used, how it shouldn't be used, what it offers, what it
>         doesn't... Or at least provide links for people to read up on this.
> 
>         Thoughts?
> 
>         Davanum Srinivas wrote:
> 
>             One clarification: Since
>             https://pypi.python.org/pypi/etcd3gw
>             <https://pypi.python.org/pypi/etcd3gw> just
>             uses the HTTP API (/v3alpha) it will work under both
>             eventlet and
>             non-eventlet environments.
> 
>             Thanks,
>             Dims
> 
> 
>             On Wed, Jun 7, 2017 at 6:47 AM, Davanum
>             Srinivas<davanum at gmail.com <mailto:davanum at gmail.com>>  wrote:
> 
>                 Team,
> 
>                 Here's the update to the base services resolution from
>                 the TC:
>                 https://governance.openstack.org/tc/reference/base-services.html
>                 <https://governance.openstack.org/tc/reference/base-services.html>
> 
>                 First request is to Distros, Packagers, Deployers,
>                 anyone who
>                 installs/configures OpenStack:
>                 Please make sure you have latest etcd 3.x available in your
>                 environment for Services to use, Fedora already does, we
>                 need help in
>                 making sure all distros and architectures are covered.
> 
>                 Any project who want to use etcd v3 API via grpc, please
>                 use:
>                 https://pypi.python.org/pypi/etcd3
>                 <https://pypi.python.org/pypi/etcd3> (works only for
>                 non-eventlet services)
> 
>                 Those that depend on eventlet, please use the etcd3
>                 v3alpha HTTP API using:
>                 https://pypi.python.org/pypi/etcd3gw
>                 <https://pypi.python.org/pypi/etcd3gw>
> 
>                 If you use tooz, there are 2 driver choices for you:
>                 https://github.com/openstack/tooz/blob/master/setup.cfg#L29
>                 <https://github.com/openstack/tooz/blob/master/setup.cfg#L29>
>                 https://github.com/openstack/tooz/blob/master/setup.cfg#L30
>                 <https://github.com/openstack/tooz/blob/master/setup.cfg#L30>
> 
>                 If you use oslo.cache, there is a driver for you:
>                 https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33
>                 <https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33>
> 
>                 Devstack installs etcd3 by default and points cinder to it:
>                 http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3
>                 <http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3>
>                 http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356
>                 <http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356>
> 
> 
>                 Review in progress for keystone to use etcd3 for caching:
>                 https://review.openstack.org/#/c/469621/
>                 <https://review.openstack.org/#/c/469621/>
> 
>                 Doug is working on proposal(s) for oslo.config to store some
>                 configuration in etcd3:
>                 https://review.openstack.org/#/c/454897/
>                 <https://review.openstack.org/#/c/454897/>
> 
>                 So, feel free to turn on / test with etcd3 and report
>                 issues.
> 
>                 Thanks,
>                 Dims
> 
>                 -- 
>                 Davanum Srinivas :: https://twitter.com/dims
> 
> 
> 
> 
> 
>         __________________________________________________________________________
>         OpenStack Development Mailing List (not for usage questions)
>         Unsubscribe:
>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>         <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> 
>     __________________________________________________________________________
>     OpenStack Development Mailing List (not for usage questions)
>     Unsubscribe:
>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe>
>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
> 
> 
> 
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>