Open Stack

On Fri, Jun 9, 2017 at 9:57 AM, Mike Bayer <mbayer at redhat.com> wrote:

>
>
> On 06/08/2017 01:34 PM, Lance Bragstad wrote:
>
>> After digging into etcd a bit, one place this might be help deployer
>> experience would be the handling of fernet keys for token encryption in
>> keystone. Currently, all keys used to encrypt and decrypt tokens are kept
>> on disk for each keystone node in the deployment. While simple, it requires
>> operators to perform rotation on a single node and then push, or sync, the
>> new key set to the rest of the nodes. This must be done in lock step in
>> order to prevent early token invalidation and inconsistent token responses.
>>
>> An alternative would be to keep the keys in etcd and make the fernet bits
>> pluggable so that it's possible to read keys from disk or etcd (pending
>> configuration). The advantage would be that operators could initiate key
>> rotations from any keystone node in the deployment (or using etcd directly)
>> and not have to worry about distributing the new key set. Since etcd
>> associates metadata to the key-value pairs, we might be able to simplify
>> the rotation strategy as well.
>>
>
> Interesting, I had the mis-conception that "fernet" keys no longer
> required any server-side storage (how is "kept-on-disk" now implemented?) .


Currently - the keys used to encrypt and decrypt fernet tokens are stored
as files on the keystone server. The repositories default location is in
`/etc/keystone/fernet-keys`. The size of this repository is regulated by
the rotation process we provide in keystone-manage tooling [0].


> We've had continuous issues with the pre-fernet Keystone tokens filling up
> databases, even when operators were correctly expunging old tokens; some
> environments just did so many requests that the keystone-token table still
> blew up to where MySQL can no longer delete from it without producing a
> too-large transaction for Galera.
>

Yep - we actually just fixed a bug related to this [1].


>
> So after all the "finally fernet solves this problem" we propose, hey lets
> put them *back* in the database :).  That's great.  But, lets please not
> leave "cleaning out old tokens" as some kind of cron/worry-about-it-later
> thing.  that was a terrible architectural decision, with apologies to
> whoever made it.    if you're putting some kind of "we create an infinite,
> rapidly growing, turns-to-garbage-in-30-seconds" kind of data in a
> database, removing that data robustly and ASAP needs to be part of the
> process.
>
>
I should have clarified. The idea was to put the keys used to encrypt and
decrypt the tokens in etcd so that synchronizing the repository across a
cluster for keystone nodes is easier for operators (but not without other
operator pain as Kevin pointed out). The tokens themselves will remain
completely non-persistent. Fernet key creation is explicitly controlled by
operators and isn't something that end users generate.

[0]
https://github.com/openstack/keystone/blob/c528539879e824b8e6d5654292a85ccbee6dcf89/keystone/conf/fernet_tokens.py#L44-L54
[1] https://launchpad.net/bugs/1649616


>
>
>
>
>
>> On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <mbayer at redhat.com <mailto:
>> mbayer at redhat.com>> wrote:
>>
>>
>>
>>     On 06/08/2017 12:47 AM, Joshua Harlow wrote:
>>
>>         So just out of curiosity, but do people really even know what
>>         etcd is good for? I am thinking that there should be some
>>         guidance from folks in the community as to where etcd should be
>>         used and where it shouldn't (otherwise we just all end up in a
>>         mess).
>>
>>
>>     So far I've seen a proposal of etcd3 as a replacement for memcached
>>     in keystone, and a new dogpile connector was added to oslo.cache to
>>     handle referring to etcd3 as a cache backend.  This is a really
>>     simplistic / minimal kind of use case for a key-store.
>>
>>     But, keeping in mind I don't know anything about etcd3 other than
>>     "it's another key-store", it's the only database used by Kubernetes
>>     as a whole, which suggests it's doing a better job than Redis in
>>     terms of "durable".   So I wouldn't be surprised if new / existing
>>     openstack applications express some gravitational pull towards using
>>     it as their own datastore as well.    I'll be trying to hang onto
>>     the etcd3 track as much as possible so that if/when that happens I
>>     still have a job :).
>>
>>
>>
>>
>>
>>         Perhaps a good idea to actually give examples of how it should
>>         be used, how it shouldn't be used, what it offers, what it
>>         doesn't... Or at least provide links for people to read up on
>> this.
>>
>>         Thoughts?
>>
>>         Davanum Srinivas wrote:
>>
>>             One clarification: Since
>>             https://pypi.python.org/pypi/etcd3gw
>>             <https://pypi.python.org/pypi/etcd3gw> just
>>             uses the HTTP API (/v3alpha) it will work under both
>>             eventlet and
>>             non-eventlet environments.
>>
>>             Thanks,
>>             Dims
>>
>>
>>             On Wed, Jun 7, 2017 at 6:47 AM, Davanum
>>             Srinivas<davanum at gmail.com <mailto:davanum at gmail.com>>
>> wrote:
>>
>>                 Team,
>>
>>                 Here's the update to the base services resolution from
>>                 the TC:
>>                 https://governance.openstack.o
>> rg/tc/reference/base-services.html
>>                 <https://governance.openstack.
>> org/tc/reference/base-services.html>
>>
>>                 First request is to Distros, Packagers, Deployers,
>>                 anyone who
>>                 installs/configures OpenStack:
>>                 Please make sure you have latest etcd 3.x available in
>> your
>>                 environment for Services to use, Fedora already does, we
>>                 need help in
>>                 making sure all distros and architectures are covered.
>>
>>                 Any project who want to use etcd v3 API via grpc, please
>>                 use:
>>                 https://pypi.python.org/pypi/etcd3
>>                 <https://pypi.python.org/pypi/etcd3> (works only for
>>                 non-eventlet services)
>>
>>                 Those that depend on eventlet, please use the etcd3
>>                 v3alpha HTTP API using:
>>                 https://pypi.python.org/pypi/etcd3gw
>>                 <https://pypi.python.org/pypi/etcd3gw>
>>
>>                 If you use tooz, there are 2 driver choices for you:
>>                 https://github.com/openstack/t
>> ooz/blob/master/setup.cfg#L29
>>                 <https://github.com/openstack/
>> tooz/blob/master/setup.cfg#L29>
>>                 https://github.com/openstack/t
>> ooz/blob/master/setup.cfg#L30
>>                 <https://github.com/openstack/
>> tooz/blob/master/setup.cfg#L30>
>>
>>                 If you use oslo.cache, there is a driver for you:
>>                 https://github.com/openstack/o
>> slo.cache/blob/master/setup.cfg#L33
>>                 <https://github.com/openstack/
>> oslo.cache/blob/master/setup.cfg#L33>
>>
>>                 Devstack installs etcd3 by default and points cinder to
>> it:
>>                 http://git.openstack.org/cgit/
>> openstack-dev/devstack/tree/lib/etcd3
>>                 <http://git.openstack.org/cgit
>> /openstack-dev/devstack/tree/lib/etcd3>
>>                 http://git.openstack.org/cgit/
>> openstack-dev/devstack/tree/lib/cinder#n356
>>                 <http://git.openstack.org/cgit
>> /openstack-dev/devstack/tree/lib/cinder#n356>
>>
>>
>>                 Review in progress for keystone to use etcd3 for caching:
>>                 https://review.openstack.org/#/c/469621/
>>                 <https://review.openstack.org/#/c/469621/>
>>
>>                 Doug is working on proposal(s) for oslo.config to store
>> some
>>                 configuration in etcd3:
>>                 https://review.openstack.org/#/c/454897/
>>                 <https://review.openstack.org/#/c/454897/>
>>
>>                 So, feel free to turn on / test with etcd3 and report
>>                 issues.
>>
>>                 Thanks,
>>                 Dims
>>
>>                 --                 Davanum Srinivas ::
>> https://twitter.com/dims
>>
>>
>>
>>
>>
>>         ____________________________________________________________
>> ______________
>>         OpenStack Development Mailing List (not for usage questions)
>>         Unsubscribe:
>>         OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>         <http://OpenStack-dev-request@lists.openstack.org?subject:un
>> subscribe>
>>         http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>     ____________________________________________________________
>> ______________
>>     OpenStack Development Mailing List (not for usage questions)
>>     Unsubscribe:
>>     OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>     <http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe
>> >
>>     http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>     <http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev>
>>
>>
>>
>>
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20170609/84640565/attachment.html>