<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Jun 9, 2017 at 9:57 AM, Mike Bayer <span dir="ltr"><<a href="mailto:mbayer@redhat.com" target="_blank">mbayer@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span class="gmail-"><br>
<br>
On 06/08/2017 01:34 PM, Lance Bragstad wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
After digging into etcd a bit, one place this might be help deployer experience would be the handling of fernet keys for token encryption in keystone. Currently, all keys used to encrypt and decrypt tokens are kept on disk for each keystone node in the deployment. While simple, it requires operators to perform rotation on a single node and then push, or sync, the new key set to the rest of the nodes. This must be done in lock step in order to prevent early token invalidation and inconsistent token responses.<br>
<br>
An alternative would be to keep the keys in etcd and make the fernet bits pluggable so that it's possible to read keys from disk or etcd (pending configuration). The advantage would be that operators could initiate key rotations from any keystone node in the deployment (or using etcd directly) and not have to worry about distributing the new key set. Since etcd associates metadata to the key-value pairs, we might be able to simplify the rotation strategy as well.<br>
</blockquote>
<br></span>
Interesting, I had the mis-conception that "fernet" keys no longer required any server-side storage (how is "kept-on-disk" now implemented?) . </blockquote><div><br></div><div>Currently - the keys used to encrypt and decrypt fernet tokens are stored as files on the keystone server. The repositories default location is in `/etc/keystone/fernet-keys`. The size of this repository is regulated by the rotation process we provide in keystone-manage tooling [0].</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> We've had continuous issues with the pre-fernet Keystone tokens filling up databases, even when operators were correctly expunging old tokens; some environments just did so many requests that the keystone-token table still blew up to where MySQL can no longer delete from it without producing a too-large transaction for Galera.<br></blockquote><div><br></div><div>Yep - we actually just fixed a bug related to this [1].</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
So after all the "finally fernet solves this problem" we propose, hey lets put them *back* in the database :). That's great. But, lets please not leave "cleaning out old tokens" as some kind of cron/worry-about-it-later thing. that was a terrible architectural decision, with apologies to whoever made it. if you're putting some kind of "we create an infinite, rapidly growing, turns-to-garbage-in-30-seconds<wbr>" kind of data in a database, removing that data robustly and ASAP needs to be part of the process.<br>
<br></blockquote><div><br></div><div>I should have clarified. The idea was to put the keys used to encrypt and decrypt the tokens in etcd so that synchronizing the repository across a cluster for keystone nodes is easier for operators (but not without other operator pain as Kevin pointed out). The tokens themselves will remain completely non-persistent. Fernet key creation is explicitly controlled by operators and isn't something that end users generate.</div><div><br></div><div>[0] <a href="https://github.com/openstack/keystone/blob/c528539879e824b8e6d5654292a85ccbee6dcf89/keystone/conf/fernet_tokens.py#L44-L54">https://github.com/openstack/keystone/blob/c528539879e824b8e6d5654292a85ccbee6dcf89/keystone/conf/fernet_tokens.py#L44-L54</a></div><div>[1] <a href="https://launchpad.net/bugs/1649616">https://launchpad.net/bugs/1649616</a></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div class="gmail-h5">
<br>
On Thu, Jun 8, 2017 at 11:37 AM, Mike Bayer <<a href="mailto:mbayer@redhat.com" target="_blank">mbayer@redhat.com</a> <mailto:<a href="mailto:mbayer@redhat.com" target="_blank">mbayer@redhat.com</a>>> wrote:<br>
<br>
<br>
<br>
On 06/08/2017 12:47 AM, Joshua Harlow wrote:<br>
<br>
So just out of curiosity, but do people really even know what<br>
etcd is good for? I am thinking that there should be some<br>
guidance from folks in the community as to where etcd should be<br>
used and where it shouldn't (otherwise we just all end up in a<br>
mess).<br>
<br>
<br>
So far I've seen a proposal of etcd3 as a replacement for memcached<br>
in keystone, and a new dogpile connector was added to oslo.cache to<br>
handle referring to etcd3 as a cache backend. This is a really<br>
simplistic / minimal kind of use case for a key-store.<br>
<br>
But, keeping in mind I don't know anything about etcd3 other than<br>
"it's another key-store", it's the only database used by Kubernetes<br>
as a whole, which suggests it's doing a better job than Redis in<br>
terms of "durable". So I wouldn't be surprised if new / existing<br>
openstack applications express some gravitational pull towards using<br>
it as their own datastore as well. I'll be trying to hang onto<br>
the etcd3 track as much as possible so that if/when that happens I<br>
still have a job :).<br>
<br>
<br>
<br>
<br>
<br>
Perhaps a good idea to actually give examples of how it should<br>
be used, how it shouldn't be used, what it offers, what it<br>
doesn't... Or at least provide links for people to read up on this.<br>
<br>
Thoughts?<br>
<br>
Davanum Srinivas wrote:<br>
<br>
One clarification: Since<br>
<a href="https://pypi.python.org/pypi/etcd3gw" rel="noreferrer" target="_blank">https://pypi.python.org/pypi/e<wbr>tcd3gw</a><br>
<<a href="https://pypi.python.org/pypi/etcd3gw" rel="noreferrer" target="_blank">https://pypi.python.org/pypi/<wbr>etcd3gw</a>> just<br>
uses the HTTP API (/v3alpha) it will work under both<br>
eventlet and<br>
non-eventlet environments.<br>
<br>
Thanks,<br>
Dims<br>
<br>
<br>
On Wed, Jun 7, 2017 at 6:47 AM, Davanum<br></div></div><span class="gmail-">
Srinivas<<a href="mailto:davanum@gmail.com" target="_blank">davanum@gmail.com</a> <mailto:<a href="mailto:davanum@gmail.com" target="_blank">davanum@gmail.com</a>>> wrote:<br>
<br>
Team,<br>
<br>
Here's the update to the base services resolution from<br>
the TC:<br>
<a href="https://governance.openstack.org/tc/reference/base-services.html" rel="noreferrer" target="_blank">https://governance.openstack.o<wbr>rg/tc/reference/base-services.<wbr>html</a><br>
<<a href="https://governance.openstack.org/tc/reference/base-services.html" rel="noreferrer" target="_blank">https://governance.openstack.<wbr>org/tc/reference/base-services<wbr>.html</a>><br>
<br>
First request is to Distros, Packagers, Deployers,<br>
anyone who<br>
installs/configures OpenStack:<br>
Please make sure you have latest etcd 3.x available in your<br>
environment for Services to use, Fedora already does, we<br>
need help in<br>
making sure all distros and architectures are covered.<br>
<br>
Any project who want to use etcd v3 API via grpc, please<br>
use:<br>
<a href="https://pypi.python.org/pypi/etcd3" rel="noreferrer" target="_blank">https://pypi.python.org/pypi/e<wbr>tcd3</a><br>
<<a href="https://pypi.python.org/pypi/etcd3" rel="noreferrer" target="_blank">https://pypi.python.org/pypi/<wbr>etcd3</a>> (works only for<br>
non-eventlet services)<br>
<br>
Those that depend on eventlet, please use the etcd3<br>
v3alpha HTTP API using:<br>
<a href="https://pypi.python.org/pypi/etcd3gw" rel="noreferrer" target="_blank">https://pypi.python.org/pypi/e<wbr>tcd3gw</a><br>
<<a href="https://pypi.python.org/pypi/etcd3gw" rel="noreferrer" target="_blank">https://pypi.python.org/pypi/<wbr>etcd3gw</a>><br>
<br>
If you use tooz, there are 2 driver choices for you:<br>
<a href="https://github.com/openstack/tooz/blob/master/setup.cfg#L29" rel="noreferrer" target="_blank">https://github.com/openstack/t<wbr>ooz/blob/master/setup.cfg#L29</a><br></span>
<<a href="https://github.com/openstack/tooz/blob/master/setup.cfg#L29" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>tooz/blob/master/setup.cfg#L29</a><wbr>><br>
<a href="https://github.com/openstack/tooz/blob/master/setup.cfg#L30" rel="noreferrer" target="_blank">https://github.com/openstack/t<wbr>ooz/blob/master/setup.cfg#L30</a><span class="gmail-"><br>
<<a href="https://github.com/openstack/tooz/blob/master/setup.cfg#L30" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>tooz/blob/master/setup.cfg#L30</a><wbr>><br>
<br>
If you use oslo.cache, there is a driver for you:<br>
<a href="https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33" rel="noreferrer" target="_blank">https://github.com/openstack/o<wbr>slo.cache/blob/master/setup.cf<wbr>g#L33</a><br>
<<a href="https://github.com/openstack/oslo.cache/blob/master/setup.cfg#L33" rel="noreferrer" target="_blank">https://github.com/openstack/<wbr>oslo.cache/blob/master/setup.c<wbr>fg#L33</a>><br>
<br>
Devstack installs etcd3 by default and points cinder to it:<br>
<a href="http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3" rel="noreferrer" target="_blank">http://git.openstack.org/cgit/<wbr>openstack-dev/devstack/tree/li<wbr>b/etcd3</a><br></span>
<<a href="http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/etcd3" rel="noreferrer" target="_blank">http://git.openstack.org/cgit<wbr>/openstack-dev/devstack/tree/<wbr>lib/etcd3</a>><br>
<a href="http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356" rel="noreferrer" target="_blank">http://git.openstack.org/cgit/<wbr>openstack-dev/devstack/tree/li<wbr>b/cinder#n356</a><span class="gmail-"><br>
<<a href="http://git.openstack.org/cgit/openstack-dev/devstack/tree/lib/cinder#n356" rel="noreferrer" target="_blank">http://git.openstack.org/cgit<wbr>/openstack-dev/devstack/tree/<wbr>lib/cinder#n356</a>><br>
<br>
<br>
Review in progress for keystone to use etcd3 for caching:<br>
<a href="https://review.openstack.org/#/c/469621/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/469621/</a><br>
<<a href="https://review.openstack.org/#/c/469621/" rel="noreferrer" target="_blank">https://review.openstack.org/<wbr>#/c/469621/</a>><br>
<br>
Doug is working on proposal(s) for oslo.config to store some<br>
configuration in etcd3:<br>
<a href="https://review.openstack.org/#/c/454897/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/454897/</a><br>
<<a href="https://review.openstack.org/#/c/454897/" rel="noreferrer" target="_blank">https://review.openstack.org/<wbr>#/c/454897/</a>><br>
<br>
So, feel free to turn on / test with etcd3 and report<br>
issues.<br>
<br>
Thanks,<br>
Dims<br>
<br>
-- Davanum Srinivas :: <a href="https://twitter.com/dims" rel="noreferrer" target="_blank">https://twitter.com/dims</a><br>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br></span>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@<wbr>lists.openstack.org?subject:un<wbr>subscribe</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a> <<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cg<wbr>i-bin/mailman/listinfo/opensta<wbr>ck-dev</a>><span class="gmail-"><br>
<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe:<br>
<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br></span>
<<a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">http://OpenStack-dev-request@<wbr>lists.openstack.org?subject:un<wbr>subscribe</a>><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><span class="gmail-"><br>
<<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cg<wbr>i-bin/mailman/listinfo/opensta<wbr>ck-dev</a>><br>
<br>
<br>
<br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br>
</span></blockquote><div class="gmail-HOEnZb"><div class="gmail-h5">
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</div></div></blockquote></div><br></div></div>