Open Stack

On Mon, Jul 27, 2015 at 2:03 PM, Clint Byrum <clint at fewbar.com> wrote:

> Excerpts from Dolph Mathews's message of 2015-07-27 11:48:12 -0700:
> > On Mon, Jul 27, 2015 at 1:31 PM, Clint Byrum <clint at fewbar.com> wrote:
> >
> > > Excerpts from Alexander Makarov's message of 2015-07-27 10:01:34 -0700:
> > > > Greetings!
> > > >
> > > > I'd like to discuss pro's and contra's of having Fernet encryption
> keys
> > > > stored in a database backend.
> > > > The idea itself emerged during discussion about synchronizing rotated
> > > keys
> > > > in HA environment.
> > > > Now Fernet keys are stored in the filesystem that has some
> availability
> > > > issues in unstable cluster.
> > > > OTOH, making SQL highly available is considered easier than that for
> a
> > > > filesystem.
> > > >
> > >
> > > I don't think HA is the root of the problem here. The problem is
> > > synchronization. If I have 3 keystone servers (n+1), and I rotate keys
> on
> > > them, I must very carefully restart them all at the exact right time to
> > > make sure one of them doesn't issue a token which will not be validated
> > > on another. This is quite a real possibility because the validation
> > > will not come from the user, but from the service, so it's not like we
> > > can use simple persistence rules. One would need a layer 7 capable load
> > > balancer that can find the token ID and make sure it goes back to the
> > > server that issued it.
> > >
> >
> > This is not true (or if it is, I'd love see a bug report).
> keystone-manage
> > fernet_rotate uses a three phase rotation strategy (staged -> primary ->
> > secondary) that allows you to distribute a staged key (used only for
> token
> > validation) throughout your cluster before it becomes a primary key (used
> > for token creation and validation) anywhere. Secondary keys are only used
> > for token validation.
> >
> > All you have to do is atomically replace the fernet key directory with a
> > new key set.
> >
> > You also don't have to restart keystone for it to pickup new keys dropped
> > onto the filesystem beneath it.
> >
>
> That's great news! Is this documented anywhere? I dug through the
> operators guides, security guide, install guide, etc. Nothing described
> this dance, which is impressive and should be written down!
>

(BTW, your original assumption would normally have been an accurate one!)

I don't believe it's documented in any of those places, yet. The best
explanation of the three phases in tree I'm aware of is probably this
(which isn't particularly accessible..):


https://github.com/openstack/keystone/blob/6a6fcc2/keystone/cmd/cli.py#L208-L223

Lance Bragstad and I also gave a small presentation at the Vancouver summit
on the behavior and he mentions the same on one of his blog posts:

  https://www.youtube.com/watch?v=duRBlm9RtCw&feature=youtu.be
  http://lbragstad.com/?p=133


> I even tried to discern how it worked from the code but it actually
> looks like it does not work the way you describe on casual investigation.
>

I don't blame you! I'll work to improve the user-facing docs on the topic.


>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150727/4653c68b/attachment.html>