Open Stack

Excerpts from Boris Bobrov's message of 2015-08-01 14:18:21 -0700:
> On Saturday 01 August 2015 16:27:17 bdobrelia at mirantis.com wrote:
> > I suggest to use pacemaker multistate clone resource to rotate and 
> rsync
> > fernet tokens from local directories across cluster nodes. The resource
> > prototype is described here
> > https://etherpad.openstack.org/p/fernet_tokens_pacemaker> Pros: 
> Pacemaker
> > will care about CAP/split-brain stuff for us, we just design rotate and
> > rsync logic. Also no shared FS/DB involved but only Corosync CIB - to 
> store
> > few internal resource state related params, not tokens. Cons: Keystone
> > nodes hosting fernet tokens directories must be members of pacemaker
> > cluster. Also custom OCF script should be created to implement this. __
> > Regards,
> > Bogdan Dobrelya.
> > IRC: bogdando
> 
> Looks complex.
> 
> I suggest this kind of bash or python script, running on Fuel master node:
> 
> 0. Check that all controllers are online;
> 1. Go to one of the controllers, rotate keys there;
> 2. Fetch key 0 from there;
> 3. For each other controller rotate keys there and put the 0-key instead of 
> their new 0-key.
> 4. If any of the nodes fail to get new keys (because they went offline or for 
> some other reason) revert the rotate (move the key with the biggest index 
> back to 0).
> 
> The script can be launched by cron or by button in Fuel.
> 
> I don't see anything critically bad if one rotation/sync event fails.
> 

This too is overly complex and will cause failures. If you replace key 0,
you will stop validating tokens that were encrypted with the old key 0.

You simply need to run rotate on one, and then rsync that key repository
to all of the others. You _must not_ run rotate again until you rsync to
all of the others, since the key 0 from one rotation becomes the primary
token encrypting key going forward, so you need it to get pushed out to
all nodes as 0 first.

Don't over think it. Just read http://lbragstad.com/?p=133 and it will
remain simple.