Open Stack

Glad to see you weighed in on this. -d

On Sat, Aug 1, 2015 at 3:50 PM, Matt Fischer <matt at mattfischer.com> wrote:

> Agree that you guys are way over thinking this. You don't need to rotate
> keys at exactly the same time, we do it in within a one or two hours
> typically based on how our regions are setup. We do it with puppet, puppet
> runs on one keystone node at a time and drops the keys into place. The
> actual rotation and generation we handle with a script that then proposes
> the new key structure as a review which is then approved and deployed via
> the normal process. For this process I always drop keys 0, 1, 2 into place,
> I'm not bumping the numbers like the normal rotations do.
>
> We had also considered ansible which would be perfect for this, but that
> makes our ability to setup throw away environments with a single click a
> bit more complicated. If you don't have that requirement, a simple ansible
> script is what you should do.
>
>
> On Sat, Aug 1, 2015 at 3:41 PM, Clint Byrum <clint at fewbar.com> wrote:
>
>> Excerpts from Boris Bobrov's message of 2015-08-01 14:18:21 -0700:
>> > On Saturday 01 August 2015 16:27:17 bdobrelia at mirantis.com wrote:
>> > > I suggest to use pacemaker multistate clone resource to rotate and
>> > rsync
>> > > fernet tokens from local directories across cluster nodes. The
>> resource
>> > > prototype is described here
>> > > https://etherpad.openstack.org/p/fernet_tokens_pacemaker> Pros:
>> > Pacemaker
>> > > will care about CAP/split-brain stuff for us, we just design rotate
>> and
>> > > rsync logic. Also no shared FS/DB involved but only Corosync CIB - to
>> > store
>> > > few internal resource state related params, not tokens. Cons: Keystone
>> > > nodes hosting fernet tokens directories must be members of pacemaker
>> > > cluster. Also custom OCF script should be created to implement this.
>> __
>> > > Regards,
>> > > Bogdan Dobrelya.
>> > > IRC: bogdando
>> >
>> > Looks complex.
>> >
>> > I suggest this kind of bash or python script, running on Fuel master
>> node:
>> >
>> > 0. Check that all controllers are online;
>> > 1. Go to one of the controllers, rotate keys there;
>> > 2. Fetch key 0 from there;
>> > 3. For each other controller rotate keys there and put the 0-key
>> instead of
>> > their new 0-key.
>> > 4. If any of the nodes fail to get new keys (because they went offline
>> or for
>> > some other reason) revert the rotate (move the key with the biggest
>> index
>> > back to 0).
>> >
>> > The script can be launched by cron or by button in Fuel.
>> >
>> > I don't see anything critically bad if one rotation/sync event fails.
>> >
>>
>> This too is overly complex and will cause failures. If you replace key 0,
>> you will stop validating tokens that were encrypted with the old key 0.
>>
>> You simply need to run rotate on one, and then rsync that key repository
>> to all of the others. You _must not_ run rotate again until you rsync to
>> all of the others, since the key 0 from one rotation becomes the primary
>> token encrypting key going forward, so you need it to get pushed out to
>> all nodes as 0 first.
>>
>> Don't over think it. Just read http://lbragstad.com/?p=133 and it will
>> remain simple.
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe:
>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20150802/25caccd1/attachment.html>