Open Stack

> On Aug 3, 2015, at 21:14, Davanum Srinivas <davanum at gmail.com> wrote:
> 
> agree. "Native HA solution" was already ruled out in several email
> threads by keystone cores already (if i remember right). This is a
> devops issue and should be handled as such was the feedback.
> 

Correct. This is generally considered a devops issue. CMS handles this type of configuration today extremely well compared to most in-keystone solutions. Enhancements to keystone are welcome to be proposed as long as they are keeping the devops direction as the core way to manage these keys. Just like certs and PKI, placing devops focus here allows the organization to adhere to their requirements for sensitive/cryptographic keys/data. 

--Morgan


> Thanks,
> -- dims
> 
> On Mon, Aug 3, 2015 at 7:03 AM, Sergii Golovatiuk
> <sgolovatiuk at mirantis.com> wrote:
>> Hi,
>> 
>> --
>> Best regards,
>> Sergii Golovatiuk,
>> Skype #golserge
>> IRC #holser
>> 
>>> On Mon, Aug 3, 2015 at 12:44 PM, Adam Heczko <aheczko at mirantis.com> wrote:
>>> 
>>> Hi folks, we are discussing operations on sensitive data.
>>> May I ask you what security controls Pacemaker provides?
>> 
>> 
>> Pacemaker doesn't exchange any security information.
>> 
>>> 
>>> How we could audit its operations and data it is accessing?
>> 
>> 
>> Just audit all OCF scripts as they may contain some bits for storing
>> security data on CIB. If they store any data, then this data is exchanged
>> across all pacemaker nodes.
>> 
>>> 
>>> The same question arises when discussing native Keystone solution.
>>> From the security perspective, reduction of attack surface would be
>>> beneficial.
>>> IMO Keystone native solution would be the best possible, unless even today
>>> Pacemaker is accessing Keystone sensitive data (not sure about it).
>>> Bogdan, could you clarify this a bit?
>> 
>> 
>> Native HA solution is very costy which may require a lot of engineering
>> resource to make keystone ready with HA patterns (consensus algorithms,
>> network issues, split brain)
>> 
>>> 
>>> 
>>> Regards,
>>> 
>>> Adam
>>> 
>>> 
>>> On Mon, Aug 3, 2015 at 12:02 PM, Sergii Golovatiuk
>>> <sgolovatiuk at mirantis.com> wrote:
>>>> 
>>>> Hi,
>>>> 
>>>> I agree with Bogdan that key rotation procedure should be part of HA
>>>> solution. If you make a simple script then this script will be a single
>>>> point of failure. It requires operator's attention so it may lead to human
>>>> errors also. Adding monitoring around it or expiration time is not a
>>>> solution either.
>>>> 
>>>> There are couple of approaches how to make 'key rotation' HA ready.
>>>> 
>>>> 1. Make it as part of pacemaker OCF script. In this case pacemaker will
>>>> select the node which will be Master. It will be responsible for key
>>>> generations. In this case OCF script should have logic how to distribute
>>>> keys. It may be puppet or some rsync wrappers like lsyncd or special
>>>> function in OCF script itself. In this case when master is dead, pacemaker
>>>> will elect a new master while old one is down.
>>>> 
>>>> 2. Make keystone HA ready by itself. In this case, all logic of
>>>> distributed system should be covered in keystone. keystone should be able to
>>>> detect peers, it should have some consensus algorithms (PAXOS, RAFT, ZAB).
>>>> Using this algorithm master should be elected. Master should generate keys
>>>> and distribute them somehow to all other peers. Key distribution may be done
>>>> via rsync or using memcache/db as centralized storage for keys. Master may
>>>> send a event to all peers or peers may check memcache/db periodically.
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> --
>>>> Best regards,
>>>> Sergii Golovatiuk,
>>>> Skype #golserge
>>>> IRC #holser
>>>> 
>>>> On Mon, Aug 3, 2015 at 2:37 AM, David Medberry <openstack at medberry.net>
>>>> wrote:
>>>>> 
>>>>> Glad to see you weighed in on this. -d
>>>>> 
>>>>> On Sat, Aug 1, 2015 at 3:50 PM, Matt Fischer <matt at mattfischer.com>
>>>>> wrote:
>>>>>> 
>>>>>> Agree that you guys are way over thinking this. You don't need to
>>>>>> rotate keys at exactly the same time, we do it in within a one or two hours
>>>>>> typically based on how our regions are setup. We do it with puppet, puppet
>>>>>> runs on one keystone node at a time and drops the keys into place. The
>>>>>> actual rotation and generation we handle with a script that then proposes
>>>>>> the new key structure as a review which is then approved and deployed via
>>>>>> the normal process. For this process I always drop keys 0, 1, 2 into place,
>>>>>> I'm not bumping the numbers like the normal rotations do.
>>>>>> 
>>>>>> We had also considered ansible which would be perfect for this, but
>>>>>> that makes our ability to setup throw away environments with a single click
>>>>>> a bit more complicated. If you don't have that requirement, a simple ansible
>>>>>> script is what you should do.
>>>>>> 
>>>>>> 
>>>>>>> On Sat, Aug 1, 2015 at 3:41 PM, Clint Byrum <clint at fewbar.com> wrote:
>>>>>>> 
>>>>>>> Excerpts from Boris Bobrov's message of 2015-08-01 14:18:21 -0700:
>>>>>>>>> On Saturday 01 August 2015 16:27:17 bdobrelia at mirantis.com wrote:
>>>>>>>>> I suggest to use pacemaker multistate clone resource to rotate and
>>>>>>>> rsync
>>>>>>>>> fernet tokens from local directories across cluster nodes. The
>>>>>>>>> resource
>>>>>>>>> prototype is described here
>>>>>>>>> https://etherpad.openstack.org/p/fernet_tokens_pacemaker> Pros:
>>>>>>>> Pacemaker
>>>>>>>>> will care about CAP/split-brain stuff for us, we just design
>>>>>>>>> rotate and
>>>>>>>>> rsync logic. Also no shared FS/DB involved but only Corosync CIB -
>>>>>>>>> to
>>>>>>>> store
>>>>>>>>> few internal resource state related params, not tokens. Cons:
>>>>>>>>> Keystone
>>>>>>>>> nodes hosting fernet tokens directories must be members of
>>>>>>>>> pacemaker
>>>>>>>>> cluster. Also custom OCF script should be created to implement
>>>>>>>>> this. __
>>>>>>>>> Regards,
>>>>>>>>> Bogdan Dobrelya.
>>>>>>>>> IRC: bogdando
>>>>>>>> 
>>>>>>>> Looks complex.
>>>>>>>> 
>>>>>>>> I suggest this kind of bash or python script, running on Fuel master
>>>>>>>> node:
>>>>>>>> 
>>>>>>>> 0. Check that all controllers are online;
>>>>>>>> 1. Go to one of the controllers, rotate keys there;
>>>>>>>> 2. Fetch key 0 from there;
>>>>>>>> 3. For each other controller rotate keys there and put the 0-key
>>>>>>>> instead of
>>>>>>>> their new 0-key.
>>>>>>>> 4. If any of the nodes fail to get new keys (because they went
>>>>>>>> offline or for
>>>>>>>> some other reason) revert the rotate (move the key with the biggest
>>>>>>>> index
>>>>>>>> back to 0).
>>>>>>>> 
>>>>>>>> The script can be launched by cron or by button in Fuel.
>>>>>>>> 
>>>>>>>> I don't see anything critically bad if one rotation/sync event
>>>>>>>> fails.
>>>>>>> 
>>>>>>> This too is overly complex and will cause failures. If you replace key
>>>>>>> 0,
>>>>>>> you will stop validating tokens that were encrypted with the old key
>>>>>>> 0.
>>>>>>> 
>>>>>>> You simply need to run rotate on one, and then rsync that key
>>>>>>> repository
>>>>>>> to all of the others. You _must not_ run rotate again until you rsync
>>>>>>> to
>>>>>>> all of the others, since the key 0 from one rotation becomes the
>>>>>>> primary
>>>>>>> token encrypting key going forward, so you need it to get pushed out
>>>>>>> to
>>>>>>> all nodes as 0 first.
>>>>>>> 
>>>>>>> Don't over think it. Just read http://lbragstad.com/?p=133 and it will
>>>>>>> remain simple.
>>>>>>> 
>>>>>>> 
>>>>>>> __________________________________________________________________________
>>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>>> Unsubscribe:
>>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> __________________________________________________________________________
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>> 
>>>>> 
>>>>> 
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>> 
>>>> 
>>>> 
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> 
>>> 
>>> 
>>> --
>>> Adam Heczko
>>> Security Engineer @ Mirantis Inc.
>>> 
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>> 
>> 
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
> 
> 
> 
> -- 
> Davanum Srinivas :: https://twitter.com/dims
> 
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev