Open Stack

agree. "Native HA solution" was already ruled out in several email
threads by keystone cores already (if i remember right). This is a
devops issue and should be handled as such was the feedback.

Thanks,
-- dims

On Mon, Aug 3, 2015 at 7:03 AM, Sergii Golovatiuk
<sgolovatiuk at mirantis.com> wrote:
> Hi,
>
> --
> Best regards,
> Sergii Golovatiuk,
> Skype #golserge
> IRC #holser
>
> On Mon, Aug 3, 2015 at 12:44 PM, Adam Heczko <aheczko at mirantis.com> wrote:
>>
>> Hi folks, we are discussing operations on sensitive data.
>> May I ask you what security controls Pacemaker provides?
>
>
> Pacemaker doesn't exchange any security information.
>
>>
>> How we could audit its operations and data it is accessing?
>
>
> Just audit all OCF scripts as they may contain some bits for storing
> security data on CIB. If they store any data, then this data is exchanged
> across all pacemaker nodes.
>
>>
>> The same question arises when discussing native Keystone solution.
>> From the security perspective, reduction of attack surface would be
>> beneficial.
>> IMO Keystone native solution would be the best possible, unless even today
>> Pacemaker is accessing Keystone sensitive data (not sure about it).
>> Bogdan, could you clarify this a bit?
>
>
> Native HA solution is very costy which may require a lot of engineering
> resource to make keystone ready with HA patterns (consensus algorithms,
> network issues, split brain)
>
>>
>>
>> Regards,
>>
>> Adam
>>
>>
>> On Mon, Aug 3, 2015 at 12:02 PM, Sergii Golovatiuk
>> <sgolovatiuk at mirantis.com> wrote:
>>>
>>> Hi,
>>>
>>> I agree with Bogdan that key rotation procedure should be part of HA
>>> solution. If you make a simple script then this script will be a single
>>> point of failure. It requires operator's attention so it may lead to human
>>> errors also. Adding monitoring around it or expiration time is not a
>>> solution either.
>>>
>>> There are couple of approaches how to make 'key rotation' HA ready.
>>>
>>> 1. Make it as part of pacemaker OCF script. In this case pacemaker will
>>> select the node which will be Master. It will be responsible for key
>>> generations. In this case OCF script should have logic how to distribute
>>> keys. It may be puppet or some rsync wrappers like lsyncd or special
>>> function in OCF script itself. In this case when master is dead, pacemaker
>>> will elect a new master while old one is down.
>>>
>>> 2. Make keystone HA ready by itself. In this case, all logic of
>>> distributed system should be covered in keystone. keystone should be able to
>>> detect peers, it should have some consensus algorithms (PAXOS, RAFT, ZAB).
>>> Using this algorithm master should be elected. Master should generate keys
>>> and distribute them somehow to all other peers. Key distribution may be done
>>> via rsync or using memcache/db as centralized storage for keys. Master may
>>> send a event to all peers or peers may check memcache/db periodically.
>>>
>>>
>>>
>>>
>>>
>>> --
>>> Best regards,
>>> Sergii Golovatiuk,
>>> Skype #golserge
>>> IRC #holser
>>>
>>> On Mon, Aug 3, 2015 at 2:37 AM, David Medberry <openstack at medberry.net>
>>> wrote:
>>>>
>>>> Glad to see you weighed in on this. -d
>>>>
>>>> On Sat, Aug 1, 2015 at 3:50 PM, Matt Fischer <matt at mattfischer.com>
>>>> wrote:
>>>>>
>>>>> Agree that you guys are way over thinking this. You don't need to
>>>>> rotate keys at exactly the same time, we do it in within a one or two hours
>>>>> typically based on how our regions are setup. We do it with puppet, puppet
>>>>> runs on one keystone node at a time and drops the keys into place. The
>>>>> actual rotation and generation we handle with a script that then proposes
>>>>> the new key structure as a review which is then approved and deployed via
>>>>> the normal process. For this process I always drop keys 0, 1, 2 into place,
>>>>> I'm not bumping the numbers like the normal rotations do.
>>>>>
>>>>> We had also considered ansible which would be perfect for this, but
>>>>> that makes our ability to setup throw away environments with a single click
>>>>> a bit more complicated. If you don't have that requirement, a simple ansible
>>>>> script is what you should do.
>>>>>
>>>>>
>>>>> On Sat, Aug 1, 2015 at 3:41 PM, Clint Byrum <clint at fewbar.com> wrote:
>>>>>>
>>>>>> Excerpts from Boris Bobrov's message of 2015-08-01 14:18:21 -0700:
>>>>>> > On Saturday 01 August 2015 16:27:17 bdobrelia at mirantis.com wrote:
>>>>>> > > I suggest to use pacemaker multistate clone resource to rotate and
>>>>>> > rsync
>>>>>> > > fernet tokens from local directories across cluster nodes. The
>>>>>> > > resource
>>>>>> > > prototype is described here
>>>>>> > > https://etherpad.openstack.org/p/fernet_tokens_pacemaker> Pros:
>>>>>> > Pacemaker
>>>>>> > > will care about CAP/split-brain stuff for us, we just design
>>>>>> > > rotate and
>>>>>> > > rsync logic. Also no shared FS/DB involved but only Corosync CIB -
>>>>>> > > to
>>>>>> > store
>>>>>> > > few internal resource state related params, not tokens. Cons:
>>>>>> > > Keystone
>>>>>> > > nodes hosting fernet tokens directories must be members of
>>>>>> > > pacemaker
>>>>>> > > cluster. Also custom OCF script should be created to implement
>>>>>> > > this. __
>>>>>> > > Regards,
>>>>>> > > Bogdan Dobrelya.
>>>>>> > > IRC: bogdando
>>>>>> >
>>>>>> > Looks complex.
>>>>>> >
>>>>>> > I suggest this kind of bash or python script, running on Fuel master
>>>>>> > node:
>>>>>> >
>>>>>> > 0. Check that all controllers are online;
>>>>>> > 1. Go to one of the controllers, rotate keys there;
>>>>>> > 2. Fetch key 0 from there;
>>>>>> > 3. For each other controller rotate keys there and put the 0-key
>>>>>> > instead of
>>>>>> > their new 0-key.
>>>>>> > 4. If any of the nodes fail to get new keys (because they went
>>>>>> > offline or for
>>>>>> > some other reason) revert the rotate (move the key with the biggest
>>>>>> > index
>>>>>> > back to 0).
>>>>>> >
>>>>>> > The script can be launched by cron or by button in Fuel.
>>>>>> >
>>>>>> > I don't see anything critically bad if one rotation/sync event
>>>>>> > fails.
>>>>>> >
>>>>>>
>>>>>> This too is overly complex and will cause failures. If you replace key
>>>>>> 0,
>>>>>> you will stop validating tokens that were encrypted with the old key
>>>>>> 0.
>>>>>>
>>>>>> You simply need to run rotate on one, and then rsync that key
>>>>>> repository
>>>>>> to all of the others. You _must not_ run rotate again until you rsync
>>>>>> to
>>>>>> all of the others, since the key 0 from one rotation becomes the
>>>>>> primary
>>>>>> token encrypting key going forward, so you need it to get pushed out
>>>>>> to
>>>>>> all nodes as 0 first.
>>>>>>
>>>>>> Don't over think it. Just read http://lbragstad.com/?p=133 and it will
>>>>>> remain simple.
>>>>>>
>>>>>>
>>>>>> __________________________________________________________________________
>>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>>> Unsubscribe:
>>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> __________________________________________________________________________
>>>>> OpenStack Development Mailing List (not for usage questions)
>>>>> Unsubscribe:
>>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>>
>>>>
>>>>
>>>>
>>>> __________________________________________________________________________
>>>> OpenStack Development Mailing List (not for usage questions)
>>>> Unsubscribe:
>>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>>
>>>
>>>
>>>
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:
>>> OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>>
>> --
>> Adam Heczko
>> Security Engineer @ Mirantis Inc.
>>
>> __________________________________________________________________________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>



-- 
Davanum Srinivas :: https://twitter.com/dims