Open Stack

On 03/22/2016 06:27 PM, Kevin Carter wrote:
> Comments in-line.
>
> On 03/22/2016 02:46 PM, Adrian Otto wrote:
>> Team,
>>
>> This thread is a continuation of a branch of the previous High
>> Availability thread [1]. As the Magnum PTL, I’ve been aware of a number
>> of different groups who have started using Magnum in recent months. For
>> various reasons, there have been multiple requests for information about
>> how to turn off the dependency on Barbican, which we use for secure
>> storage of TLS certificates that are used to secure communications
>> between various components of the software hosted on Magnum Bay
>> resources. Examples of this are Docker Swarm, and Kubernetes, which we
>> affectionately refer to as COEs (Container Orchestration Engines). The
>> only alternative to Barbican currently offered in Magnum is a local file
>> option, which is only intended to be used for testing, as the
>> certificates are stored unencrypted on a local filesystem where the
>> conductor runs, and when you use this option, you can’t scale beyond a
>> single conductor.
>>
>> Although our whole community agrees that using Barbican is the right
>> long term solution for deployments of Magnum, we still wish to make the
>> friction of adopting Magnum to be as low as possible without completely
>> compromising all security best practices.
>
> /me wearing my deployer hat now: Many of my customers and product folks
> want Magnum but they also want magnum to be as secure and stable as
> possible. If Barbican is the best long term solution for the project it
> would make sense to me that Magnum remain on course with Barbican as the
> defacto way of deploying in production. IMHO building alternative means
> for certificate management is a distraction and will only confuse folks
> looking to deploy Magnum into production.

I'm going to agree. This reminds me of people who didn't want to run 
keystone back in the day. Those people were a distraction, and placating 
them hampered OpenStack's progress by probably several years.

>> Some ops teams are willing to
>> adopt a new service, but not two. They only want to add Magnum and not
>> Barbican.
>
> It would seem to me that once the Barbican dependency is well
> documented, which it should be at this point, Barbican is be easy to
> accept especially with the understanding of why it is needed. Many of
> the deployment projects are creating the automation needed to make the
> adoption of services simpler and I'd imagine deployment automation is
> the largest hurdle to wide spread adoption for both Barbican and Magnum.
> If the OPS team you mention does not want both services it would seem
> they can use "local file" option; this is similar to Cinder+LVM and
> Glance+file_store both of which have scale operational issues in production.

Agree.

>> We think that once those operators become familiar with
>> Magnum, adding Barbican will follow. In the mean time, we’d like to
>> offer a Barbican alternative that allows Magnum to scale beyond one
>> conductor, and allows for encrypted storage of TLC credentials needed
>> for unattended bay operations.
>
> If all of this is to simplify or provide for the developer/"someone
> kicking the tires" use case I'd imagine the "local file" storage would
> be sufficient. If the acceptance of Barbican is too much to handle or
> introduce into an active deployment (I'm not sure why that would be
> especially if they're adding Magnum), the synchronization of locally
> stored certificates across multiple hosts is manageable and can be
> handled by a very long list of other pre-existing operational means.
>
>> A blueprint [2] was recently proposed to
>> address this. We discussed this in our team meeting today [3], where we
>> used an etherpad [4] to collaborate on options that could be used as
>> alternatives besides the ones offered today. This thread is not intended
>> to answer how to make Barbican easier to adopt, but rather how to make
>> Magnum easier to adopt while keeping Barbican as the default
>> best-practice choice for certificate storage.
>
> I'd like there _NOT_ to be an "easy button" way for operators to hang
> themselves in production by following a set of "quick start
> instructions" under the guise of "easy to adopt". If Barbican is the
> best-practice lets keep it that way. If for some reason Barbican is hard
> to adopt lets identify those difficulties and get them fixed. Going down
> the path of NIH or alternative less secure solutions because someone
> (not identified here or speaking for themselves) has said they don't
> want Barbican or deploying it is hard seems like a recipe for
> fragmentation and disaster.

Agree.

>> I want to highlight that the implementation of the spec referenced by
>> Daneyon Hansen in his quoted response below was completed in the Liberty
>> release timeframe, and communication between COE components is now
>> secured using TLS. We are discussing the continued use of TLS for
>> encrypted connections between COE components, but potentially using
>> Keystone tokens for authentication between clients and COE’s rather than
>> using TLS for both encryption and authentication. Further notes on this
>> are available in the etherpad [4].
>>
>> I ask that you please review the options under consideration, note your
>> remarks in the etherpad [4], and continue discussion here as needed.
>>
>> Thanks,
>>
>> Adrian
>>
>> [1]
>> http://lists.openstack.org/pipermail/openstack-dev/2016-March/089684.html
>> [2] https://blueprints.launchpad.net/magnum/+spec/barbican-alternative-store
>> [3]
>> http://eavesdrop.openstack.org/meetings/containers/2016/containers.2016-03-22-16.01.html
>> [4] https://etherpad.openstack.org/p/magnum-barbican-alternative
>>
>>> On Mar 22, 2016, at 11:52 AM, Daneyon Hansen (danehans)
>>> <danehans at cisco.com <mailto:danehans at cisco.com>> wrote:
>>>
>>>
>>>
>>> From:Hongbin Lu <hongbin.lu at huawei.com <mailto:hongbin.lu at huawei.com>>
>>> Reply-To:"OpenStack Development Mailing List (not for usage
>>> questions)" <openstack-dev at lists.openstack.org
>>> <mailto:openstack-dev at lists.openstack.org>>
>>> Date:Monday, March 21, 2016 at 8:19 PM
>>> To:"OpenStack Development Mailing List (not for usage questions)"
>>> <openstack-dev at lists.openstack.org
>>> <mailto:openstack-dev at lists.openstack.org>>
>>> Subject:Re: [openstack-dev] [magnum] High Availability
>>>
>>>> Tim,
>>>> Thanks for your advice. I respect your point of view and we will
>>>> definitely encourage our users to try Barbican if they see fits.
>>>> However, for the sake of Magnum, I think we have to decouple from
>>>> Barbican at current stage. The coupling of Magnum and Barbican will
>>>> increase the size of the system by two (1 project -> 2 project),
>>>> which will significant increase the overall complexities.
>>>> ·For developers, it incurs significant overheads on development,
>>>> quality assurance, and maintenance.
>>>> ·For operators, it doubles the amount of efforts of deploying and
>>>> monitoring the system.
>>>> ·For users, a large system is likely to be unstable and fragile which
>>>> affects the user experience.
>>>> In my point of view, I would like to minimize the system we are going
>>>> to ship, so that we can reduce the overheads of maintenance and
>>>> provides a stable system to our users.
>>>> I noticed that there are several suggestions to “force” our users to
>>>> install Barbican, which I would respectfully disagree. Magnum is a
>>>> young project and we are struggling to increase the adoption rate. I
>>>> think we need to be nice to our users, otherwise, they will choose
>>>> our competitors (there are container service everywhere). Please
>>>> understand that we are not a mature project, like Nova, who has
>>>> thousands of users. We really don’t have the power to force our users
>>>> to do what they don’t like to do.
>>>> I also recognized there are several disagreements from the Barbican
>>>> team. Per my understanding, most of the complaints are about the
>>>> re-invention of Barbican equivalent functionality in Magnum. To
>>>> address that, I am going to propose an idea to achieve the goal
>>>> without duplicating Barbican. In particular, I suggest to add support
>>>> for additional authentication system (Keystone in particular) for our
>>>> Kubernetes bay (potentially for swarm/mesos). As a result, users can
>>>> specify how to secure their bay’s API endpoint:
>>>> ·TLS: This option requires Barbican to be installed for storing the
>>>> TLS certificates.
>>>> ·Keystone: This option doesn’t require Barbican. Users will use their
>>>> OpenStack credentials to log into Kubernetes.
>>>
>>> I believe this is a sensible option that addresses the original
>>> problem statement in [1]:
>>>
>>> "Magnum currently controls Kubernetes API services
>>> usingunauthenticated HTTP. If an attacker knows the api_address of a
>>> Kubernetes Bay, (s)he can control the cluster without any access control."
>>>
>>> The [1] problem statement is authenticating the bay API endpoint, not
>>> encrypting it. With the option you propose, we can leave the existing
>>> tls-disabled attribute alone and continue supporting encryption. Using
>>> Keystone to authenticate the Kubernetes API already exists outside of
>>> Magnum in Hypernetes [2]. We will need to investigate support for the
>>> other coe types.
>>> _
>>> _
>>> [1]
>>> https://github.com/openstack/magnum/blob/master/specs/tls-support-magnum.rst
>>> [2] http://thenewstack.io/hypernetes-brings-multi-tenancy-microservices/
>>>
>>>
>>>> I am going to send another ML to describe the details. You are
>>>> welcome to provide your inputs. Thanks.
>>>> Best regards,
>>>> Hongbin
>>>> *From:*Tim Bell [mailto:Tim.Bell at cern.ch]
>>>> *Sent:*March-19-16 5:55 AM
>>>> *To:*OpenStack Development Mailing List (not for usage questions)
>>>> *Subject:*Re: [openstack-dev] [magnum] High Availability
>>>> *From:*Hongbin Lu <hongbin.lu at huawei.com <mailto:hongbin.lu at huawei.com>>
>>>> *Reply-To:*"OpenStack Development Mailing List (not for usage
>>>> questions)" <openstack-dev at lists.openstack.org
>>>> <mailto:openstack-dev at lists.openstack.org>>
>>>> *Date:*Saturday 19 March 2016 at 04:52
>>>> *To:*"OpenStack Development Mailing List (not for usage questions)"
>>>> <openstack-dev at lists.openstack.org
>>>> <mailto:openstack-dev at lists.openstack.org>>
>>>> *Subject:*Re: [openstack-dev] [magnum] High Availability
>>>>> ...
>>>>> If you disagree, I would request you to justify why this approach
>>>>> works for Heat but not for Magnum. Also, I also wonder if Heat has a
>>>>> plan to set a hard dependency on Barbican for just protecting the
>>>>> hidden parameters.
>>>> There is a risk that we use decisions made by other projects to
>>>> justify how Magnum is implemented. Heat was created 3 years ago
>>>> according to
>>>> https://www.openstack.org/software/project-navigator/ and Barbican
>>>> only 2 years ago, thus Barbican may not have been an option (or a
>>>> high risk one).
>>>> Barbican has demonstrated that the project has corporate diversity
>>>> and good stability
>>>> (https://www.openstack.org/software/releases/liberty/components/barbican).
>>>> There are some areas that could be improved (packaging and puppet
>>>> modules are often needing some more investment).
>>>> I think it is worth a go to try it out and have concrete areas to
>>>> improve if there are problems.
>>>> Tim
>>>>> If you don’t like code duplication between Magnum and Heat, I would
>>>>> suggest to move the implementation to a oslo library to make it DRY.
>>>>> Thoughts?
>>>>> [1]https://specs.openstack.org/openstack/heat-specs/specs/juno/encrypt-hidden-parameters.html
>>>>> Best regards,
>>>>> Hongbin
>>>>> *From:*David Stanek [mailto:dstanek at dstanek.com]
>>>>> *Sent:*March-18-16 4:12 PM
>>>>> *To:*OpenStack Development Mailing List (not for usage questions)
>>>>> *Subject:*Re: [openstack-dev] [magnum] High Availability
>>>>>
>>>>> On Fri, Mar 18, 2016 at 4:03 PM Douglas Mendizábal
>>>>> <douglas.mendizabal at rackspace.com
>>>>> <mailto:douglas.mendizabal at rackspace.com>> wrote:
>>>>>>
>>>>>> [snip]
>>>>>>>
>>>>>>> Regarding the Keystone solution, I'd like to hear the Keystone team's feadback on that.  It definitely sounds to me like you're trying to put a square peg in a round hole.
>>>>>>>
>>>>>>
>>>>> I believe that using Keystone for this is a mistake. As mentioned in
>>>>> the blueprint, Keystone is not encrypting the data so magnum would
>>>>> be on the hook to do it. So that means that if security is a
>>>>> requirement you'd have to duplicate more than just code. magnum
>>>>> would start having a larger security burden. Since we have a system
>>>>> designed to securely store data I think that's the best place for
>>>>> data that needs to be secure.
>>> __________________________________________________________________________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe:OpenStack-dev-request at lists.openstack.org
>>> <mailto:OpenStack-dev-request at lists.openstack.org>?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>