[openstack-dev] [magnum] High Availability
Hongbin Lu
hongbin.lu at huawei.com
Sun Mar 20 22:33:34 UTC 2016
The Magnum team discussed Anchor several times (in the design summit/midcycle). According to what I remembered, the conclusion is to leverage Anchor though Barbican (presumably there is an Anchor backend for Barbican). Is Anchor support in Barbican still in the roadmap?
Best regards,
Hongbin
> -----Original Message-----
> From: Clark, Robert Graham [mailto:robert.clark at hpe.com]
> Sent: March-20-16 1:57 AM
> To: maishsk+openstack at maishsk.com; OpenStack Development Mailing List
> (not for usage questions)
> Subject: Re: [openstack-dev] [magnum] High Availability
>
> At the risk of muddying the waters further, I recently chatted with
> some of you about Anchor, it's an ephemeral PKI system setup to provide
> private community PKI - certificate services for internal systems, a
> lot like k8 pods.
>
> An overview of why revocation doesn't work very well in many cases and
> how ephemeral PKI helps: https://openstack-
> security.github.io/tooling/2016/01/20/ephemeral-pki.html
>
> First half of a threat analysis on Anchor, the Security Project's
> implementation of ephemeral PKI: https://openstack-
> security.github.io/threatanalysis/2016/02/07/anchorTA.html
>
> This might not solve your problem, it's certainly not a direct drop in
> for Barbican (and it never will be) but if your primary concern is
> Certificate Management for internal systems (not presenting
> certificates over the edge of the cloud) you might find some of it's
> properties valuable. Not least, it's trivial to HA being stateless and
> it's trivial to deploy being a single Pecan service.
>
> There's a reasonably complete deck on Anchor here:
> https://docs.google.com/presentation/d/1HDyEiSA5zp6HNdDZcRAYMT5GtxqkHrx
> brqDRzITuSTc/edit?usp=sharing
>
> And of course, code over here:
> http://git.openstack.org/cgit/openstack/anchor
>
> Cheers
> -Rob
>
> > -----Original Message-----
> > From: Maish Saidel-Keesing [mailto:maishsk at maishsk.com]
> > Sent: 19 March 2016 18:10
> > To: OpenStack Development Mailing List (not for usage questions)
> > Subject: Re: [openstack-dev] [magnum] High Availability
> >
> > Forgive me for the top post and also for asking the obvious (with my
> > Operator hat on)
> >
> > Relying on an external service for certificate store - is the best
> > option - assuming of course that the certificate store is actually
> > also highly available.
> >
> > Is that the case today with Barbican?
> >
> > According to the architecture docs [1] I see that they are using a
> > relational database. MySQL? PostgreSQL? Does that now mean we have an
> > additional database to maintain, backup, provide HA for as an
> Operator?
> >
> > The only real reference I can see to anything remotely HA is this [2]
> > and this [3]
> >
> > An overall solution is highly available *only* if all of the parts it
> > relies are also highly available as well.
> >
> >
> > [1]
> >
> http://docs.openstack.org/developer/barbican/contribute/architecture.h
> > tml#overall-architecture [2]
> > https://github.com/cloudkeep-ops/barbican-vagrant-zero
> > [3]
> > http://lists.openstack.org/pipermail/openstack/2014-March/006100.html
> >
> > Some food for thought
> >
> > --
> > Best Regards,
> > Maish Saidel-Keesing
> >
> >
> > On 03/18/16 17:18, Hongbin Lu wrote:
> > > Douglas,
> > >
> > > I am not opposed to adopt Barbican in Magnum (In fact, we already
> > > adopted Barbican). What I am opposed to is a Barbican lock-in,
> which
> > already has a negative impact on Magnum adoption based on our
> > feedback. I also want to see an increase of Barbican adoption in the
> future, and all our users have Barbican installed in their clouds. If
> that happens, I have no problem to have a hard dependency on Barbican.
> > >
> > > Best regards,
> > > Hongbin
> > >
> > > -----Original Message-----
> > > From: Douglas Mendizábal [mailto:douglas.mendizabal at rackspace.com]
> > > Sent: March-18-16 9:45 AM
> > > To: openstack-dev at lists.openstack.org
> > > Subject: Re: [openstack-dev] [magnum] High Availability
> > >
> > > Hongbin,
> > >
> > > I think Adrian makes some excellent points regarding the adoption
> of
> > > Barbican. As the PTL for Barbican, it's frustrating to me to
> > constantly hear from other projects that securing their sensitive
> data
> > is a requirement but then turn around and say that deploying Barbican
> is a problem.
> > >
> > > I guess I'm having a hard time understanding the operator persona
> > > that is willing to deploy new services with security features but
> > unwilling to also deploy the service that is meant to secure
> sensitive data across all of OpenStack.
> > >
> > > I understand one barrier to entry for Barbican is the high cost of
> > > Hardware Security Modules, which we recommend as the best option
> for
> > the Storage and Crypto backends for Barbican. But there are also
> > other options for securing Barbican using open source software like
> DogTag or SoftHSM.
> > >
> > > I also expect Barbican adoption to increase in the future, and I
> was
> > > hoping that Magnum would help drive that adoption. There are also
> > other projects that are actively developing security features like
> > Swift Encryption, and DNSSEC support in Desginate. Eventually these
> > features will also require Barbican, so I agree with Adrian that we
> as a community should be encouraging deployers to adopt the best
> security practices.
> > >
> > > Regarding the Keystone solution, I'd like to hear the Keystone
> > > team's feadback on that. It definitely sounds to me like you're
> > > trying to put
> > a square peg in a round hole.
> > >
> > > - Doug
> > >
> > > On 3/17/16 8:45 PM, Hongbin Lu wrote:
> > >> Thanks Adrian,
> > >>
> > >>
> > >>
> > >> I think the Keystone approach will work. For others, please speak
> > >> up if it doesn't work for you.
> > >>
> > >>
> > >>
> > >> Best regards,
> > >>
> > >> Hongbin
> > >>
> > >>
> > >>
> > >> *From:*Adrian Otto [mailto:adrian.otto at rackspace.com]
> > >> *Sent:* March-17-16 9:28 PM
> > >> *To:* OpenStack Development Mailing List (not for usage questions)
> > >> *Subject:* Re: [openstack-dev] [magnum] High Availability
> > >>
> > >>
> > >>
> > >> Hongbin,
> > >>
> > >>
> > >>
> > >> I tweaked the blueprint in accordance with this approach, and
> > >> approved it for Newton:
> > >>
> > >> https://blueprints.launchpad.net/magnum/+spec/barbican-
> alternative-
> > >> sto
> > >> re
> > >>
> > >>
> > >>
> > >> I think this is something we can all agree on as a middle ground,
> > >> If not, I'm open to revisiting the discussion.
> > >>
> > >>
> > >>
> > >> Thanks,
> > >>
> > >>
> > >>
> > >> Adrian
> > >>
> > >>
> > >>
> > >> On Mar 17, 2016, at 6:13 PM, Adrian Otto
> <adrian.otto at rackspace.com
> > >> <mailto:adrian.otto at rackspace.com>> wrote:
> > >>
> > >>
> > >>
> > >> Hongbin,
> > >>
> > >> One alternative we could discuss as an option for operators
> that
> > >> have a good reason not to use Barbican, is to use Keystone.
> > >>
> > >> Keystone credentials store:
> > >>
> > >> http://specs.openstack.org/openstack/keystone-
> specs/api/v3/identity
> > >> -ap i-v3.html#credentials-v3-credentials
> > >>
> > >> The contents are stored in plain text in the Keystone DB, so
> we
> > >> would want to generate an encryption key per bay, encrypt the
> > >> certificate and store it in keystone. We would then use the
> same key
> > >> to decrypt it upon reading the key back. This might be an
> acceptable
> > >> middle ground for clouds that will not or can not run Barbican.
> This
> > >> should work for any OpenStack cloud since Grizzly. The total
> amount
> > >> of code in Magnum would be small, as the API already exists.
> We
> > >> would need a library function to encrypt and decrypt the data,
> and
> > >> ideally a way to select different encryption algorithms in
> case one
> > >> is judged weak at some point in the future, justifying the use
> of an
> > >> alternate.
> > >>
> > >> Adrian
> > >>
> > >>
> > >> On Mar 17, 2016, at 4:55 PM, Adrian Otto
> <adrian.otto at rackspace.com
> > >> <mailto:adrian.otto at rackspace.com>> wrote:
> > >>
> > >> Hongbin,
> > >>
> > >>
> > >> On Mar 17, 2016, at 2:25 PM, Hongbin Lu <hongbin.lu at huawei.com
> > >> <mailto:hongbin.lu at huawei.com>> wrote:
> > >>
> > >> Adrian,
> > >>
> > >> I think we need a boarder set of inputs in this matter, so I
> moved
> > >> the discussion from whiteboard back to here. Please check my
> replies
> > >> inline.
> > >>
> > >>
> > >> I would like to get a clear problem statement written for this.
> > >> As I see it, the problem is that there is no safe place to put
> > >> certificates in clouds that do not run Barbican.
> > >> It seems the solution is to make it easy to add Barbican such
> that
> > >> it's included in the setup for Magnum.
> > >>
> > >> No, the solution is to explore an non-Barbican solution to
> store
> > >> certificates securely.
> > >>
> > >>
> > >> I am seeking more clarity about why a non-Barbican solution is
> > >> desired. Why is there resistance to adopting both Magnum and
> > >> Barbican together? I think the answer is that people think
> they can
> > >> make Magnum work with really old clouds that were set up
> before
> > >> Barbican was introduced. That expectation is simply not
> reasonable.
> > >> If there were a way to easily add Barbican to older clouds,
> perhaps
> > >> this reluctance would melt away.
> > >>
> > >>
> > >> Magnum should not be in the business of credential storage
> when
> > >> there is an existing service focused on that need.
> > >>
> > >> Is there an issue with running Barbican on older clouds?
> > >> Anyone can choose to use the builtin option with Magnum if
> hey
> > >> don't have Barbican.
> > >> A known limitation of that approach is that certificates
> are not
> > >> replicated.
> > >>
> > >> I guess the *builtin* option you referred is simply placing
> the
> > >> certificates to local file system. A few of us had concerns on
> this
> > >> approach (In particular, Tom Cammann has gave -2 on the review
> [1])
> > >> because it cannot scale beyond a single conductor. Finally, we
> made
> > >> a compromise to land this option and use it for
> testing/debugging
> > >> only. In other words, this option is not for production. As a
> > >> result, Barbican becomes the only option for production which
> is the
> > >> root of the problem. It basically forces everyone to install
> > >> Barbican in order to use Magnum.
> > >>
> > >> [1] https://review.openstack.org/#/c/212395/
> > >>
> > >>
> > >> It's probably a bad idea to replicate them.
> > >> That's what Barbican is for. --adrian_otto
> > >>
> > >> Frankly, I am surprised that you disagreed here. Back to July
> 2015,
> > >> we all agreed to have two phases of implementation and the
> statement
> > >> was made by you [2].
> > >>
> > >>
> ================================================================
> > >> #agreed Magnum will use Barbican for an initial implementation
> for
> > >> certificate generation and secure storage/retrieval. We will
> commit
> > >> to a second phase of development to eliminating the hard
> requirement
> > >> on Barbican with an alternate implementation that implements
> the
> > >> functional equivalent implemented in Magnum, which may depend
> on
> > >> libraries, but not Barbican.
> > >>
> > >> ================================================================
> > >>
> > >> [2]
> > >>
> > >> http://lists.openstack.org/pipermail/openstack-dev/2015-
> July/069130
> > >> .ht
> > >> ml
> > >>
> > >>
> > >> The context there is important. Barbican was considered for
> two
> > >> purposes: (1) CA signing capability, and (2) certificate
> storage. My
> > >> willingness to implement an alternative was based on our need
> to get
> > >> a certificate generation and signing solution that actually
> worked,
> > >> as Barbican did not work for that at the time. I have always
> viewed
> > >> Barbican as a suitable solution for certificate storage, as
> that was
> > >> what it was first designed for. Since then, we have
> implemented
> > >> certificate generation and signing logic within a library that
> does
> > >> not depend on Barbican, and we can use that safely in
> production use
> > >> cases. What we don't have built in is what Barbican is best at,
> > >> secure storage for our certificates that will allow multi-
> conductor
> > >> operation.
> > >>
> > >> I am opposed to the idea that Magnum should re-implement
> Barbican
> > >> for certificate storage just because operators are reluctant
> to
> > >> adopt it. If we need to ship a Barbican instance along with
> each
> > >> Magnum control plane, so be it, but I don't see the value in
> > >> re-inventing the wheel. I promised the OpenStack community
> that we
> > >> were out to integrate with and enhance OpenStack not to
> replace it.
> > >>
> > >> Now, with all that said, I do recognize that not all clouds
> are
> > >> motivated to use all available security best practices. They
> may be
> > >> operating in environments that they believe are already secure
> > >> (because of a secure perimeter), and that it's okay to run
> > >> fundamentally insecure software within those environments. As
> > >> misguided as this viewpoint may be, it's common. My belief is
> that
> > >> it's best to offer the best practice by default, and only
> allow
> > >> insecure operation when someone deliberately turns of
> fundamental
> > >> security features.
> > >>
> > >> With all this said, I also care about Magnum adoption as much
> as all
> > >> of us, so I'd like us to think creatively about how to strike
> the
> > >> right balance between re-implementing existing technology, and
> > >> making that technology easily accessible.
> > >>
> > >> Thanks,
> > >>
> > >> Adrian
> > >>
> > >>
> > >>
> > >> Best regards,
> > >> Hongbin
> > >>
> > >> -----Original Message-----
> > >> From: Adrian Otto [mailto:adrian.otto at rackspace.com]
> > >> Sent: March-17-16 4:32 PM
> > >> To: OpenStack Development Mailing List (not for usage
> questions)
> > >> Subject: Re: [openstack-dev] [magnum] High Availability
> > >>
> > >> I have trouble understanding that blueprint. I will put some
> remarks
> > >> on the whiteboard. Duplicating Barbican sounds like a mistake
> to me.
> > >>
> > >> --
> > >> Adrian
> > >>
> > >>
> > >> On Mar 17, 2016, at 12:01 PM, Hongbin Lu
> <hongbin.lu at huawei.com
> > >> <mailto:hongbin.lu at huawei.com>> wrote:
> > >>
> > >> The problem of missing Barbican alternative implementation has
> been
> > >> raised several times by different people. IMO, this is a very
> > >> serious issue that will hurt Magnum adoption. I created a
> blueprint
> > >> for that [1] and set the PTL as approver. It will be picked up
> by a
> > >> contributor once it is approved.
> > >>
> > >> [1]
> > >> https://blueprints.launchpad.net/magnum/+spec/barbican-
> alternative-sto
> > >> re
> > >>
> > >> Best regards,
> > >> Hongbin
> > >>
> > >> -----Original Message-----
> > >> From: Ricardo Rocha [mailto:rocha.porto at gmail.com]
> > >> Sent: March-17-16 2:39 PM
> > >> To: OpenStack Development Mailing List (not for usage
> questions)
> > >> Subject: Re: [openstack-dev] [magnum] High Availability
> > >>
> > >> Hi.
> > >>
> > >> We're on the way, the API is using haproxy load balancing in
> the
> > >> same way all openstack services do here - this part seems to
> work fine.
> > >>
> > >> For the conductor we're stopped due to bay certificates - we
> don't
> > >> currently have barbican so local was the only option. To get
> them
> > >> accessible on all nodes we're considering two options:
> > >> - store bay certs in a shared filesystem, meaning a new set of
> > >> credentials in the boxes (and a process to renew fs tokens)
> > >> - deploy barbican (some bits of puppet missing we're sorting
> > >> out)
> > >>
> > >> More news next week.
> > >>
> > >> Cheers,
> > >> Ricardo
> > >>
> > >>
> > >> On Thu, Mar 17, 2016 at 6:46 PM, Daneyon Hansen (danehans)
> > >> <danehans at cisco.com <mailto:danehans at cisco.com>> wrote:
> > >> All,
> > >>
> > >> Does anyone have experience deploying Magnum in a highly-
> available
> > >> fashion?
> > >> If so, I'm interested in learning from your experience. My
> biggest
> > >> unknown is the Conductor service. Any insight you can provide
> is
> > >> greatly appreciated.
> > >>
> > >> Regards,
> > >> Daneyon Hansen
> > >>
> > >>
> > >>
> ___________________________________________________________________
> > >> __
> > >>
> >
> >
> >
> ______________________________________________________________________
> > ____ OpenStack Development Mailing List (not for usage questions)
> > Unsubscribe:
> > OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
> _______________________________________________________________________
> ___
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-
> request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
More information about the OpenStack-dev
mailing list