<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body text="#000000" bgcolor="#FFFFFF">
<p><br>
</p>
<br>
<div class="moz-cite-prefix">On 19/07/17 01:23, Colleen Murphy
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:CAJkgcEnQFLYRaMamnjAqaTKdOJ=ueCE0H-3xVktt9xGnxF4oyA@mail.gmail.com">
<div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote">On Tue, Jul 18, 2017 at 1:39 AM, Zane
Bitter <span dir="ltr"><<a
href="mailto:zbitter@redhat.com" target="_blank"
moz-do-not-send="true">zbitter@redhat.com</a>></span>
wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px
0.8ex;border-left:1px solid
rgb(204,204,204);padding-left:1ex">So the application
credentials spec has merged - huge thanks to Monty and the
Keystone team for getting this done:<br>
<br>
<a href="https://review.openstack.org/#/c/450415/"
rel="noreferrer" target="_blank" moz-do-not-send="true">https://review.openstack.org/#<wbr>/c/450415/</a><br>
<a
href="http://specs.openstack.org/openstack/keystone-specs/specs/keystone/pike/application-credentials.html"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://specs.openstack.org/ope<wbr>nstack/keystone-specs/specs/<wbr>keystone/pike/application-<wbr>credentials.html</a><br>
<br>
However, it appears that there was a disconnect in how two
groups of folks were reading the spec that only became
apparent towards the end of the process. Specifically, at
this exact moment:<br>
<br>
<a
href="http://eavesdrop.openstack.org/irclogs/%23openstack-keystone/%23openstack-keystone.2017-06-09.log.html#t2017-06-09T17:43:59"
rel="noreferrer" target="_blank" moz-do-not-send="true">http://eavesdrop.openstack.org<wbr>/irclogs/%23openstack-keystone<wbr>/%23openstack-keystone.2017-<wbr>06-09.log.html#t2017-06-09T17:<wbr>43:59</a><br>
<br>
To summarise, Keystone folks are uncomfortable with the
idea of application credentials that share the lifecycle
of the project (rather than the user that created them),
because a consumer could surreptitiously create an
application credential and continue to use that to access
the OpenStack APIs even after their User account is
deleted. The agreed solution was to delete the application
credentials when the User that created them is deleted,
thus tying the lifecycle to that of the User.<br>
<br>
This means that teams using this feature will need to
audit all of their applications for credential usage and
rotate any credentials created by a soon-to-be-former team
member *before* removing said team member's User account,
or risk breakage. Basically we're relying on users to do
the Right Thing (bad), but when they don't we're
defaulting to breaking [some of] their apps over leaving
them insecure (all things being equal, good).<br>
<br>
Unfortunately, if we do regard this as a serious problem,
I don't think this solution is sufficient. Assuming that
application credentials are stored on VMs in the project
for use by the applications running on them, then anyone
with access to those servers can obtain the credentials
and continue to use them even if their own account is
deleted. The solution to this is to rotate *all*
application keys when a user is deleted. So really we're
relying on users to do the Right Thing (bad), but when
they don't we're defaulting to breaking [some of] their
apps *and* [potentially] leaving them insecure (worst
possible combination).<br>
<br>
(We're also being inconsistent, because according to the
spec if you revoke a role from a User then any application
credentials they've created that rely on that role
continue to work. It's only if you delete the User that
they're revoked.)<br>
<br>
<br>
As far as I can see, there are only two solutions to the
fundamental problem:<br>
<br>
1) Fine-grained user-defined access control. We can
minimise the set of things that the application
credentials are authorised to do. That's out of scope for
this spec, but something we're already planning as a
future enhancement.<br>
2) Automated regular rotation of credentials. We can make
sure that whatever a departing team member does manage to
hang onto quickly becomes useless.<br>
<br>
By way of comparison, AWS does both. There's fine-grained
defined access control in the form of IAM Roles, and these
Roles can be associated with EC2 servers. The servers have
an account with rotating keys provided through the
metadata server. I can't find the exact period of rotation
documented, but it's on the order of magnitude of 1 hour.<br>
<br>
There's plenty not to like about this design.
Specifically, it's 2017 not 2007 and the idea that there's
no point offering to segment permissions at a finer
grained level than that of a VM no longer holds water
IMHO, thanks to SELinux and containers. It'd be nice to be
able to provide multiple sets of credentials to different
services running on a VM, and it's probably essential to
our survival that we find a way to provide individual
credentials to containers. Nevertheless, what they have
does solve the problem.<br>
<br>
Note that there's pretty much no sane way for the user to
automate credential rotation themselves, because it's
turtles all the way down. e.g. it's easy in principle to
set up a Heat template with a Mistral workflow that will
rotate the credentials for you, but they'll do so using
trusts that are, in turn, tied back to the consumer who
created the stack. (It suddenly occurs to me that this is
a problem that all services using trusts are going to need
to solve.) Somewhere it all has to be tied back to
something that survives the entire lifecycle of the
project.<br>
<br>
Would Keystone folks be happy to allow persistent
credentials once we have a way to hand out only the
minimum required privileges?<br>
</blockquote>
<div><br>
</div>
<div>I agree that in the haste of getting this approved
before the spec freeze deadline we took this in the wrong
direction. I think that this spec was fine before the
addition of "Will be deleted when the associated User is
deleted" constraint.</div>
<div><br>
</div>
<div>As I understand it, the worry coming from the team is
that a user who sneakily copies the secret keys to an
offsite location before their user is deleted will still
have access to the resources the credential had access to.
This worry is irrational for two reasons:</div>
<div><br>
</div>
<div>1) Once the application credential is created, it is
going to be shared with the whole team. Therefore if ANY
member of the team is deleted, whether or not they were
the one to create the credential, they could still have
copied this key and still gain access to those resources
after their user was terminated. Tying the lifecycle of
the credential to its creating user does nothing to
prevent other teammates from regaining this access.</div>
<div><br>
</div>
<div>2) The ability to rotate the secret, whether it's
automated or not, is already an infinitely better
situation than where we currently are, which is that a
user must live forever and share their personal passwords
with the whole team in order to keep applications running.
We can encourage immediate credential rotation upon user
termination and provide helpful tooling and perhaps Heat
integration, but even without all of that it's still
better than forcing users to share their passwords with
one another and being unable to delete users who have left
the organization.</div>
<div><br>
</div>
<div>I'm supportive of revising the spec to undo this
constraint.</div>
<div><br>
</div>
</div>
</div>
</div>
</blockquote>
While I understand worries behind these security problems, I'd like
to add they they aren't really our problem. They can't be solved
with code on our end. These are procedural problems, and this
problem persists in any organisation, in any form of shared user
passwords, in any form of automation. If an application credential
is created for a service used/maintained by a team, there is nothing
stopping anyone from the team from storing that key elsewhere
regardless of who created it, because once created, that key is in
the 'shared' team domain where anyone can see it, exactly as Colleen
points out. If it wasn't shared with more than one person, then it's
useless since it means the service using the key itself would
entirely have to be maintained/controlled by that one person and no
one else would have access. Teams don't work like that.<br>
<br>
The problem is then entirely procedural within a team. Do they
rotate all keys when one person leaves? Anything less is the same
problem. All we can do is make rotation less of a pain, but it will
still be painful no matter what, and depending on the situation the
team makes the choice of how to handle rotation if at all.<br>
<br>
The sole reason for project level ownership of these application
credentials is so that a user leaving/being deleted isn't a scramble
to replace keys, and a team has the option/time to do it if they
care about the possibility of that person having known the keys
(again, not our problem, not a security flaw in code). Anything
else, pretty much makes this feature useless for teams. :(<br>
<br>
Having both options (owned by project vs user) is useful, but the
'security issues' are kind of implied by using project owned app
creds. It's a very useful feature with some 'use at your own risk'
attached.<br>
<br>
<br>
</body>
</html>