Open Stack

On 19/07/17 23:19, Monty Taylor wrote:
> 
> Instance users do not solve this. Instance users can be built with this- 
> but instance users are themselves not sufficient. Instance users are 
> only sufficient in single-cloud ecosystems where it is possible to grant 
> permissions on all the resources in the single-cloud ecosystem to an 
> instance. We are not a single-cloud ecosystem.

Good point. Actually, nobody lives in a single-cloud ecosystem any more. 
So the 'public' side of any hybrid-cloud arrangement (including the big 
3 public clouds, not just OpenStack) will always need a way to deal with 
this.

> Nodepool runs in Rackspace's DFW region. It has accounts across nine 
> different clouds. If this were only solved with Instance users we'd have 
> to boot a VM in each cloud so that we could call the publicly-accessible 
> REST APIs of the clouds to boot VMs in each cloud.

I'm glad you're here, because I don't spend a lot of time thinking about 
such use cases (if we can get cloud applications to work on even one 
cloud then I can retire to my goat farm happy) and this one would have 
escaped me :)

So let's boil this down to 4 types of 'users' who need to authenticate 
to a given cloud:

1) Actual, corporeal humans
2) Services that are part of the cloud itself (e.g. autoscaling)
3) Hybrid-cloud applications running elsewhere (e.g. nodepool)
4) Applications running in the cloud

Looking at how AWS handles these cases AIUI:

1) For each tenant there is a 'root' account with access to billing &c. 
Best practice is not to create API credentials for this account at all. 
Instead, you create IAM Users for all of the humans who need to access 
the tenant and give permissions to them (bootstrapped by the root 
account) using IAM Policies. To make management easier, you can 
aggregate Users into Groups. If a user leaves the organisation, you 
delete their IAM User. If the owner leaves the organisation, somebody 
else becomes the owner and you rotate the root password.

2) Cloud services can be named as principals in IAM policies, so 
permissions can be given to them in the same way that they are to human 
users.

3) You create an IAM User for the application and give it the 
appropriate permissions. The credential they get is actually a private 
key, not a password, so in theory you could store it in an HSM that just 
signs stuff with it and not provide it directly to the application. 
Otherwise, the credentials are necessarily disclosed to the team 
maintaining the application. If somebody who has/had access to private 
key leaves, you need to rotate the credentials. It's possible to 
automate the mechanics of this, but ultimately it has to be triggered by 
a human using their own credentials otherwise it's turtles all the way 
down. The AWS cloud has no way of ensuring that you rotate the 
credentials at appropriate times, or even knowing when those times are.

4) Instance users. You can give permissions to a VM that you have 
created in the cloud. It automatically receives credentials in its 
metadata. The credentials expire quite rapidly and are automatically 
replaced with new ones, also accessible through the metadata server. The 
application just reads the latest credentials from metadata and uses 
them. If someone leaves the organisation, you don't care. If an attacker 
breaches your server, the damage is limited to a relatively short window 
once you've evicted them again. There's no way to do the Wrong Thing 
even if you're trying.

And in OpenStack:

1) Works great provided you only have one user per project. Your 
password may, and probably will, be shared with your billing account 
(public cloud), or will be shared with pretty much your whole life 
(private cloud). If multiple humans need to work on the project, you'll 
generally need to share passwords or do something out-of-band to set it 
up (e.g. open a ticket with IT). If somebody leaves the organisation, 
same deal.

Application credentials could greatly improve this in the public cloud 
scenario.

2) Cloud services can create trusts that allow them to act on behalf of 
a particular user. If that user leaves the organisation, your 
application is hosed until someone else redeploys it to get a new trust.

Persistent application credentials could potentially replace trusts and 
solve this problem, although they'd need to be stored somewhere more 
secure (i.e. Barbican) than trust IDs are currently stored. A better 
solution might be to allow the service user to be granted permissions by 
the forthcoming fine-grained authorisation mechanism (independently of 
an application credential) - but this would require changes to the 
Keystone policies, because it would currently be blocked by the 
Scoped-RBAC system.

3) The credentials are necessarily disclosed to the team maintaining the 
application. Your password may, and probably will, be shared with your 
billing account. If somebody leaves the organisation, you have to rotate 
the password. This has to be done manually. The cloud has no way of 
ensuring that you rotate the credentials at appropriate times, or even 
knowing when those times are.

Application credentials would solve the part where the shared password 
is the same one as for your billing account.

4) No general solution. Some services (Swift, Zaqar) have signed URLs. 
Servers deployed by Heat have a way of phoning back to it, and only it, 
to report deployment progress (using Keystone User accounts that it 
creates in a separate domain for the purpose). Otherwise, you're putting 
your password on the server. Your password may, and probably will, be 
shared with your billing account (public cloud), or will be shared with 
pretty much your whole life (private cloud); thus, you can't allow 
access to anyone else on your team. If you do and someone leaves the 
organisation, you have to rotate your password which is a manual 
process. If you leave the organisation, the application is hosed. If an 
attacker breaches your server, _you_ are hosed.

Application credentials would solve the problem of having to put your 
own password on the server. If somebody leaves the organisation, you 
have to rotate the password. This has to be done manually. The cloud has 
no way of ensuring that you rotate the credentials at appropriate times, 
or even knowing when those times are. If you don't rotate the 
credentials then the application is hosed. With persistent application 
credentials, the application won't break but you'd also need to audit 
all of the credentials to make sure you rotated them all (i.e. updating 
every application is not sufficient; you have to check for credentials 
not associated with an application).


Some thoughts:

* The existence of persistent application credentials would not really 
be any worse security-wise than IAM Users. If there's a difference it's 
that IAM Users are the mechanism for everyday interaction with AWS, so 
users are more likely to notice if something is amiss. Application 
Credentials as proposed are kind of a bolt-on API that might be easier 
to overlook.

* Assuming that most public clouds are single-user-per-project, (1) and 
(3) can be greatly improved by application credentials without them 
needing to be persistent beyond the lifetime of the Keystone User.

* (2) is better solved by having the authorisation mechanism be able to 
authorise service users, rather than trying to replace trusts with 
application credentials.

* If Keystone supported either a public-key or a Kerberos-style 
authentication mechanism to get a token, then for (3) you could store 
the passwords in an HSM (or software equivalent) instead of in the 
config file and have it do request signing instead of allowing access to 
the key, which would eliminate the risk of exposing your billing 
credentials on a publicly-accessible server. This seems like the kind of 
thing the Barbican API should support, although at first glance it 
doesn't appear to (yet?). If it did then you'd just need to authenticate 
to Barbican to get it to sign stuff for you, essentially making (3) a 
subset of (the admittedly more difficult) (4) instead of a disjoint 
problem. This is probably the most secure solution to the problem 
independently of whether Application Credentials are a thing or not.

* Unless we have instance users, (4) is going to remain problematic, 
especially for public clouds (or any cloud not behind a firewall), 
because it requires rotating credentials manually.


> Application Credentials, on the other hand, would allow us improve the situation without imposing an architecture on us that is a copy of an architecture from some clouds that have a vested interest in keeping people thinking about problems in a uni-cloud manner. 

I'm all for us doing things differently when the world has moved on and 
we have an opportunity to do things better than other clouds, and there 
are definitely things that are more challenging when you're thinking 
about a fleet of interoperable clouds instead of just one (e.g. 
versioning is much harder when time doesn't only run forwards). But for 
every decision like that, I've seen 10 where we did something different 
just because we didn't understand the use case until it was too late. 
Some clouds have had solutions to the above 4 problems for 10 years; 
we're just now talking about the problems AFAIK for the first time after 
6. Let's exercise an appropriate degree of humility: 
https://en.wikipedia.org/wiki/G._K._Chesterton#Chesterton.27s_fence

cheers,
Zane.