On Fri, Jan 25, 2019 at 3:02 PM James Penick <jpenick@gmail.com> wrote:

Hey Lance, We'd definitely be interested in helping with the work. I'll grab some volunteers from my team and get them in touch within the next few days.

Awesome, that sounds great! I'm open to using this thread for more technical communication if needed. Otherwise, #openstack-keystone is always open for folks to swing by if they want to discuss things there. FWIW - we brought this up in the keystone meeting today and there several other people interested in this work. There is probably going to be an opportunity to break the work up a bit.

-James

On Fri, Jan 25, 2019 at 11:16 AM Lance Bragstad <lbragstad@gmail.com> wrote:

...

Hi all,

We've been going over keystone gaps that need to be addressed for edge use cases every Tuesday. Since Berlin, Oath has open-sourced some of their custom authentication plugins for keystone that help them address these gaps.

The basic idea is that users authenticate to some external identity provider (Athenz in Oath's case), and then present an Athenz token to keystone. The custom plugins decode the token from Athenz to determine the user, project, roles assignments, and other useful bits of information. After that, it creates any resources that don't exist in keystone already. Ultimately, a user can authenticate against a keystone node and have specific resources provisioned automatically. In Berlin, engineers from Oath were saying they'd like to move away from Athenz tokens altogether and use x509 certificates issued by Athenz instead. The auto-provisioning approach is very similar to a feature we have in keystone already. In Berlin, and shortly after, there was general agreement that if we could support x509 authentication with auto-provisioning via keystone federation, that would pretty much solve Oath's use case without having to maintain custom keystone plugins.

Last week, Colleen started digging into keystone's existing x509 authentication support. I'll start with the good news, which is x509 authentication works, for the most part. It's been a feature in keystone for a long time, and it landed after we implemented federation support around the Kilo release. Chances are there won't be a need for a keystone specification like we were initially thinking in the edge meetings. Unfortunately, the implementation for x509 authentication has outdated documentation, is extremely fragile, hard to set up, and hasn't been updated with improvements we've made to the federation API since the original implementation (like shadow users or auto-provisioning, which work with other federated protocols like OpenID Connect and SAML). We've started tracking the gaps with bugs [0] so that we have things written down.

I think the good thing is that once we get this cleaned up, we'll be able to re-use some of the newer federation features with x509 authentication/federation. These updates would make x509 a first-class federated protocol. The approach, pending the bug fixes, would remove the need for Oath's custom authentication plugins. It could be useful for edge deployments, or even deployments with many regions, by allowing users to be auto-provisioned in each region. Although, it doesn't necessarily solve the network partition issue.

Now that we have an idea of where to start and some bug reports [0], I'm wondering if anyone is interested in helping with the update or refactor. Because this won't require a specification, we can get started on it sooner, instead of having to wait for Train development and a new specification. I'm also curious if anyone has comments or questions about the approach.

Thanks,

Lance

[0] https://bugs.launchpad.net/keystone/+bugs?field.tag=x509 _______________________________________________ Edge-computing mailing list Edge-computing@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/edge-computing

-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA256 Hi Lance, I'd be glad to help out with the docs update. -----BEGIN PGP SIGNATURE----- Version: FlowCrypt 6.6.1 Gmail Encryption Comment: Seamlessly send and receive encrypted email wsFcBAEBCAAGBQJcY0KNAAoJEKhrJ5xxCfXWfsQQAJJktzu0kwIYBs8wi+QY Szd4wxRPf2GhKfYXsCgtUKAf5QWRXJBWEVpwai3ZbEDyCXa9UkEqSZY4s9l6 iizIuGW07qw/vTVJL9GX1q/6lpoYO5YqiL5cepCronKbzQtHvNL8TP2SPovG BXBTTiyj5LxKdQ7nojePpOQlINkTVWPVyVEwyVGqzXWh6/Dm7ws3pMUM5E5q SYcmmyiBxTbttV7csLqvB4WMpwM4Ucxd/1b9ojdaxeqkXy6uKA6fIOktP7QK uq+P7h9TYeuZ0x4wkw3dodJaRDLL4bvp+1sFtLXQELPQSEWFQ8SX5SMH9dQs lCTpkNjwmqTHXQ0/f69rjWc9zWIG/Y6S+f6I9fPwVV/L6bEkqoC9B3wz7K/y 2emAi96XwER4uLMrEpcQnQVi8aDczSfZtSw355Gxdp0h+A2FBmGC7pGU6Vn3 o1UUOV/HE49jWeqo+suNCoMBqz52+pAQ76fY81QAiUsnYcHGF6rL5yQJJZBG 6aF6pYa5A3iNeaSLIiZKNC2QEItV0GjmbJg7LHIsJDQwls2ITRa/WGpbakmW Jisgr/VxIIjrwp2z9+kOTQNptDbYANuyu6KQp/DORuDzNPGoCUedtZALebC7 2cJjzlSo1ZqjFbZiTw6wpZBTlfsGrJmrv0uRZYII21Zf4KLRmr4PXb83VRFo +rYF =gitY -----END PGP SIGNATURE-----

Hey Lance,

I like the plan.

Just a clarifying question on *“Although, it doesn't necessarily solve the network partition issue.”* .

- I’m assuming this is in a scenario where after the network partition the Edge Cloud and local client(s) do not have access to the Identity Provider ?

Correct. Using x509 certificates to authenticate to keystone still requires access to keystone. Keystone doesn't necessarily need a link to

On Tue, Jan 29, 2019 at 8:06 AM Waines, Greg <Greg.Waines@windriver.com> wrote: the "identity provider" in this case, since the authentication path doesn't require online validation. This is different from using SAML assertions, where the entire flow establishes a connection between keystone acting as the service provider (at the edge) and an identity provider somewhere authenticating the user.

- - And in this case, it doesn’t work because ? - For a new local client (without any cached tokens), even if there are local shadow users already configured, the authentication still requires communication with the Identity Provider ?

I guess it depends on the architecture you're considering [0]. There isn't

a hard requirement to talk to the identity provider of an x509 certificate, but token validation still needs to work. If you're deploying the architecture with the distributed control plane, you can authenticate with an x509 certificate against any keystone. For example, only using a centralized data center and medium/large edge cluster would make keystone available everywhere, so a network partition might not be an issue. Conversely, if you authenticate for a token with an x509 certificate and use it to spin up compute resources in a small edge cluster, which doesn't have a keystone deployment, the network partition is going to make online token validation impossible, if you're calling APIs in the small edge directly. The same issue is going to be present for deployments following the centralized control plane architecture since keystone is only available in the central data center and isn't available at the large, medium, or small edge sites. Validating the token online from edge sites to the centralized data center is going to be susceptible to network partitions. In my opinion, the big difference between x509 and other federated protocols is that it doesn't really have a hard requirement on linking back to the identity provider. In the case of SAML, the identity provider is anything that has the ability to issue SAML assertions proving the identity of its users (e.g., keystone acting as an identity provider, ADFS, etc.) With x509 certificates, the identity provider is a certificate authority that issues and signs user certificates. Keystone needs to be configured to "trust" certificated signed by that certificate authority. When a user authenticates, keystone relies on SSL plugin libraries to validate the certificate against the root certificate authority, but this is done offline since the SSL configuration has a copy of the root certificates.

From there, the plan is to treat each trusted certificate authority as its own identity provider, so all users with certificates signed by the same authority get mapped into the same namespace/identity provider. Once the users have their signed certificate, they can authenticate for tokens without a link being established between keystone and whatever certificate authority issued the certificate.

The downside is that certificate revocation and certificate distribution is now a thing you need to worry about. James might have more input there since it sounds like this is the approach they are shooting for with Athenz (which is the certificate authority in their case.) I hope that helps clear things up? [0] https://wiki.openstack.org/wiki/Edge_Computing_Group/Edge_Reference_Architec...

- -

Is this correct ?

Greg.

*From: *Lance Bragstad <lbragstad@gmail.com> *Date: *Friday, January 25, 2019 at 2:16 PM *To: *"edge-computing@lists.openstack.org" < edge-computing@lists.openstack.org>, " openstack-discuss@lists.openstack.org" < openstack-discuss@lists.openstack.org> *Subject: *[Edge-computing] [keystone] x509 authentication

Hi all,

We've been going over keystone gaps that need to be addressed for edge use cases every Tuesday. Since Berlin, Oath has open-sourced some of their custom authentication plugins for keystone that help them address these gaps.

The basic idea is that users authenticate to some external identity provider (Athenz in Oath's case), and then present an Athenz token to keystone. The custom plugins decode the token from Athenz to determine the user, project, roles assignments, and other useful bits of information. After that, it creates any resources that don't exist in keystone already. Ultimately, a user can authenticate against a keystone node and have specific resources provisioned automatically. In Berlin, engineers from Oath were saying they'd like to move away from Athenz tokens altogether and use x509 certificates issued by Athenz instead. The auto-provisioning approach is very similar to a feature we have in keystone already. In Berlin, and shortly after, there was general agreement that if we could support x509 authentication with auto-provisioning via keystone federation, that would pretty much solve Oath's use case without having to maintain custom keystone plugins.

Last week, Colleen started digging into keystone's existing x509 authentication support. I'll start with the good news, which is x509 authentication works, for the most part. It's been a feature in keystone for a long time, and it landed after we implemented federation support around the Kilo release. Chances are there won't be a need for a keystone specification like we were initially thinking in the edge meetings. Unfortunately, the implementation for x509 authentication has outdated documentation, is extremely fragile, hard to set up, and hasn't been updated with improvements we've made to the federation API since the original implementation (like shadow users or auto-provisioning, which work with other federated protocols like OpenID Connect and SAML). We've started tracking the gaps with bugs [0] so that we have things written down.

I think the good thing is that once we get this cleaned up, we'll be able to re-use some of the newer federation features with x509 authentication/federation. These updates would make x509 a first-class federated protocol. The approach, pending the bug fixes, would remove the need for Oath's custom authentication plugins. It could be useful for edge deployments, or even deployments with many regions, by allowing users to be auto-provisioned in each region. Although, it doesn't necessarily solve the network partition issue.

Now that we have an idea of where to start and some bug reports [0], I'm wondering if anyone is interested in helping with the update or refactor. Because this won't require a specification, we can get started on it sooner, instead of having to wait for Train development and a new specification. I'm also curious if anyone has comments or questions about the approach.

Thanks,

Lance

[0] https://bugs.launchpad.net/keystone/+bugs?field.tag=x509