Open Stack

James, thanks for all the information! I will study, try some of this out
and get back to the thread with any more questions or findings.

One topic that may be worth discussing now, is that if we do pass ssh host
keys (including private) via vendordata, then there needs to be a way to
signal to Nova either:
- that specific data is private and should not be exposed via metadata API
- or for each vendordata choose what cloud-init data source it should be
offered on.

The goal would be to make sure that the private key is only offered to the
instance via config drive.

Does that make sense?

thanks,
Pino




On Thu, Oct 5, 2017 at 3:47 PM, James Penick <jpenick at gmail.com> wrote:

> Hey Pino,
>
> mriedem pointed me to the vendordata code [1] which shows some fields are
> passed (such as project ID) and that SSL is supported. So that's good.
>
> The docs on vendordata suck. But I think it'll do what you're looking for.
> Michael Still wrote up a helpful post titled "Nova vendordata deployment,
> an excessively detailed guide"[2] and he's written a vendordata service
> example[3] which even shows keystone integration.
>
> At Oath, we have a system that provides a unique x509 certificate for each
> host, including the ability to sign host SSH keys against an HSM. In our
> case what we do is have Nova call the service, which generates and returns
> a signed (and time limited) host bootstrap document, which is injected into
> the instance. When the instance boots it calls our identity service and
> provides its bootstrap document as a bearer certificate. The identity
> service trusts this one-time document to attest the instance, and will then
> provide an x509 certificate as well as sign the hosts SSH keys. After the
> initial bootstrap the host will rotate its keys frequently, by providing
> its last certificate in exchange for a new one. The service tracks all host
> document and certificate IDs which have been exchanged until their expiry,
> so that a cert cannot be re-used.
>
> This infrastructure relies on Athenz [4] as the AuthNG system for all
> principals (users, services, roles, domains, etc) as well as an internal
> signatory service which signs x509 certificates and SSH host keys using an
> HSM infrastructure.
>
> Instead, you could write a vendordata service which, when called, would
> generate an ssh host keypair, sign it, and return those files as encoded
> data, which can be expanded into files in the correct locations on first
> boot. I strongly suggest using not only using keystone auth, but that you
> ensure all calls from vendordata to the microservice are encrypted with TLS
> mutual auth.
>
> -James
>
>
> 1: https://github.com/openstack/nova/blob/master/nova/api/
> metadata/vendordata_dynamic.py#L77
> 2: https://www.stillhq.com/openstack/000022.html
> 3: https://github.com/mikalstill/vendordata
> 4: https://athenz.io
>
>
> On Fri, Sep 29, 2017 at 5:17 PM, Fox, Kevin M <Kevin.Fox at pnnl.gov> wrote:
>
>> https://review.openstack.org/#/c/222293/
>> ------------------------------
>> *From:* Giuseppe de Candia [giuseppe.decandia at gmail.com]
>> *Sent:* Friday, September 29, 2017 1:05 PM
>> *To:* OpenStack Development Mailing List (not for usage questions)
>> *Subject:* Re: [openstack-dev] Supporting SSH host certificates
>>
>> Ihar, thanks for pointing that out - I'll definitely take a close look.
>>
>> Jon, I'm not very familiar with Barbican, but I did assume the full
>> implementation would use Barbican to store private keys. However, in terms
>> of actually getting a private key (or SSH host cert) into a VM instance,
>> Barbican doesn't help. The instance needs permission to access secrets
>> stored in Barbican. The main question of my e-mail is: how do you inject a
>> credential in an automated but secure way? I'd love to hear ideas - in the
>> meantime I'll study Ihar's link.
>>
>> thanks,
>> Pino
>>
>>
>>
>> On Fri, Sep 29, 2017 at 2:49 PM, Ihar Hrachyshka <ihrachys at redhat.com>
>> wrote:
>>
>>> What you describe (at least the use case) seems to resemble
>>> https://review.openstack.org/#/c/456394/ This work never moved
>>> anywhere since the spec was posted though. You may want to revive the
>>> discussion in scope of the spec.
>>>
>>> Ihar
>>>
>>> On Fri, Sep 29, 2017 at 12:21 PM, Giuseppe de Candia
>>> <giuseppe.decandia at gmail.com> wrote:
>>> > Hi Folks,
>>> >
>>> >
>>> >
>>> > My intent in this e-mail is to solicit advice for how to inject SSH
>>> host
>>> > certificates into VM instances, with minimal or no burden on users.
>>> >
>>> >
>>> >
>>> > Background (skip if you're already familiar with SSH certificates):
>>> without
>>> > host certificates, when clients ssh to a host for the first time (or
>>> after
>>> > the host has been re-installed), they have to hope that there's no man
>>> in
>>> > the middle and that the public key being presented actually belongs to
>>> the
>>> > host they're trying to reach. The host's public key is stored in the
>>> > client's known_hosts file. SSH host certicates eliminate the
>>> possibility of
>>> > Man-in-the-Middle attack: a Certificate Authority public key is
>>> distributed
>>> > to clients (and written to their known_hosts file with a special
>>> syntax and
>>> > options); the host public key is signed by the CA, generating an SSH
>>> > certificate that contains the hostname and validity period (among other
>>> > things). When negotiating the ssh connection, the host presents its
>>> SSH host
>>> > certificate and the client verifies that it was signed by the CA.
>>> >
>>> >
>>> >
>>> > How to support SSH host certificates in OpenStack?
>>> >
>>> >
>>> >
>>> > First, let's consider doing it by hand, instance by instance. The only
>>> > solution I can think of is to VNC to the instance, copy the public key
>>> to my
>>> > CA server, sign it, and then write the certificate back into the host
>>> (again
>>> > via VNC). I cannot ssh without risking a MITM attack. What about using
>>> Nova
>>> > user-data? User-data is exposed via the metadata service. Metadata is
>>> > queried via http (reply transmitted in the clear, susceptible to
>>> snooping),
>>> > and any compute node can query for any instance's meta-data/user-data.
>>> >
>>> >
>>> >
>>> > At this point I have to admit I'm ignorant of details of cloud-init. I
>>> know
>>> > cloud-init allows specifying SSH private keys (both for users and for
>>> SSH
>>> > service). I have not yet studied how such information is securely
>>> injected
>>> > into an instance. I assume it should only be made available via
>>> ConfigDrive
>>> > rather than metadata-service (again, that service transmits in the
>>> clear).
>>> >
>>> >
>>> >
>>> > What about providing SSH host certificates as a service in OpenStack?
>>> Let's
>>> > keep out of scope issues around choosing and storing the CA keys, but
>>> the CA
>>> > key is per project. What design supports setting up the SSH host
>>> certificate
>>> > automatically for every VM instance?
>>> >
>>> >
>>> >
>>> > I have looked at Vendor Data and I don't see a way to use that, mainly
>>> > because 1) it doesn't take parameters, so you can't pass the public
>>> key out;
>>> > and 2) it's queried over http, not https.
>>> >
>>> >
>>> >
>>> > Just as a feasibility argument, one solution would be to modify Nova
>>> compute
>>> > instance boot code. Nova compute can securely query a CA service
>>> asking for
>>> > a triplet (private key, public key, SSH certificate) for the specific
>>> > hostname. It can then inject the triplet using ConfigDrive. I believe
>>> this
>>> > securely gets the private key into the instance.
>>> >
>>> >
>>> >
>>> > I cannot figure out how to get the equivalent functionality without
>>> > modifying Nova compute and the boot process. Every solution I can
>>> think of
>>> > risks either exposing the private key or vulnerability to a MITM attack
>>> > during the signing process.
>>> >
>>> >
>>> >
>>> > Your help is appreciated.
>>> >
>>> >
>>> >
>>> > --Pino
>>> >
>>> >
>>> > ____________________________________________________________
>>> ______________
>>> > OpenStack Development Mailing List (not for usage questions)
>>> > Unsubscribe: OpenStack-dev-request at lists.op
>>> enstack.org?subject:unsubscribe
>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>> >
>>>
>>> ____________________________________________________________
>>> ______________
>>> OpenStack Development Mailing List (not for usage questions)
>>> Unsubscribe: OpenStack-dev-request at lists.op
>>> enstack.org?subject:unsubscribe
>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>>
>>
>>
>> ____________________________________________________________
>> ______________
>> OpenStack Development Mailing List (not for usage questions)
>> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscrib
>> e
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>>
>>
>
> __________________________________________________________________________
> OpenStack Development Mailing List (not for usage questions)
> Unsubscribe: OpenStack-dev-request at lists.openstack.org?subject:unsubscribe
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-dev/attachments/20171006/38e26d58/attachment.html>