<div dir="ltr">James, thanks for all the information! I will study, try some of this out and get back to the thread with any more questions or findings.<div><br></div><div>One topic that may be worth discussing now, is that if we do pass ssh host keys (including private) via vendordata, then there needs to be a way to signal to Nova either:</div><div>- that specific data is private and should not be exposed via metadata API</div><div>- or for each vendordata choose what cloud-init data source it should be offered on.</div><div><br></div><div>The goal would be to make sure that the private key is only offered to the instance via config drive.</div><div><br></div><div>Does that make sense?</div><div><br></div><div>thanks,</div><div>Pino</div><div><br></div><div><br></div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Oct 5, 2017 at 3:47 PM, James Penick <span dir="ltr"><<a href="mailto:jpenick@gmail.com" target="_blank">jpenick@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hey Pino,<div><br></div><div><div>mriedem pointed me to the vendordata code [1] which shows some fields are passed (such as project ID) and that SSL is supported. So that's good.</div></div><div><br></div><div>The docs on vendordata suck. But I think it'll do what you're looking for. Michael Still wrote up a helpful post titled "Nova vendordata deployment, an excessively detailed guide"[2] and he's written a vendordata service example[3] which even shows keystone integration.<br></div><div><br></div><div>At Oath, we have a system that provides a unique x509 certificate for each host, including the ability to sign host SSH keys against an HSM. In our case what we do is have Nova call the service, which generates and returns a signed (and time limited) host bootstrap document, which is injected into the instance. When the instance boots it calls our identity service and provides its bootstrap document as a bearer certificate. The identity service trusts this one-time document to attest the instance, and will then provide an x509 certificate as well as sign the hosts SSH keys. After the initial bootstrap the host will rotate its keys frequently, by providing its last certificate in exchange for a new one. The service tracks all host document and certificate IDs which have been exchanged until their expiry, so that a cert cannot be re-used.</div><div><br></div><div>This infrastructure relies on Athenz [4] as the AuthNG system for all principals (users, services, roles, domains, etc) as well as an internal signatory service which signs x509 certificates and SSH host keys using an HSM infrastructure.</div><div><br></div><div>Instead, you could write a vendordata service which, when called, would generate an ssh host keypair, sign it, and return those files as encoded data, which can be expanded into files in the correct locations on first boot. I strongly suggest using not only using keystone auth, but that you ensure all calls from vendordata to the microservice are encrypted with TLS mutual auth.</div><div><br></div><div>-James</div><div><br></div><div><br></div><div>1: <a href="https://github.com/openstack/nova/blob/master/nova/api/metadata/vendordata_dynamic.py#L77" target="_blank">https://github.com/openstack/<wbr>nova/blob/master/nova/api/<wbr>metadata/vendordata_dynamic.<wbr>py#L77</a></div><div>2: <a href="https://www.stillhq.com/openstack/000022.html" target="_blank">https://www.stillhq.com/<wbr>openstack/000022.html</a></div><div>3: <a href="https://github.com/mikalstill/vendordata" target="_blank">https://github.com/<wbr>mikalstill/vendordata</a></div><div>4: <a href="https://athenz.io" target="_blank">https://athenz.io</a></div><div><br></div></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Fri, Sep 29, 2017 at 5:17 PM, Fox, Kevin M <span dir="ltr"><<a href="mailto:Kevin.Fox@pnnl.gov" target="_blank">Kevin.Fox@pnnl.gov</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div style="direction:ltr;font-family:Tahoma;color:#000000;font-size:10pt"><a href="https://review.openstack.org/#/c/222293/" target="_blank">https://review.openstack.org/#<wbr>/c/222293/</a><br>
<div style="font-family:Times New Roman;color:#000000;font-size:16px">
<hr>
<div id="m_-3244657552731646408m_9041349648992642493divRpF766623" style="direction:ltr"><font face="Tahoma" color="#000000" size="2"><b>From:</b> Giuseppe de Candia [<a href="mailto:giuseppe.decandia@gmail.com" target="_blank">giuseppe.decandia@gmail.com</a>]<br>
<b>Sent:</b> Friday, September 29, 2017 1:05 PM<br>
<b>To:</b> OpenStack Development Mailing List (not for usage questions)<br>
<b>Subject:</b> Re: [openstack-dev] Supporting SSH host certificates<br>
</font><br>
</div><div><div class="m_-3244657552731646408h5">
<div></div>
<div>
<div dir="ltr">Ihar, thanks for pointing that out - I'll definitely take a close look.
<div><br>
</div>
<div>Jon, I'm not very familiar with Barbican, but I did assume the full implementation would use Barbican to store private keys. However, in terms of actually getting a private key (or SSH host cert) into a VM instance, Barbican doesn't help. The instance
needs permission to access secrets stored in Barbican. The main question of my e-mail is: how do you inject a credential in an automated but secure way? I'd love to hear ideas - in the meantime I'll study Ihar's link.</div>
<div><br>
</div>
<div>thanks,</div>
<div>Pino</div>
<div><br>
</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Fri, Sep 29, 2017 at 2:49 PM, Ihar Hrachyshka <span dir="ltr">
<<a href="mailto:ihrachys@redhat.com" target="_blank">ihrachys@redhat.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
What you describe (at least the use case) seems to resemble<br>
<a href="https://review.openstack.org/#/c/456394/" rel="noreferrer" target="_blank">https://review.openstack.org/#<wbr>/c/456394/</a> This work never moved<br>
anywhere since the spec was posted though. You may want to revive the<br>
discussion in scope of the spec.<br>
<span class="m_-3244657552731646408m_9041349648992642493HOEnZb"><font color="#888888"><br>
Ihar<br>
</font></span>
<div class="m_-3244657552731646408m_9041349648992642493HOEnZb">
<div class="m_-3244657552731646408m_9041349648992642493h5"><br>
On Fri, Sep 29, 2017 at 12:21 PM, Giuseppe de Candia<br>
<<a href="mailto:giuseppe.decandia@gmail.com" target="_blank">giuseppe.decandia@gmail.com</a>> wrote:<br>
> Hi Folks,<br>
><br>
><br>
><br>
> My intent in this e-mail is to solicit advice for how to inject SSH host<br>
> certificates into VM instances, with minimal or no burden on users.<br>
><br>
><br>
><br>
> Background (skip if you're already familiar with SSH certificates): without<br>
> host certificates, when clients ssh to a host for the first time (or after<br>
> the host has been re-installed), they have to hope that there's no man in<br>
> the middle and that the public key being presented actually belongs to the<br>
> host they're trying to reach. The host's public key is stored in the<br>
> client's known_hosts file. SSH host certicates eliminate the possibility of<br>
> Man-in-the-Middle attack: a Certificate Authority public key is distributed<br>
> to clients (and written to their known_hosts file with a special syntax and<br>
> options); the host public key is signed by the CA, generating an SSH<br>
> certificate that contains the hostname and validity period (among other<br>
> things). When negotiating the ssh connection, the host presents its SSH host<br>
> certificate and the client verifies that it was signed by the CA.<br>
><br>
><br>
><br>
> How to support SSH host certificates in OpenStack?<br>
><br>
><br>
><br>
> First, let's consider doing it by hand, instance by instance. The only<br>
> solution I can think of is to VNC to the instance, copy the public key to my<br>
> CA server, sign it, and then write the certificate back into the host (again<br>
> via VNC). I cannot ssh without risking a MITM attack. What about using Nova<br>
> user-data? User-data is exposed via the metadata service. Metadata is<br>
> queried via http (reply transmitted in the clear, susceptible to snooping),<br>
> and any compute node can query for any instance's meta-data/user-data.<br>
><br>
><br>
><br>
> At this point I have to admit I'm ignorant of details of cloud-init. I know<br>
> cloud-init allows specifying SSH private keys (both for users and for SSH<br>
> service). I have not yet studied how such information is securely injected<br>
> into an instance. I assume it should only be made available via ConfigDrive<br>
> rather than metadata-service (again, that service transmits in the clear).<br>
><br>
><br>
><br>
> What about providing SSH host certificates as a service in OpenStack? Let's<br>
> keep out of scope issues around choosing and storing the CA keys, but the CA<br>
> key is per project. What design supports setting up the SSH host certificate<br>
> automatically for every VM instance?<br>
><br>
><br>
><br>
> I have looked at Vendor Data and I don't see a way to use that, mainly<br>
> because 1) it doesn't take parameters, so you can't pass the public key out;<br>
> and 2) it's queried over http, not https.<br>
><br>
><br>
><br>
> Just as a feasibility argument, one solution would be to modify Nova compute<br>
> instance boot code. Nova compute can securely query a CA service asking for<br>
> a triplet (private key, public key, SSH certificate) for the specific<br>
> hostname. It can then inject the triplet using ConfigDrive. I believe this<br>
> securely gets the private key into the instance.<br>
><br>
><br>
><br>
> I cannot figure out how to get the equivalent functionality without<br>
> modifying Nova compute and the boot process. Every solution I can think of<br>
> risks either exposing the private key or vulnerability to a MITM attack<br>
> during the signing process.<br>
><br>
><br>
><br>
> Your help is appreciated.<br>
><br>
><br>
><br>
> --Pino<br>
><br>
><br>
</div>
</div>
<div class="m_-3244657552731646408m_9041349648992642493HOEnZb">
<div class="m_-3244657552731646408m_9041349648992642493h5">> ______________________________<wbr>______________________________<wbr>______________<br>
> OpenStack Development Mailing List (not for usage questions)<br>
> Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
> <a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">
http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
><br>
<br>
______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">
OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div></div></div>
</div>
</div>
<br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.op<wbr>enstack.org?subject:unsubscrib<wbr>e</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi<wbr>-bin/mailman/listinfo/openstac<wbr>k-dev</a><br>
<br></blockquote></div><br></div>
</div></div><br>______________________________<wbr>______________________________<wbr>______________<br>
OpenStack Development Mailing List (not for usage questions)<br>
Unsubscribe: <a href="http://OpenStack-dev-request@lists.openstack.org?subject:unsubscribe" rel="noreferrer" target="_blank">OpenStack-dev-request@lists.<wbr>openstack.org?subject:<wbr>unsubscribe</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev" rel="noreferrer" target="_blank">http://lists.openstack.org/<wbr>cgi-bin/mailman/listinfo/<wbr>openstack-dev</a><br>
<br></blockquote></div><br></div>