[release][infra] Supporting rget in our release process
Hi, A colleague at Red Hat is working on an effort to record signatures of release artifacts. Essentially it's a way to help users verify release artifacts (or determine if they have been changed) independent of PGP signatures. You can read about it here: https://github.com/merklecounty/rget#rget It sounds like an interesting and useful effort, and I think we can support it at little cost. If we wanted to do so, I think we would need to do the following things: 1) Generate SHA256SUMS of our release artifacts. These could even include the GPG signature files. 2) Run "rget submit" on the resulting files after publication. That's it. Both of those would be changes to the release publication jobs, and wouldn't require any other changes to our processes. As mentioned in the README this is very early stages and the author, Brandon Philips, welcomes both further testing and feedback on the process in general. Thoughts? -Jim
On Mon, Jul 29, 2019, at 1:52 PM, James E. Blair wrote:
Hi,
A colleague at Red Hat is working on an effort to record signatures of release artifacts. Essentially it's a way to help users verify release artifacts (or determine if they have been changed) independent of PGP signatures. You can read about it here: https://github.com/merklecounty/rget#rget
It sounds like an interesting and useful effort, and I think we can support it at little cost. If we wanted to do so, I think we would need to do the following things:
1) Generate SHA256SUMS of our release artifacts. These could even include the GPG signature files.
We'll also need to publish the sha256sums file. We are already publishing the other files so this should be easy.
2) Run "rget submit" on the resulting files after publication.
That's it.
There is also a step 0) install rget. Unfortunately their docs don't mention how to verify the installation of rget (self bootstrapping) though I think you would download rget, hash it before running it, then run it to get the hashes of rget and then compare? Though a modified rget could just tell you it was the unmodified version. Maybe that is why they don't bother telling you what to do there. We may also want to set up some sort of periodic audit of our "certs" in the certificate transparency logs. Just to ensure there are no unexpected changes.
Both of those would be changes to the release publication jobs, and wouldn't require any other changes to our processes.
As mentioned in the README this is very early stages and the author, Brandon Philips, welcomes both further testing and feedback on the process in general.
Thoughts?
Overall seems like a good way for people (including ourselves) to sanity check that our releases are not changing unexpectedly.
-Jim
On 7/29/19 5:52 PM, Clark Boylan wrote:
On Mon, Jul 29, 2019, at 1:52 PM, James E. Blair wrote:
Hi,
A colleague at Red Hat is working on an effort to record signatures of release artifacts. Essentially it's a way to help users verify release artifacts (or determine if they have been changed) independent of PGP signatures. You can read about it here: https://github.com/merklecounty/rget#rget
It sounds like an interesting and useful effort, and I think we can support it at little cost. If we wanted to do so, I think we would need to do the following things:
1) Generate SHA256SUMS of our release artifacts. These could even include the GPG signature files.
We'll also need to publish the sha256sums file. We are already publishing the other files so this should be easy.
2) Run "rget submit" on the resulting files after publication.
That's it.
There is also a step 0) install rget. Unfortunately their docs don't mention how to verify the installation of rget (self bootstrapping) though I think you would download rget, hash it before running it, then run it to get the hashes of rget and then compare? Though a modified rget could just tell you it was the unmodified version. Maybe that is why they don't bother telling you what to do there. Actually - it turns out this is just doing a single submit to a URL, so it will likely be much easier to just use curl or an ansible URI call to do the submission step. (it's great that rget has this for folks, but in our case I think we don't need to use rget just to do the submission)
We may also want to set up some sort of periodic audit of our "certs" in the certificate transparency logs. Just to ensure there are no unexpected changes.
++
Both of those would be changes to the release publication jobs, and wouldn't require any other changes to our processes.
As mentioned in the README this is very early stages and the author, Brandon Philips, welcomes both further testing and feedback on the process in general.
Thoughts?
Overall seems like a good way for people (including ourselves) to sanity check that our releases are not changing unexpectedly.
-Jim
On 2019-07-29 13:52:20 -0700 (-0700), James E. Blair wrote:
A colleague at Red Hat is working on an effort to record signatures of release artifacts. Essentially it's a way to help users verify release artifacts (or determine if they have been changed) independent of PGP signatures. You can read about it here: https://github.com/merklecounty/rget#rget [...] As mentioned in the README this is very early stages and the author, Brandon Philips, welcomes both further testing and feedback on the process in general.
Thoughts?
I really like the way it leverages RFC 6962 Certificate Transparency logs for checksum distribution; the decision not to fall into the blockchain-all-things trap lends a lot of additional credibility to the idea. I agree this would be fairly trivial to integrate into our release publication jobs, and even to backfill with our existing archives. It would grant consumers of our release artifacts the ability to validate them against an open third-party registry, separately from checking the cryptographic release signatures we currently provide alongside them... it could even be used to detect tampering with the signatures themselves in the event of a signing key compromise. This seems like a great idea for URLs of artifacts we host, and I'm happy to hack on the implementation in OpenDev's CI system, likely via a new role in Zuul's standard library of job components. For artifacts we upload to third-party services like PyPI and Docker Hub on the other hand, assuming I've digested (pun intended) the relevant literature correctly, it might make more sense for the maintainers of those services to do something similar as they tend to perform a fair amount of URL indirection and so trying to keep up historical data for those URLs ourselves could be tricky. On the other hand if those third-party services were to integrate rget updating as part of their infrastructure it would be a lot more seamless (especially if they similarly integrated CT checks into the corresponding client-side tooling). Another challenge I see is that, due to the fact that most of what we host is source code, and most consumers of our source code are obtaining it via Git rather than release artifacts, rget wouldn't really do much for them as far as I can see... though once Git completes its planned transition to SHA2-256 in the coming years, I could see a call for some solution to publish known object hashes to a CT log in a similar fashion. I suppose it could be done now by re-checksumming all content over a Git object and submitting a certificate for that, but it seems a bit heavy-weight and I'll admit to not having thought through it completely so there are likely hidden gotchas with that idea. -- Jeremy Stanley
Jeremy Stanley wrote:
[...] For artifacts we upload to third-party services like PyPI and Docker Hub on the other hand, assuming I've digested (pun intended) the relevant literature correctly, it might make more sense for the maintainers of those services to do something similar as they tend to perform a fair amount of URL indirection and so trying to keep up historical data for those URLs ourselves could be tricky. On the other hand if those third-party services were to integrate rget updating as part of their infrastructure it would be a lot more seamless (especially if they similarly integrated CT checks into the corresponding client-side tooling).
Another challenge I see is that, due to the fact that most of what we host is source code, and most consumers of our source code are obtaining it via Git rather than release artifacts, rget wouldn't really do much for them as far as I can see... though once Git completes its planned transition to SHA2-256 in the coming years, I could see a call for some solution to publish known object hashes to a CT log in a similar fashion. I suppose it could be done now by re-checksumming all content over a Git object and submitting a certificate for that, but it seems a bit heavy-weight and I'll admit to not having thought through it completely so there are likely hidden gotchas with that idea.
I agree with Jeremy, it seems to cover a limited amount of use cases (people who download tarball source releases from releases.o.o). But covering only a few use cases is not a reason not to do it: we should support it for the same reason we provide signatures for released artifacts right now. Furthermore it is an initiative I'm fine being early adopters of this idea, if only so that one day we may find it covering other ways to retrieve our deliverables (pypi, git). -- Thierry Carrez (ttx)
participants (5)
-
Clark Boylan
-
corvus＠inaugust.com
-
Jeremy Stanley
-
Monty Taylor
-
Thierry Carrez