Thank you to everyone that has participated in this discussion. My hope was that the community could have a discussion around the issues and come up with some ideas beyond the "single maintainer" or "out of OpenStack" specified in the etherpad[1]. As I mentioned, and was highlighted by a few responses, removing additional maintainers from the projects doesn't fully solve the problem for our community. In fact it raises some new problems that were not previously discussed. Let me summarize some of the issues/proposals for easier comment/discussion (please add/correct if I missed something): 1. Issue: There is currently no way to validate packages on PyPi (tar or wheel) are official OpenStack packages. a) PEP 458[2][3] - TUF for PyPi/pip does not exist yet. Development has slowed. b) PyPi has removed the capability to associate signatures with packages(beyond uploading an asc file) and does not display them in any way via the web UI. c) Some packagers include the pgp signature as ".asc" files associated with each package file uploaded. i) OpenStack does not do this today. Currently OpenStack only publishes pgp signatures on tarballs.opendev.org. ii) The tooling for working with PyPi doesn't use these signature files, so they are of minimal value. d) Question raised: Should we as the OpenStack community get involved in the PEP458 development to help move this forward? e) Question raised: Should we stop using PyPi as it does not provide software traceability? PyPi has had a lot of press recently about issues with look-alike packes, etc.[4][5][6] f) Question raised: Should we develop some OpenStack tooling that provides the required packaged validation? Either via pulling signatures from tarballs.o.o or pulling OpenStack packages from tarballs.o.o directly. g) Question raised: Should we use lockfiles (pipfile/pipfile.lock)[4] to provide hash validation of packages? h) Question raised: Should we start providing hashes in requirements.txt to allow "hash-checking mode"[5]? 2. Issue: There is no documentation for how to add a service to OpenStack on the OpenStack website any longer and no policy on what it means for artifacts transitioning to OpenStack management. a) Historically not all OpenStack services were required to be managed via the release team process. It was not a problem for PTLs to push tags and publish releases via alternate processes. b) Question raised: Should this change and all OpenStack services be required to use the release process? c) Question raised: Should we clearly call out all associated services (PyPi, github, snyk, twitter, grafana, docker hub, quay, etc.) will become solely managed by OpenStack? d) Question raised: Should we bring back a link to the "project creator" section of the docs.opendev.org or do we need an OpenStack specific guide? 3. Issue: Current and historical documentation[9] for project teams management of PyPi projects has stated that "openstackci" should not have "Owner" access to the project, just "Maintainer". a) The "Maintainer" role allows "openstackci" to publish packages.[10[ b) The "Maintainer" role does not allow the deletion of packages, the project, or the removal of other maintainers.[10] c) Question raised: Does "openstackci" even have the required permission to remove the other maintainers? d) Question raised: Do we need to create/correct proper documentation for project management on PyPi? 4. Issue: Removing maintainers other than "openstackci" creates a single point of management. a) Resources available in OpenStack are declining and response times for infrastructure issues are rising[11][12]. b) Even with current openstackci access, many of these PyPi projects have not been maintained by "openstackci" resources, but at least some have been maintained by the other maintainers. i) Some projects still have "openstackjenkins" maintainers[13] that were never cleaned up. ii) Have all projects with "openstackci" access been updated to require 2FA? (I turned this on for the Octavia projects I maintain long ago) iii) The issue with another group having access to the project highlights that we might not have resources to actively manage all of these projects[14]. c) If (when if you are a security person) the "openstackci" account is compromised (insider or other), we would be locked out of all of these projects. d) In OpenStack we distribute the work and responsibility to trusted parties (PTLs, core reviews, liaisons, etc.), moving to a single account for PyPi would not allow this pattern. e) Question raised: Could we provide an option 3 where we leave access for active maintainers if the PTL approves? f) Question raised: Should we have secondary accounts for services like this that are held in escrow in case we have a compromise of the primary account? g) Question raised: Can we setup an automated system where the current PTL can get access to the project on PyPi for a period of time? h) Why isn't the release/"openstackci" account a lower permission account used for the release process only instead of a "root" style account? (related to question "f") above. i) The good news is, even with the above issues, someone did see a notification of a maintainer list change to one of these projects. We don't get notifications of releases, but we do get notifications of maintainer list changes. 5. Issue: We have no audit checks in place to identify rogue releases on PyPi. a) Even with the change to only allowing "openstackci" as a maintainer, a compromised "openstackci" or PyPi could still release compromised packages. b) Proposal: Update the existing release team audit script to also audit the PyPi packages for unexpected packages (tar, wheel, etc.). There are a lot of questions here to work through and I hope they are a topic for future TC/community discussion. Here is what I would do: 1. Give the PTLs the option to leave some maintainers on the packages as appropriate until we have additional infrastructure in place. Remove the other "maintainers/owners" from the projects. 2. Immediately audit the "openstackci" access to see what level of permission we have on the projects. We may need to find people or escalate with the PyPi admins to fix this. It's not a simple "maintainer" or not. 3. Create OpenStack documentation for how access to PyPi projects is expected to work (either in the release team docs, or a new section to replace the project creator guide). If we want to use the opendev.org docs, we need to relink those into the OpenStack docs and correct the issues with the instructions. 4. We should create another account in all of the OpenStack managed projects that will assume the "owner" role. Access to this account should be separate (escrowed?) from the other maintainer accounts. 5. The "openstackci" accounts should all be dropped to "maintainer" level permissions, allowing releases, but no destructive actions. 6. Create a documented action plan for who/how a package can be pulled from PyPi in the case of a compromised package was released or the automated release process had a failure. 7. Update the release team audit script to also check PyPi for rogue releases, preferably by also checking the hash. This may uncover existing problem releases we do not know about yet. 8. Automate the above audit script to run regularly or document a schedule it will be run by the release team. 9. Audit that all of the OpenStack projects on PyPi have 2FA required. This is a per-project setting on PyPi. 10. Start work on developing an automated system where PTLs can get owner level access, for a period of time, to address the issues the individual maintainers have done historically. This would at least provide an audit log and allow us to spread the work. 11. Form a task team to evaluate if we should get involved in the PEP 458 work or if using one of the hash validation tools would be feasible as part of our requirements process. Michael [1] https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L17 [2] https://peps.python.org/pep-0458/ [3] https://github.com/pypi/warehouse/issues/10672 [4] https://thenewstack.io/poisoned-lolip0p-pypi-packages/ [5] https://www.theregister.com/2023/01/09/pypi_aws_malware_key/ [6] https://www.securityweek.com/security-firms-find-over-20-malicious-pypi-pack... [7] https://pypi.org/project/pipfile/ [8] https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode [9] https://docs.opendev.org/opendev/infra-manual/latest/creators.html#give-open... [10] https://pypi.org/help/#collaborator-roles [11] https://review.opendev.org/c/opendev/system-config/+/795596 [12] https://review.opendev.org/c/openinfra/openstack-map/+/840774 [13] https://pypi.org/user/openstackjenkins/ [14] https://github.com/openstack/xstatic-font-awesome/pull/2 On Thu, Jan 26, 2023 at 10:31 AM Radosław Piliszek <radoslaw.piliszek@gmail.com> wrote:
I agree with Jay's summary below wholeheartedly.
It is good that this issue has also highlighted other issues with the current maintainers&owners lists. The bottom line anyways is that the current state of affairs *must* change and a policy be created and kept in check going forward.
Radek -yoctozepto
On Thu, 26 Jan 2023 at 18:53, Jay Faulkner <jay@gr-oss.io> wrote:
I'm going to be honest, I'm a little frustrated that this isn't being treated as a severe security issue; because it is. Regardless of what people anecdotally say, there are a large number of operators who are installing and running openstack from pypi. Even larger still are developers -- like you and I -- installing openstack packages as our user anytime we run tox tests.
I'm not personally worried too much about the cases where an existing core or PTL is listed. I'm happy to have a policy argument in that context. That's not what's happening in the vast majority of these cases. For Ironic, of the 18 pypi artifacts we have; 14 have maintainers who are not currently core reviewers or openstack contributors on them -- and a handful of them have maintainers who I personally have never even heard of (we even have a `login.launchpad.net_154` as a maintainer on IPA). At any point in time, one of these people could[1] upload a malicious version of the package up to pypi and compromise any one of us who are unlucky enough to run `tox -r` locally before it's caught and rolled back.
I don't think packages that are managed by the OpenStack governance process should ever have a backdoor open to humans of any kind; but that's not the case we're arguing here. Right now, we have a large group of untrusted people with the ability to compromise our developers and users if they are not securing their pypi accounts well. This is especially true in an open source climate that is becoming more and more focused on supply chain risks. That is where my focus lies, and voting to remove those erroneous maintainers hastily is exactly the sort of decision I'd expect to be made by the Technical Committee -- even before I was elected to it (and if you disagree; I'm happy to only serve one term). We have to protect our users and our contributors.
Our focus must remain on closing this security hole. It does raise policy questions, but right now I'd rather us put out the fire rather than arguing about fire prevention tactics.
Thanks, Jay Faulkner TC Member Ironic PTL
Footnote: 1 - Arguably this has already happened once, in the case of xstatic-font-awesome -- we were just lucky that it was a good-faith release and not an actual compromised.