Open Stack

Thank you to everyone that has participated in this discussion. My
hope was that the community could have a discussion around the issues
and come up with some ideas beyond the "single maintainer" or "out of
OpenStack" specified in the etherpad[1].

As I mentioned, and was highlighted by a few responses, removing
additional maintainers from the projects doesn't fully solve the
problem for our community. In fact it raises some new problems that
were not previously discussed.

Let me summarize some of the issues/proposals for easier
comment/discussion (please add/correct if I missed something):

1. Issue: There is currently no way to validate packages on PyPi (tar
or wheel) are official OpenStack packages.
  a) PEP 458[2][3] - TUF for PyPi/pip does not exist yet. Development
has slowed.
  b) PyPi has removed the capability to associate signatures with
packages(beyond uploading an asc file) and does not display them in
any way via the web UI.
  c) Some packagers include the pgp signature as ".asc" files
associated with each package file uploaded.
    i) OpenStack does not do this today. Currently OpenStack only
publishes pgp signatures on tarballs.opendev.org.
    ii) The tooling for working with PyPi doesn't use these signature
files, so they are of minimal value.
  d) Question raised: Should we as the OpenStack community get
involved in the PEP458 development to help move this forward?
  e) Question raised: Should we stop using PyPi as it does not provide
software traceability? PyPi has had a lot of press recently about
issues with look-alike packes, etc.[4][5][6]
  f) Question raised: Should we develop some OpenStack tooling that
provides the required packaged validation? Either via pulling
signatures from tarballs.o.o or pulling OpenStack packages from
tarballs.o.o directly.
  g) Question raised: Should we use lockfiles
(pipfile/pipfile.lock)[4] to provide hash validation of packages?
  h) Question raised: Should we start providing hashes in
requirements.txt to allow "hash-checking mode"[5]?

2. Issue: There is no documentation for how to add a service to
OpenStack on the OpenStack website any longer and no policy on what it
means for artifacts transitioning to OpenStack management.
  a) Historically not all OpenStack services were required to be
managed via the release team process. It was not a problem for PTLs to
push tags and publish releases via alternate processes.
  b) Question raised: Should this change and all OpenStack services be
required to use the release process?
  c) Question raised: Should we clearly call out all associated
services (PyPi, github, snyk, twitter, grafana, docker hub, quay,
etc.) will become solely managed by OpenStack?
  d) Question raised: Should we bring back a link to the "project
creator" section of the docs.opendev.org or do we need an OpenStack
specific guide?

3. Issue: Current and historical documentation[9] for project teams
management of PyPi projects has stated that "openstackci" should not
have "Owner" access to the project, just "Maintainer".
  a) The "Maintainer" role allows "openstackci" to publish packages.[10[
  b) The "Maintainer" role does not allow the deletion of packages,
the project, or the removal of other maintainers.[10]
  c) Question raised: Does "openstackci" even have the required
permission to remove the other maintainers?
  d) Question raised: Do we need to create/correct proper
documentation for project management on PyPi?

4. Issue: Removing maintainers other than "openstackci" creates a
single point of management.
  a) Resources available in OpenStack are declining and response times
for infrastructure issues are rising[11][12].
  b) Even with current openstackci access, many of these PyPi projects
have not been maintained by "openstackci" resources, but at least some
have been maintained by the other maintainers.
    i) Some projects still have "openstackjenkins" maintainers[13]
that were never cleaned up.
    ii) Have all projects with "openstackci" access been updated to
require 2FA? (I turned this on for the Octavia projects I maintain
long ago)
    iii) The issue with another group having access to the project
highlights that we might not have resources to actively manage all of
these projects[14].
  c) If (when if you are a security person) the "openstackci" account
is compromised (insider or other), we would be locked out of all of
these projects.
  d) In OpenStack we distribute the work and responsibility to trusted
parties (PTLs, core reviews, liaisons, etc.), moving to a single
account for PyPi would not allow this pattern.
  e) Question raised: Could we provide an option 3 where we leave
access for active maintainers if the PTL approves?
  f) Question raised: Should we have secondary accounts for services
like this that are held in escrow in case we have a compromise of the
primary account?
  g) Question raised: Can we setup an automated system where the
current PTL can get access to the project on PyPi for a period of
time?
  h) Why isn't the release/"openstackci" account a lower permission
account used for the release process only instead of a "root" style
account? (related to question "f") above.
  i) The good news is, even with the above issues, someone did see a
notification of a maintainer list change to one of these projects. We
don't get notifications of releases, but we do get notifications of
maintainer list changes.

5. Issue: We have no audit checks in place to identify rogue releases on PyPi.
  a) Even with the change to only allowing "openstackci" as a
maintainer, a compromised "openstackci" or PyPi could still release
compromised packages.
  b) Proposal: Update the existing release team audit script to also
audit the PyPi packages for unexpected packages (tar, wheel, etc.).

There are a lot of questions here to work through and I hope they are
a topic for future TC/community discussion.

Here is what I would do:
1. Give the PTLs the option to leave some maintainers on the packages
as appropriate until we have additional infrastructure in place.
Remove the other "maintainers/owners" from the projects.
2. Immediately audit the "openstackci" access to see what level of
permission we have on the projects. We may need to find people or
escalate with the PyPi admins to fix this. It's not a simple
"maintainer" or not.
3. Create OpenStack documentation for how access to PyPi projects is
expected to work (either in the release team docs, or a new section to
replace the project creator guide). If we want to use the opendev.org
docs, we need to relink those into the OpenStack docs and correct the
issues with the instructions.
4. We should create another account in all of the OpenStack managed
projects that will assume the "owner" role. Access to this account
should be separate (escrowed?) from the other maintainer accounts.
5. The "openstackci" accounts should all be dropped to "maintainer"
level permissions, allowing releases, but no destructive actions.
6. Create a documented action plan for who/how a package can be pulled
from PyPi in the case of a compromised package was released or the
automated release process had a failure.
7. Update the release team audit script to also check PyPi for rogue
releases, preferably by also checking the hash. This may uncover
existing problem releases we do not know about yet.
8. Automate the above audit script to run regularly or document a
schedule it will be run by the release team.
9. Audit that all of the OpenStack projects on PyPi have 2FA required.
This is a per-project setting on PyPi.
10. Start work on developing an automated system where PTLs can get
owner level access, for a period of time, to address the issues the
individual maintainers have done historically. This would at least
provide an audit log and allow us to spread the work.
11. Form a task team to evaluate if we should get involved in the PEP
458 work or if using one of the hash validation tools would be
feasible as part of our requirements process.

Michael

[1] https://etherpad.opendev.org/p/openstack-pypi-maintainers-cleanup#L17
[2] https://peps.python.org/pep-0458/
[3] https://github.com/pypi/warehouse/issues/10672
[4] https://thenewstack.io/poisoned-lolip0p-pypi-packages/
[5] https://www.theregister.com/2023/01/09/pypi_aws_malware_key/
[6] https://www.securityweek.com/security-firms-find-over-20-malicious-pypi-packages-designed-data-theft/
[7] https://pypi.org/project/pipfile/
[8] https://pip.pypa.io/en/stable/topics/secure-installs/#hash-checking-mode
[9] https://docs.opendev.org/opendev/infra-manual/latest/creators.html#give-opendev-permission-to-publish-releases
[10] https://pypi.org/help/#collaborator-roles
[11] https://review.opendev.org/c/opendev/system-config/+/795596
[12] https://review.opendev.org/c/openinfra/openstack-map/+/840774
[13] https://pypi.org/user/openstackjenkins/
[14] https://github.com/openstack/xstatic-font-awesome/pull/2


On Thu, Jan 26, 2023 at 10:31 AM Radosław Piliszek
<radoslaw.piliszek at gmail.com> wrote:
>
> I agree with Jay's summary below wholeheartedly.
>
> It is good that this issue has also highlighted other issues with the
> current maintainers&owners lists. The bottom line anyways is that the
> current state of affairs *must* change and a policy be created and
> kept in check going forward.
>
> Radek
> -yoctozepto
>
> On Thu, 26 Jan 2023 at 18:53, Jay Faulkner <jay at gr-oss.io> wrote:
> >
> >
> >
> > I'm going to be honest, I'm a little frustrated that this isn't being treated as a severe security issue; because it is. Regardless of what people anecdotally say, there are a large number of operators who are installing and running openstack from pypi. Even larger still are developers -- like you and I -- installing openstack packages as our user anytime we run tox tests.
> >
> > I'm not personally worried too much about the cases where an existing core or PTL is listed. I'm happy to have a policy argument in that context. That's not what's happening in the vast majority of these cases. For Ironic, of the 18 pypi artifacts we have; 14 have maintainers who are not currently core reviewers or openstack contributors on them -- and a handful of them have maintainers who I personally have never even heard of (we even have a `login.launchpad.net_154` as a maintainer on IPA). At any point in time, one of these people could[1] upload a malicious version of the package up to pypi and compromise any one of us who are unlucky enough to run `tox -r` locally before it's caught and rolled back.
> >
> > I don't think packages that are managed by the OpenStack governance process should ever have a backdoor open to humans of any kind; but that's not the case we're arguing here. Right now, we have a large group of untrusted people with the ability to compromise our developers and users if they are not securing their pypi accounts well. This is especially true in an open source climate that is becoming more and more focused on supply chain risks. That is where my focus lies, and voting to remove those erroneous maintainers hastily is exactly the sort of decision I'd expect to be made by the Technical Committee -- even before I was elected to it (and if you disagree; I'm happy to only serve one term). We have to protect our users and our contributors.
> >
> > Our focus must remain on closing this security hole. It does raise policy questions, but right now I'd rather us put out the fire rather than arguing about fire prevention tactics.
> >
> > Thanks,
> > Jay Faulkner
> > TC Member
> > Ironic PTL
> >
> > Footnote:
> > 1 - Arguably this has already happened once, in the case of xstatic-font-awesome -- we were just lucky that it was a good-faith release and not an actual compromised.
>