October 2020 - openstack-discuss - lists.openstack.org

Re: [service-announce] October 20 Gerrit Outage Update
by Slawek Kaplonski 21 Oct '20

21 Oct '20

Hi, Hi, Both Lajos and me checked today all neutron and neutron stadium changes there. And all looks ok for us. I didn't notice anything wrong or suspicious there. Thx Lajos for help with that :) Dnia środa, 21 października 2020 09:49:49 CEST Slawek Kaplonski pisze: > On Wed, Oct 21, 2020 at 11:33:14AM +1100, Ian Wienand wrote: > > As of this mail, Gerrit access has been restored. Please read on for > > important information, especially around change verification. > > > > Background > > ----------- > > > > On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval > > to a change that they were not expected to have access to. At 02:06 > > UTC an alert was raised via IRC. Administrators found the account had > > added themselves to a core group and made the +W vote. The account > > was disabled, and removed from the groups it had added itself to by > > 02:55 UTC. Administrators began to analyse the situation and Gerrit > > was taken offline at 04:02 UTC to preserve state and allow for > > analysis. > > > > From this time, administrators were working on log collection and > > analysis, along with restoring backups for comparison purposes. > > > > By around 08:45 UTC it was clear that the privilege escalation had > > been achieved by gaining control of a Launchpad SSO account with > > Gerrit administrator privileges. By this time, we had ruled out > > software vulnerabilities. Logs showed the first unauthorized access > > of the administrator account in Gerrit on 2020-10-06. Communication > > with Launchpad admins agrees with this analysis. We saw one session > > opened as the administrator user to StoryBoard on this same day, but > > logs show no data was modified or hidden stories viewed. > > > > Analysis has been performed on the Gerrit database and git trees from > > October 1st, pre-dating any known unauthorized access. > > > > Access was restored at around 2020-10-21 00:00 UTC > > > > Outcomes > > ----------- > > > > The following has been verified: > > The administrator account used has been disabled and credentials > > updated > > > > We have verified that all group and user addition/removals since > > Oct 1 are valid. The only invalid additions were made by the > > compromised administrator account to add a single user account to > > the Administrators group; and then that account added itself to > > another known group. > > > > The account given administrator privilege has been removed from > > the groups it added itself to and is disabled. > > > > There is no evidence of any unauthorized access via methods other > > than Gerrit HTTP and Gerrit SSH access. > > > > No commits have been pushed to git trees bypassing code review. > > Every git tree has been compared to the Oct 1 version and all > > commits have been correctly inserted via Gerrit changes. > > > > The version of Gerrit we use stores HTTP API passwords in > > plain-text. We know that a limited number of passwords were > > gathered via the HTTP API and it is possible passwords were > > gathered via the database. We thus have assumed that all HTTP API > > passwords have been disclosed. This password needs to be > > explicitly enabled by users, and many users do not have it > > enabled. > > > > Remediation > > ----------- > > > > This leaves us with the following remediation actions: > > Users should double-check their Launchpad recent activity at > > https://login.launchpad.net/activity for any suspicious logins. If > > found, please notify the OpenDev admins in Freenode #opendev and > > Launchpad admins in #launchpad immediately. > > > > All HTTP API passwords have been cleared. If you push changes via > > HTTPS (instead of typical SSH), are a gertty user, or run a CI > > system or something else that communicates with the Gerrit HTTP > > API, you will need to regenerate a password. > > > > Any SSH keys added to accounts since 2020-10-01 have been removed. > > This affects only a limited number of accounts. This is done in > > an abundance of caution, and we do not believe any accounts had > > unauthorized SSH keys added > > > > We should audit all changes for projects since 2020-10-01. > > > > We have no evidence that any account had its ssh keys compromised, > > thus we can rule out any unauthorized changes being uploaded via SSH. > > However we can not conclusively rule out that compromised HTTP API > > passwords were used to push a change through Gerrit. For example, a > > change could be uploaded that looks like it came from a user, or the > > API key of a core team member may have been used to approve a change > > without authorization. > > > > Given our extensive analysis we consider it exceedingly unlikely that > > this vector was used. We have had no notifications of users seeing > > unexpected changes either uploaded by them, or approved by them in > > projects they work on. This said, we believe it is important to > > inform the community of this very unlikely, but still possible, > > vulnerability of the source code. > > > > To this end, we have prepared a list of all changes from the known > > affected period which should be audited for correctness. These are > > available at > > > > https://static.opendev.org/project/opendev.org/gerrit-diffs/ > > > > Team members should browse these changes and make sure they were > > correctly approved in Gerrit. If any change looks suspicious you > > should notify OpenDev administrators in Freenode #opendev immediately. > > > > Further actions > > ---------------- > > > > We are planning the following for the short term future: > > The Opendev administrators will be looking at alternative models > > for Gerrit admin account management. > > > > We are already well into planning and testing a coming upgrade to > > a version of Gerrit which does not store plain-text API keys. > > > > Longer term, we've written a spec for replacing Launchpad SSO as > > our authentication provider. > > > > We thank you for your patience during this trying time, and we look > > forward to returning to supporting the community doing what it does > > best -- working together to create great things. > > > > > > _______________________________________________ > > service-announce mailing list > > service-announce(a)lists.opendev.org > > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce -- Slawek Kaplonski Principal Software Engineer Red Hat

1 0

[storlets] Gerrit breach and commit audit
by Takashi Kajinami 21 Oct '20

21 Oct '20

Hi, As suggested by the recent announcement[1], I have audited all commits (actually, 1 commit) in storlets repository, and confirmed that everything looks good now. [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html Thank you, Takashi

1 0

[masakari] Branches reviewed post-gerrit-breach
by Radosław Piliszek 21 Oct '20

21 Oct '20

Hi Folks, I have reviewed Masakari repos and no suspicious commits were found. Thank you, OpenDev team, for handling the situation and providing patches list. -yoctozepto

1 0

[cloudkitty] Gerrit breach and commit audit
by Pierre Riteau 21 Oct '20

21 Oct '20

Hello, Following the announcement of the Gerrit breach [1], I have audited all commits in repositories of the CloudKitty project. Everything looks good. Thank you to the OpenDev admins for dealing with this issue efficiently. Pierre [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

[blazar] Gerrit breach and commit audit
by Pierre Riteau 21 Oct '20

21 Oct '20

Hello, Following the announcement of the Gerrit breach [1], I have audited all commits in repositories of the Blazar project. Everything looks good. Thank you to the OpenDev admins for dealing with this issue efficiently. Pierre [1] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

1 0

Openstack client can't load modules
by Cristina Mayo 21 Oct '20

21 Oct '20

Hi everyone, Openstack client installed on an Ubuntu 20.04 is showing these errors when I launch as non root user; indeed it is fine with sudo: Could not load 'metric_metric_create': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricCreate' Could not load 'metric_metric_delete': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricDelete' Could not load 'metric_metric_list': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricList' Could not load 'metric_metric_show': module 'gnocchiclient.v1.metric_cli' has no attribute 'DeprecatedCliMetricShow' I don't have noticed any bad behaviour using it, but I'm not sure. Any help? Thanks

3 2

[barbican] gerrit breach and auditing
by Moises Guimaraes de Medeiros 21 Oct '20

21 Oct '20

Hi all, I'm putting up a review spreadsheet[0] for the barbican changes since October 1st based on Julia's spreadsheet for Ironic[1]. [0]: https://docs.google.com/spreadsheets/d/1WdhxDsAZCHlzhRpcH0jKfyY2kzO2h2jRzni… [1]: http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018148.… -- Moisés Guimarães Software Engineer Red Hat <https://www.redhat.com> <https://red.ht/sig>

1 0

[glance] Wallaby PTG schedule
by Abhishek Kekane 21 Oct '20

21 Oct '20

Hello All, Greetings!!! Wallaby PTG is around the corner and if you haven't already registered, please do so as soon as possible [1]. I have created a Virtual PTG planning etherpad [2] and also added day wise topics along with timings we are going to discuss. Kindly let me know if you have any concerns with allotted time slots. We also have some slots open on Thursday and Friday for unplanned discussions. So please feel free to add your topics if you still haven't added yet. As a reminder, these are the time slots for our discussion. Tuesday 27 October 2020 1400 UTC to 1700 UTC Folsom Wednesday 28 October 2020 1400 UTC to 1700 UTC Folsom Thursday 29 October 2020 1400 UTC to 1700 UTC Folsom Friday 30 October 2020 1400 UTC to 1700 UTC Folsom We will be using bluejeans for our discussion, kindly try to use it once before the actual discussion. The meeting URL is mentioned in etherpad [2] and will be the same throughout the PTG. [1] https://october2020ptg.eventbrite.com/ [2] https://etherpad.opendev.org/p/glance-wallaby-ptg Thank you, Abhishek

1 0

[kolla] Branches reviewed post-gerrit-breach
by Radosław Piliszek 21 Oct '20

21 Oct '20

Hi Folks, The Kolla team has reviewed its projects and no suspicious commits were found. Thank you, OpenDev team, for handling the situation and providing patches list. Thank you, Mark, Michał, Marcin, Pierre and myself for reviewing. Let's roll with development. :-) -yoctozepto

1 0

[release][core] gerrit breach and auditing all releases deliverables [1] since Oct 01
by Herve Beraud 21 Oct '20

21 Oct '20

Hello, As every team we are also concerned by the gerrit breach and we must take a look at our changes during this time frame on all our deliverables [1]. The majority of our changes during this period are changes related to releasing new deliverables, that mean changes on yaml files. Due to the criticality of the problem I propose the following action plan: - first, focus us on changes against our scripts and executable files; - second, inspect yaml files if teams identify possible issues in their code base. I don't think we need to inspect our `deliverable/*/*.yaml` files for now, it could be done later; - third, inspect documentation changes. They could be easily ignored for now and analyzed later too. I wrote a script [2][3] to help us to extract relevant changes (*.py, *.sh), all the rest ((deliverable|data)*.yaml, *.rst) have been ignored for now. I ran this script against the repos that we own, here is the result: - openstack/release-test - No changes merged since more than 3 months - openstack/releases - https://review.opendev.org/#/q/9e527f9e0e0cac29699171813b45c65b487010dc - https://review.opendev.org/#/q/9c4f3c43039fa686cee6d03fb6d9894de05c89a6 - https://review.opendev.org/#/q/a8ab91eebc7a0b9d0be863772ea286627e6b57d9 - openstack/reno - No changes merged since more than 2 months - openstack/specs-cookiecutter - No changes merged since more than 4 months Example of script usage against our openstack/releases repos: ``` $ cd releases $ curl https://gist.githubusercontent.com/4383/511359cc2080e06295944c5f40bd1033/ra… | sh ``` I'll take a look at the identified commits but double checking would be greatly appreciated. Please ensure to double check your account activity [4] and make sure nothing is off. Special congrats to Julia Kreger and for her excellent job [5]. Thank you in advance for your help, [1] https://governance.openstack.org/tc/reference/projects/release-management.h… [2] https://gist.github.com/4383/511359cc2080e06295944c5f40bd1033 [3] https://gist.githubusercontent.com/4383/511359cc2080e06295944c5f40bd1033/ra… [4] http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html [5] http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018148.… -- Hervé Beraud Senior Software Engineer Red Hat - Openstack Oslo irc: hberaud -----BEGIN PGP SIGNATURE----- wsFcBAABCAAQBQJb4AwCCRAHwXRBNkGNegAALSkQAHrotwCiL3VMwDR0vcja10Q+ Kf31yCutl5bAlS7tOKpPQ9XN4oC0ZSThyNNFVrg8ail0SczHXsC4rOrsPblgGRN+ RQLoCm2eO1AkB0ubCYLaq0XqSaO+Uk81QxAPkyPCEGT6SRxXr2lhADK0T86kBnMP F8RvGolu3EFjlqCVgeOZaR51PqwUlEhZXZuuNKrWZXg/oRiY4811GmnvzmUhgK5G 5+f8mUg74hfjDbR2VhjTeaLKp0PhskjOIKY3vqHXofLuaqFDD+WrAy/NgDGvN22g glGfj472T3xyHnUzM8ILgAGSghfzZF5Skj2qEeci9cB6K3Hm3osj+PbvfsXE/7Kw m/xtm+FjnaywZEv54uCmVIzQsRIm1qJscu20Qw6Q0UiPpDFqD7O6tWSRKdX11UTZ hwVQTMh9AKQDBEh2W9nnFi9kzSSNu4OQ1dRMcYHWfd9BEkccezxHwUM4Xyov5Fe0 qnbfzTB1tYkjU78loMWFaLa00ftSxP/DtQ//iYVyfVNfcCwfDszXLOqlkvGmY1/Y F1ON0ONekDZkGJsDoS6QdiUSn8RZ2mHArGEWMV00EV5DCIbCXRvywXV43ckx8Z+3 B8qUJhBqJ8RS2F+vTs3DTaXqcktgJ4UkhYC2c1gImcPRyGrK9VY0sCT+1iA+wp/O v6rDpkeNksZ9fFSyoY2o =ECSj -----END PGP SIGNATURE-----

2 1