<div dir="ltr"><div>Hello Zorillas, </div><div><br></div><div>I join the list of contributors to thank the infra team for their work in isolating, protecting and fixing the recent vulnerability. I looked through the code changes committed to repositories we maintain and didn't find any malicious code submissions or approvals. If you would like to take a look, here is an Ethercalc with the changes: <a href="https://ethercalc.openstack.org/uscp1x7i8ewn">https://ethercalc.openstack.org/uscp1x7i8ewn</a></div><div>Please do respond here in case you notice anything amiss.</div><div><br></div><div>Thank you!</div><div>Goutham</div><div><br></div><div><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">---------- Forwarded message ---------<br>From: <strong class="gmail_sendername" dir="auto">Ian Wienand</strong> <span dir="auto"><<a href="mailto:iwienand@redhat.com">iwienand@redhat.com</a>></span><br>Date: Tue, Oct 20, 2020 at 5:33 PM<br>Subject: [service-announce] October 20 Gerrit Outage Update<br>To: <<a href="mailto:service-announce@lists.opendev.org">service-announce@lists.opendev.org</a>><br></div><br><br>As of this mail, Gerrit access has been restored. Please read on for<br>
important information, especially around change verification.<br>
<br>
Background<br>
-----------<br>
<br>
On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval<br>
to a change that they were not expected to have access to. At 02:06<br>
UTC an alert was raised via IRC. Administrators found the account had<br>
added themselves to a core group and made the +W vote. The account<br>
was disabled, and removed from the groups it had added itself to by<br>
02:55 UTC. Administrators began to analyse the situation and Gerrit<br>
was taken offline at 04:02 UTC to preserve state and allow for<br>
analysis.<br>
<br>
>From this time, administrators were working on log collection and<br>
analysis, along with restoring backups for comparison purposes.<br>
<br>
By around 08:45 UTC it was clear that the privilege escalation had<br>
been achieved by gaining control of a Launchpad SSO account with<br>
Gerrit administrator privileges. By this time, we had ruled out<br>
software vulnerabilities. Logs showed the first unauthorized access<br>
of the administrator account in Gerrit on 2020-10-06. Communication<br>
with Launchpad admins agrees with this analysis. We saw one session<br>
opened as the administrator user to StoryBoard on this same day, but<br>
logs show no data was modified or hidden stories viewed.<br>
<br>
Analysis has been performed on the Gerrit database and git trees from<br>
October 1st, pre-dating any known unauthorized access.<br>
<br>
Access was restored at around 2020-10-21 00:00 UTC<br>
<br>
Outcomes<br>
-----------<br>
<br>
The following has been verified:<br>
<br>
The administrator account used has been disabled and credentials<br>
updated<br>
<br>
We have verified that all group and user addition/removals since<br>
Oct 1 are valid. The only invalid additions were made by the<br>
compromised administrator account to add a single user account to<br>
the Administrators group; and then that account added itself to<br>
another known group.<br>
<br>
The account given administrator privilege has been removed from<br>
the groups it added itself to and is disabled.<br>
<br>
There is no evidence of any unauthorized access via methods other<br>
than Gerrit HTTP and Gerrit SSH access.<br>
<br>
No commits have been pushed to git trees bypassing code review.<br>
Every git tree has been compared to the Oct 1 version and all<br>
commits have been correctly inserted via Gerrit changes.<br>
<br>
The version of Gerrit we use stores HTTP API passwords in<br>
plain-text. We know that a limited number of passwords were<br>
gathered via the HTTP API and it is possible passwords were<br>
gathered via the database. We thus have assumed that all HTTP API<br>
passwords have been disclosed. This password needs to be<br>
explicitly enabled by users, and many users do not have it<br>
enabled.<br>
<br>
Remediation<br>
-----------<br>
<br>
This leaves us with the following remediation actions:<br>
<br>
Users should double-check their Launchpad recent activity at<br>
<a href="https://login.launchpad.net/activity" rel="noreferrer" target="_blank">https://login.launchpad.net/activity</a> for any suspicious logins. If<br>
found, please notify the OpenDev admins in Freenode #opendev and<br>
Launchpad admins in #launchpad immediately.<br>
<br>
All HTTP API passwords have been cleared. If you push changes via<br>
HTTPS (instead of typical SSH), are a gertty user, or run a CI<br>
system or something else that communicates with the Gerrit HTTP<br>
API, you will need to regenerate a password.<br>
<br>
Any SSH keys added to accounts since 2020-10-01 have been removed.<br>
This affects only a limited number of accounts. This is done in<br>
an abundance of caution, and we do not believe any accounts had<br>
unauthorized SSH keys added<br>
<br>
We should audit all changes for projects since 2020-10-01.<br>
<br>
We have no evidence that any account had its ssh keys compromised,<br>
thus we can rule out any unauthorized changes being uploaded via SSH.<br>
However we can not conclusively rule out that compromised HTTP API<br>
passwords were used to push a change through Gerrit. For example, a<br>
change could be uploaded that looks like it came from a user, or the<br>
API key of a core team member may have been used to approve a change<br>
without authorization.<br>
<br>
Given our extensive analysis we consider it exceedingly unlikely that<br>
this vector was used. We have had no notifications of users seeing<br>
unexpected changes either uploaded by them, or approved by them in<br>
projects they work on. This said, we believe it is important to<br>
inform the community of this very unlikely, but still possible,<br>
vulnerability of the source code.<br>
<br>
To this end, we have prepared a list of all changes from the known<br>
affected period which should be audited for correctness. These are<br>
available at<br>
<br>
<a href="https://static.opendev.org/project/opendev.org/gerrit-diffs/" rel="noreferrer" target="_blank">https://static.opendev.org/project/opendev.org/gerrit-diffs/</a><br>
<br>
Team members should browse these changes and make sure they were<br>
correctly approved in Gerrit. If any change looks suspicious you<br>
should notify OpenDev administrators in Freenode #opendev immediately.<br>
<br>
Further actions<br>
----------------<br>
<br>
We are planning the following for the short term future:<br>
<br>
The Opendev administrators will be looking at alternative models<br>
for Gerrit admin account management.<br>
<br>
We are already well into planning and testing a coming upgrade to<br>
a version of Gerrit which does not store plain-text API keys.<br>
<br>
Longer term, we've written a spec for replacing Launchpad SSO as<br>
our authentication provider.<br>
<br>
We thank you for your patience during this trying time, and we look<br>
forward to returning to supporting the community doing what it does<br>
best -- working together to create great things.<br>
<br>
<br>
_______________________________________________<br>
service-announce mailing list<br>
<a href="mailto:service-announce@lists.opendev.org" target="_blank">service-announce@lists.opendev.org</a><br>
<a href="http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce" rel="noreferrer" target="_blank">http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce</a><br>
</div></div>