Open Stack

Hi,

Hi,

Both Lajos and me checked today all neutron and neutron
stadium changes there. And all looks ok for us. I didn't notice anything
wrong or suspicious there.
Thx Lajos for help with that :)

Dnia środa, 21 października 2020 09:49:49 CEST Slawek Kaplonski pisze:
> On Wed, Oct 21, 2020 at 11:33:14AM +1100, Ian Wienand wrote:
> > As of this mail, Gerrit access has been restored.  Please read on for
> > important information, especially around change verification.
> > 
> > Background
> > -----------
> > 
> > On 2020-10-20 at 01:30 a user unexpectedly added a workflow approval
> > to a change that they were not expected to have access to.  At 02:06
> > UTC an alert was raised via IRC.  Administrators found the account had
> > added themselves to a core group and made the +W vote.  The account
> > was disabled, and removed from the groups it had added itself to by
> > 02:55 UTC.  Administrators began to analyse the situation and Gerrit
> > was taken offline at 04:02 UTC to preserve state and allow for
> > analysis.
> > 
> > From this time, administrators were working on log collection and
> > analysis, along with restoring backups for comparison purposes.
> > 
> > By around 08:45 UTC it was clear that the privilege escalation had
> > been achieved by gaining control of a Launchpad SSO account with
> > Gerrit administrator privileges.  By this time, we had ruled out
> > software vulnerabilities.  Logs showed the first unauthorized access
> > of the administrator account in Gerrit on 2020-10-06.  Communication
> > with Launchpad admins agrees with this analysis.  We saw one session
> > opened as the administrator user to StoryBoard on this same day, but
> > logs show no data was modified or hidden stories viewed.
> > 
> > Analysis has been performed on the Gerrit database and git trees from
> > October 1st, pre-dating any known unauthorized access.
> > 
> > Access was restored at around 2020-10-21 00:00 UTC
> > 
> > Outcomes
> > -----------
> > 
> > The following has been verified:
> >  The administrator account used has been disabled and credentials
> >  updated
> >  
> >  We have verified that all group and user addition/removals since
> >  Oct 1 are valid.  The only invalid additions were made by the
> >  compromised administrator account to add a single user account to
> >  the Administrators group; and then that account added itself to
> >  another known group.
> >  
> >  The account given administrator privilege has been removed from
> >  the groups it added itself to and is disabled.
> >  
> >  There is no evidence of any unauthorized access via methods other
> >  than Gerrit HTTP and Gerrit SSH access.
> >  
> >  No commits have been pushed to git trees bypassing code review.
> >  Every git tree has been compared to the Oct 1 version and all
> >  commits have been correctly inserted via Gerrit changes.
> >  
> >  The version of Gerrit we use stores HTTP API passwords in
> >  plain-text.  We know that a limited number of passwords were
> >  gathered via the HTTP API and it is possible passwords were
> >  gathered via the database.  We thus have assumed that all HTTP API
> >  passwords have been disclosed.  This password needs to be
> >  explicitly enabled by users, and many users do not have it
> >  enabled.
> > 
> > Remediation
> > -----------
> > 
> > This leaves us with the following remediation actions:
> >  Users should double-check their Launchpad recent activity at
> >  https://login.launchpad.net/activity for any suspicious logins.  If
> >  found, please notify the OpenDev admins in Freenode #opendev and
> >  Launchpad admins in #launchpad immediately.
> >  
> >  All HTTP API passwords have been cleared.  If you push changes via
> >  HTTPS (instead of typical SSH), are a gertty user, or run a CI
> >  system or something else that communicates with the Gerrit HTTP
> >  API, you will need to regenerate a password.
> >  
> >  Any SSH keys added to accounts since 2020-10-01 have been removed.
> >  This affects only a limited number of accounts.  This is done in
> >  an abundance of caution, and we do not believe any accounts had
> >  unauthorized SSH keys added
> >  
> >  We should audit all changes for projects since 2020-10-01.
> > 
> > We have no evidence that any account had its ssh keys compromised,
> > thus we can rule out any unauthorized changes being uploaded via SSH.
> > However we can not conclusively rule out that compromised HTTP API
> > passwords were used to push a change through Gerrit. For example, a
> > change could be uploaded that looks like it came from a user, or the
> > API key of a core team member may have been used to approve a change
> > without authorization.
> > 
> > Given our extensive analysis we consider it exceedingly unlikely that
> > this vector was used.  We have had no notifications of users seeing
> > unexpected changes either uploaded by them, or approved by them in
> > projects they work on.  This said, we believe it is important to
> > inform the community of this very unlikely, but still possible,
> > vulnerability of the source code.
> > 
> > To this end, we have prepared a list of all changes from the known
> > affected period which should be audited for correctness.  These are
> > available at
> > 
> >     https://static.opendev.org/project/opendev.org/gerrit-diffs/
> > 
> > Team members should browse these changes and make sure they were
> > correctly approved in Gerrit.  If any change looks suspicious you
> > should notify OpenDev administrators in Freenode #opendev immediately.
> > 
> > Further actions
> > ----------------
> > 
> > We are planning the following for the short term future:
> >     The Opendev administrators will be looking at alternative models
> >     for Gerrit admin account management.
> >     
> >     We are already well into planning and testing a coming upgrade to
> >     a version of Gerrit which does not store plain-text API keys.
> >     
> >     Longer term, we've written a spec for replacing Launchpad SSO as
> >     our authentication provider.
> > 
> > We thank you for your patience during this trying time, and we look
> > forward to returning to supporting the community doing what it does
> > best -- working together to create great things.
> > 
> > 
> > _______________________________________________
> > service-announce mailing list
> > service-announce at lists.opendev.org
> > http://lists.opendev.org/cgi-bin/mailman/listinfo/service-announce
-- 
Slawek Kaplonski                                                                                                                                                                                                                                                               
Principal Software Engineer                                                                                                                                                                                                     
Red Hat