On Wed, Oct 21, 2020 at 2:42 PM Giulio Fidente gfidente@redhat.com wrote:

On 10/21/20 9:15 AM, Marios Andreou wrote:

...

Hi folks,

as you are undoubtedly aware, gerrit was down yesterday. There was this email to service-announce [1] with more information about what happened (kudos Julia Kreger who sent [2] where I saw that). There is a list of changes [3] since October 1st that we should audit out of precaution and to be responsible and accountable to our community and users.

As you can expect there are a great number of changes. I put a full commit list at [5]. I mined those from [3] - see [4] for info about the 'mining' and even better if someone has time to verify that I didn't miss any repos or commits.

Please I need help from all core reviewers. We need to check that the commits in [5] appear valid and correct - remember the concern is for any changes that may have been merged by a compromised account. I propose that we do this via Gerrit and that we leave a comment - 'CHECKED' - on each review that we check? Hopefully we can cover all of these before the end of the week by distributing our efforts. I am open to other suggestions though if folks feel this is better done via some document/spreadsheet etc.

Of course as stated in [1] it is a good idea for everyone to double check their account activity and make sure nothing is off,

Thank you in advance for your help,

marios

[1]

http://lists.opendev.org/pipermail/service-announce/2020-October/000011.html

...

[2]

http://lists.openstack.org/pipermail/openstack-discuss/2020-October/018148.h...

...

[3] https://static.opendev.org/project/opendev.org/gerrit-diffs/ [4] https://gist.github.com/marios/a44a55998531354dc3d634dddeadf1c0 [5] https://gist.github.com/marios/d1b774c827769373b67d3988105140dd

thanks a lot Marios for looking into this and organizing activities

do I understand correctly that our most immediate responsibility is to go through the list of commits in [5] and compare what is actually in the git repos with what was proposed in gerrit?

I don't think we need to worry that it was 'one of our accounts' that was compromised, at least I expect we would have known by now if there was any indication that this is the case.

The main concern is if the compromised admin account made any commits at all. So the immediate check is to make sure that all those commits were in fact merged by 'one of us' and not by any unknown account. For example with the compromised account they may have updated a review and merged it without us noticing. Unlikely I know, especially since we are quite an active project but I think it is better we make sure.

Of course I may be wrong in my assessment here in which case I fully expect that you will let me know ! I mean there is nothing wrong with doing what you suggested but I don't know if there is a need to go that far in this case. Verifying the person(s) that +2 and +A the review should be enough for now, making sure we don't have any rogue merges.

thanks ;)

marios

-- Giulio Fidente GPG KEY: 08D733BA

On Wednesday, October 21, 2020, Jeremy Stanley fungi@yuggoth.org wrote:

On 2020-10-21 15:02:54 +0300 (+0300), Marios Andreou wrote: [...]

...

I don't think we need to worry that it was 'one of our accounts' that was compromised, at least I expect we would have known by now if there was any indication that this is the case.

The main concern is if the compromised admin account made any commits at all. So the immediate check is to make sure that all those commits were in fact merged by 'one of us' and not by any unknown account.

[...]

Not quite. The main concern is that the attacker had access (via an account in Gerrit's Administrators group) to add their own SSH key or view/add/change the REST API key for any user of the service, so could in theory have proposed a change masquerading as a regular member of your team, +2'd it as another member of your team, and approved it as yet a third member of your team, without necessarily raising suspicion. While we consider this unlikely, it was entirely possible for the first few weeks of this month.

Per my other reply on this thread, we already checked that every commit corresponds to a change in Gerrit, so it should be sufficient to just skim the last few week's changes and make sure you remember reviewing/approving them.

I see.... hm potentially much more malicious than I thought then. Thanks for the clarification - I've mainly been checking that the merges were from known tripleo cores.

Rather it should be that each core should check the reviews merged by their account ID and make sure it corresponds to a valid +A that they (possibly) recall doing.

I think we should be mostly done for tripleo ... my original list at https://gist.github.com/marios/d1b774c827769373b67d3988105140dd contains duplicates as i found so far and I know a number of folks have jumped in and started checking today. Thanks to everyone for doing that

thanks again Jeremy for clarifying what we should focus on

On 2020-10-21 13:42:18 +0200 (+0200), Giulio Fidente wrote: [...]

do I understand correctly that our most immediate responsibility is to go through the list of commits in [5] and compare what is actually in the git repos with what was proposed in gerrit?

You don't need to compare them with what's in Gerrit (that was already done automatically with a script). You need to double-check those commits to make sure they're legitimate changes to the software, likely ones you recall reviewing/approving over the past few weeks.