[OpenStack-Infra] Wiki.o.o sustaining spam attack
Paul Belanger
pabelanger at redhat.com
Tue Mar 22 13:38:11 UTC 2016
On Tue, Mar 22, 2016 at 08:23:08AM -0500, JP Maxwell wrote:
> If anyone wants to approve this I am still happy to help.
>
> https://review.openstack.org/#/c/285641/1
>
Looking at the review, Jim would like to see more history of collaborting in git
first. Which is a fair requirement.
> I don't think you are ever going to be successful at blocking accounts or
> IPs. You must block the creation of the spam by the bots. IMHO focusing on
> improving the captcha or understanding the bypass path around the captcha
> is the best short term path to accomplish this.
>
I think we're also wanting to upgrade to ubuntu trusty next and bump our
version of mediawiki too. I understand you want to look into why captcha is
getting bypassed but I think we are in 2 camps currently, we have a security
issue (which gets resolved by patching / upgrading) or they've cracked our
captcha with humans (which means rotate our questions more often).
Something else on the plate has been the discussion to move from launchpad.net
to openstackid.org as our SSO provider, which in theory will have less spammers.
So, moving forward. We'll land 287232[1], then purge / block users created with
no passwords (these are mobile users), use SmiteSpam to clean up current
content.
Then, plan migration to ubuntu trusty sometime after summit. With latest version
of mediawiki.
[1] https://review.openstack.org/#/c/287232/
> J.P. Maxwell | tipit.net | fibercove.com
> On Mar 22, 2016 8:15 AM, "Paul Belanger" <pabelanger at redhat.com> wrote:
>
> > On Tue, Mar 22, 2016 at 03:32:23PM +0800, Tom Fifield wrote:
> > > Hi all,
> > >
> > >
> > > I'm sad to say that:
> > >
> > > * spammers are back - 100-odd pages have gone in over the weekend
> > > https://wiki.openstack.org/wiki/Special:NewPages
> > >
> > > * Cleanup was ineffective, with many spam pages still existing on the
> > wiki
> > > (scroll through the NewPages link above)
> > >
> > So, we are still working through the clean up of the wiki. Right now,
> > we've
> > only stopped the creation of new accounts. Both from openid and mobile
> > users.
> >
> > We're going to be adding SmitSpam[1] to allow admins to run some cleanup
> > tools.
> > But that hasn't landed yet.
> >
> > Until now, I am going into the wiki every few days to ban existing
> > accounts that
> > have already been created manually.
> >
> > [1] https://review.openstack.org/#/c/287232/
> > >
> > >
> > > Regards,
> > >
> > >
> > > Tom
> > >
> > >
> > > On 28/02/16 01:11, JP Maxwell wrote:
> > > >Elizabeth
> > > >
> > > >I hope you feel better.
> > > >
> > > >Just FYI, this is going full force in IRC right now. I’ve bowed out as
> > > >the approach I was suggesting didn’t get traction.
> > > >
> > > >I proposed to manually iterate on this to confirm precisely which change
> > > >solves the spam problem. Once that has been identified we can revert
> > > >and come up with a proper patch. Right now the assumption is that
> > > >disabling manual accounts will solve the problem (and it might). As a
> > > >result the team is trying to solve for the consequences of not having
> > > >manual accounts. Some bots currently use manual accounts among other
> > > >issues. If the assumption is correct, these efforts will be worth it.
> > > > However, if it isn’t it will have been a great waste of energy.
> > > >
> > > >In any case have a good weekend everyone. I’m off to eat some delicious
> > > >central Texas BBQ!
> > > >
> > > >
> > > >*J.P. Maxwell* | tipit.net <http://tipit.net> | fibercove.com
> > > ><http://www.fibercove.com>
> > > >
> > > >On Sat, Feb 27, 2016 at 10:15 AM, Elizabeth K. Joseph
> > > ><lyz at princessleia.com> wrote:
> > > >
> > > > We'll be getting together on Monday around 1700 UTC to work through
> > > > this together in a debug session in #openstack-infra (I'm too sick
> > > > this weekend, plus we need a time when more infra-root folks with
> > > > the institutional knowledge are around).
> > > >
> > > > On Feb 27, 2016 05:37, "Marton Kiss" <marton.kiss at gmail.com
> > > > <mailto:marton.kiss at gmail.com>> wrote:
> > > >
> > > > Yeah, the Settings.php was overriden by the latest puppet run.
> > > > We need to wait for some infra guys to approve my patches and
> > > > make it permanent:
> > > > https://review.openstack.org/285669 Disable standard password
> > > > based auth
> > > > https://review.openstack.org/285672 Disable mobile frontend
> > > >
> > > > M.
> > > >
> > > > On Sat, Feb 27, 2016 at 2:27 PM JP Maxwell <jp at tipit.net
> > > > <mailto:jp at tipit.net>> wrote:
> > > >
> > > > FYI. Still seeing the mobile view...
> > > >
> > > > J.P. Maxwell | tipit.net <http://tipit.net> | fibercove.com
> > > > <http://fibercove.com>
> > > >
> > > > On Feb 27, 2016 6:53 AM, "Marton Kiss"
> > > > <marton.kiss at gmail.com <mailto:marton.kiss at gmail.com>>
> > wrote:
> > > >
> > > > Yes, applied them manually. Let's wait a few hours, and
> > > > check for new spam content / user accounts.
> > > >
> > > > M.
> > > > JP Maxwell <jp at tipit.net <mailto:jp at tipit.net>>
> > > > (időpont: 2016. febr. 27., Szo, 13:50) ezt írta:
> > > >
> > > > Cool. Are these applied? Any indication it has
> > > > stopped the spam? Should we clear out these non
> > > > launchpad accounts from the DB?
> > > >
> > > > J.P. Maxwell | tipit.net <http://tipit.net> |
> > > > fibercove.com <http://fibercove.com>
> > > >
> > > > On Feb 27, 2016 6:47 AM, "Marton Kiss"
> > > > <marton.kiss at gmail.com
> > > > <mailto:marton.kiss at gmail.com>> wrote:
> > > >
> > > > And the mobile frontend will be disabled
> > > > permanently with this patch:
> > > > https://review.openstack.org/285672 Disable
> > > > mobile frontend
> > > >
> > > > M.
> > > >
> > > > On Sat, Feb 27, 2016 at 1:39 PM Marton Kiss
> > > > <marton.kiss at gmail.com
> > > > <mailto:marton.kiss at gmail.com>> wrote:
> > > >
> > > > I made some investigation, and it seems to
> > > > be that the spam pages are created by
> > > > accounts registered with password accounts,
> > > > and the launchpad openid auth is not
> > > > affected at all.
> > > >
> > > > So the spam script is creating accounts like
> > > > this:
> > > > mysql> select * from user where user_name =
> > > > 'CedricJamieson'\G;
> > > > *************************** 1. row
> > > > ***************************
> > > > user_id: 7494
> > > > user_name: CedricJamieson
> > > > user_real_name: Cedric Jamieson
> > > > user_password:
> > > >
> > :pbkdf2:sha256:10000:128:Mlo9tdaP+38niZrrEka7Ow==:jEVnrTclkwIpE1RzJywDlrSvkY5G3idYwOwYRkv5O0J/MSHjY+gdhtKmArQ53v6/w7o8E1wXb2QOR6HdL5TPfOI1bswS/fYXVVYjPjkEEdxqZ8q9L5p2f3N6rEYpMfT5tk+wDiy+j5aimrHrGSga44hndAHgX9/SnqUyxlutDVY=
> > > > user_newpassword:
> > > > user_newpass_time: NULL
> > > > user_email: balashkina.evdokiya at mail.ru
> > > > <mailto:balashkina.evdokiya at mail.ru>
> > > > user_touched: 20160227052454
> > > > user_token: 7c39e44e849fb0e2bfae8790d6cc1379
> > > > user_email_authenticated: NULL
> > > > user_email_token:
> > > > be963ac3bd43e70ff2f323063c61e320
> > > > user_email_token_expires: 20160305052441
> > > > user_registration: 20160227052441
> > > > user_editcount: 2
> > > > user_password_expires: NULL
> > > >
> > > > The user_password field is always filled
> > > > with a value, meanwhile this field of
> > > > non-infected user accounts with openid
> > > > logins is empty.
> > > > We have 423 total accounts with passwords:
> > > > mysql> select count(*) from user where
> > > > user_password != '';
> > > > +----------+
> > > > | count(*) |
> > > > +----------+
> > > > | 423 |
> > > > +----------+
> > > > 1 row in set (0.00 sec)
> > > >
> > > > Mediawiki logs-in the newly created users
> > > > without any preliminary email confirmation,
> > > > right after the registration. I disabled the
> > > > standard user login page, as described here:
> > > >
> > https://www.mediawiki.org/wiki/Manual:Special_pages#Disabling_Special:UserLogin_and_Special:UserLogout_pages
> > > >
> > > > And I made this patch to make it permanent:
> > > > https://review.openstack.org/285669 Disable
> > > > standard password based auth
> > > >
> > > > Just for the record, the last spam user
> > account:
> > > > 7536 | EarthaChester22
> > > >
> > > > Marton
> > > >
> > > >
> > > > On Sat, Feb 27, 2016 at 8:31 AM Marton Kiss
> > > > <marton.kiss at gmail.com
> > > > <mailto:marton.kiss at gmail.com>> wrote:
> > > >
> > > > Hi,
> > > >
> > > > I created the following patch, infra
> > > > cores must approve that:
> > > > https://review.openstack.org/285641 Add
> > > > ssh key of JP Maxwell to wiki.o.o
> > > >
> > > > Marton
> > > >
> > > > On Sat, Feb 27, 2016 at 6:41 AM JP
> > > > Maxwell <jp at tipit.net
> > > > <mailto:jp at tipit.net>> wrote:
> > > >
> > > > Marton has SSH access and applied a
> > > > patch earlier today. It appears the
> > > > spam continues to flow:
> > > >
> > > >
> > https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen
> > > >
> > > > Marton let me know if you can look
> > > > at it some more or Infra if you want
> > > > to give me SSH I'll do so as well in
> > > > the morning (public key attached).
> > > >
> > > >
> > > >
> > > > ssh-rsa
> > > >
> > AAAAB3NzaC1yc2EAAAABIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q==
> > > > jpmaxman at tipit.net
> > > > <mailto:jpmaxman at tipit.net>
> > > >
> > > >
> > > >
> > > >
> > > > J.P. Maxwell / tipit.net
> > > > <http://www.tipit.net>
> > > >
> > > >
> > > > On Fri, Feb 26, 2016 at 12:09 PM,
> > > > Jimmy McArthur <jimmy at openstack.org
> > > > <mailto:jimmy at openstack.org>>
> > wrote:
> > > >
> > > > Super thankful for all the folks
> > > > that have jumped in over the
> > > > last couple of days to help with
> > > > the puppetization, etc... I just
> > > > feel like we're taking a very
> > > > wrong approach here.
> > > >
> > > > Paul Belanger wrote:
> > > >
> > > > Right, and I don't have an
> > issue with that approach. Based on the work we did
> > > > yesterday, anybody can do
> > that via our workflow. Please submit a patch to
> > > > puppet-mediawiki[1] and
> > ping an infra-root in #openstack-infra IRC.
> > > >
> > > > What I'm proposing is the
> > > > workflow is really meant for
> > > > software, not for web
> > > > applications. It's tedious and
> > > > time consuming when what's
> > > > needed here is a set of tests on
> > > > the server. Submitting a patch,
> > > > waiting for a +1, then getting
> > > > on IRC to find someone with
> > > > access (and time) to paste the
> > > > logs is a pretty time consuming
> > > > process for what should be a
> > > > series of rapid-fire
> > > > changes/fixes on the server.
> > > > Especially when we're dealign
> > > > with an active attack.
> > > >
> > > > We can then have somebody
> > look at the logs. I think it is more about scheduling
> > > > the task since more
> > infra-root as travling back from the mid-cycle last night
> > > > and today.
> > > >
> > > > Right, this is my point. This
> > > > has been going on for 3 weeks
> > > > (or more). Tom Fifeldt was
> > > > asking for help without
> > > > response. And here we are
> > > > through another week and no
> > > > closer to stemming the flow.
> > > >
> > > > I'm fully aware what I'm
> > > > proposing goes against what
> > > > Infra and the OpenStack workflow
> > > > is all about, but I'd ask you
> > > > all to look at this from a web
> > > > development perspective instead
> > > > of a software development
> > > > perspective.
> > > >
> > > > Jimmy
> > > >
> > > > Last email from me, just on
> > a plane. Will follow up when I land.
> > > >
> > > > [1]
> > https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki
> > > >
> > > >
> > > > J.P. Maxwell |tipit.net
> > <http://tipit.net> [http://tipit.net] |fibercove.com
> > > > <http://fibercove.com>
> > > > [
> > http://www.fibercove.com]
> > > > On Fri, Feb 26, 2016 at
> > 11:25 AM, Paul Belanger<pabelanger at redhat.com>
> > > > <mailto:
> > pabelanger at redhat.com>
> > > > wrote:
> > > > On Fri, Feb 26, 2016 at
> > 11:08:18AM -0600, Jimmy McArthur wrote:
> > > >
> > > > Given the state of
> > the wiki a the moment, I think taking the quickest path
> > > > to get it fixed
> > would be prudent. Is there a way we can get JP root access
> > > > to this server,
> > even temporarily? We get 25% of our website traffic (2
> > > > million visitors)
> > to the wiki. I realize we're all after the same thing,
> > > >
> > > > but
> > > >
> > > > spammers are not
> > going to hit the dev environment, so there's really no
> > > >
> > > > way
> > > >
> > > > to tell if teh
> > problem is fixed without actually working directly on the
> > > > production machine.
> > This should be a 30 minute fix.
> > > >
> > > > I am still unclear what
> > the 30min fix is. If really 30mins, then it
> > > > shouldn't be
> > > > hard to get the fix
> > into our workflow. Could somebody please elaborate.
> > > >
> > > > If we are talking about
> > deploying new versions of php or mediawiki manually,
> > > > I
> > > > not be in-favor of
> > this. To me, while the attack sucks, we should be working
> > > > on
> > > > 2 fronts. Getting the
> > help needed to mitigate the attack, then adding the
> > > > changes into -infra
> > workflow in parallel.
> > > >
> > > > I realize there is
> > a lot of risk in giving ssh access to infra machines,
> > > >
> > > > but
> > > >
> > > > I think it's worth
> > taking a look at either putting this machine in a place
> > > > where a different
> > level of admin could access it without giving away the
> > > > keys to the entire
> > OpenStack infrastructure or figuring out a way to set
> > > >
> > > > up
> > > >
> > > > credentials with
> > varying levels of access.
> > > >
> > > > As a note, all the work
> > I've been doing to help with the attack hasn't
> > > > require
> > > > SSH access for me to
> > wiki.o.o. I did need infra-root help to expose our
> > > > configuration safely.
> > I'd rather take some time to see what the fixes are,
> > > > having infra-root apply
> > changes, then move them into puppet.
> > > >
> > > > It also has been
> > discussed to simply disable write access to the wiki if we
> > > > really want spamming to
> > stop, obviously that will affect normal usage.
> > > >
> > > > Jimmy
> > > >
> > > > Paul Belanger wrote:
> > > >
> > > > On Fri, Feb 26,
> > 2016 at 10:12:12AM -0600, JP Maxwell wrote:
> > > >
> > > > But if you
> > wanted to upgrade everything, remove the mobile view
> > > >
> > > > extension,
> > > >
> > > > test in a
> > dev/staging environment then deploy to production fingers
> > > > crossed, I
> > think that would be a valid approach as well.
> > > >
> > > > Current review
> > up[1]. I'll launch a node tonight / tomorrow locally to
> > > >
> > > > see
> > > > how
> > > >
> > > > puppet reacts.
> > I suspect there will be some issues.
> > > >
> > > > If infra-roots
> > are fine with this approach, we can use that box to test
> > > >
> > > > against.
> > > >
> > > > [1]
> > https://review.openstack.org/#/c/285405/
> > > >
> > > > J.P.
> > Maxwell |tipit.net
> > > > <
> > http://tipit.net> |fibercove.com <http://fibercove.com>
> > > > On Feb 26,
> > 2016 10:08 AM, "JP Maxwell"<jp at tipit.net>
> > > > <mailto:
> > jp at tipit.net> wrote:
> > > >
> > > > Plus
> > one except in this case it is much easier to know if our efforts
> > > >
> > > > are
> > > >
> > > > working
> > on production because the spam either stops or not.
> > > >
> > > > J.P.
> > Maxwell |tipit.net <http://tipit.net> |fibercove.com
> > > > <
> > http://fibercove.com>
> > > > On Feb
> > 26, 2016 9:48 AM, "Paul Belanger"<pabelanger at redhat.com>
> > > > <mailto:
> > pabelanger at redhat.com> wrote:
> > > >
> > > > On
> > Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> > > >
> > > >
> > I really think you might consider the option that there is a
> > > >
> > > >
> > vulnerability
> > > >
> > > >
> > in one of the extensions. If that is the case black listing IPs will
> > > >
> > > > be
> > > >
> > > > an
> > > >
> > > >
> > ongoing wild goose chase.
> > > >
> > > >
> > I think this would be easily proven or disproven by making the questy
> > > >
> > question impossible and see if the spam continues.
> > > >
> > > >
> > We'll have to let an infra-root make that call. Since nobody would be
> > > >
> > able to
> > > > use
> > the wiki. Honestly, I'd rather spend the time standing up a mirror
> > > >
> > > > dev
> > > >
> > > >
> > instance for us to work on, rather then production.
> > > >
> > > >
> > J.P. Maxwell |tipit.net
> > > >
> > <http://tipit.net> |fibercove.com
> > > >
> > <http://fibercove.com>
> > > >
> > On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelanger at redhat.com>
> > > >
> > <mailto:pabelanger at redhat.com>
> > > >
> > > > wrote:
> > > >
> > > >
> > On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
> > > >
> > > >
> > On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<fungi at yuggoth.org>
> > > >
> > <mailto:fungi at yuggoth.org>
> > > >
> > > >
> > wrote:
> > > >
> > > >
> > On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> > > >
> > > >
> > Please be aware that you can now create accounts under the
> > mobile
> > > >
> > view in the wiki native user table. I just created an
> > account for
> > > >
> > JpMaxMan. Not sure if this matters but wanted to make sure
> > you
> > > >
> > were aware.
> > > >
> > > >
> > Oh, yes I think having a random garbage question/answer was in
> > > >
> > > > fact
> > > >
> > > >
> > previously preventing account creation under the mobile view. We
> > > >
> > probably need a way to disable mobile view account creation as
> > it
> > > >
> > bypasses OpenID authentication entirely.
> > > >
> > > >
> > So that's what it was doing! We'll have to tackle the mobile view
> > > >
> > > >
> > issue.
> > > >
> > > >
> > Otherwise, quick update here:
> > > >
> > > >
> > The captcha didn't appear to help stem the spam tide. We'll want to
> > > >
> > explore and start implementing some of the other solutions.
> > > >
> > > >
> > I did some database poking around today and it does seem like all
> > > >
> > > > the
> > > >
> > > >
> > users do have launchpad accounts and email addresses.
> > > >
> > > >
> > So, I have a few hours before jumping on my plane and checked into
> > > >
> > > >
> > this.
> > > >
> > > >
> > We are
> > > >
> > using QuestyCaptcha which according to docs, should almost be
> > > >
> > > >
> > impossible
> > > >
> > > >
> > for
> > > >
> > spammers to by pass in an automated fashion. So, either our captcha
> > > >
> > > > is
> > too
> > > >
> > > >
> > easy, or we didn't set it up properly. I don't have SSH on wiki.o.o
> > > >
> > > > so
> > > >
> > > >
> > others
> > > >
> > will have to check logs. I did test new pages and edits, and was
> > > >
> > > >
> > promoted
> > > >
> > > >
> > by
> > > >
> > captcha.
> > > >
> > > >
> > As a next step, we might need to add additional apache2
> > > >
> > > > configuration
> > > >
> > > > to
> > > >
> > > >
> > blacklist IPs. I am reading up on that now.
> > > >
> > > >
> > --
> > > >
> > Elizabeth Krumbach Joseph || Lyz || pleia2
> > > >
> > > >
> > _______________________________________________
> > > >
> > OpenStack-Infra mailing list
> > > >
> > OpenStack-Infra at lists.openstack.org
> > > >
> > <mailto:OpenStack-Infra at lists.openstack.org>
> > > >
> > > >
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > > >
> > _______________________________________________
> > > >
> > OpenStack-Infra mailing list
> > > >
> > OpenStack-Infra at lists.openstack.org
> > > >
> > <mailto:OpenStack-Infra at lists.openstack.org>
> > > >
> > > >
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > > >
> > > >
> > _______________________________________________
> > > > OpenStack-Infra
> > mailing list
> > > >
> > OpenStack-Infra at lists.openstack.org
> > > > <mailto:
> > OpenStack-Infra at lists.openstack.org>
> > > >
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > > >
> > > >
> > _______________________________________________
> > > > OpenStack-Infra
> > mailing list
> > > >
> > OpenStack-Infra at lists.openstack.org
> > > > <mailto:
> > OpenStack-Infra at lists.openstack.org>
> > > >
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > > >
> > > >
> > _______________________________________________
> > > > OpenStack-Infra mailing
> > list
> > > >
> > OpenStack-Infra at lists.openstack.org
> > > > <mailto:
> > OpenStack-Infra at lists.openstack.org>
> > > >
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > > >
> > > >
> > > >
> > > >
> > _______________________________________________
> > > > OpenStack-Infra mailing list
> > > > OpenStack-Infra at lists.openstack.org
> > > > <mailto:
> > OpenStack-Infra at lists.openstack.org>
> > > >
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > > >
> > > >
> > > > _______________________________________________
> > > > OpenStack-Infra mailing list
> > > > OpenStack-Infra at lists.openstack.org
> > > > <mailto:OpenStack-Infra at lists.openstack.org>
> > > >
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > > >
> > > >
> > > >
> > > >_______________________________________________
> > > >OpenStack-Infra mailing list
> > > >OpenStack-Infra at lists.openstack.org
> > > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > > >
> > >
> > > _______________________________________________
> > > OpenStack-Infra mailing list
> > > OpenStack-Infra at lists.openstack.org
> > > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> >
> > _______________________________________________
> > OpenStack-Infra mailing list
> > OpenStack-Infra at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> >
More information about the OpenStack-Infra
mailing list