[OpenStack-Infra] Wiki.o.o sustaining spam attack
Tom Fifield
tom at openstack.org
Tue Mar 22 07:32:23 UTC 2016
Hi all,
I'm sad to say that:
* spammers are back - 100-odd pages have gone in over the weekend
https://wiki.openstack.org/wiki/Special:NewPages
* Cleanup was ineffective, with many spam pages still existing on the
wiki (scroll through the NewPages link above)
Regards,
Tom
On 28/02/16 01:11, JP Maxwell wrote:
> Elizabeth
>
> I hope you feel better.
>
> Just FYI, this is going full force in IRC right now. I’ve bowed out as
> the approach I was suggesting didn’t get traction.
>
> I proposed to manually iterate on this to confirm precisely which change
> solves the spam problem. Once that has been identified we can revert
> and come up with a proper patch. Right now the assumption is that
> disabling manual accounts will solve the problem (and it might). As a
> result the team is trying to solve for the consequences of not having
> manual accounts. Some bots currently use manual accounts among other
> issues. If the assumption is correct, these efforts will be worth it.
> However, if it isn’t it will have been a great waste of energy.
>
> In any case have a good weekend everyone. I’m off to eat some delicious
> central Texas BBQ!
>
>
> *J.P. Maxwell* | tipit.net <http://tipit.net> | fibercove.com
> <http://www.fibercove.com>
>
> On Sat, Feb 27, 2016 at 10:15 AM, Elizabeth K. Joseph
> <lyz at princessleia.com> wrote:
>
> We'll be getting together on Monday around 1700 UTC to work through
> this together in a debug session in #openstack-infra (I'm too sick
> this weekend, plus we need a time when more infra-root folks with
> the institutional knowledge are around).
>
> On Feb 27, 2016 05:37, "Marton Kiss" <marton.kiss at gmail.com
> <mailto:marton.kiss at gmail.com>> wrote:
>
> Yeah, the Settings.php was overriden by the latest puppet run.
> We need to wait for some infra guys to approve my patches and
> make it permanent:
> https://review.openstack.org/285669 Disable standard password
> based auth
> https://review.openstack.org/285672 Disable mobile frontend
>
> M.
>
> On Sat, Feb 27, 2016 at 2:27 PM JP Maxwell <jp at tipit.net
> <mailto:jp at tipit.net>> wrote:
>
> FYI. Still seeing the mobile view...
>
> J.P. Maxwell | tipit.net <http://tipit.net> | fibercove.com
> <http://fibercove.com>
>
> On Feb 27, 2016 6:53 AM, "Marton Kiss"
> <marton.kiss at gmail.com <mailto:marton.kiss at gmail.com>> wrote:
>
> Yes, applied them manually. Let's wait a few hours, and
> check for new spam content / user accounts.
>
> M.
> JP Maxwell <jp at tipit.net <mailto:jp at tipit.net>>
> (időpont: 2016. febr. 27., Szo, 13:50) ezt írta:
>
> Cool. Are these applied? Any indication it has
> stopped the spam? Should we clear out these non
> launchpad accounts from the DB?
>
> J.P. Maxwell | tipit.net <http://tipit.net> |
> fibercove.com <http://fibercove.com>
>
> On Feb 27, 2016 6:47 AM, "Marton Kiss"
> <marton.kiss at gmail.com
> <mailto:marton.kiss at gmail.com>> wrote:
>
> And the mobile frontend will be disabled
> permanently with this patch:
> https://review.openstack.org/285672 Disable
> mobile frontend
>
> M.
>
> On Sat, Feb 27, 2016 at 1:39 PM Marton Kiss
> <marton.kiss at gmail.com
> <mailto:marton.kiss at gmail.com>> wrote:
>
> I made some investigation, and it seems to
> be that the spam pages are created by
> accounts registered with password accounts,
> and the launchpad openid auth is not
> affected at all.
>
> So the spam script is creating accounts like
> this:
> mysql> select * from user where user_name =
> 'CedricJamieson'\G;
> *************************** 1. row
> ***************************
> user_id: 7494
> user_name: CedricJamieson
> user_real_name: Cedric Jamieson
> user_password:
> :pbkdf2:sha256:10000:128:Mlo9tdaP+38niZrrEka7Ow==:jEVnrTclkwIpE1RzJywDlrSvkY5G3idYwOwYRkv5O0J/MSHjY+gdhtKmArQ53v6/w7o8E1wXb2QOR6HdL5TPfOI1bswS/fYXVVYjPjkEEdxqZ8q9L5p2f3N6rEYpMfT5tk+wDiy+j5aimrHrGSga44hndAHgX9/SnqUyxlutDVY=
> user_newpassword:
> user_newpass_time: NULL
> user_email: balashkina.evdokiya at mail.ru
> <mailto:balashkina.evdokiya at mail.ru>
> user_touched: 20160227052454
> user_token: 7c39e44e849fb0e2bfae8790d6cc1379
> user_email_authenticated: NULL
> user_email_token:
> be963ac3bd43e70ff2f323063c61e320
> user_email_token_expires: 20160305052441
> user_registration: 20160227052441
> user_editcount: 2
> user_password_expires: NULL
>
> The user_password field is always filled
> with a value, meanwhile this field of
> non-infected user accounts with openid
> logins is empty.
> We have 423 total accounts with passwords:
> mysql> select count(*) from user where
> user_password != '';
> +----------+
> | count(*) |
> +----------+
> | 423 |
> +----------+
> 1 row in set (0.00 sec)
>
> Mediawiki logs-in the newly created users
> without any preliminary email confirmation,
> right after the registration. I disabled the
> standard user login page, as described here:
> https://www.mediawiki.org/wiki/Manual:Special_pages#Disabling_Special:UserLogin_and_Special:UserLogout_pages
>
> And I made this patch to make it permanent:
> https://review.openstack.org/285669 Disable
> standard password based auth
>
> Just for the record, the last spam user account:
> 7536 | EarthaChester22
>
> Marton
>
>
> On Sat, Feb 27, 2016 at 8:31 AM Marton Kiss
> <marton.kiss at gmail.com
> <mailto:marton.kiss at gmail.com>> wrote:
>
> Hi,
>
> I created the following patch, infra
> cores must approve that:
> https://review.openstack.org/285641 Add
> ssh key of JP Maxwell to wiki.o.o
>
> Marton
>
> On Sat, Feb 27, 2016 at 6:41 AM JP
> Maxwell <jp at tipit.net
> <mailto:jp at tipit.net>> wrote:
>
> Marton has SSH access and applied a
> patch earlier today. It appears the
> spam continues to flow:
>
> https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen
>
> Marton let me know if you can look
> at it some more or Infra if you want
> to give me SSH I'll do so as well in
> the morning (public key attached).
>
>
>
> ssh-rsa
> AAAAB3NzaC1yc2EAAAABIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q==
> jpmaxman at tipit.net
> <mailto:jpmaxman at tipit.net>
>
>
>
>
> J.P. Maxwell / tipit.net
> <http://www.tipit.net>
>
>
> On Fri, Feb 26, 2016 at 12:09 PM,
> Jimmy McArthur <jimmy at openstack.org
> <mailto:jimmy at openstack.org>> wrote:
>
> Super thankful for all the folks
> that have jumped in over the
> last couple of days to help with
> the puppetization, etc... I just
> feel like we're taking a very
> wrong approach here.
>
> Paul Belanger wrote:
>
> Right, and I don't have an issue with that approach. Based on the work we did
> yesterday, anybody can do that via our workflow. Please submit a patch to
> puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.
>
> What I'm proposing is the
> workflow is really meant for
> software, not for web
> applications. It's tedious and
> time consuming when what's
> needed here is a set of tests on
> the server. Submitting a patch,
> waiting for a +1, then getting
> on IRC to find someone with
> access (and time) to paste the
> logs is a pretty time consuming
> process for what should be a
> series of rapid-fire
> changes/fixes on the server.
> Especially when we're dealign
> with an active attack.
>
> We can then have somebody look at the logs. I think it is more about scheduling
> the task since more infra-root as travling back from the mid-cycle last night
> and today.
>
> Right, this is my point. This
> has been going on for 3 weeks
> (or more). Tom Fifeldt was
> asking for help without
> response. And here we are
> through another week and no
> closer to stemming the flow.
>
> I'm fully aware what I'm
> proposing goes against what
> Infra and the OpenStack workflow
> is all about, but I'd ask you
> all to look at this from a web
> development perspective instead
> of a software development
> perspective.
>
> Jimmy
>
> Last email from me, just on a plane. Will follow up when I land.
>
> [1]https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki
>
>
> J.P. Maxwell |tipit.net <http://tipit.net> [http://tipit.net] |fibercove.com
> <http://fibercove.com>
> [http://www.fibercove.com]
> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger<pabelanger at redhat.com>
> <mailto:pabelanger at redhat.com>
> wrote:
> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
>
> Given the state of the wiki a the moment, I think taking the quickest path
> to get it fixed would be prudent. Is there a way we can get JP root access
> to this server, even temporarily? We get 25% of our website traffic (2
> million visitors) to the wiki. I realize we're all after the same thing,
>
> but
>
> spammers are not going to hit the dev environment, so there's really no
>
> way
>
> to tell if teh problem is fixed without actually working directly on the
> production machine. This should be a 30 minute fix.
>
> I am still unclear what the 30min fix is. If really 30mins, then it
> shouldn't be
> hard to get the fix into our workflow. Could somebody please elaborate.
>
> If we are talking about deploying new versions of php or mediawiki manually,
> I
> not be in-favor of this. To me, while the attack sucks, we should be working
> on
> 2 fronts. Getting the help needed to mitigate the attack, then adding the
> changes into -infra workflow in parallel.
>
> I realize there is a lot of risk in giving ssh access to infra machines,
>
> but
>
> I think it's worth taking a look at either putting this machine in a place
> where a different level of admin could access it without giving away the
> keys to the entire OpenStack infrastructure or figuring out a way to set
>
> up
>
> credentials with varying levels of access.
>
> As a note, all the work I've been doing to help with the attack hasn't
> require
> SSH access for me to wiki.o.o. I did need infra-root help to expose our
> configuration safely. I'd rather take some time to see what the fixes are,
> having infra-root apply changes, then move them into puppet.
>
> It also has been discussed to simply disable write access to the wiki if we
> really want spamming to stop, obviously that will affect normal usage.
>
> Jimmy
>
> Paul Belanger wrote:
>
> On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
>
> But if you wanted to upgrade everything, remove the mobile view
>
> extension,
>
> test in a dev/staging environment then deploy to production fingers
> crossed, I think that would be a valid approach as well.
>
> Current review up[1]. I'll launch a node tonight / tomorrow locally to
>
> see
> how
>
> puppet reacts. I suspect there will be some issues.
>
> If infra-roots are fine with this approach, we can use that box to test
>
> against.
>
> [1]https://review.openstack.org/#/c/285405/
>
> J.P. Maxwell |tipit.net
> <http://tipit.net> |fibercove.com <http://fibercove.com>
> On Feb 26, 2016 10:08 AM, "JP Maxwell"<jp at tipit.net>
> <mailto:jp at tipit.net> wrote:
>
> Plus one except in this case it is much easier to know if our efforts
>
> are
>
> working on production because the spam either stops or not.
>
> J.P. Maxwell |tipit.net <http://tipit.net> |fibercove.com
> <http://fibercove.com>
> On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelanger at redhat.com>
> <mailto:pabelanger at redhat.com> wrote:
>
> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
>
> I really think you might consider the option that there is a
>
> vulnerability
>
> in one of the extensions. If that is the case black listing IPs will
>
> be
>
> an
>
> ongoing wild goose chase.
>
> I think this would be easily proven or disproven by making the questy
> question impossible and see if the spam continues.
>
> We'll have to let an infra-root make that call. Since nobody would be
> able to
> use the wiki. Honestly, I'd rather spend the time standing up a mirror
>
> dev
>
> instance for us to work on, rather then production.
>
> J.P. Maxwell |tipit.net
> <http://tipit.net> |fibercove.com
> <http://fibercove.com>
> On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelanger at redhat.com>
> <mailto:pabelanger at redhat.com>
>
> wrote:
>
> On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
>
> On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<fungi at yuggoth.org>
> <mailto:fungi at yuggoth.org>
>
> wrote:
>
> On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
>
> Please be aware that you can now create accounts under the mobile
> view in the wiki native user table. I just created an account for
> JpMaxMan. Not sure if this matters but wanted to make sure you
> were aware.
>
> Oh, yes I think having a random garbage question/answer was in
>
> fact
>
> previously preventing account creation under the mobile view. We
> probably need a way to disable mobile view account creation as it
> bypasses OpenID authentication entirely.
>
> So that's what it was doing! We'll have to tackle the mobile view
>
> issue.
>
> Otherwise, quick update here:
>
> The captcha didn't appear to help stem the spam tide. We'll want to
> explore and start implementing some of the other solutions.
>
> I did some database poking around today and it does seem like all
>
> the
>
> users do have launchpad accounts and email addresses.
>
> So, I have a few hours before jumping on my plane and checked into
>
> this.
>
> We are
> using QuestyCaptcha which according to docs, should almost be
>
> impossible
>
> for
> spammers to by pass in an automated fashion. So, either our captcha
>
> is too
>
> easy, or we didn't set it up properly. I don't have SSH on wiki.o.o
>
> so
>
> others
> will have to check logs. I did test new pages and edits, and was
>
> promoted
>
> by
> captcha.
>
> As a next step, we might need to add additional apache2
>
> configuration
>
> to
>
> blacklist IPs. I am reading up on that now.
>
> --
> Elizabeth Krumbach Joseph || Lyz || pleia2
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> <mailto:OpenStack-Infra at lists.openstack.org>
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> <mailto:OpenStack-Infra at lists.openstack.org>
>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> <mailto:OpenStack-Infra at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> <mailto:OpenStack-Infra at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> <mailto:OpenStack-Infra at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> <mailto:OpenStack-Infra at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> <mailto:OpenStack-Infra at lists.openstack.org>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
More information about the OpenStack-Infra
mailing list