Open Stack

Sat Feb 27 17:08:04 UTC 2016

Ok, I'll be there.

M.

On Sat, Feb 27, 2016 at 5:15 PM Elizabeth K. Joseph <lyz at princessleia.com>
wrote:

> We'll be getting together on Monday around 1700 UTC to work through this
> together in a debug session in #openstack-infra (I'm too sick this weekend,
> plus we need a time when more infra-root folks with the institutional
> knowledge are around).
> On Feb 27, 2016 05:37, "Marton Kiss" <marton.kiss at gmail.com> wrote:
>
>> Yeah, the Settings.php was overriden by the latest puppet run. We need to
>> wait for some infra guys to approve my patches and make it permanent:
>> https://review.openstack.org/285669 Disable standard password based auth
>> https://review.openstack.org/285672 Disable mobile frontend
>>
>> M.
>>
>> On Sat, Feb 27, 2016 at 2:27 PM JP Maxwell <jp at tipit.net> wrote:
>>
>>> FYI. Still seeing the mobile view...
>>>
>>> J.P. Maxwell | tipit.net | fibercove.com
>>> On Feb 27, 2016 6:53 AM, "Marton Kiss" <marton.kiss at gmail.com> wrote:
>>>
>>>> Yes, applied them manually. Let's wait a few hours, and check for new
>>>> spam content / user accounts.
>>>>
>>>> M.
>>>> JP Maxwell <jp at tipit.net> (időpont: 2016. febr. 27., Szo, 13:50) ezt
>>>> írta:
>>>>
>>>>> Cool. Are these applied? Any indication it has stopped the spam?
>>>>> Should we clear out these non launchpad accounts from the DB?
>>>>>
>>>>> J.P. Maxwell | tipit.net | fibercove.com
>>>>> On Feb 27, 2016 6:47 AM, "Marton Kiss" <marton.kiss at gmail.com> wrote:
>>>>>
>>>>>> And the mobile frontend will be disabled permanently with this patch:
>>>>>> https://review.openstack.org/285672 Disable mobile frontend
>>>>>>
>>>>>> M.
>>>>>>
>>>>>> On Sat, Feb 27, 2016 at 1:39 PM Marton Kiss <marton.kiss at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> I made some investigation, and it seems to be that the spam pages
>>>>>>> are created by accounts registered with password accounts, and the
>>>>>>> launchpad openid auth is not affected at all.
>>>>>>>
>>>>>>> So the spam script is creating accounts like this:
>>>>>>> mysql> select * from user where user_name = 'CedricJamieson'\G;
>>>>>>> *************************** 1. row ***************************
>>>>>>>                  user_id: 7494
>>>>>>>                user_name: CedricJamieson
>>>>>>>           user_real_name: Cedric Jamieson
>>>>>>>            user_password:
>>>>>>> :pbkdf2:sha256:10000:128:Mlo9tdaP+38niZrrEka7Ow==:jEVnrTclkwIpE1RzJywDlrSvkY5G3idYwOwYRkv5O0J/MSHjY+gdhtKmArQ53v6/w7o8E1wXb2QOR6HdL5TPfOI1bswS/fYXVVYjPjkEEdxqZ8q9L5p2f3N6rEYpMfT5tk+wDiy+j5aimrHrGSga44hndAHgX9/SnqUyxlutDVY=
>>>>>>>         user_newpassword:
>>>>>>>        user_newpass_time: NULL
>>>>>>>               user_email: balashkina.evdokiya at mail.ru
>>>>>>>             user_touched: 20160227052454
>>>>>>>               user_token: 7c39e44e849fb0e2bfae8790d6cc1379
>>>>>>> user_email_authenticated: NULL
>>>>>>>         user_email_token: be963ac3bd43e70ff2f323063c61e320
>>>>>>> user_email_token_expires: 20160305052441
>>>>>>>        user_registration: 20160227052441
>>>>>>>           user_editcount: 2
>>>>>>>    user_password_expires: NULL
>>>>>>>
>>>>>>> The user_password field is always filled with a value, meanwhile
>>>>>>> this field of non-infected user accounts with openid logins is empty.
>>>>>>> We have 423 total accounts with passwords:
>>>>>>> mysql> select count(*) from user where user_password != '';
>>>>>>> +----------+
>>>>>>> | count(*) |
>>>>>>> +----------+
>>>>>>> |      423 |
>>>>>>> +----------+
>>>>>>> 1 row in set (0.00 sec)
>>>>>>>
>>>>>>> Mediawiki logs-in the newly created users without any preliminary
>>>>>>> email confirmation, right after the registration. I disabled the standard
>>>>>>> user login page, as described here:
>>>>>>> https://www.mediawiki.org/wiki/Manual:Special_pages#Disabling_Special:UserLogin_and_Special:UserLogout_pages
>>>>>>>
>>>>>>> And I made this patch to make it permanent:
>>>>>>> https://review.openstack.org/285669 Disable standard password based
>>>>>>> auth
>>>>>>>
>>>>>>> Just for the record, the last spam user account:
>>>>>>> 7536 | EarthaChester22
>>>>>>>
>>>>>>> Marton
>>>>>>>
>>>>>>>
>>>>>>> On Sat, Feb 27, 2016 at 8:31 AM Marton Kiss <marton.kiss at gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> I created the following patch, infra cores must approve that:
>>>>>>>> https://review.openstack.org/285641 Add ssh key of JP Maxwell to
>>>>>>>> wiki.o.o
>>>>>>>>
>>>>>>>> Marton
>>>>>>>>
>>>>>>>> On Sat, Feb 27, 2016 at 6:41 AM JP Maxwell <jp at tipit.net> wrote:
>>>>>>>>
>>>>>>>>> Marton has SSH access and applied a patch earlier today.  It
>>>>>>>>> appears the spam continues to flow:
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen
>>>>>>>>>
>>>>>>>>> Marton let me know if you can look at it some more or Infra if you
>>>>>>>>> want to give me SSH I'll do so as well in the morning (public key
>>>>>>>>> attached).
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ssh-rsa
>>>>>>>>> AAAAB3NzaC1yc2EAAAABIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q==
>>>>>>>>> jpmaxman at tipit.net
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> J.P. Maxwell / tipit.net <http://www.tipit.net>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Feb 26, 2016 at 12:09 PM, Jimmy McArthur <
>>>>>>>>> jimmy at openstack.org> wrote:
>>>>>>>>>
>>>>>>>>>> Super thankful for all the folks that have jumped in over the
>>>>>>>>>> last couple of days to help with the puppetization, etc... I just feel like
>>>>>>>>>> we're taking a very wrong approach here.
>>>>>>>>>>
>>>>>>>>>> Paul Belanger wrote:
>>>>>>>>>>
>>>>>>>>>> Right, and I don't have an issue with that approach.  Based on the work we did
>>>>>>>>>> yesterday, anybody can do that via our workflow. Please submit a patch to
>>>>>>>>>> puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.
>>>>>>>>>>
>>>>>>>>>> What I'm proposing is the workflow is really meant for software,
>>>>>>>>>> not for web applications. It's tedious and time consuming when what's
>>>>>>>>>> needed here is a set of tests on the server. Submitting a patch, waiting
>>>>>>>>>> for a +1, then getting on IRC to find someone with access (and time) to
>>>>>>>>>> paste the logs is a pretty time consuming process for what should be a
>>>>>>>>>> series of rapid-fire changes/fixes on the server. Especially when we're
>>>>>>>>>> dealign with an active attack.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We can then have somebody look at the logs.  I think it is more about scheduling
>>>>>>>>>> the task since more infra-root as travling back from the mid-cycle last night
>>>>>>>>>> and today.
>>>>>>>>>>
>>>>>>>>>> Right, this is my point. This has been going on for 3 weeks (or
>>>>>>>>>> more). Tom Fifeldt was asking for help without response. And here we are
>>>>>>>>>> through another week and no closer to stemming the flow.
>>>>>>>>>>
>>>>>>>>>> I'm fully aware what I'm proposing goes against what Infra and
>>>>>>>>>> the OpenStack workflow is all about, but I'd ask you all to look at this
>>>>>>>>>> from a web development perspective instead of a software development
>>>>>>>>>> perspective.
>>>>>>>>>>
>>>>>>>>>> Jimmy
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Last email from me, just on a plane.  Will follow up when I land.
>>>>>>>>>>
>>>>>>>>>> [1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki
>>>>>>>>>>
>>>>>>>>>>  J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com
>>>>>>>>>> [http://www.fibercove.com]
>>>>>>>>>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <pabelanger at redhat.com> <pabelanger at redhat.com>
>>>>>>>>>> wrote:
>>>>>>>>>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
>>>>>>>>>>
>>>>>>>>>> Given the state of the wiki a the moment, I think taking the quickest path
>>>>>>>>>> to get it fixed would be prudent. Is there a way we can get JP root access
>>>>>>>>>> to this server, even temporarily? We get 25% of our website traffic (2
>>>>>>>>>> million visitors) to the wiki. I realize we're all after the same thing,
>>>>>>>>>>
>>>>>>>>>> but
>>>>>>>>>>
>>>>>>>>>> spammers are not going to hit the dev environment, so there's really no
>>>>>>>>>>
>>>>>>>>>> way
>>>>>>>>>>
>>>>>>>>>> to tell if teh problem is fixed without actually working directly on the
>>>>>>>>>> production machine. This should be a 30 minute fix.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I am still unclear what the 30min fix is. If really 30mins, then it
>>>>>>>>>> shouldn't be
>>>>>>>>>> hard to get the fix into our workflow. Could somebody please elaborate.
>>>>>>>>>>
>>>>>>>>>> If we are talking about deploying new versions of php or mediawiki manually,
>>>>>>>>>> I
>>>>>>>>>> not be in-favor of this. To me, while the attack sucks, we should be working
>>>>>>>>>> on
>>>>>>>>>> 2 fronts. Getting the help needed to mitigate the attack, then adding the
>>>>>>>>>> changes into -infra workflow in parallel.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> I realize there is a lot of risk in giving ssh access to infra machines,
>>>>>>>>>>
>>>>>>>>>> but
>>>>>>>>>>
>>>>>>>>>> I think it's worth taking a look at either putting this machine in a place
>>>>>>>>>> where a different level of admin could access it without giving away the
>>>>>>>>>> keys to the entire OpenStack infrastructure or figuring out a way to set
>>>>>>>>>>
>>>>>>>>>> up
>>>>>>>>>>
>>>>>>>>>> credentials with varying levels of access.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> As a note, all the work I've been doing to help with the attack hasn't
>>>>>>>>>> require
>>>>>>>>>> SSH access for me to wiki.o.o. I did need infra-root help to expose our
>>>>>>>>>> configuration safely. I'd rather take some time to see what the fixes are,
>>>>>>>>>> having infra-root apply changes, then move them into puppet.
>>>>>>>>>>
>>>>>>>>>> It also has been discussed to simply disable write access to the wiki if we
>>>>>>>>>> really want spamming to stop, obviously that will affect normal usage.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Jimmy
>>>>>>>>>>
>>>>>>>>>> Paul Belanger wrote:
>>>>>>>>>>
>>>>>>>>>> On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
>>>>>>>>>>
>>>>>>>>>> But if you wanted to upgrade everything, remove the mobile view
>>>>>>>>>>
>>>>>>>>>> extension,
>>>>>>>>>>
>>>>>>>>>> test in a dev/staging environment then deploy to production fingers
>>>>>>>>>> crossed, I think that would be a valid approach as well.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Current review up[1]. I'll launch a node tonight / tomorrow locally to
>>>>>>>>>>
>>>>>>>>>> see
>>>>>>>>>> how
>>>>>>>>>>
>>>>>>>>>> puppet reacts. I suspect there will be some issues.
>>>>>>>>>>
>>>>>>>>>> If infra-roots are fine with this approach, we can use that box to test
>>>>>>>>>>
>>>>>>>>>> against.
>>>>>>>>>>
>>>>>>>>>> [1] https://review.openstack.org/#/c/285405/
>>>>>>>>>>
>>>>>>>>>> J.P. Maxwell | tipit.net | fibercove.com
>>>>>>>>>> On Feb 26, 2016 10:08 AM, "JP Maxwell"<jp at tipit.net> <jp at tipit.net> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Plus one except in this case it is much easier to know if our efforts
>>>>>>>>>>
>>>>>>>>>> are
>>>>>>>>>>
>>>>>>>>>> working on production because the spam either stops or not.
>>>>>>>>>>
>>>>>>>>>> J.P. Maxwell | tipit.net | fibercove.com
>>>>>>>>>> On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelanger at redhat.com> <pabelanger at redhat.com> wrote:
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
>>>>>>>>>>
>>>>>>>>>> I really think you might consider the option that there is a
>>>>>>>>>>
>>>>>>>>>> vulnerability
>>>>>>>>>>
>>>>>>>>>> in one of the extensions. If that is the case black listing IPs will
>>>>>>>>>>
>>>>>>>>>> be
>>>>>>>>>>
>>>>>>>>>> an
>>>>>>>>>>
>>>>>>>>>> ongoing wild goose chase.
>>>>>>>>>>
>>>>>>>>>> I think this would be easily proven or disproven by making the questy
>>>>>>>>>> question impossible and see if the spam continues.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> We'll have to let an infra-root make that call. Since nobody would be
>>>>>>>>>> able to
>>>>>>>>>> use the wiki. Honestly, I'd rather spend the time standing up a mirror
>>>>>>>>>>
>>>>>>>>>> dev
>>>>>>>>>>
>>>>>>>>>> instance for us to work on, rather then production.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> J.P. Maxwell | tipit.net | fibercove.com
>>>>>>>>>> On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelanger at redhat.com> <pabelanger at redhat.com>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
>>>>>>>>>>
>>>>>>>>>> On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<fungi at yuggoth.org> <fungi at yuggoth.org>
>>>>>>>>>>
>>>>>>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
>>>>>>>>>>
>>>>>>>>>> Please be aware that you can now create accounts under the mobile
>>>>>>>>>> view in the wiki native user table. I just created an account for
>>>>>>>>>> JpMaxMan. Not sure if this matters but wanted to make sure you
>>>>>>>>>> were aware.
>>>>>>>>>>
>>>>>>>>>> Oh, yes I think having a random garbage question/answer was in
>>>>>>>>>>
>>>>>>>>>> fact
>>>>>>>>>>
>>>>>>>>>> previously preventing account creation under the mobile view. We
>>>>>>>>>> probably need a way to disable mobile view account creation as it
>>>>>>>>>> bypasses OpenID authentication entirely.
>>>>>>>>>>
>>>>>>>>>> So that's what it was doing! We'll have to tackle the mobile view
>>>>>>>>>>
>>>>>>>>>> issue.
>>>>>>>>>>
>>>>>>>>>> Otherwise, quick update here:
>>>>>>>>>>
>>>>>>>>>> The captcha didn't appear to help stem the spam tide. We'll want to
>>>>>>>>>> explore and start implementing some of the other solutions.
>>>>>>>>>>
>>>>>>>>>> I did some database poking around today and it does seem like all
>>>>>>>>>>
>>>>>>>>>> the
>>>>>>>>>>
>>>>>>>>>> users do have launchpad accounts and email addresses.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> So, I have a few hours before jumping on my plane and checked into
>>>>>>>>>>
>>>>>>>>>> this.
>>>>>>>>>>
>>>>>>>>>> We are
>>>>>>>>>> using QuestyCaptcha which according to docs, should almost be
>>>>>>>>>>
>>>>>>>>>> impossible
>>>>>>>>>>
>>>>>>>>>> for
>>>>>>>>>> spammers to by pass in an automated fashion. So, either our captcha
>>>>>>>>>>
>>>>>>>>>> is too
>>>>>>>>>>
>>>>>>>>>> easy, or we didn't set it up properly. I don't have SSH on wiki.o.o
>>>>>>>>>>
>>>>>>>>>> so
>>>>>>>>>>
>>>>>>>>>> others
>>>>>>>>>> will have to check logs. I did test new pages and edits, and was
>>>>>>>>>>
>>>>>>>>>> promoted
>>>>>>>>>>
>>>>>>>>>> by
>>>>>>>>>> captcha.
>>>>>>>>>>
>>>>>>>>>> As a next step, we might need to add additional apache2
>>>>>>>>>>
>>>>>>>>>> configuration
>>>>>>>>>>
>>>>>>>>>> to
>>>>>>>>>>
>>>>>>>>>> blacklist IPs. I am reading up on that now.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Elizabeth Krumbach Joseph || Lyz || pleia2
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OpenStack-Infra mailing listOpenStack-Infra at lists.openstack.org
>>>>>>>>>>
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OpenStack-Infra mailing listOpenStack-Infra at lists.openstack.org
>>>>>>>>>>
>>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OpenStack-Infra mailing listOpenStack-Infra at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OpenStack-Infra mailing listOpenStack-Infra at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> OpenStack-Infra mailing listOpenStack-Infra at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> OpenStack-Infra mailing list
>>>>>>>>> OpenStack-Infra at lists.openstack.org
>>>>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>>>>>>>
>>>>>>>>
>> _______________________________________________
>> OpenStack-Infra mailing list
>> OpenStack-Infra at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20160227/9ae04ec9/attachment-0001.html>

Open Stack

[OpenStack-Infra] Wiki.o.o sustaining spam attack

OpenStack

Community

Documentation

Branding & Legal