Open Stack

On phone but patch puppet-mediawiki and enable captcha for all pages. We only did edit and create On Feb 26, 2016 10:38 AM, Marton Kiss <marton.kiss at gmail.com> wrote:
I see a ton of incoming post requests:

POST
/w/index.php?title=Special%3ARunJobs&tasks=jobs&maxjobs=1&sigexpiry=1456508270&signature=571cfb216f944b15d2eee1c0253d08b77003328e

M.

On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss <marton.kiss at gmail.com> wrote:

> Oh, I can login. So what we need?
>
> M.
>
> On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell <jp at tipit.net> wrote:
>
>> I think what Jimmy is referring to is what I was suggesting by removing
>> the extensions / making the question impossible to answer.  Basically a
>> series of rapid fire changes while tailing the logs and seeing what stops
>> the spam.  Once you know what worked then you can submit as an official
>> patch.  But being able to quickly try these things on a server actually
>> under attack is the fastest path toward identifying the fix.
>>
>> *J.P. Maxwell* | tipit.net | fibercove.com <http://www.fibercove.com>
>>
>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <pabelanger at redhat.com>
>> wrote:
>>
>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
>> > Given the state of the wiki a the moment, I think taking the quickest
>> path
>> > to get it fixed would be prudent. Is there a way we can get JP root
>> access
>> > to this server, even temporarily? We get 25% of our website traffic (2
>> > million visitors) to the wiki. I realize we're all after the same
>> thing, but
>> > spammers are not going to hit the dev environment, so there's really no
>> way
>> > to tell if teh problem is fixed without actually working directly on the
>> > production machine. This should be a 30 minute fix.
>> >
>> I am still unclear what the 30min fix is. If really 30mins, then it
>> shouldn't be
>> hard to get the fix into our workflow. Could somebody please elaborate.
>>
>> If we are talking about deploying new versions of php or mediawiki
>> manually, I
>> not be in-favor of this. To me, while the attack sucks, we should be
>> working on
>> 2 fronts. Getting the help needed to mitigate the attack, then adding the
>> changes into -infra workflow in parallel.
>>
>> > I realize there is a lot of risk in giving ssh access to infra
>> machines, but
>> > I think it's worth taking a look at either putting this machine in a
>> place
>> > where a different level of admin could access it without giving away the
>> > keys to the entire OpenStack infrastructure or figuring out a way to
>> set up
>> > credentials with varying levels of access.
>> >
>> As a note, all the work I've been doing to help with the attack hasn't
>> require
>> SSH access for me to wiki.o.o. I did need infra-root help to expose our
>> configuration safely. I'd rather take some time to see what the fixes are,
>> having infra-root apply changes, then move them into puppet.
>>
>> It also has been discussed to simply disable write access to the wiki if
>> we
>> really want spamming to stop, obviously that will affect normal usage.
>>
>> > Jimmy
>> >
>> > Paul Belanger wrote:
>> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
>> > >>But if you wanted to upgrade everything, remove the mobile view
>> extension,
>> > >>test in a dev/staging environment then deploy to production fingers
>> > >>crossed, I think that would be a valid approach as well.
>> > >>
>> > >Current review up[1]. I'll launch a node tonight / tomorrow locally to
>> see how
>> > >puppet reacts. I suspect there will be some issues.
>> > >
>> > >If infra-roots are fine with this approach, we can use that box to
>> test against.
>> > >
>> > >[1] https://review.openstack.org/#/c/285405/
>> > >
>> > >>J.P. Maxwell | tipit.net | fibercove.com
>> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell"<jp at tipit.net> wrote:
>> > >>
>> > >>>Plus one except in this case it is much easier to know if our
>> efforts are
>> > >>>working on production because the spam either stops or not.
>> > >>>
>> > >>>J.P. Maxwell | tipit.net | fibercove.com
>> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelanger at redhat.com>
>> wrote:
>> > >>>
>> > >>>>On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
>> > >>>>>I really think you might consider the option that there is a
>> > >>>>vulnerability
>> > >>>>>in one of the extensions. If that is the case black listing IPs
>> will be
>> > >>>>an
>> > >>>>>ongoing wild goose chase.
>> > >>>>>
>> > >>>>>I think this would be easily proven or disproven by making the
>> questy
>> > >>>>>question impossible and see if the spam continues.
>> > >>>>>
>> > >>>>We'll have to let an infra-root make that call. Since nobody would
>> be
>> > >>>>able to
>> > >>>>use the wiki. Honestly, I'd rather spend the time standing up a
>> mirror dev
>> > >>>>instance for us to work on, rather then production.
>> > >>>>
>> > >>>>>J.P. Maxwell | tipit.net | fibercove.com
>> > >>>>>On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelanger at redhat.com>
>> wrote:
>> > >>>>>
>> > >>>>>>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph
>> wrote:
>> > >>>>>>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<
>> fungi at yuggoth.org>
>> > >>>>>>wrote:
>> > >>>>>>>>On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
>> > >>>>>>>>>Please be aware that you can now create accounts under the
>> mobile
>> > >>>>>>>>>view in the wiki native user table. I just created an account
>> for
>> > >>>>>>>>>JpMaxMan. Not sure if this matters but wanted to make sure you
>> > >>>>>>>>>were aware.
>> > >>>>>>>>Oh, yes I think having a random garbage question/answer was in
>> > >>>>fact
>> > >>>>>>>>previously preventing account creation under the mobile view. We
>> > >>>>>>>>probably need a way to disable mobile view account creation as
>> it
>> > >>>>>>>>bypasses OpenID authentication entirely.
>> > >>>>>>>So that's what it was doing! We'll have to tackle the mobile view
>> > >>>>issue.
>> > >>>>>>>Otherwise, quick update here:
>> > >>>>>>>
>> > >>>>>>>The captcha didn't appear to help stem the spam tide. We'll want
>> to
>> > >>>>>>>explore and start implementing some of the other solutions.
>> > >>>>>>>
>> > >>>>>>>I did some database poking around today and it does seem like all
>> > >>>>the
>> > >>>>>>>users do have launchpad accounts and email addresses.
>> > >>>>>>>
>> > >>>>>>So, I have a few hours before jumping on my plane and checked into
>> > >>>>this.
>> > >>>>>>We are
>> > >>>>>>using QuestyCaptcha which according to docs, should almost be
>> > >>>>impossible
>> > >>>>>>for
>> > >>>>>>spammers to by pass in an automated fashion. So, either our
>> captcha
>> > >>>>is too
>> > >>>>>>easy, or we didn't set it up properly. I don't have SSH on
>> wiki.o.o
>> > >>>>so
>> > >>>>>>others
>> > >>>>>>will have to check logs. I did test new pages and edits, and was
>> > >>>>promoted
>> > >>>>>>by
>> > >>>>>>captcha.
>> > >>>>>>
>> > >>>>>>As a next step, we might need to add additional apache2
>> configuration
>> > >>>>to
>> > >>>>>>blacklist IPs. I am reading up on that now.
>> > >>>>>>
>> > >>>>>>>--
>> > >>>>>>>Elizabeth Krumbach Joseph || Lyz || pleia2
>> > >>>>>>>
>> > >>>>>>>_______________________________________________
>> > >>>>>>>OpenStack-Infra mailing list
>> > >>>>>>>OpenStack-Infra at lists.openstack.org
>> > >>>>>>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>> > >>>>>>_______________________________________________
>> > >>>>>>OpenStack-Infra mailing list
>> > >>>>>>OpenStack-Infra at lists.openstack.org
>> > >>>>>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>> > >>>>>>
>> > >
>> > >_______________________________________________
>> > >OpenStack-Infra mailing list
>> > >OpenStack-Infra at lists.openstack.org
>> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>> >
>>
>> > _______________________________________________
>> > OpenStack-Infra mailing list
>> > OpenStack-Infra at lists.openstack.org
>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>>
>> _______________________________________________
>> OpenStack-Infra mailing list
>> OpenStack-Infra at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>> _______________________________________________
>> OpenStack-Infra mailing list
>> OpenStack-Infra at lists.openstack.org
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20160226/c8aefba9/attachment.html>