Open Stack

Yeah, I checked it and it is internal job runner:
https://www.mediawiki.org/wiki/Manual:Job_queue

M.

On Fri, Feb 26, 2016 at 7:00 PM JP Maxwell <jp at tipit.net> wrote:

> A quick google indicates this may be an unrelated issue that should be
> fixed, but I don’t *think* it is related to the spam.
>
> *J.P. Maxwell* | tipit.net | fibercove.com <http://www.fibercove.com>
>
> On Fri, Feb 26, 2016 at 11:56 AM, Marton Kiss <marton.kiss at gmail.com>
> wrote:
>
> I'm going to get a dinner, but I'll be on irc after, so if I can help
> somehow, I will be here. #openstack-infra mrmartin
>
> M.
>
> On Fri, Feb 26, 2016 at 6:51 PM Paul Belanger <pbelange at redhat.com> wrote:
>
>> On phone but patch puppet-mediawiki and enable captcha for all pages. We
>> only did edit and create
>> On Feb 26, 2016 10:38 AM, Marton Kiss <marton.kiss at gmail.com> wrote:
>>
>> I see a ton of incoming post requests:
>>
>> POST
>> /w/index.php?title=Special%3ARunJobs&tasks=jobs&maxjobs=1&sigexpiry=1456508270&signature=571cfb216f944b15d2eee1c0253d08b77003328e
>>
>> M.
>>
>> On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss <marton.kiss at gmail.com>
>> wrote:
>>
>>> Oh, I can login. So what we need?
>>>
>>> M.
>>>
>>> On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell <jp at tipit.net> wrote:
>>>
>>>> I think what Jimmy is referring to is what I was suggesting by removing
>>>> the extensions / making the question impossible to answer. Basically a
>>>> series of rapid fire changes while tailing the logs and seeing what stops
>>>> the spam. Once you know what worked then you can submit as an official
>>>> patch. But being able to quickly try these things on a server actually
>>>> under attack is the fastest path toward identifying the fix.
>>>>
>>>> *J.P. Maxwell* | tipit.net | fibercove.com <http://www.fibercove.com>
>>>>
>>>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <pabelanger at redhat.com>
>>>> wrote:
>>>>
>>>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
>>>> > Given the state of the wiki a the moment, I think taking the quickest
>>>> path
>>>> > to get it fixed would be prudent. Is there a way we can get JP root
>>>> access
>>>> > to this server, even temporarily? We get 25% of our website traffic (2
>>>> > million visitors) to the wiki. I realize we're all after the same
>>>> thing, but
>>>> > spammers are not going to hit the dev environment, so there's really
>>>> no way
>>>> > to tell if teh problem is fixed without actually working directly on
>>>> the
>>>> > production machine. This should be a 30 minute fix.
>>>> >
>>>> I am still unclear what the 30min fix is. If really 30mins, then it
>>>> shouldn't be
>>>> hard to get the fix into our workflow. Could somebody please elaborate.
>>>>
>>>> If we are talking about deploying new versions of php or mediawiki
>>>> manually, I
>>>> not be in-favor of this. To me, while the attack sucks, we should be
>>>> working on
>>>> 2 fronts. Getting the help needed to mitigate the attack, then adding
>>>> the
>>>> changes into -infra workflow in parallel.
>>>>
>>>> > I realize there is a lot of risk in giving ssh access to infra
>>>> machines, but
>>>> > I think it's worth taking a look at either putting this machine in a
>>>> place
>>>> > where a different level of admin could access it without giving away
>>>> the
>>>> > keys to the entire OpenStack infrastructure or figuring out a way to
>>>> set up
>>>> > credentials with varying levels of access.
>>>> >
>>>> As a note, all the work I've been doing to help with the attack hasn't
>>>> require
>>>> SSH access for me to wiki.o.o. I did need infra-root help to expose our
>>>> configuration safely. I'd rather take some time to see what the fixes
>>>> are,
>>>> having infra-root apply changes, then move them into puppet.
>>>>
>>>> It also has been discussed to simply disable write access to the wiki
>>>> if we
>>>> really want spamming to stop, obviously that will affect normal usage.
>>>>
>>>> > Jimmy
>>>> >
>>>> > Paul Belanger wrote:
>>>> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
>>>> > >>But if you wanted to upgrade everything, remove the mobile view
>>>> extension,
>>>> > >>test in a dev/staging environment then deploy to production fingers
>>>> > >>crossed, I think that would be a valid approach as well.
>>>> > >>
>>>> > >Current review up[1]. I'll launch a node tonight / tomorrow locally
>>>> to see how
>>>> > >puppet reacts. I suspect there will be some issues.
>>>> > >
>>>> > >If infra-roots are fine with this approach, we can use that box to
>>>> test against.
>>>> > >
>>>> > >[1] https://review.openstack.org/#/c/285405/
>>>> > >
>>>> > >>J.P. Maxwell | tipit.net | fibercove.com
>>>> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell"<jp at tipit.net> wrote:
>>>> > >>
>>>> > >>>Plus one except in this case it is much easier to know if our
>>>> efforts are
>>>> > >>>working on production because the spam either stops or not.
>>>> > >>>
>>>> > >>>J.P. Maxwell | tipit.net | fibercove.com
>>>> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelanger at redhat.com>
>>>> wrote:
>>>> > >>>
>>>> > >>>>On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
>>>> > >>>>>I really think you might consider the option that there is a
>>>> > >>>>vulnerability
>>>> > >>>>>in one of the extensions. If that is the case black listing IPs
>>>> will be
>>>> > >>>>an
>>>> > >>>>>ongoing wild goose chase.
>>>> > >>>>>
>>>> > >>>>>I think this would be easily proven or disproven by making the
>>>> questy
>>>> > >>>>>question impossible and see if the spam continues.
>>>> > >>>>>
>>>> > >>>>We'll have to let an infra-root make that call. Since nobody
>>>> would be
>>>> > >>>>able to
>>>> > >>>>use the wiki. Honestly, I'd rather spend the time standing up a
>>>> mirror dev
>>>> > >>>>instance for us to work on, rather then production.
>>>> > >>>>
>>>> > >>>>>J.P. Maxwell | tipit.net | fibercove.com
>>>> > >>>>>On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelanger at redhat.com>
>>>> wrote:
>>>> > >>>>>
>>>> > >>>>>>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph
>>>> wrote:
>>>> > >>>>>>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<
>>>> fungi at yuggoth.org>
>>>> > >>>>>>wrote:
>>>> > >>>>>>>>On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
>>>> > >>>>>>>>>Please be aware that you can now create accounts under the
>>>> mobile
>>>> > >>>>>>>>>view in the wiki native user table. I just created an
>>>> account for
>>>> > >>>>>>>>>JpMaxMan. Not sure if this matters but wanted to make sure
>>>> you
>>>> > >>>>>>>>>were aware.
>>>> > >>>>>>>>Oh, yes I think having a random garbage question/answer was in
>>>> > >>>>fact
>>>> > >>>>>>>>previously preventing account creation under the mobile view.
>>>> We
>>>> > >>>>>>>>probably need a way to disable mobile view account creation
>>>> as it
>>>> > >>>>>>>>bypasses OpenID authentication entirely.
>>>> > >>>>>>>So that's what it was doing! We'll have to tackle the mobile
>>>> view
>>>> > >>>>issue.
>>>> > >>>>>>>Otherwise, quick update here:
>>>> > >>>>>>>
>>>> > >>>>>>>The captcha didn't appear to help stem the spam tide. We'll
>>>> want to
>>>> > >>>>>>>explore and start implementing some of the other solutions.
>>>> > >>>>>>>
>>>> > >>>>>>>I did some database poking around today and it does seem like
>>>> all
>>>> > >>>>the
>>>> > >>>>>>>users do have launchpad accounts and email addresses.
>>>> > >>>>>>>
>>>> > >>>>>>So, I have a few hours before jumping on my plane and checked
>>>> into
>>>> > >>>>this.
>>>> > >>>>>>We are
>>>> > >>>>>>using QuestyCaptcha which according to docs, should almost be
>>>> > >>>>impossible
>>>> > >>>>>>for
>>>> > >>>>>>spammers to by pass in an automated fashion. So, either our
>>>> captcha
>>>> > >>>>is too
>>>> > >>>>>>easy, or we didn't set it up properly. I don't have SSH on
>>>> wiki.o.o
>>>> > >>>>so
>>>> > >>>>>>others
>>>> > >>>>>>will have to check logs. I did test new pages and edits, and was
>>>> > >>>>promoted
>>>> > >>>>>>by
>>>> > >>>>>>captcha.
>>>> > >>>>>>
>>>> > >>>>>>As a next step, we might need to add additional apache2
>>>> configuration
>>>> > >>>>to
>>>> > >>>>>>blacklist IPs. I am reading up on that now.
>>>> > >>>>>>
>>>> > >>>>>>>--
>>>> > >>>>>>>Elizabeth Krumbach Joseph || Lyz || pleia2
>>>> > >>>>>>>
>>>> > >>>>>>>_______________________________________________
>>>> > >>>>>>>OpenStack-Infra mailing list
>>>> > >>>>>>>OpenStack-Infra at lists.openstack.org
>>>> > >>>>>>>
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>> > >>>>>>_______________________________________________
>>>> > >>>>>>OpenStack-Infra mailing list
>>>> > >>>>>>OpenStack-Infra at lists.openstack.org
>>>> > >>>>>>
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>> > >>>>>>
>>>> > >
>>>> > >_______________________________________________
>>>> > >OpenStack-Infra mailing list
>>>> > >OpenStack-Infra at lists.openstack.org
>>>> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>> >
>>>>
>>>> > _______________________________________________
>>>> > OpenStack-Infra mailing list
>>>> > OpenStack-Infra at lists.openstack.org
>>>> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>>
>>>>
>>>> _______________________________________________
>>>> OpenStack-Infra mailing list
>>>> OpenStack-Infra at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>>
>>>> _______________________________________________
>>>> OpenStack-Infra mailing list
>>>> OpenStack-Infra at lists.openstack.org
>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>>>
>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20160226/4c9445bd/attachment.html>