<div dir="ltr">I see a ton of incoming post requests:<div><br></div><div>POST /w/index.php?title=Special%3ARunJobs&tasks=jobs&maxjobs=1&sigexpiry=1456508270&signature=571cfb216f944b15d2eee1c0253d08b77003328e<br><div><br></div><div>M.</div></div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss <<a href="mailto:marton.kiss@gmail.com">marton.kiss@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Oh, I can login. So what we need?</div><div dir="ltr"><div><br></div><div>M.</div></div><br><div class="gmail_quote"><div dir="ltr">On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell <<a href="mailto:jp@tipit.net" target="_blank">jp@tipit.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="auto"><span>I think what Jimmy is referring to is what I was
suggesting by removing the extensions / making the question impossible to
answer. Basically a series of rapid fire changes while tailing the
logs and seeing what stops the spam. Once you know what worked
then you can submit as an official patch. But being able to quickly
try these things on a server actually under attack is the fastest path
toward identifying the fix. <br></span><br><div><div><b>J.P.
Maxwell</b> | <a href="http://tipit.net" target="_blank">tipit.net</a> | <a href="http://www.fibercove.com" target="_blank">fibercove.com</a></div></div></div><div dir="auto"><span><br></span><div><div>On Fri, Feb 26, 2016 at
11:25 AM, Paul Belanger <<a href="mailto:pabelanger@redhat.com" target="_blank">pabelanger@redhat.com</a>> wrote:<br><div style="overflow:visible"><blockquote style="margin:0;border-left:#d6d6d6 1px solid;padding-left:10px">On Fri,
Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:<br>> Given the
state of the wiki a the moment, I think taking the quickest path<br>> to
get it fixed would be prudent. Is there a way we can get JP root
access<br>> to this server, even temporarily? We get 25% of our website
traffic (2<br>> million visitors) to the wiki. I realize we're all after
the same thing, but<br>> spammers are not going to hit the dev
environment, so there's really no way<br>> to tell if teh problem is
fixed without actually working directly on the<br>> production machine.
This should be a 30 minute fix.<br>> <br>I am still unclear what the
30min fix is. If really 30mins, then it shouldn't be<br>hard to get the fix
into our workflow. Could somebody please elaborate.<br><br>If we are
talking about deploying new versions of php or mediawiki manually, I<br>not
be in-favor of this. To me, while the attack sucks, we should be working
on<br>2 fronts. Getting the help needed to mitigate the attack, then
adding the<br>changes into -infra workflow in parallel.<br><br>> I
realize there is a lot of risk in giving ssh access to infra machines,
but<br>> I think it's worth taking a look at either putting this machine
in a place<br>> where a different level of admin could access it without
giving away the<br>> keys to the entire OpenStack infrastructure or
figuring out a way to set up<br>> credentials with varying levels of
access.<br>> <br>As a note, all the work I've been doing to help with
the attack hasn't require<br>SSH access for me to wiki.o.o. I did need
infra-root help to expose our<br>configuration safely. I'd rather take
some time to see what the fixes are,<br>having infra-root apply changes,
then move them into puppet.<br><br>It also has been discussed to simply
disable write access to the wiki if we<br>really want spamming to stop,
obviously that will affect normal usage.<br><br>> Jimmy<br>> <br>>
Paul Belanger wrote:<br>> >On Fri, Feb 26, 2016 at 10:12:12AM -0600,
JP Maxwell wrote:<br>> >>But if you wanted to upgrade everything,
remove the mobile view extension,<br>> >>test in a dev/staging
environment then deploy to production fingers<br>> >>crossed, I
think that would be a valid approach as well.<br>> >><br>>
>Current review up[1]. I'll launch a node tonight / tomorrow locally to
see how<br>> >puppet reacts. I suspect there will be some
issues.<br>> ><br>> >If infra-roots are fine with this
approach, we can use that box to test against.<br>> ><br>> >[1]
<a href="https://review.openstack.org/#/c/285405/" target="_blank">https://review.openstack.org/#/c/285405/</a><br>> ><br>> >>J.P.
Maxwell | <a href="http://tipit.net" target="_blank">tipit.net</a> | <a href="http://fibercove.com" target="_blank">fibercove.com</a><br>> >>On Feb 26, 2016 10:08
AM, "JP Maxwell"<<a href="mailto:jp@tipit.net" target="_blank">jp@tipit.net</a>> wrote:<br>> >><br>>
>>>Plus one except in this case it is much easier to know if our
efforts are<br>> >>>working on production because the spam
either stops or not.<br>> >>><br>> >>>J.P. Maxwell
| <a href="http://tipit.net" target="_blank">tipit.net</a> | <a href="http://fibercove.com" target="_blank">fibercove.com</a><br>> >>>On Feb 26, 2016 9:48 AM,
"Paul Belanger"<<a href="mailto:pabelanger@redhat.com" target="_blank">pabelanger@redhat.com</a>> wrote:<br>>
>>><br>> >>>>On Fri, Feb 26, 2016 at 09:18:00AM
-0600, JP Maxwell wrote:<br>> >>>>>I really think you
might consider the option that there is a<br>>
>>>>vulnerability<br>> >>>>>in one of the
extensions. If that is the case black listing IPs will be<br>>
>>>>an<br>> >>>>>ongoing wild goose
chase.<br>> >>>>><br>> >>>>>I think
this would be easily proven or disproven by making the questy<br>>
>>>>>question impossible and see if the spam
continues.<br>> >>>>><br>> >>>>We'll have
to let an infra-root make that call. Since nobody would be<br>>
>>>>able to<br>> >>>>use the wiki. Honestly, I'd
rather spend the time standing up a mirror dev<br>>
>>>>instance for us to work on, rather then production.<br>>
>>>><br>> >>>>>J.P. Maxwell | <a href="http://tipit.net" target="_blank">tipit.net</a> |
<a href="http://fibercove.com" target="_blank">fibercove.com</a><br>> >>>>>On Feb 26, 2016 9:12 AM, "Paul
Belanger"<<a href="mailto:pabelanger@redhat.com" target="_blank">pabelanger@redhat.com</a>> wrote:<br>>
>>>>><br>> >>>>>>On Thu, Feb 25, 2016
at 08:10:34PM -0800, Elizabeth K. Joseph wrote:<br>>
>>>>>>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy
Stanley<<a href="mailto:fungi@yuggoth.org" target="_blank">fungi@yuggoth.org</a>><br>>
>>>>>>wrote:<br>> >>>>>>>>On
2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:<br>>
>>>>>>>>>Please be aware that you can now create
accounts under the mobile<br>> >>>>>>>>>view
in the wiki native user table. I just created an account for<br>>
>>>>>>>>>JpMaxMan. Not sure if this matters but
wanted to make sure you<br>> >>>>>>>>>were
aware.<br>> >>>>>>>>Oh, yes I think having a
random garbage question/answer was in<br>> >>>>fact<br>>
>>>>>>>>previously preventing account creation
under the mobile view. We<br>> >>>>>>>>probably
need a way to disable mobile view account creation as it<br>>
>>>>>>>>bypasses OpenID authentication
entirely.<br>> >>>>>>>So that's what it was doing!
We'll have to tackle the mobile view<br>> >>>>issue.<br>>
>>>>>>>Otherwise, quick update here:<br>>
>>>>>>><br>> >>>>>>>The
captcha didn't appear to help stem the spam tide. We'll want to<br>>
>>>>>>>explore and start implementing some of the
other solutions.<br>> >>>>>>><br>>
>>>>>>>I did some database poking around today and it
does seem like all<br>> >>>>the<br>>
>>>>>>>users do have launchpad accounts and email
addresses.<br>> >>>>>>><br>>
>>>>>>So, I have a few hours before jumping on my plane
and checked into<br>> >>>>this.<br>>
>>>>>>We are<br>> >>>>>>using
QuestyCaptcha which according to docs, should almost be<br>>
>>>>impossible<br>> >>>>>>for<br>>
>>>>>>spammers to by pass in an automated fashion. So,
either our captcha<br>> >>>>is too<br>>
>>>>>>easy, or we didn't set it up properly. I don't
have SSH on wiki.o.o<br>> >>>>so<br>>
>>>>>>others<br>> >>>>>>will have to
check logs. I did test new pages and edits, and was<br>>
>>>>promoted<br>> >>>>>>by<br>>
>>>>>>captcha.<br>> >>>>>><br>>
>>>>>>As a next step, we might need to add additional
apache2 configuration<br>> >>>>to<br>>
>>>>>>blacklist IPs. I am reading up on that
now.<br>> >>>>>><br>>
>>>>>>>--<br>>
>>>>>>>Elizabeth Krumbach Joseph || Lyz ||
pleia2<br>> >>>>>>><br>>
>>>>>>>_______________________________________________<br>>
>>>>>>>OpenStack-Infra mailing list<br>>
>>>>>>><a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a><br>>
>>>>>>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a><br>>
>>>>>>_______________________________________________<br>>
>>>>>>OpenStack-Infra mailing list<br>>
>>>>>><a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a><br>>
>>>>>><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a><br>>
>>>>>><br>> ><br>>
>_______________________________________________<br>>
>OpenStack-Infra mailing list<br>>
><a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a><br>>
><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a><br>>
<br><br>> _______________________________________________<br>>
OpenStack-Infra mailing list<br>>
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a><br>>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a><br><br><br>_______________________________________________<br>OpenStack-Infra
mailing
list<br><a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a><br><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a><br></blockquote></div></div></div></div>_______________________________________________<br>
OpenStack-Infra mailing list<br>
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a><br>
</blockquote></div></blockquote></div>