<div dir="ltr">Yeah, the Settings.php was overriden by the latest puppet run. We need to wait for some infra guys to approve my patches and make it permanent:<div><a href="https://review.openstack.org/285669">https://review.openstack.org/285669</a> Disable standard password based auth<br></div><div><div><a href="https://review.openstack.org/285672">https://review.openstack.org/285672</a> Disable mobile frontend</div></div><div><br></div><div>M.</div></div><br><div class="gmail_quote"><div dir="ltr">On Sat, Feb 27, 2016 at 2:27 PM JP Maxwell <<a href="mailto:jp@tipit.net">jp@tipit.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">FYI. Still seeing the mobile view...</p>
<p dir="ltr">J.P. Maxwell | <a href="http://tipit.net" target="_blank">tipit.net</a> | <a href="http://fibercove.com" target="_blank">fibercove.com</a></p>
<div class="gmail_quote">On Feb 27, 2016 6:53 AM, "Marton Kiss" <<a href="mailto:marton.kiss@gmail.com" target="_blank">marton.kiss@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Yes, applied them manually. Let's wait a few hours, and check for new spam content / user accounts.<br><br>M.<br><div class="gmail_quote"><div dir="ltr">JP Maxwell <<a href="mailto:jp@tipit.net" target="_blank">jp@tipit.net</a>> (időpont: 2016. febr. 27., Szo, 13:50) ezt írta:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><p dir="ltr">Cool. Are these applied? Any indication it has stopped the spam? Should we clear out these non launchpad accounts from the DB?</p>
<p dir="ltr">J.P. Maxwell | <a href="http://tipit.net" target="_blank">tipit.net</a> | <a href="http://fibercove.com" target="_blank">fibercove.com</a></p>
<div class="gmail_quote">On Feb 27, 2016 6:47 AM, "Marton Kiss" <<a href="mailto:marton.kiss@gmail.com" target="_blank">marton.kiss@gmail.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">And the mobile frontend will be disabled permanently with this patch:<div><a href="https://review.openstack.org/285672" target="_blank">https://review.openstack.org/285672</a> Disable mobile frontend<br></div><div><br></div><div>M.</div></div><br><div class="gmail_quote"><div dir="ltr">On Sat, Feb 27, 2016 at 1:39 PM Marton Kiss <<a href="mailto:marton.kiss@gmail.com" target="_blank">marton.kiss@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">I made some investigation, and it seems to be that the spam pages are created by accounts registered with password accounts, and the launchpad openid auth is not affected at all.<div><br></div><div>So the spam script is creating accounts like this:</div><div><div>mysql> select * from user where user_name = 'CedricJamieson'\G;</div><div>*************************** 1. row ***************************</div><div> user_id: 7494</div><div> user_name: CedricJamieson</div><div> user_real_name: Cedric Jamieson</div><div> user_password: :pbkdf2:sha256:10000:128:Mlo9tdaP+38niZrrEka7Ow==:jEVnrTclkwIpE1RzJywDlrSvkY5G3idYwOwYRkv5O0J/MSHjY+gdhtKmArQ53v6/w7o8E1wXb2QOR6HdL5TPfOI1bswS/fYXVVYjPjkEEdxqZ8q9L5p2f3N6rEYpMfT5tk+wDiy+j5aimrHrGSga44hndAHgX9/SnqUyxlutDVY=</div><div> user_newpassword:</div><div> user_newpass_time: NULL</div><div> user_email: <a href="mailto:balashkina.evdokiya@mail.ru" target="_blank">balashkina.evdokiya@mail.ru</a></div><div> user_touched: 20160227052454</div><div> user_token: 7c39e44e849fb0e2bfae8790d6cc1379</div><div>user_email_authenticated: NULL</div><div> user_email_token: be963ac3bd43e70ff2f323063c61e320</div><div>user_email_token_expires: 20160305052441</div><div> user_registration: 20160227052441</div><div> user_editcount: 2</div><div> user_password_expires: NULL</div></div><div><br></div><div>The user_password field is always filled with a value, meanwhile this field of non-infected user accounts with openid logins is empty.</div><div>We have 423 total accounts with passwords:</div><div><div>mysql> select count(*) from user where user_password != '';</div><div>+----------+</div><div>| count(*) |</div><div>+----------+</div><div>| 423 |</div><div>+----------+</div><div>1 row in set (0.00 sec)</div></div><div><br></div><div>Mediawiki logs-in the newly created users without any preliminary email confirmation, right after the registration. I disabled the standard user login page, as described here: <a href="https://www.mediawiki.org/wiki/Manual:Special_pages#Disabling_Special:UserLogin_and_Special:UserLogout_pages" target="_blank">https://www.mediawiki.org/wiki/Manual:Special_pages#Disabling_Special:UserLogin_and_Special:UserLogout_pages</a></div><div><br></div><div>And I made this patch to make it permanent:</div><div><a href="https://review.openstack.org/285669" target="_blank">https://review.openstack.org/285669</a> Disable standard password based auth<br></div><div><br></div><div>Just for the record, the last spam user account:</div><div>7536 | EarthaChester22<br></div></div><div dir="ltr"><div><br></div><div>Marton</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Sat, Feb 27, 2016 at 8:31 AM Marton Kiss <<a href="mailto:marton.kiss@gmail.com" target="_blank">marton.kiss@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi,<div><br></div><div>I created the following patch, infra cores must approve that:</div><div><a href="https://review.openstack.org/285641" target="_blank">https://review.openstack.org/285641</a> Add ssh key of JP Maxwell to wiki.o.o<br></div></div><div dir="ltr"><div><br></div><div>Marton</div></div><br><div class="gmail_quote"><div dir="ltr">On Sat, Feb 27, 2016 at 6:41 AM JP Maxwell <<a href="mailto:jp@tipit.net" target="_blank">jp@tipit.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Marton has SSH access and applied a patch earlier today. It appears the spam continues to flow:<div><br></div><div><a href="https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen" target="_blank">https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen</a><br></div><div><br></div><div>Marton let me know if you can look at it some more or Infra if you want to give me SSH I'll do so as well in the morning (public key attached). </div><div><br></div><div><br></div><div><br></div><div>
<p><span>ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q== <a href="mailto:jpmaxman@tipit.net" target="_blank">jpmaxman@tipit.net</a></span></p><p><span><br></span></p><p><span><br></span></p></div></div><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><span style="font-family:arial,helvetica,sans-serif">J.P. Maxwell / <a href="http://www.tipit.net" target="_blank">tipit.net</a> <br><br></span></div></div></div></div></div></div><div class="gmail_extra">
<br><div class="gmail_quote">On Fri, Feb 26, 2016 at 12:09 PM, Jimmy McArthur <span dir="ltr"><<a href="mailto:jimmy@openstack.org" target="_blank">jimmy@openstack.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">Super thankful for all the
folks that have jumped in over the last couple of days to help with the
puppetization, etc... I just feel like we're taking a very wrong
approach here. <br><span>
<br>
<span>Paul Belanger wrote:</span><br>
<blockquote type="cite">
<pre>Right, and I don't have an issue with that approach. Based on the work we did
yesterday, anybody can do that via our workflow. Please submit a patch to
puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.</pre>
</blockquote></span>
What I'm proposing is the workflow is really meant for software, not for
web applications. It's tedious and time consuming when what's needed
here is a set of tests on the server. Submitting a patch, waiting for a
+1, then getting on IRC to find someone with access (and time) to paste
the logs is a pretty time consuming process for what should be a series
of rapid-fire changes/fixes on the server. Especially when we're dealign
with an active attack. <br><span>
<blockquote type="cite">
<pre>
We can then have somebody look at the logs. I think it is more about scheduling
the task since more infra-root as travling back from the mid-cycle last night
and today.</pre>
</blockquote></span>
Right, this is my point. This has been going on for 3 weeks (or more).
Tom Fifeldt was asking for help without response. And here we are
through another week and no closer to stemming the flow. <br>
<br>
I'm fully aware what I'm proposing goes against what Infra and the
OpenStack workflow is all about, but I'd ask you all to look at this
from a web development perspective instead of a software development
perspective. <br><span><font color="#888888">
<br>
Jimmy</font></span><div><div><br>
<blockquote type="cite">
<pre>
Last email from me, just on a plane. Will follow up when I land.
[1] <a href="https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki" target="_blank">https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki</a>
</pre>
<blockquote type="cite"><pre>J.P. Maxwell | <a href="http://tipit.net" target="_blank">tipit.net</a> [<a href="http://tipit.net" target="_blank">http://tipit.net</a>] | <a href="http://fibercove.com" target="_blank">fibercove.com</a>
[<a href="http://www.fibercove.com" target="_blank">http://www.fibercove.com</a>]
On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <a href="mailto:pabelanger@redhat.com" target="_blank"><pabelanger@redhat.com></a>
wrote:
On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
</pre><blockquote type="cite"><pre>Given the state of the wiki a the moment, I think taking the quickest path
to get it fixed would be prudent. Is there a way we can get JP root access
to this server, even temporarily? We get 25% of our website traffic (2
million visitors) to the wiki. I realize we're all after the same thing,
</pre></blockquote><pre>but
</pre><blockquote type="cite"><pre>spammers are not going to hit the dev environment, so there's really no
</pre></blockquote><pre>way
</pre><blockquote type="cite"><pre>to tell if teh problem is fixed without actually working directly on the
production machine. This should be a 30 minute fix.
</pre></blockquote><pre>I am still unclear what the 30min fix is. If really 30mins, then it
shouldn't be
hard to get the fix into our workflow. Could somebody please elaborate.
If we are talking about deploying new versions of php or mediawiki manually,
I
not be in-favor of this. To me, while the attack sucks, we should be working
on
2 fronts. Getting the help needed to mitigate the attack, then adding the
changes into -infra workflow in parallel.
</pre><blockquote type="cite"><pre>I realize there is a lot of risk in giving ssh access to infra machines,
</pre></blockquote><pre>but
</pre><blockquote type="cite"><pre>I think it's worth taking a look at either putting this machine in a place
where a different level of admin could access it without giving away the
keys to the entire OpenStack infrastructure or figuring out a way to set
</pre></blockquote><pre>up
</pre><blockquote type="cite"><pre>credentials with varying levels of access.
</pre></blockquote><pre>As a note, all the work I've been doing to help with the attack hasn't
require
SSH access for me to wiki.o.o. I did need infra-root help to expose our
configuration safely. I'd rather take some time to see what the fixes are,
having infra-root apply changes, then move them into puppet.
It also has been discussed to simply disable write access to the wiki if we
really want spamming to stop, obviously that will affect normal usage.
</pre><blockquote type="cite"><pre>Jimmy
Paul Belanger wrote:
</pre><blockquote type="cite"><pre>On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
</pre><blockquote type="cite"><pre>But if you wanted to upgrade everything, remove the mobile view
</pre></blockquote></blockquote></blockquote><pre>extension,
</pre><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre>test in a dev/staging environment then deploy to production fingers
crossed, I think that would be a valid approach as well.
</pre></blockquote><pre>Current review up[1]. I'll launch a node tonight / tomorrow locally to
</pre></blockquote></blockquote><pre>see
how
</pre><blockquote type="cite"><blockquote type="cite"><pre>puppet reacts. I suspect there will be some issues.
If infra-roots are fine with this approach, we can use that box to test
</pre></blockquote></blockquote><pre>against.
</pre><blockquote type="cite"><blockquote type="cite"><pre>[1] <a href="https://review.openstack.org/#/c/285405/" target="_blank">https://review.openstack.org/#/c/285405/</a>
</pre><blockquote type="cite"><pre>J.P. Maxwell | <a href="http://tipit.net" target="_blank">tipit.net</a> | <a href="http://fibercove.com" target="_blank">fibercove.com</a>
On Feb 26, 2016 10:08 AM, "JP Maxwell"<a href="mailto:jp@tipit.net" target="_blank"><jp@tipit.net></a> wrote:
</pre><blockquote type="cite"><pre>Plus one except in this case it is much easier to know if our efforts
</pre></blockquote></blockquote></blockquote></blockquote><pre>are
</pre><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre>working on production because the spam either stops or not.
J.P. Maxwell | <a href="http://tipit.net" target="_blank">tipit.net</a> | <a href="http://fibercove.com" target="_blank">fibercove.com</a>
On Feb 26, 2016 9:48 AM, "Paul Belanger"<a href="mailto:pabelanger@redhat.com" target="_blank"><pabelanger@redhat.com></a> wrote:
</pre><blockquote type="cite"><pre>On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
</pre><blockquote type="cite"><pre>I really think you might consider the option that there is a
</pre></blockquote><pre>vulnerability
</pre><blockquote type="cite"><pre>in one of the extensions. If that is the case black listing IPs will
</pre></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><pre>be
</pre><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre>an
</pre><blockquote type="cite"><pre>ongoing wild goose chase.
I think this would be easily proven or disproven by making the questy
question impossible and see if the spam continues.
</pre></blockquote><pre>We'll have to let an infra-root make that call. Since nobody would be
able to
use the wiki. Honestly, I'd rather spend the time standing up a mirror
</pre></blockquote></blockquote></blockquote></blockquote></blockquote><pre>dev
</pre><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre>instance for us to work on, rather then production.
</pre><blockquote type="cite"><pre>J.P. Maxwell | <a href="http://tipit.net" target="_blank">tipit.net</a> | <a href="http://fibercove.com" target="_blank">fibercove.com</a>
On Feb 26, 2016 9:12 AM, "Paul Belanger"<a href="mailto:pabelanger@redhat.com" target="_blank"><pabelanger@redhat.com></a>
</pre></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><pre>wrote:
</pre><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
</pre><blockquote type="cite"><pre>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<a href="mailto:fungi@yuggoth.org" target="_blank"><fungi@yuggoth.org></a>
</pre></blockquote><pre>wrote:
</pre><blockquote type="cite"><blockquote type="cite"><pre>On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
</pre><blockquote type="cite"><pre>Please be aware that you can now create accounts under the mobile
view in the wiki native user table. I just created an account for
JpMaxMan. Not sure if this matters but wanted to make sure you
were aware.
</pre></blockquote><pre>Oh, yes I think having a random garbage question/answer was in
</pre></blockquote></blockquote></blockquote></blockquote><pre>fact
</pre><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre>previously preventing account creation under the mobile view. We
probably need a way to disable mobile view account creation as it
bypasses OpenID authentication entirely.
</pre></blockquote><pre>So that's what it was doing! We'll have to tackle the mobile view
</pre></blockquote></blockquote></blockquote><pre>issue.
</pre><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre>Otherwise, quick update here:
The captcha didn't appear to help stem the spam tide. We'll want to
explore and start implementing some of the other solutions.
I did some database poking around today and it does seem like all
</pre></blockquote></blockquote></blockquote><pre>the
</pre><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre>users do have launchpad accounts and email addresses.
</pre></blockquote><pre>So, I have a few hours before jumping on my plane and checked into
</pre></blockquote></blockquote><pre>this.
</pre><blockquote type="cite"><blockquote type="cite"><pre>We are
using QuestyCaptcha which according to docs, should almost be
</pre></blockquote></blockquote><pre>impossible
</pre><blockquote type="cite"><blockquote type="cite"><pre>for
spammers to by pass in an automated fashion. So, either our captcha
</pre></blockquote></blockquote><pre>is too
</pre><blockquote type="cite"><blockquote type="cite"><pre>easy, or we didn't set it up properly. I don't have SSH on wiki.o.o
</pre></blockquote></blockquote><pre>so
</pre><blockquote type="cite"><blockquote type="cite"><pre>others
will have to check logs. I did test new pages and edits, and was
</pre></blockquote></blockquote><pre>promoted
</pre><blockquote type="cite"><blockquote type="cite"><pre>by
captcha.
As a next step, we might need to add additional apache2
</pre></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><pre>configuration
</pre><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre>to
</pre><blockquote type="cite"><blockquote type="cite"><pre>blacklist IPs. I am reading up on that now.
</pre><blockquote type="cite"><pre>--
Elizabeth Krumbach Joseph || Lyz || pleia2
_______________________________________________
OpenStack-Infra mailing list
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a>
</pre></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a>
_______________________________________________
OpenStack-Infra mailing list
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a>
</pre></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><pre><a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a>
</pre></blockquote></blockquote></blockquote></blockquote><pre>_______________________________________________
OpenStack-Infra mailing list
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a>
</pre></blockquote></blockquote><blockquote type="cite"><pre>_______________________________________________
OpenStack-Infra mailing list
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a>
</pre></blockquote><pre>_______________________________________________
OpenStack-Infra mailing list
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a>
</pre></blockquote>
</blockquote>
<br>
</div></div></div>
</blockquote></div><br></div>
_______________________________________________<br>
OpenStack-Infra mailing list<br>
<a href="mailto:OpenStack-Infra@lists.openstack.org" target="_blank">OpenStack-Infra@lists.openstack.org</a><br>
<a href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra" rel="noreferrer" target="_blank">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a><br>
</blockquote></div></blockquote></div></blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>