<html><head>
<meta content="text/html; charset=ISO-8859-1" http-equiv="Content-Type">
</head><body bgcolor="#FFFFFF" text="#000000">Super thankful for all the
folks that have jumped in over the last couple of days to help with the
puppetization, etc... I just feel like we're taking a very wrong
approach here. <br>
<br>
<span>Paul Belanger wrote:</span><br>
<blockquote cite="mid:20160226173610.GB11120@localhost.localdomain"
type="cite">
<pre wrap=""><!---->Right, and I don't have an issue with that approach. Based on the work we did
yesterday, anybody can do that via our workflow. Please submit a patch to
puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.</pre>
</blockquote>
What I'm proposing is the workflow is really meant for software, not for
web applications. It's tedious and time consuming when what's needed
here is a set of tests on the server. Submitting a patch, waiting for a
+1, then getting on IRC to find someone with access (and time) to paste
the logs is a pretty time consuming process for what should be a series
of rapid-fire changes/fixes on the server. Especially when we're dealign
with an active attack. <br>
<blockquote cite="mid:20160226173610.GB11120@localhost.localdomain"
type="cite">
<pre wrap="">
We can then have somebody look at the logs. I think it is more about scheduling
the task since more infra-root as travling back from the mid-cycle last night
and today.</pre>
</blockquote>
Right, this is my point. This has been going on for 3 weeks (or more).
Tom Fifeldt was asking for help without response. And here we are
through another week and no closer to stemming the flow. <br>
<br>
I'm fully aware what I'm proposing goes against what Infra and the
OpenStack workflow is all about, but I'd ask you all to look at this
from a web development perspective instead of a software development
perspective. <br>
<br>
Jimmy<br>
<blockquote cite="mid:20160226173610.GB11120@localhost.localdomain"
type="cite">
<pre wrap="">
Last email from me, just on a plane. Will follow up when I land.
[1] <a class="moz-txt-link-freetext" href="https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki">https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki</a>
</pre>
<blockquote type="cite"><pre wrap="">J.P. Maxwell | tipit.net [<a class="moz-txt-link-freetext" href="http://tipit.net">http://tipit.net</a>] | fibercove.com
[<a class="moz-txt-link-freetext" href="http://www.fibercove.com">http://www.fibercove.com</a>]
On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <a class="moz-txt-link-rfc2396E" href="mailto:pabelanger@redhat.com"><pabelanger@redhat.com></a>
wrote:
On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
</pre><blockquote type="cite"><pre wrap="">Given the state of the wiki a the moment, I think taking the quickest path
to get it fixed would be prudent. Is there a way we can get JP root access
to this server, even temporarily? We get 25% of our website traffic (2
million visitors) to the wiki. I realize we're all after the same thing,
</pre></blockquote><pre wrap="">but
</pre><blockquote type="cite"><pre wrap="">spammers are not going to hit the dev environment, so there's really no
</pre></blockquote><pre wrap="">way
</pre><blockquote type="cite"><pre wrap="">to tell if teh problem is fixed without actually working directly on the
production machine. This should be a 30 minute fix.
</pre></blockquote><pre wrap="">I am still unclear what the 30min fix is. If really 30mins, then it
shouldn't be
hard to get the fix into our workflow. Could somebody please elaborate.
If we are talking about deploying new versions of php or mediawiki manually,
I
not be in-favor of this. To me, while the attack sucks, we should be working
on
2 fronts. Getting the help needed to mitigate the attack, then adding the
changes into -infra workflow in parallel.
</pre><blockquote type="cite"><pre wrap="">I realize there is a lot of risk in giving ssh access to infra machines,
</pre></blockquote><pre wrap="">but
</pre><blockquote type="cite"><pre wrap="">I think it's worth taking a look at either putting this machine in a place
where a different level of admin could access it without giving away the
keys to the entire OpenStack infrastructure or figuring out a way to set
</pre></blockquote><pre wrap="">up
</pre><blockquote type="cite"><pre wrap="">credentials with varying levels of access.
</pre></blockquote><pre wrap="">As a note, all the work I've been doing to help with the attack hasn't
require
SSH access for me to wiki.o.o. I did need infra-root help to expose our
configuration safely. I'd rather take some time to see what the fixes are,
having infra-root apply changes, then move them into puppet.
It also has been discussed to simply disable write access to the wiki if we
really want spamming to stop, obviously that will affect normal usage.
</pre><blockquote type="cite"><pre wrap="">Jimmy
Paul Belanger wrote:
</pre><blockquote type="cite"><pre wrap="">On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
</pre><blockquote type="cite"><pre wrap="">But if you wanted to upgrade everything, remove the mobile view
</pre></blockquote></blockquote></blockquote><pre wrap="">extension,
</pre><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><pre wrap="">test in a dev/staging environment then deploy to production fingers
crossed, I think that would be a valid approach as well.
</pre></blockquote><pre wrap="">Current review up[1]. I'll launch a node tonight / tomorrow locally to
</pre></blockquote></blockquote><pre wrap="">see
how
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">puppet reacts. I suspect there will be some issues.
If infra-roots are fine with this approach, we can use that box to test
</pre></blockquote></blockquote><pre wrap="">against.
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">[1] <a class="moz-txt-link-freetext" href="https://review.openstack.org/#/c/285405/">https://review.openstack.org/#/c/285405/</a>
</pre><blockquote type="cite"><pre wrap="">J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 10:08 AM, "JP Maxwell"<a class="moz-txt-link-rfc2396E" href="mailto:jp@tipit.net"><jp@tipit.net></a> wrote:
</pre><blockquote type="cite"><pre wrap="">Plus one except in this case it is much easier to know if our efforts
</pre></blockquote></blockquote></blockquote></blockquote><pre wrap="">are
</pre><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><blockquote type="cite"><pre wrap="">working on production because the spam either stops or not.
J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 9:48 AM, "Paul Belanger"<a class="moz-txt-link-rfc2396E" href="mailto:pabelanger@redhat.com"><pabelanger@redhat.com></a> wrote:
</pre><blockquote type="cite"><pre wrap="">On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
</pre><blockquote type="cite"><pre wrap="">I really think you might consider the option that there is a
</pre></blockquote><pre wrap="">vulnerability
</pre><blockquote type="cite"><pre wrap="">in one of the extensions. If that is the case black listing IPs will
</pre></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><pre wrap="">be
</pre><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><blockquote type="cite"><blockquote type="cite"><pre wrap="">an
</pre><blockquote type="cite"><pre wrap="">ongoing wild goose chase.
I think this would be easily proven or disproven by making the questy
question impossible and see if the spam continues.
</pre></blockquote><pre wrap="">We'll have to let an infra-root make that call. Since nobody would be
able to
use the wiki. Honestly, I'd rather spend the time standing up a mirror
</pre></blockquote></blockquote></blockquote></blockquote></blockquote><pre wrap="">dev
</pre><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><blockquote type="cite"><blockquote type="cite"><pre wrap="">instance for us to work on, rather then production.
</pre><blockquote type="cite"><pre wrap="">J.P. Maxwell | tipit.net | fibercove.com
On Feb 26, 2016 9:12 AM, "Paul Belanger"<a class="moz-txt-link-rfc2396E" href="mailto:pabelanger@redhat.com"><pabelanger@redhat.com></a>
</pre></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><pre wrap="">wrote:
</pre><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><blockquote type="cite"><pre wrap="">On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
</pre><blockquote type="cite"><pre wrap="">On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<a class="moz-txt-link-rfc2396E" href="mailto:fungi@yuggoth.org"><fungi@yuggoth.org></a>
</pre></blockquote><pre wrap="">wrote:
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
</pre><blockquote type="cite"><pre wrap="">Please be aware that you can now create accounts under the mobile
view in the wiki native user table. I just created an account for
JpMaxMan. Not sure if this matters but wanted to make sure you
were aware.
</pre></blockquote><pre wrap="">Oh, yes I think having a random garbage question/answer was in
</pre></blockquote></blockquote></blockquote></blockquote><pre wrap="">fact
</pre><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><blockquote type="cite"><pre wrap="">previously preventing account creation under the mobile view. We
probably need a way to disable mobile view account creation as it
bypasses OpenID authentication entirely.
</pre></blockquote><pre wrap="">So that's what it was doing! We'll have to tackle the mobile view
</pre></blockquote></blockquote></blockquote><pre wrap="">issue.
</pre><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><pre wrap="">Otherwise, quick update here:
The captcha didn't appear to help stem the spam tide. We'll want to
explore and start implementing some of the other solutions.
I did some database poking around today and it does seem like all
</pre></blockquote></blockquote></blockquote><pre wrap="">the
</pre><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><pre wrap="">users do have launchpad accounts and email addresses.
</pre></blockquote><pre wrap="">So, I have a few hours before jumping on my plane and checked into
</pre></blockquote></blockquote><pre wrap="">this.
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">We are
using QuestyCaptcha which according to docs, should almost be
</pre></blockquote></blockquote><pre wrap="">impossible
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">for
spammers to by pass in an automated fashion. So, either our captcha
</pre></blockquote></blockquote><pre wrap="">is too
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">easy, or we didn't set it up properly. I don't have SSH on wiki.o.o
</pre></blockquote></blockquote><pre wrap="">so
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">others
will have to check logs. I did test new pages and edits, and was
</pre></blockquote></blockquote><pre wrap="">promoted
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">by
captcha.
As a next step, we might need to add additional apache2
</pre></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><pre wrap="">configuration
</pre><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><blockquote type="cite"><blockquote type="cite"><pre wrap="">to
</pre><blockquote type="cite"><blockquote type="cite"><pre wrap="">blacklist IPs. I am reading up on that now.
</pre><blockquote type="cite"><pre wrap="">--
Elizabeth Krumbach Joseph || Lyz || pleia2
_______________________________________________
OpenStack-Infra mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-Infra@lists.openstack.org">OpenStack-Infra@lists.openstack.org</a>
</pre></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote
type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><blockquote type="cite"><blockquote type="cite"><pre wrap=""><a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a>
_______________________________________________
OpenStack-Infra mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-Infra@lists.openstack.org">OpenStack-Infra@lists.openstack.org</a>
</pre></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><blockquote
type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote
type="cite"><blockquote type="cite"><pre wrap=""><a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a>
</pre></blockquote></blockquote></blockquote></blockquote><pre wrap="">_______________________________________________
OpenStack-Infra mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-Infra@lists.openstack.org">OpenStack-Infra@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a>
</pre></blockquote></blockquote><blockquote type="cite"><pre wrap="">_______________________________________________
OpenStack-Infra mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-Infra@lists.openstack.org">OpenStack-Infra@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a>
</pre></blockquote><pre wrap="">
_______________________________________________
OpenStack-Infra mailing list
<a class="moz-txt-link-abbreviated" href="mailto:OpenStack-Infra@lists.openstack.org">OpenStack-Infra@lists.openstack.org</a>
<a class="moz-txt-link-freetext" href="http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra">http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra</a>
</pre></blockquote>
</blockquote>
<br>
</body></html>