Open Stack

On Fri, Feb 26, 2016 at 11:29:31AM -0600, JP Maxwell wrote:
> I think what Jimmy is referring to is what I was suggesting by removing the
> extensions / making the question impossible to answer. Basically a series of
> rapid fire changes while tailing the logs and seeing what stops the spam.
> Once
> you know what worked then you can submit as an official patch. But being
> able to
> quickly try these things on a server actually under attack is the fastest
> path
> toward identifying the fix.
> 
Right, and I don't have an issue with that approach.  Based on the work we did
yesterday, anybody can do that via our workflow. Please submit a patch to
puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.

We can then have somebody look at the logs.  I think it is more about scheduling
the task since more infra-root as travling back from the mid-cycle last night
and today.

Last email from me, just on a plane.  Will follow up when I land.

[1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki


> J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com
> [http://www.fibercove.com]
> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <pabelanger at redhat.com>
> wrote:
> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
> > Given the state of the wiki a the moment, I think taking the quickest path
> > to get it fixed would be prudent. Is there a way we can get JP root access
> > to this server, even temporarily? We get 25% of our website traffic (2
> > million visitors) to the wiki. I realize we're all after the same thing,
> but
> > spammers are not going to hit the dev environment, so there's really no
> way
> > to tell if teh problem is fixed without actually working directly on the
> > production machine. This should be a 30 minute fix.
> >
> I am still unclear what the 30min fix is. If really 30mins, then it
> shouldn't be
> hard to get the fix into our workflow. Could somebody please elaborate.
> 
> If we are talking about deploying new versions of php or mediawiki manually,
> I
> not be in-favor of this. To me, while the attack sucks, we should be working
> on
> 2 fronts. Getting the help needed to mitigate the attack, then adding the
> changes into -infra workflow in parallel.
> 
> > I realize there is a lot of risk in giving ssh access to infra machines,
> but
> > I think it's worth taking a look at either putting this machine in a place
> > where a different level of admin could access it without giving away the
> > keys to the entire OpenStack infrastructure or figuring out a way to set
> up
> > credentials with varying levels of access.
> >
> As a note, all the work I've been doing to help with the attack hasn't
> require
> SSH access for me to wiki.o.o. I did need infra-root help to expose our
> configuration safely. I'd rather take some time to see what the fixes are,
> having infra-root apply changes, then move them into puppet.
> 
> It also has been discussed to simply disable write access to the wiki if we
> really want spamming to stop, obviously that will affect normal usage.
> 
> > Jimmy
> >
> > Paul Belanger wrote:
> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
> > >>But if you wanted to upgrade everything, remove the mobile view
> extension,
> > >>test in a dev/staging environment then deploy to production fingers
> > >>crossed, I think that would be a valid approach as well.
> > >>
> > >Current review up[1]. I'll launch a node tonight / tomorrow locally to
> see
> how
> > >puppet reacts. I suspect there will be some issues.
> > >
> > >If infra-roots are fine with this approach, we can use that box to test
> against.
> > >
> > >[1] https://review.openstack.org/#/c/285405/
> > >
> > >>J.P. Maxwell | tipit.net | fibercove.com
> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell"<jp at tipit.net> wrote:
> > >>
> > >>>Plus one except in this case it is much easier to know if our efforts
> are
> > >>>working on production because the spam either stops or not.
> > >>>
> > >>>J.P. Maxwell | tipit.net | fibercove.com
> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelanger at redhat.com> wrote:
> > >>>
> > >>>>On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> > >>>>>I really think you might consider the option that there is a
> > >>>>vulnerability
> > >>>>>in one of the extensions. If that is the case black listing IPs will
> be
> > >>>>an
> > >>>>>ongoing wild goose chase.
> > >>>>>
> > >>>>>I think this would be easily proven or disproven by making the questy
> > >>>>>question impossible and see if the spam continues.
> > >>>>>
> > >>>>We'll have to let an infra-root make that call. Since nobody would be
> > >>>>able to
> > >>>>use the wiki. Honestly, I'd rather spend the time standing up a mirror
> dev
> > >>>>instance for us to work on, rather then production.
> > >>>>
> > >>>>>J.P. Maxwell | tipit.net | fibercove.com
> > >>>>>On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelanger at redhat.com>
> wrote:
> > >>>>>
> > >>>>>>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
> > >>>>>>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<fungi at yuggoth.org>
> > >>>>>>wrote:
> > >>>>>>>>On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> > >>>>>>>>>Please be aware that you can now create accounts under the mobile
> > >>>>>>>>>view in the wiki native user table. I just created an account for
> > >>>>>>>>>JpMaxMan. Not sure if this matters but wanted to make sure you
> > >>>>>>>>>were aware.
> > >>>>>>>>Oh, yes I think having a random garbage question/answer was in
> > >>>>fact
> > >>>>>>>>previously preventing account creation under the mobile view. We
> > >>>>>>>>probably need a way to disable mobile view account creation as it
> > >>>>>>>>bypasses OpenID authentication entirely.
> > >>>>>>>So that's what it was doing! We'll have to tackle the mobile view
> > >>>>issue.
> > >>>>>>>Otherwise, quick update here:
> > >>>>>>>
> > >>>>>>>The captcha didn't appear to help stem the spam tide. We'll want to
> > >>>>>>>explore and start implementing some of the other solutions.
> > >>>>>>>
> > >>>>>>>I did some database poking around today and it does seem like all
> > >>>>the
> > >>>>>>>users do have launchpad accounts and email addresses.
> > >>>>>>>
> > >>>>>>So, I have a few hours before jumping on my plane and checked into
> > >>>>this.
> > >>>>>>We are
> > >>>>>>using QuestyCaptcha which according to docs, should almost be
> > >>>>impossible
> > >>>>>>for
> > >>>>>>spammers to by pass in an automated fashion. So, either our captcha
> > >>>>is too
> > >>>>>>easy, or we didn't set it up properly. I don't have SSH on wiki.o.o
> > >>>>so
> > >>>>>>others
> > >>>>>>will have to check logs. I did test new pages and edits, and was
> > >>>>promoted
> > >>>>>>by
> > >>>>>>captcha.
> > >>>>>>
> > >>>>>>As a next step, we might need to add additional apache2
> configuration
> > >>>>to
> > >>>>>>blacklist IPs. I am reading up on that now.
> > >>>>>>
> > >>>>>>>--
> > >>>>>>>Elizabeth Krumbach Joseph || Lyz || pleia2
> > >>>>>>>
> > >>>>>>>_______________________________________________
> > >>>>>>>OpenStack-Infra mailing list
> > >>>>>>>OpenStack-Infra at lists.openstack.org
> >
> >>>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > >>>>>>_______________________________________________
> > >>>>>>OpenStack-Infra mailing list
> > >>>>>>OpenStack-Infra at lists.openstack.org
> >
> >>>>>>http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > >>>>>>
> > >
> > >_______________________________________________
> > >OpenStack-Infra mailing list
> > >OpenStack-Infra at lists.openstack.org
> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> >
> 
> > _______________________________________________
> > OpenStack-Infra mailing list
> > OpenStack-Infra at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> 
> 
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra