Open Stack

Oh, I can login. So what we need?

M.

On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell <jp at tipit.net> wrote:

> I think what Jimmy is referring to is what I was suggesting by removing
> the extensions / making the question impossible to answer.  Basically a
> series of rapid fire changes while tailing the logs and seeing what stops
> the spam.  Once you know what worked then you can submit as an official
> patch.  But being able to quickly try these things on a server actually
> under attack is the fastest path toward identifying the fix.
>
> *J.P. Maxwell* | tipit.net | fibercove.com <http://www.fibercove.com>
>
> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <pabelanger at redhat.com>
> wrote:
>
> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
> > Given the state of the wiki a the moment, I think taking the quickest
> path
> > to get it fixed would be prudent. Is there a way we can get JP root
> access
> > to this server, even temporarily? We get 25% of our website traffic (2
> > million visitors) to the wiki. I realize we're all after the same thing,
> but
> > spammers are not going to hit the dev environment, so there's really no
> way
> > to tell if teh problem is fixed without actually working directly on the
> > production machine. This should be a 30 minute fix.
> >
> I am still unclear what the 30min fix is. If really 30mins, then it
> shouldn't be
> hard to get the fix into our workflow. Could somebody please elaborate.
>
> If we are talking about deploying new versions of php or mediawiki
> manually, I
> not be in-favor of this. To me, while the attack sucks, we should be
> working on
> 2 fronts. Getting the help needed to mitigate the attack, then adding the
> changes into -infra workflow in parallel.
>
> > I realize there is a lot of risk in giving ssh access to infra machines,
> but
> > I think it's worth taking a look at either putting this machine in a
> place
> > where a different level of admin could access it without giving away the
> > keys to the entire OpenStack infrastructure or figuring out a way to set
> up
> > credentials with varying levels of access.
> >
> As a note, all the work I've been doing to help with the attack hasn't
> require
> SSH access for me to wiki.o.o. I did need infra-root help to expose our
> configuration safely. I'd rather take some time to see what the fixes are,
> having infra-root apply changes, then move them into puppet.
>
> It also has been discussed to simply disable write access to the wiki if we
> really want spamming to stop, obviously that will affect normal usage.
>
> > Jimmy
> >
> > Paul Belanger wrote:
> > >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
> > >>But if you wanted to upgrade everything, remove the mobile view
> extension,
> > >>test in a dev/staging environment then deploy to production fingers
> > >>crossed, I think that would be a valid approach as well.
> > >>
> > >Current review up[1]. I'll launch a node tonight / tomorrow locally to
> see how
> > >puppet reacts. I suspect there will be some issues.
> > >
> > >If infra-roots are fine with this approach, we can use that box to test
> against.
> > >
> > >[1] https://review.openstack.org/#/c/285405/
> > >
> > >>J.P. Maxwell | tipit.net | fibercove.com
> > >>On Feb 26, 2016 10:08 AM, "JP Maxwell"<jp at tipit.net> wrote:
> > >>
> > >>>Plus one except in this case it is much easier to know if our efforts
> are
> > >>>working on production because the spam either stops or not.
> > >>>
> > >>>J.P. Maxwell | tipit.net | fibercove.com
> > >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelanger at redhat.com>
> wrote:
> > >>>
> > >>>>On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> > >>>>>I really think you might consider the option that there is a
> > >>>>vulnerability
> > >>>>>in one of the extensions. If that is the case black listing IPs
> will be
> > >>>>an
> > >>>>>ongoing wild goose chase.
> > >>>>>
> > >>>>>I think this would be easily proven or disproven by making the
> questy
> > >>>>>question impossible and see if the spam continues.
> > >>>>>
> > >>>>We'll have to let an infra-root make that call. Since nobody would be
> > >>>>able to
> > >>>>use the wiki. Honestly, I'd rather spend the time standing up a
> mirror dev
> > >>>>instance for us to work on, rather then production.
> > >>>>
> > >>>>>J.P. Maxwell | tipit.net | fibercove.com
> > >>>>>On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelanger at redhat.com>
> wrote:
> > >>>>>
> > >>>>>>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph
> wrote:
> > >>>>>>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<fungi at yuggoth.org
> >
> > >>>>>>wrote:
> > >>>>>>>>On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> > >>>>>>>>>Please be aware that you can now create accounts under the
> mobile
> > >>>>>>>>>view in the wiki native user table. I just created an account
> for
> > >>>>>>>>>JpMaxMan. Not sure if this matters but wanted to make sure you
> > >>>>>>>>>were aware.
> > >>>>>>>>Oh, yes I think having a random garbage question/answer was in
> > >>>>fact
> > >>>>>>>>previously preventing account creation under the mobile view. We
> > >>>>>>>>probably need a way to disable mobile view account creation as it
> > >>>>>>>>bypasses OpenID authentication entirely.
> > >>>>>>>So that's what it was doing! We'll have to tackle the mobile view
> > >>>>issue.
> > >>>>>>>Otherwise, quick update here:
> > >>>>>>>
> > >>>>>>>The captcha didn't appear to help stem the spam tide. We'll want
> to
> > >>>>>>>explore and start implementing some of the other solutions.
> > >>>>>>>
> > >>>>>>>I did some database poking around today and it does seem like all
> > >>>>the
> > >>>>>>>users do have launchpad accounts and email addresses.
> > >>>>>>>
> > >>>>>>So, I have a few hours before jumping on my plane and checked into
> > >>>>this.
> > >>>>>>We are
> > >>>>>>using QuestyCaptcha which according to docs, should almost be
> > >>>>impossible
> > >>>>>>for
> > >>>>>>spammers to by pass in an automated fashion. So, either our captcha
> > >>>>is too
> > >>>>>>easy, or we didn't set it up properly. I don't have SSH on wiki.o.o
> > >>>>so
> > >>>>>>others
> > >>>>>>will have to check logs. I did test new pages and edits, and was
> > >>>>promoted
> > >>>>>>by
> > >>>>>>captcha.
> > >>>>>>
> > >>>>>>As a next step, we might need to add additional apache2
> configuration
> > >>>>to
> > >>>>>>blacklist IPs. I am reading up on that now.
> > >>>>>>
> > >>>>>>>--
> > >>>>>>>Elizabeth Krumbach Joseph || Lyz || pleia2
> > >>>>>>>
> > >>>>>>>_______________________________________________
> > >>>>>>>OpenStack-Infra mailing list
> > >>>>>>>OpenStack-Infra at lists.openstack.org
> > >>>>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > >>>>>>_______________________________________________
> > >>>>>>OpenStack-Infra mailing list
> > >>>>>>OpenStack-Infra at lists.openstack.org
> > >>>>>>
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> > >>>>>>
> > >
> > >_______________________________________________
> > >OpenStack-Infra mailing list
> > >OpenStack-Infra at lists.openstack.org
> > >http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
> >
>
> > _______________________________________________
> > OpenStack-Infra mailing list
> > OpenStack-Infra at lists.openstack.org
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20160226/91912dd9/attachment-0001.html>