Open Stack

Marton
Where are you seeing the logs?
Paul
The point is that to comment out a line in VI and watch the logs in another
window takes about a minute or two. To submit a patch, get approval, push, ask
someone to share the logs takes a lot longer and relies on other people.
I understand the need for openness and I appreciate the process and the
automation. I think Jimmy’s point was if we want a quick fix….

J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com [http://www.fibercove.com]
On Fri, Feb 26, 2016 at 11:38 AM, Marton Kiss <marton.kiss at gmail.com> wrote:
I see a ton of incoming post requests:
POST
/w/index.php?title=Special%3ARunJobs&tasks=jobs&maxjobs=1&sigexpiry=1456508270&signature=571cfb216f944b15d2eee1c0253d08b77003328e

M.
On Fri, Feb 26, 2016 at 6:35 PM Marton Kiss < marton.kiss at gmail.com [marton.kiss at gmail.com] > wrote:
Oh, I can login. So what we need?
M.
On Fri, Feb 26, 2016 at 6:33 PM JP Maxwell < jp at tipit.net [jp at tipit.net] > wrote:
I think what Jimmy is referring to is what I was suggesting by removing the
extensions / making the question impossible to answer. Basically a series of
rapid fire changes while tailing the logs and seeing what stops the spam. Once
you know what worked then you can submit as an official patch. But being able to
quickly try these things on a server actually under attack is the fastest path
toward identifying the fix.

J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com [http://www.fibercove.com]
On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger < pabelanger at redhat.com [pabelanger at redhat.com] > wrote:
On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
> Given the state of the wiki a the moment, I think taking the quickest path
> to get it fixed would be prudent. Is there a way we can get JP root access
> to this server, even temporarily? We get 25% of our website traffic (2
> million visitors) to the wiki. I realize we're all after the same thing, but
> spammers are not going to hit the dev environment, so there's really no way
> to tell if teh problem is fixed without actually working directly on the
> production machine. This should be a 30 minute fix.
>
I am still unclear what the 30min fix is. If really 30mins, then it shouldn't be
hard to get the fix into our workflow. Could somebody please elaborate.

If we are talking about deploying new versions of php or mediawiki manually, I
not be in-favor of this. To me, while the attack sucks, we should be working on
2 fronts. Getting the help needed to mitigate the attack, then adding the
changes into -infra workflow in parallel.

> I realize there is a lot of risk in giving ssh access to infra machines, but
> I think it's worth taking a look at either putting this machine in a place
> where a different level of admin could access it without giving away the
> keys to the entire OpenStack infrastructure or figuring out a way to set up
> credentials with varying levels of access.
>
As a note, all the work I've been doing to help with the attack hasn't require
SSH access for me to wiki.o.o. I did need infra-root help to expose our
configuration safely. I'd rather take some time to see what the fixes are,
having infra-root apply changes, then move them into puppet.

It also has been discussed to simply disable write access to the wiki if we
really want spamming to stop, obviously that will affect normal usage.

> Jimmy
>
> Paul Belanger wrote:
> >On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
> >>But if you wanted to upgrade everything, remove the mobile view extension,
> >>test in a dev/staging environment then deploy to production fingers
> >>crossed, I think that would be a valid approach as well.
> >>
> >Current review up[1]. I'll launch a node tonight / tomorrow locally to see
how
> >puppet reacts. I suspect there will be some issues.
> >
> >If infra-roots are fine with this approach, we can use that box to test
against.
> >
> >[1] https://review.openstack.org/#/c/285405/
[https://review.openstack.org/#/c/285405/]
> >
> >>J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com [http://fibercove.com]
> >>On Feb 26, 2016 10:08 AM, "JP Maxwell"< jp at tipit.net [jp at tipit.net] > wrote:
> >>
> >>>Plus one except in this case it is much easier to know if our efforts are
> >>>working on production because the spam either stops or not.
> >>>
> >>>J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com [http://fibercove.com]
> >>>On Feb 26, 2016 9:48 AM, "Paul Belanger"< pabelanger at redhat.com [pabelanger at redhat.com] > wrote:
> >>>
> >>>>On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
> >>>>>I really think you might consider the option that there is a
> >>>>vulnerability
> >>>>>in one of the extensions. If that is the case black listing IPs will be
> >>>>an
> >>>>>ongoing wild goose chase.
> >>>>>
> >>>>>I think this would be easily proven or disproven by making the questy
> >>>>>question impossible and see if the spam continues.
> >>>>>
> >>>>We'll have to let an infra-root make that call. Since nobody would be
> >>>>able to
> >>>>use the wiki. Honestly, I'd rather spend the time standing up a mirror dev
> >>>>instance for us to work on, rather then production.
> >>>>
> >>>>>J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com [http://fibercove.com]
> >>>>>On Feb 26, 2016 9:12 AM, "Paul Belanger"< pabelanger at redhat.com [pabelanger at redhat.com] > wrote:
> >>>>>
> >>>>>>On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
> >>>>>>>On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley< fungi at yuggoth.org [fungi at yuggoth.org] >
> >>>>>>wrote:
> >>>>>>>>On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
> >>>>>>>>>Please be aware that you can now create accounts under the mobile
> >>>>>>>>>view in the wiki native user table. I just created an account for
> >>>>>>>>>JpMaxMan. Not sure if this matters but wanted to make sure you
> >>>>>>>>>were aware.
> >>>>>>>>Oh, yes I think having a random garbage question/answer was in
> >>>>fact
> >>>>>>>>previously preventing account creation under the mobile view. We
> >>>>>>>>probably need a way to disable mobile view account creation as it
> >>>>>>>>bypasses OpenID authentication entirely.
> >>>>>>>So that's what it was doing! We'll have to tackle the mobile view
> >>>>issue.
> >>>>>>>Otherwise, quick update here:
> >>>>>>>
> >>>>>>>The captcha didn't appear to help stem the spam tide. We'll want to
> >>>>>>>explore and start implementing some of the other solutions.
> >>>>>>>
> >>>>>>>I did some database poking around today and it does seem like all
> >>>>the
> >>>>>>>users do have launchpad accounts and email addresses.
> >>>>>>>
> >>>>>>So, I have a few hours before jumping on my plane and checked into
> >>>>this.
> >>>>>>We are
> >>>>>>using QuestyCaptcha which according to docs, should almost be
> >>>>impossible
> >>>>>>for
> >>>>>>spammers to by pass in an automated fashion. So, either our captcha
> >>>>is too
> >>>>>>easy, or we didn't set it up properly. I don't have SSH on wiki.o.o
> >>>>so
> >>>>>>others
> >>>>>>will have to check logs. I did test new pages and edits, and was
> >>>>promoted
> >>>>>>by
> >>>>>>captcha.
> >>>>>>
> >>>>>>As a next step, we might need to add additional apache2 configuration
> >>>>to
> >>>>>>blacklist IPs. I am reading up on that now.
> >>>>>>
> >>>>>>>--
> >>>>>>>Elizabeth Krumbach Joseph || Lyz || pleia2
> >>>>>>>
> >>>>>>>_______________________________________________
> >>>>>>>OpenStack-Infra mailing list
> >>>>>>> OpenStack-Infra at lists.openstack.org [OpenStack-Infra at lists.openstack.org]
> >>>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra]
> >>>>>>_______________________________________________
> >>>>>>OpenStack-Infra mailing list
> >>>>>> OpenStack-Infra at lists.openstack.org [OpenStack-Infra at lists.openstack.org]
> >>>>>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra]
> >>>>>>
> >
> >_______________________________________________
> >OpenStack-Infra mailing list
> > OpenStack-Infra at lists.openstack.org [OpenStack-Infra at lists.openstack.org]
> > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra]
>

> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org [OpenStack-Infra at lists.openstack.org]
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra]


_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra at lists.openstack.org [OpenStack-Infra at lists.openstack.org]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra]
_______________________________________________
OpenStack-Infra mailing list
OpenStack-Infra at lists.openstack.org [OpenStack-Infra at lists.openstack.org]
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
[http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra]
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20160226/bf5c4c4a/attachment.html>