Open Stack

Hi,

I created the following patch, infra cores must approve that:
https://review.openstack.org/285641 Add ssh key of JP Maxwell to wiki.o.o

Marton

On Sat, Feb 27, 2016 at 6:41 AM JP Maxwell <jp at tipit.net> wrote:

> Marton has SSH access and applied a patch earlier today.  It appears the
> spam continues to flow:
>
>
> https://wiki.openstack.org/wiki/40_Thoughts_Of_Using_Open_Shelves_On_A_Kitchen
>
> Marton let me know if you can look at it some more or Infra if you want to
> give me SSH I'll do so as well in the morning (public key attached).
>
>
>
> ssh-rsa
> AAAAB3NzaC1yc2EAAAABIwAAAQEA2b5I7Yff9FCrtRmSjpILUePi54Vbc8zqJTbzrIAQZGFLBi3xd2MLlhV5QVgpDBC9H3lGjbdnc81D3aFd3HwHT4dvvvyedT12PR3VDEpftdW84vw3jzdtALcayOQznjbGnScwvX5SgnRhNxuX9Rkh8qNvOsjYPUafRr9azkQoomJFkdNVI4Vb5DbLhTpt18FPeOf0UuqDt/J2tHI4SjZ3kjzr7Nbwpg8xGgANPNE0+2pJbwCA8YDt4g3bzfzvVafQs5o9Gfc9tudkR9ugQG1M+EWCgu42CleOwMTd/rYEB2fgNNPsZAWqwQfdPajVuk70EBKUEQSyoA09eEZX+xJN9Q==
> jpmaxman at tipit.net
>
>
>
>
> J.P. Maxwell / tipit.net <http://www.tipit.net>
>
>
> On Fri, Feb 26, 2016 at 12:09 PM, Jimmy McArthur <jimmy at openstack.org>
> wrote:
>
>> Super thankful for all the folks that have jumped in over the last couple
>> of days to help with the puppetization, etc... I just feel like we're
>> taking a very wrong approach here.
>>
>> Paul Belanger wrote:
>>
>> Right, and I don't have an issue with that approach.  Based on the work we did
>> yesterday, anybody can do that via our workflow. Please submit a patch to
>> puppet-mediawiki[1] and ping an infra-root in #openstack-infra IRC.
>>
>> What I'm proposing is the workflow is really meant for software, not for
>> web applications. It's tedious and time consuming when what's needed here
>> is a set of tests on the server. Submitting a patch, waiting for a +1, then
>> getting on IRC to find someone with access (and time) to paste the logs is
>> a pretty time consuming process for what should be a series of rapid-fire
>> changes/fixes on the server. Especially when we're dealign with an active
>> attack.
>>
>>
>> We can then have somebody look at the logs.  I think it is more about scheduling
>> the task since more infra-root as travling back from the mid-cycle last night
>> and today.
>>
>> Right, this is my point. This has been going on for 3 weeks (or more).
>> Tom Fifeldt was asking for help without response. And here we are through
>> another week and no closer to stemming the flow.
>>
>> I'm fully aware what I'm proposing goes against what Infra and the
>> OpenStack workflow is all about, but I'd ask you all to look at this from a
>> web development perspective instead of a software development perspective.
>>
>> Jimmy
>>
>>
>> Last email from me, just on a plane.  Will follow up when I land.
>>
>> [1] https://git.openstack.org/cgit/openstack-infra/puppet-mediawiki
>>
>>  J.P. Maxwell | tipit.net [http://tipit.net] | fibercove.com
>> [http://www.fibercove.com]
>> On Fri, Feb 26, 2016 at 11:25 AM, Paul Belanger <pabelanger at redhat.com> <pabelanger at redhat.com>
>> wrote:
>> On Fri, Feb 26, 2016 at 11:08:18AM -0600, Jimmy McArthur wrote:
>>
>> Given the state of the wiki a the moment, I think taking the quickest path
>> to get it fixed would be prudent. Is there a way we can get JP root access
>> to this server, even temporarily? We get 25% of our website traffic (2
>> million visitors) to the wiki. I realize we're all after the same thing,
>>
>> but
>>
>> spammers are not going to hit the dev environment, so there's really no
>>
>> way
>>
>> to tell if teh problem is fixed without actually working directly on the
>> production machine. This should be a 30 minute fix.
>>
>>
>> I am still unclear what the 30min fix is. If really 30mins, then it
>> shouldn't be
>> hard to get the fix into our workflow. Could somebody please elaborate.
>>
>> If we are talking about deploying new versions of php or mediawiki manually,
>> I
>> not be in-favor of this. To me, while the attack sucks, we should be working
>> on
>> 2 fronts. Getting the help needed to mitigate the attack, then adding the
>> changes into -infra workflow in parallel.
>>
>>
>> I realize there is a lot of risk in giving ssh access to infra machines,
>>
>> but
>>
>> I think it's worth taking a look at either putting this machine in a place
>> where a different level of admin could access it without giving away the
>> keys to the entire OpenStack infrastructure or figuring out a way to set
>>
>> up
>>
>> credentials with varying levels of access.
>>
>>
>> As a note, all the work I've been doing to help with the attack hasn't
>> require
>> SSH access for me to wiki.o.o. I did need infra-root help to expose our
>> configuration safely. I'd rather take some time to see what the fixes are,
>> having infra-root apply changes, then move them into puppet.
>>
>> It also has been discussed to simply disable write access to the wiki if we
>> really want spamming to stop, obviously that will affect normal usage.
>>
>>
>> Jimmy
>>
>> Paul Belanger wrote:
>>
>> On Fri, Feb 26, 2016 at 10:12:12AM -0600, JP Maxwell wrote:
>>
>> But if you wanted to upgrade everything, remove the mobile view
>>
>> extension,
>>
>> test in a dev/staging environment then deploy to production fingers
>> crossed, I think that would be a valid approach as well.
>>
>>
>> Current review up[1]. I'll launch a node tonight / tomorrow locally to
>>
>> see
>> how
>>
>> puppet reacts. I suspect there will be some issues.
>>
>> If infra-roots are fine with this approach, we can use that box to test
>>
>> against.
>>
>> [1] https://review.openstack.org/#/c/285405/
>>
>> J.P. Maxwell | tipit.net | fibercove.com
>> On Feb 26, 2016 10:08 AM, "JP Maxwell"<jp at tipit.net> <jp at tipit.net> wrote:
>>
>>
>> Plus one except in this case it is much easier to know if our efforts
>>
>> are
>>
>> working on production because the spam either stops or not.
>>
>> J.P. Maxwell | tipit.net | fibercove.com
>> On Feb 26, 2016 9:48 AM, "Paul Belanger"<pabelanger at redhat.com> <pabelanger at redhat.com> wrote:
>>
>>
>> On Fri, Feb 26, 2016 at 09:18:00AM -0600, JP Maxwell wrote:
>>
>> I really think you might consider the option that there is a
>>
>> vulnerability
>>
>> in one of the extensions. If that is the case black listing IPs will
>>
>> be
>>
>> an
>>
>> ongoing wild goose chase.
>>
>> I think this would be easily proven or disproven by making the questy
>> question impossible and see if the spam continues.
>>
>>
>> We'll have to let an infra-root make that call. Since nobody would be
>> able to
>> use the wiki. Honestly, I'd rather spend the time standing up a mirror
>>
>> dev
>>
>> instance for us to work on, rather then production.
>>
>>
>> J.P. Maxwell | tipit.net | fibercove.com
>> On Feb 26, 2016 9:12 AM, "Paul Belanger"<pabelanger at redhat.com> <pabelanger at redhat.com>
>>
>> wrote:
>>
>> On Thu, Feb 25, 2016 at 08:10:34PM -0800, Elizabeth K. Joseph wrote:
>>
>> On Thu, Feb 25, 2016 at 6:35 AM, Jeremy Stanley<fungi at yuggoth.org> <fungi at yuggoth.org>
>>
>> wrote:
>>
>> On 2016-02-25 02:46:13 -0600 (-0600), JP Maxwell wrote:
>>
>> Please be aware that you can now create accounts under the mobile
>> view in the wiki native user table. I just created an account for
>> JpMaxMan. Not sure if this matters but wanted to make sure you
>> were aware.
>>
>> Oh, yes I think having a random garbage question/answer was in
>>
>> fact
>>
>> previously preventing account creation under the mobile view. We
>> probably need a way to disable mobile view account creation as it
>> bypasses OpenID authentication entirely.
>>
>> So that's what it was doing! We'll have to tackle the mobile view
>>
>> issue.
>>
>> Otherwise, quick update here:
>>
>> The captcha didn't appear to help stem the spam tide. We'll want to
>> explore and start implementing some of the other solutions.
>>
>> I did some database poking around today and it does seem like all
>>
>> the
>>
>> users do have launchpad accounts and email addresses.
>>
>>
>> So, I have a few hours before jumping on my plane and checked into
>>
>> this.
>>
>> We are
>> using QuestyCaptcha which according to docs, should almost be
>>
>> impossible
>>
>> for
>> spammers to by pass in an automated fashion. So, either our captcha
>>
>> is too
>>
>> easy, or we didn't set it up properly. I don't have SSH on wiki.o.o
>>
>> so
>>
>> others
>> will have to check logs. I did test new pages and edits, and was
>>
>> promoted
>>
>> by
>> captcha.
>>
>> As a next step, we might need to add additional apache2
>>
>> configuration
>>
>> to
>>
>> blacklist IPs. I am reading up on that now.
>>
>>
>> --
>> Elizabeth Krumbach Joseph || Lyz || pleia2
>>
>> _______________________________________________
>> OpenStack-Infra mailing listOpenStack-Infra at lists.openstack.org
>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>> _______________________________________________
>> OpenStack-Infra mailing listOpenStack-Infra at lists.openstack.org
>>
>> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>> _______________________________________________
>> OpenStack-Infra mailing listOpenStack-Infra at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>> _______________________________________________
>> OpenStack-Infra mailing listOpenStack-Infra at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>> _______________________________________________
>> OpenStack-Infra mailing listOpenStack-Infra at lists.openstack.orghttp://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>>
>>
>>
> _______________________________________________
> OpenStack-Infra mailing list
> OpenStack-Infra at lists.openstack.org
> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-infra
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-infra/attachments/20160227/a1c9eb31/attachment-0001.html>