Open Stack

I hadn't tried blocking with robots.txt, here's our current copy:

https://github.com/openstack/openstack-manuals/blob/master/www/robots.txt

What didn't work is eliminating it from our custom search engine.

I think we have seen evidence of users linking to whatever they want to
despite our careful efforts. :)


On Sun, Jun 22, 2014 at 12:32 AM, Roger Luethi <rl at patchworkscience.org>
wrote:

> On Sun, 22 Jun 2014 07:20:33 +0200, Gauvain Pocentek wrote:
> > Le 2014-06-22 07:10, Roger Luethi a écrit :
> > >On Sat, 21 Jun 2014 17:44:21 -0500, Anne Gentle wrote:
> > >>I like this idea and have tried to have /trunk be unsearchable. If we
> > >>switch /icehouse to be /icehouse and /current and then create a /dev,
> > >>and never point to /dev links, could it be hidden?
> > >
> > >How about blocking /trunk in robots.txt? Is that what you mean with
> > >"tried to have /trunk be unsearchable"? Did that not work?
> > >
> > >robots.txt should also take care of our "Custom Search" (which throws a
> > >variety of releases at the poor user). Presumably, that's how most
> > >people end up reading trunk.
> >
> > I don't know much about search engines, but I guess that having
> > links to a dev/ branch in bugs, or emails, would mean that the URL
> > will still discovered and visible in these engines. Or should a
> > modified robots.txt take care of that?
>
> Indeed it should. robots.txt doesn't keep search engines from discovering
> pages, and it doesn't have to. It is just a widely observed convention
> used to keep the crawler out of parts of the website, regardless of
> any links pointing to those parts. Otherwise, any Internet user could
> break a carefully crafted robots.txt simply by linking to blocked pages.
>
> Roger
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openstack.org/pipermail/openstack-docs/attachments/20140622/dd62e84a/attachment.html>