Open Stack

On Sun, 22 Jun 2014 07:20:33 +0200, Gauvain Pocentek wrote:
> Le 2014-06-22 07:10, Roger Luethi a écrit :
> >On Sat, 21 Jun 2014 17:44:21 -0500, Anne Gentle wrote:
> >>I like this idea and have tried to have /trunk be unsearchable. If we
> >>switch /icehouse to be /icehouse and /current and then create a /dev,
> >>and never point to /dev links, could it be hidden?
> >
> >How about blocking /trunk in robots.txt? Is that what you mean with
> >"tried to have /trunk be unsearchable"? Did that not work?
> >
> >robots.txt should also take care of our "Custom Search" (which throws a
> >variety of releases at the poor user). Presumably, that's how most
> >people end up reading trunk.
> 
> I don't know much about search engines, but I guess that having
> links to a dev/ branch in bugs, or emails, would mean that the URL
> will still discovered and visible in these engines. Or should a
> modified robots.txt take care of that?

Indeed it should. robots.txt doesn't keep search engines from discovering
pages, and it doesn't have to. It is just a widely observed convention
used to keep the crawler out of parts of the website, regardless of
any links pointing to those parts. Otherwise, any Internet user could
break a carefully crafted robots.txt simply by linking to blocked pages.

Roger