probably should have cross-posted to ~devs and maybe even ~Design ; after all, the "robots file" is deliberately also human-readable... I often read the robots file before wandering through sites because so many these days have a terrible navigation labyrinth.
probably should have cross-posted to ~devs and maybe even ~Design ; after all, the "robots file" is deliberately also human-readable... I often read the robots file before wandering through sites because so many these days have a terrible navigation labyrinth.