trying to get a good robots.txt setup, need help
|
nate_gallery
Joined: 2009-08-12
Posts: 1 |
Posted: Thu, 2009-08-13 04:32
|
|
So far I have the below. Now I would like to prevent spiders from rss and login. I see spiders will try the login link from multiple hierarchy's which seems stupid. I'm going to add rss like this "Disallow: /v/slideshow/". But how the heck do I add login and registration. User-agent: * |
|
