building a very large robots.txt file.
Mr_Gadget
says
why so?
Danizita
says
geeky.
jameswilliams90
says
we've undergone a major update to one of my websites, and I need to direct browsers not to follow dead links.