web
n.
     net created by a spider; net, mesh; World Wide Web, network of HTML
 documents which are linked together and located all over the world 
(Computers, Internet); weave, fabric; network (of lies, etc.)
    
robots.txt, containing rules for the spidering of that server that the bot is supposed to obey.