web
n.
net created by a spider; net, mesh; World Wide Web, network of HTML
documents which are linked together and located all over the world
(Computers, Internet); weave, fabric; network (of lies, etc.)
robots.txt
, containing rules for the spidering of that server that the bot is supposed to obey.