robots.txt

What is robots.txt?

robots.txt is a file in the root directory of a website use to restrict and control the behavior of search engine spiders.

 

source: –

Tags: