Robots.txt

This is a file placed in the root directory of your site that tells search engines how to crawl your site. It tells them which content to crawl and which content to ignore. Even if you don’t have content you want to block from search engines you should still have this file on your server.