robots.txt file, as in robots exclusion protocol its a robots exclusion standard. It helps search engine crawlers to exclude your listed pages in conjunction with xml sitemap to guide them to crawl specified pages to be indexed in search engines. Also its our choice to decide to which crawlers we want to allow the permission to crawl our portal.
I hope, I have answered your query.
Thankx And Regards
Keep posting ....Continue Sharing....Multiply your knowledge