Category Archives: Robots

Robots TXT File

What is Robots TXT File Robots TXT file is a simple text file that contain commands that enables the search engine to find the pages that are needed to be crawled or not. It is a file that can stop specified pages from appearing in search engines but it doesn’t stop to make them unavailable. By using the robots txt command, “Disallow :/cgi-bin/” without the quotes professionals can make the… Read More »