A program that automatically crawls the web and downloads pages. It is used by search engines (Googlebot), website analysis tools and AI services. What it may visit is determined by robots.txt. See also AI crawlers.
FAQ / Glossary
FAQ / Glossary
A program that automatically crawls the web and downloads pages. It is used by search engines (Googlebot), website analysis tools and AI services. What it may visit is determined by robots.txt. See also AI crawlers.