Hosting · Domains · Licensing · Support · Sales
Crawl Commander

See which crawlers are visiting your site

The same crawler keeps turning up in your logs. Use Crawl Commander to see whether it’s checking new pages or requesting the same feed over and over.
1

Find the busy crawler

Open Crawl Commander’s Live Log or Log. Look for repeated requests and note the reported crawler, URL and time.
2

Check which pages it visits

Use Pages and Resources to see whether the crawler is reading useful product pages, your sitemap or repetitive feed URLs.
3

Review the current rules

Open the robots.txt manager. Check the rule for the specific crawler and preview the change. Preserve access to the pages you want search engines to discover.
4

Test and monitor

Use the fetch simulator to check the intended URL and review later visits in the log. Robots.txt gives instructions to cooperative crawlers. Ask support about enforcement if requests continue.

Need help with a crawler rule?

Include the crawler name and one URL you want it to reach or leave alone.