Step 1 - Select Search Engine Spiders

Robot-Manager Standard and Professional Editions

When creating your robots.txt file for your web site, the first thing to do is select the search engine spiders you wish to set specific rules for. For most web sites the default selection of "All Spiders" is adequate. For multi lingual sites and sites with more specific needs, you may want to select more search engine spiders. The search engine spiders selected here are the ones used in Step 2 to when setting disallow rules.