The Fediral crawler, commonly called a bot, is a polite web crawler and indexer.

The bot will make no more than one request to any single domain every three seconds and it will respect the robots.txt file of the domain it is visiting.

If you wish to explicitly allow the crawler to index your service (the default) you can add the following to your root robots.txt file:

User-agent: fediral
Allow: /

If you wish to stop the crawler from indexing your service you can add the following to your root robots.txt file:

User-agent: fediral
Disallow: /

Note it may take up to a day for us to refresh changes to your robots.txt file. You can contact us at [email protected] if you have any questions or concerns.