Line Dancing Directory uses a small web crawler called LineDancingDirectoryBot to keep line dancing club listings accurate. This page explains what it does and how to stop it visiting your site.
How to recognise it
It identifies itself with this user agent:
LineDancingDirectoryBot/1.0 (+https://linedancing.directory/crawler)What it looks at
It only visits the public websites of line dancing clubs and classes, such as a club's own site or a u3a group page. On each site it reads a handful of pages, like the home, classes, venue and contact pages, looking for information clubs publish so dancers can find them:
- the club name, venue and postcode
- class days, times and levels
- prices, the instructor's name and whether drop-ins are welcome
- links to the club's Facebook page, and the club's contact email and phone number
It doesn't log in, fill in forms, download files or visit social networks such as Facebook, and it doesn't copy other directories or listings sites.
Nothing is published automatically
What the crawler finds goes to our team as a suggestion. A person checks it against your site before anything appears on the directory, and club email addresses and phone numbers are never shown publicly. If your club is already listed, the crawler can only suggest filling in details that are missing. It never overwrites what a club admin has entered.
Run the club? Claim your listing to manage it yourself.
How it behaves
- It follows your
robots.txtrules, including anyCrawl-delay. - It waits at least 5 seconds between requests to the same site, and reads no more than 6 pages per visit.
- It visits each site about once every 30 days.
How to stop it
Add this to the robots.txt file at the root of your site:
User-agent: LineDancingDirectoryBot
Disallow: /Or email [email protected] with your site's address and we'll take it off the list. You can also ask us to remove or correct anything we've listed about your club.