InternetAtlasBot is a web crawler of Cysmos, a software company in Austria. Its requests carry the User-Agent
InternetAtlasBot/1.0 (+https://cysmos.bot/internetatlasbot; crawl)
The homepage (/) of websites that appear in public domain popularity rankings, to build a map of
the web by topic. Plain HTTP requests only (no browser, no JavaScript). It follows redirects, reads at most 5 MB
of a page and fetches nothing else from the site besides /robots.txt. It does not log in, fill in
forms or follow links.
Not on a schedule: runs are started by hand. A run skips sites it already has and sends each other site two
requests, /robots.txt and then the homepage (plus one per redirect).
InternetAtlasBot reads /robots.txt before it fetches a homepage and honours it. Its robots.txt
token is InternetAtlasBot: rules in a User-agent: InternetAtlasBot group apply to it,
otherwise those for User-agent: *. A robots.txt that answers with a server error (5xx) or cannot be
reached counts as "disallow everything"; one that answers 404 or another 4xx allows everything.
Crawl-delay is not read.
Add to your robots.txt
User-agent: InternetAtlasBot
Disallow: /or write to us, and we remove your site from the list.
Questions, problems or abuse reports: bot@cysmos.email. Please include the time (with time zone) and the URL that was requested.