Skip to content

New accounts get 1.5M free tokens. You get a sebtember.app account too.

Crawler

PeakPickBot

Last updated

PeakPick’s website crawler. If you found it in your server logs, this page explains what it is, what it reads and how to control it.

User agent
PeakPickBot/1.0 (+https://peakpick.ai/bot)
robots.txt name
PeakPickBot

What it is

PeakPick shows companies how AI assistants describe and recommend them. To compare what AI says with what a company’s own website says, PeakPick reads that website.

PeakPickBot visits a site only when a PeakPick user has added it as their company’s website: to build that company’s profile, again when they refresh a check, to see which pages changed, and when they press Check again, to see whether a setting they fixed is read as fixed. It doesn’t crawl the web.

It also visits a site when a PeakPick user’s Autopilot writes them an email to that site: a directory, a list, a newspaper or a brand the company sells, asking to be listed or mentioned. Then it reads the site’s homepage and up to two contact pages it links to, to find the address to write to, and keeps that address and nothing else.

What it reads

  • The homepage and up to 19 more pages, 20 at most, picked from your sitemaps and homepage links: the pages most likely to say what the company offers, such as products, services, applications and about.
  • When the company asks for PeakPick’s Pro reading: up to 150 pages, following the links between your site’s pages, and up to 20 PDF files they link to, such as datasheets (10 MB each at most).
  • Only HTML pages and those PDFs, on your own site. It doesn’t follow links to other sites, submit forms, run scripts or sign in.
  • Check again reads the same pages once more, for how they’re set up for search engines (their titles, descriptions, language and the settings that keep pages out of search), and keeps none of their text. It runs at most once every 10 minutes per company.
  • Your robots.txt, to tell the company whether the crawlers AI assistants search with may read the site: OAI-SearchBot and ChatGPT-User (ChatGPT), PerplexityBot and Perplexity-User (Perplexity), Claude-SearchBot and Claude-User (Claude), Applebot (Siri), Googlebot (Google, and its AI answers) and Bingbot (Bing, and Copilot). It also shows what you tell the crawlers that only gather text to train AI models: GPTBot, ClaudeBot, Google-Extended and Applebot-Extended.
  • Your homepage once more for each of five AI assistants’ crawlers, to tell the company whether a firewall turns AI assistants away even where robots.txt lets them in: one request each, with that crawler’s user agent followed by PeakPickBot-AccessTest/1.0 (+https://peakpick.ai/bot), for OAI-SearchBot, ChatGPT-User, PerplexityBot, Perplexity-User and Claude-SearchBot. Only after robots.txt lets PeakPickBot read the homepage; it keeps only how your site answered.

How it behaves

  • It obeys robots.txt: the rules for PeakPickBot, or for * when there are none for it. With no robots.txt, it reads the pages above.
  • If your robots.txt exists but can’t be read, because of a server error or a timeout, it asks once more after 5 seconds; if it still can’t, it reads nothing.
  • It makes one request at a time, at least a quarter of a second apart, or further apart if your Crawl-delay asks (honoured up to 5 seconds).
  • It uses public web addresses only (http and https, ports 80 and 443), and follows at most 5 redirects, all within your site.
  • It gives up on a request after 10 seconds, and reads at most 2 MB of a page.

Block it or slow it down

To block it entirely, add this to your robots.txt:

User-agent: PeakPickBot
Disallow: /

To keep it out of part of your site, or to space out its requests:

User-agent: PeakPickBot
Disallow: /private/
Crawl-delay: 5

It reads robots.txt again on every visit, so changes apply the next time it comes.

What happens to the pages

PeakPick keeps a copy of the pages it reads for the company that asked, and analyses their text with AI models (OpenAI) to see what the site says about the company, for example whether it mentions a product or a city. The copies are deleted when that company is deleted from PeakPick, and they aren’t used to train AI models. See our privacy policy.

Where requests come from

From Google Cloud’s europe-west1 region, in Belgium. PeakPickBot has no fixed IP addresses, so recognise it by its user agent.

Contact

Questions about PeakPickBot, or think it misbehaved on your site? Email hello@peakpick.ai.