PepFinderBot

The crawler that keeps supplier profiles and prices on PepFinder up to date.

It identifies itself as PepFinderBot/1.0 (+https://pepfinder.com/about/bot).

What it reads

Public pages of stores that sell research peptides: the homepage, product pages and policy pages (shipping, returns, terms, contact).
Public product data the store already publishes for machines: its platform’s product feed, schema.org product data, and the product data its own pages load. Prices are taken only from that data.
Some stores only show products once their JavaScript runs. For those it opens the page in a headless browser, without images, fonts or video, and reads the same public product data.

It reads robots.txt first and follows it, including Crawl-delay. Paths it is asked not to visit are not requested, by the crawler or by a page it opens.
It reads one page at a time per site, at least two seconds apart.
It never signs in, fills in forms, adds to a basket or buys anything.
It stops at the first bot check or CAPTCHA and does not try to get past it.

Add a group for PepFinderBot to your robots.txt. To keep it out entirely:

User-agent: PepFinderBot
Disallow: /

Store owners can also claim their profile and keep it up to date themselves.

If the crawler causes a problem on your site, contact us and say which domain.