How we crawl shops
We build a price-comparison catalogue by reading the public product pages of shops that do not publish a product feed. We do that from our own addresses, at a pace a shop would not notice, and only within the terms each shop has set. This page describes how we behave as a visitor and where we draw the line.
Who is asking
Our crawler is called EnebaPriceBot. Every request identifies it, and every request comes from an address we publish. A shop that wants to admit us, slow us down or refuse us has one place to look.
EnebaPriceBot/1.0 (+https://www.eneba.com/lt/kaina/documentation/how-we-get-data). Every request we make carries this exact name and the address of this page, so a shop can find out who is asking without blocking anyone else.
EnebaPriceBot. Add this token as a User-agent group to speak to us alone. Disallow: / opts the whole shop out, and Crawl-delay under the same group sets the pace we keep.
https://www.eneba.com/lt/kaina/scraper-ip.json. This list is the authoritative source. Requests to shops behind bot protection leave from these addresses directly, never through a proxy, so the address in your log is one of them.
dsa@eneba.com. Write to us to ask for a slower rate, to opt out entirely, or to report a request that misbehaved. We answer and we act on it.
To be left out of the catalogue entirely, add User-agent: EnebaPriceBot and Disallow: / to robots.txt, or write to dsa@eneba.com.
How we crawl politely
We are guests on every shop we read. These are our default crawling practices, designed so that a shop barely notices we are there and our traffic cannot be mistaken for load.
Every shop is contacted with a request for its robots.txt before any product page. The file is the shop stating what a crawler may read and how fast, and we treat it as the first word on the subject. We read it as the token, so a rule written for us applies even where a rule for everyone else does not.
When a shop asks for a pause between requests, we take the slower of its number and ours. If a shop asks for more than two minutes between requests, we do not crawl it at all rather than crawl it faster than it asked.
If robots.txt cannot be fetched, we treat the shop as off-limits and try again later. Not knowing the rules is not permission to proceed. A 404, meaning no file exists, is the one case that reads as allow-all.
Two requests per second and at most six in flight against a single shop, with a random delay added so our traffic does not arrive in a predictable rhythm.
Shops in a run start at least a quarter of a second apart. A run covering several hundred shops ramps up over minutes instead of opening every connection in the same instant.
When a shop answers 429 or 503, we wait for as long as it asks. The pause applies to the whole shop, not just the one URL that triggered it.
A failed request is retried a bounded number of times with growing delays. A shop that keeps failing is left alone rather than retried until it answers.
If throttling survives every retry, we stop touching the shop for an hour, then six hours, then a day, then three days. After repeated lockouts the shop is flagged for a person to review instead of being crawled again automatically.
Many unrelated shops sit behind one shared provider. When the same throttle response appears across several of them, we treat the group as a single endpoint and share one budget, so we slow down for all of them at once.
Catalogue pages are visited as part of a scheduled run rather than polled continuously. Between runs the data we already hold is what we publish.
If a page does not show a value, the field stays empty. A missing price is recorded as missing, never estimated or carried over from another shop.
What we will not do
These limits also apply when a shop is not cooperating. Any exception to the robots.txt and Crawl-delay rules requires specific, recorded approval for that shop, as described below.
Disallow rules and Crawl-delay are enforced by default. A run that would ignore them is refused unless a specific, recorded approval covers that shop.
A block is treated as a stop sign. We do not retry around it, rotate through other addresses to get past it, or treat it as a puzzle to solve.
A 429 or a Retry-After is an instruction to wait. Speeding up, retrying immediately or spreading the requests elsewhere to avoid the wait is not something we do.
Rate limits and concurrency caps are enforced by the runner, so an individual shop recipe cannot opt out of them or ask for a larger budget.
We read product pages a visitor can reach. We do not log in, use an account, or reach for anything behind an access decision.
Requests name us in the User-Agent and leave from the addresses in our published list. A shop that does not want our crawler can name it in robots.txt and refuse it, and that refusal is the end of the matter.
Shops with bot protection
Some shops put their catalogue behind a bot-protection service, most commonly Cloudflare. From our side this looks like a page asking the visitor to prove it is a person before the shop will serve anything.
We do not treat that as a wall to get around. A protection layer is the shop saying it wants to know who is reading and on what terms, and the honest answer to that is to identify ourselves and ask, not to find a way past the check.
So for a protected shop we take the route that keeps the crawl legal and keeps the shop in control:
- We approach it as a normal visitor would, from the addresses published above and never through a proxy, so the request a shop sees belongs to us and nobody else.
- If the shop will not serve us, we stop and leave it alone rather than look for another way in.
- If the shop does want to be included, the agreement is made with the shop — not with its protection layer.
- Where a shop cannot be read this way, it stays out of the catalogue until there is a legitimate arrangement to include it.
A shop we cannot read politely is a shop we do not read. Coverage is never worth more than the terms the shop has set.