Bot
The searchnapply crawler: what it is, what it reads, how to keep a page out.
Who is calling
Every request carries this User-Agent:
searchnapply/1.0 (+https://searchnapply.com/bot)Where a host refuses plain agents, the crawler sends a browser string that ends in the same token:
Mozilla/5.0 [the browser string] searchnapply/1.0 (+https://searchnapply.com/bot)What it reads
- Public job postings on employers' own career pages, kept current: a posting that closes leaves the index.
- The application form: its questions, as any visitor sees them. Never the answers, never a page behind a login, never personal data.
- Nothing is submitted. The crawler reads. It never fills a form and never applies.
How it paces
One request stream per host, slowed down further when a site answers 429 or 503.
Keep a page out
Write to info@searchnapply.com with the career page URL. The crawler stops reading it and its postings are gone within three to five business days.