About our crawler

LoveYourNextJobBot

You may have seen LoveYourNextJobBot/1.0 in your server logs. That's us. This page explains who we are and how to control what we do on your site.

Who we are

LoveYourNextJob.com is a job-search and apply-automation platform. We index public careers pages and company hiring platforms (Greenhouse, Lever, Ashby, Workday, etc.) so candidates can find openings across companies from one search.

What we do on your site

  • Read your robots.txt and honor every Disallow directive — both the wildcard User-agent: * rules and any rules that target LoveYourNextJobBot specifically.
  • Probe a small set of common careers paths (e.g. /careers, /jobs) only when our database has identified you as a company that might be hiring. Most sites see at most a handful of HEAD requests per week.
  • Render your careers page once it's discovered, and parse schema.org JobPosting JSON-LD when present (the same data Google Jobs reads). When there's no JSON-LD, we extract job listings from the rendered HTML using AI-learned CSS selectors.
  • Rate-limit ourselves to ≤ 1 request per second per host with a small burst allowance.

What we never do

  • We do not bypass anti-bot challenges or rate limits.
  • We do not crawl LinkedIn, Indeed, or Glassdoor. (Their terms prohibit it.)
  • We do not store or republish your job description text without your data being publicly available via your own careers page.
  • We do not resell scraped data to third parties.

How to block us

The fastest way to stop our crawler is your robots.txt:

User-agent: LoveYourNextJobBot
Disallow: /

We re-fetch robots.txt at most every 6 hours, so a change should take effect within that window.

If you need a faster removal or want to opt out of being indexed in our search results entirely, email hello@loveyournextjob.com with your domain. We'll add it to a hard exclusion list and respond within 2 business days.

Why we exist

We help people find jobs without spending hours navigating dozens of careers sites. The companies whose pages we index benefit from broader candidate reach, the same way they benefit from being indexed by Google Jobs. If that exchange isn't something you want, the opt-out above takes effect immediately.

Last updated 2026-05-04 · Operator: hello@loveyournextjob.com