Googlebot vs the AI search crawlers on agency job pages
Google runs the scripts that build a page, with delays; in a December 2024 study by Vercel and MERJ, the crawlers used by OpenAI, Anthropic and Perplexity did not. So a script-built job can reach Google and miss AI search. In The Staffing Career Site Report 2027, 17% of the 1,094 agency job pages we could check in October 2026 had no job text in the page on arrival.
Benjamin BartakovicsCo-Founder & CPO6 min read
Ask AI

In short
- Google runs the scripts that build a page, with delays; in a December 2024 study, the crawlers used by OpenAI, Anthropic and Perplexity did not.
- On 17% of the 1,094 job pages we could check in October 2026, the server's HTML held no job text, leaving AI crawlers that skip scripts nothing to read.
- robots.txt kept Googlebot out of 3% of 1,585 agency job boards, the AI search crawlers out of 4% and the GPTBot training crawler out of 6%.
- Rendering job pages on the server is the one fix that works for every crawler here; Google for Jobs also needs JobPosting data.
On this page
- How do Googlebot and the AI search crawlers compare?
- Who renders JavaScript?
- Who follows which robots.txt rules?
- Where do these crawler claims come from, and when?
- How can I see my job page the way each crawler does?
- Verdict: build for the crawler that does the least
- When to choose the other route: keeping a crawler out
How do Googlebot and the AI search crawlers compare?
The crawlers differ in two ways that affect agency job pages: whether they run scripts, and which name they answer to in robots.txt. Here a crawler means the software Google or an AI company uses to fetch your job pages and read them. An AI search crawler is the one an AI assistant, a tool built on a large language model, uses to find pages it can quote in an answer.
| Googlebot | AI search crawlers | GPTBot | |
|---|---|---|---|
| Who sends it | OpenAI (OAI-SearchBot), Perplexity (PerplexityBot), Anthropic (Claude-SearchBot) | OpenAI | |
| What it feeds | Google Search, Google for Jobs and Google's AI features | The pages AI assistants find and quote in answers | Training of OpenAI's models, not search |
| Runs JavaScript | Yes, with delays (Vercel and MERJ, 2024) | The crawlers used by OpenAI, Anthropic and Perplexity did not (Vercel and MERJ, December 2024) | Not named by the report |
| What a job page needs | Own address and job text, or time to render; JobPosting data for Google for Jobs | The job text in the HTML; for ChatGPT search, OAI-SearchBot allowed (OpenAI) | Nothing for search |
| Kept out by robots.txt on agency boards, October 2026 | 3% of 1,585 boards | 4% of 1,585 boards | 6% of 1,585 boards |
Putting the job text in the HTML the server sends, so the job is already in the page when it arrives rather than added by a script, is the one fix that serves every column.
Google for Jobs, the job search box Google shows above normal results, also needs JobPosting data, a block of fields inside the page, unseen by candidates, that hands Google the title, pay, location, employer and closing date as data.
The two questions for whoever runs your board: are job pages rendered on the server, and does every job page carry JobPosting data? Which jobs an assistant then quotes is covered in our study of what ChatGPT answers.
Who renders JavaScript?
Google does; in the December 2024 study, the crawlers used by OpenAI, Anthropic and Perplexity did not. JavaScript rendering means a script in the browser builds the page after it loads. People see the job; a crawler that does not run scripts sees an empty page.
On the 17% of the 1,094 checked job pages with no job text in the HTML the server sends, the job waits for Google to render it. A forklift job built by a script still reaches a candidate on Google once Google renders it, and stays blank to a crawler that skips scripts.
Across the 1,195 agency job pages, the results by board type show where this bites. On the 267 ATS-hosted boards, 47% of job pages were readable without JavaScript, against 98% on the 281 boards from staffing website vendors (margins about ±5 to 9 points). A board that renders in the browser has the most to gain.
Google's AI features draw on its normal index, so a script-built job can still reach them once Google has rendered it. The report's checklist adds AI agents, software that applies for jobs on a candidate's behalf: without the job text in the HTML, AI crawlers and agents cannot see the job.
Ask your web vendor or ATS provider for server rendering on every job page.
For your developer
Fetch the job page without running scripts and look for the job title:
curl -s https://jobs.example.com/jobs/123 | grep -i "forklift operator"No match means the first response carries no job, so render the job page on the server or pre-render it.
Who follows which robots.txt rules?
robots.txt rules are written per crawler name, which is how one file can let Google in and keep every AI crawler out. robots.txt is a short text file at the root of a website that tells crawlers which parts they may read.
Few agency boards use it to keep these crawlers out: 4% of the 1,585 boards we checked in October 2026 did so for the AI search crawlers. So for AI search, check the job text in the HTML first. The report adds that these rules were often the platform's doing rather than the agency's.
Bot protection is the other gate: a security layer that turns away visitors that do not behave like a normal browser. 8% of the 1,195 job pages we tested in October 2026 refused requests that did not come from a full browser, which the report warns can keep out crawlers the agency wants.
Ask whoever runs the board for the robots.txt file on the domain your jobs use, and whether a managed robots.txt setting at Cloudflare, a security and delivery service, is switched on.
For your developer
This is the pattern the report found on a CRM-based site platform's shared domains, simplified to the rules described:
User-agent: Googlebot
Allow: /
User-agent: *
Disallow: /Under the robots.txt standard (RFC 9309, September 2022) and Google's robots.txt documentation (checked 6 November 2026), a crawler follows the group that names it. A crawler with no group of its own falls back to User-agent: *. So Googlebot is let in, and every other crawler, AI crawlers included, is kept out.
Where do these crawler claims come from, and when?
The JavaScript finding comes from one December 2024 study, and the rules from the operators' own documentation, which changes:
- Google renders JavaScript, with delays: Vercel and MERJ, 2024.
- The crawlers used by OpenAI, Anthropic and Perplexity did not run JavaScript: Vercel and MERJ, "The rise of the AI crawler", 17 December 2024.
- ChatGPT search needs OAI-SearchBot allowed: OpenAI, publishers and developers FAQ (cited in the report) and OpenAI's crawler documentation at developers.openai.com/api/docs/bots, checked 6 November 2026.
- robots.txt group matching: the Robots Exclusion Protocol (RFC 9309, IETF, September 2022) and Google's robots.txt documentation, checked 6 November 2026.
Short answers to the follow-up questions, such as llms.txt and Google-Extended, are in the findability FAQ.

How can I see my job page the way each crawler does?
Two browser checks and a question show what each crawler gets.
Verdict: build for the crawler that does the least
Choose rendering on the server first if your job text is missing from the HTML. It puts the job text in front of the AI search crawlers and removes Google's wait. Choose a robots.txt and bot protection review first if the text is already there, and check the JobPosting data at the same time, since Google for Jobs lists jobs only from it.
Declared interest: we build a job board for staffing agencies, connected to Bullhorn and run on the agency's own domain or subdomain. It renders its job pages and first listing page on the server. The free job board audit checks crawler access for Google and AI assistants on your own board.
When to choose the other route: keeping a crawler out
Keeping a crawler out can be the right call, as long as someone decides it by name. robots.txt kept GPTBot out of 6% of the 1,585 agency boards we checked in October 2026. OpenAI treats GPTBot and OAI-SearchBot as separate settings (OpenAI crawler documentation, checked 6 November 2026). So a rule for GPTBot alone keeps your pages out of OpenAI's model training and leaves your jobs open to ChatGPT search.