7 reasons agency jobs miss Google and AI search, most common first
The most common reason agency jobs miss Google's job search is a missing hidden label. 45% of the 1,195 job pages we tested in October 2026 for The Staffing Career Site Report 2027 had no JobPosting data. Below are the seven reasons we found, ranked by the share the report prints for each. Some shares count job pages and others count boards, so each one names its base.
Daniel BartakovicsCo-Founder & CDO6 min read
Ask AI

In short
- The most common reason is missing JobPosting data: 45% of the 1,195 agency job pages we tested in October 2026 had none, a label that comes from the job template.
- 17% of the 1,094 job pages we could check had no job text in the HTML, so a crawler that does not run scripts finds no job.
- 63 of the 1,195 job pages carried JobPosting code with a format error (not valid JSON), which Google ignores; the Rich Results Test shows it as an error.
- Outright blocks were rare: noindex on 3% of job pages, Googlebot blocked on 3% of 1,585 boards, and the blocking rule often came from the platform.
On this page
- No JobPosting data: 45% of job pages
- No job text in the HTML: 17% of the job pages we could check
- Hash URLs: 11% of job pages
- Bot protection that may turn crawlers away: 8% of job pages
- JobPosting code that is not valid JSON: 63 job pages (5%)
- robots.txt blocks an AI search crawler: 4% of job boards
- Told to stay out: noindex on 3% of job pages, Googlebot blocked on 3% of boards
- What should I do with this list?
Why agency jobs miss Google and AI search
Bot protection refusals are counted as unknown, not failed. Each share has its own base: job pages tested, job pages we could check, or job boards whose robots.txt we read.
1. No JobPosting data: 45% of job pages
45% of the 1,195 job pages we tested had no JobPosting structured data, a hidden label that tells Google the job title, pay, location, employer and closing date in a form a machine can read. Google for Jobs, the job search box Google shows above normal results, lists jobs from that label, so for those jobs Google's job search does not exist. Where the label was present, salary appeared in 50% of pages and an expiry date in 72%, two of the fields agency job pages leave out.
The label comes from the board's job template, so the cause is the setup. ATS-hosted boards carried it on 23% of their 267 tested job pages and boards from staffing website vendors on 80% of 281. What we found on 1,195 job pages splits every reason here by board type.
To check, run step 3 of the ten-minute self-audit; its steps cover every reason below except bot protection. Ask your web agency or board provider to add JobPosting data to the job template, in the words of the questions to send your web vendor.
2. No job text in the HTML: 17% of the job pages we could check
Of the 1,094 job pages where bot protection did not stop the check, 17% had no job text in the HTML the server sends. The job text is missing when the page arrives, and a script in the browser fills it in afterwards. A recruiter posts a Warehouse Associate job in Bullhorn at nine, a candidate reads it on her phone at lunch, and an AI crawler that opens the same page finds nothing to read.
Google runs the script later. In a study by Vercel and MERJ published 17 December 2024, the crawlers used by OpenAI, Anthropic and Perplexity did not run scripts. The cause is how the board builds its pages, and the report found that ATS-hosted boards more often render jobs in the browser.
To check, view the page source and search for the job title, as in step 2 of the self-audit. Ask your board provider for job pages rendered on the server, so the job text is in the page when it arrives.
3. Hash URLs: 11% of job pages
11% of the 1,195 job pages lived at an address with a # before the job's part of the address, like agency.com/#/jobs/123. Google generally ignores everything after the #, so every job on the board looks like the same page.
Hash URLs cluster on ATS-hosted boards; job board statistics by board type give the share for each type.
To check, open any job and read the address bar: a # before the job's part of the address is this reason. Ask for one clean address per job, such as agency.com/jobs/warehouse-associate-columbus; the fix on an ATS-hosted board starts with how the board was set up.
4. Bot protection that may turn crawlers away: 8% of job pages
8% of the job pages we tested refused requests that did not come from a full browser. Bot protection is a security layer that blocks visitors that do not behave like a normal browser. Unless it lets search and AI crawlers through by name, it can keep them out too.
People see the page as normal, so the self-audit cannot catch it. The report counted these pages' job-text and sitemap checks as unknown, not failed, so how many of them crawlers cannot read is not known. Ask your web agency which crawlers the bot protection lets through.

5. JobPosting code that is not valid JSON: 63 job pages (5%)
63 of the 1,195 job pages (5%) carried JobPosting code that is not valid JSON, the strict format the label must be written in. Google ignores such code, and a single broken character, such as a missing comma, is enough to make it ignore the whole label.
To check, run the job through Google's Rich Results Test, as in step 3 of the self-audit: this problem shows as an error where a working page shows a job posting. Ask your web agency to find where the broken code comes from; if it is the job template, one fix repairs every job.
6. robots.txt blocks an AI search crawler: 4% of job boards
robots.txt blocked at least one AI search crawler on 4% of the 1,585 job boards we checked. The file sits at the root of the website and lists which parts each crawler may read. AI search crawlers are the programs AI assistants send to find pages they can quote: OpenAI's OAI-SearchBot, PerplexityBot and Claude-SearchBot. ChatGPT search needs OAI-SearchBot allowed, so where it is blocked, the jobs cannot appear there.
The rule often came from the platform. On one CRM-based site platform's shared domains (a CRM is the customer database a sales team uses), robots.txt lets Googlebot in and closes the site to every other crawler.
Several agency sites block AI crawlers through the managed robots.txt settings of Cloudflare, a security service many sites sit behind, instead of a rule written for the site. A rule against OpenAI's training crawler, GPTBot, is a separate matter, covered in Googlebot vs the AI search crawlers.
To check, open robots.txt as in step 4 of the self-audit, which shows how to read the blocking line. If a rule names OAI-SearchBot, PerplexityBot or Claude-SearchBot, ask your board provider to let it in, and ask your hosting provider whether a Cloudflare setting adds the rule.
7. Told to stay out: noindex on 3% of job pages, Googlebot blocked on 3% of boards
3% of the job pages we tested carried a noindex tag, a line in the page that tells Google not to show it in search. robots.txt kept Googlebot, Google's own crawler, out of 3% of the 1,585 job boards. Either one keeps the job out of Google.
The report found that few agencies block search engines on purpose. On one ATS's shared job domain, robots.txt disallows every crawler, Google included, and every board on that domain carries the rule.
To find a noindex tag, open the page source as in step 2 of the self-audit and search for the word noindex; for the Googlebot rule, read robots.txt as in step 4. If you find either, ask who set it before anyone removes it.
What should I do with this list?
Run the self-audit on one live job, write down each fail next to the company that runs that part: web agency, board provider or hosting provider. Send each one the matching question from the vendor brief linked under reason 1.
Our own product is a job board for staffing agencies: agencies buy it, connect it to Bullhorn and run it on their own domain. So we have a stake in this list, and the tests treated every board type the same way. To have the checks run for you, request a free job board audit.
