Making a Bilingual Arabic and English Site Readable to AI Crawlers

Qatar businesses that ship Arabic and English pages still lose AI citations when crawlers meet cookie language switches, client-only renders, or a single robots.txt ban on every AI user agent. In Doha and Lusail, readable AI access means separate crawlable URLs for each language, server-rendered HTML with correct `lang` and `dir`, reciprocal language links, and robots rules that split training bots from search bots. Googlebot and AI search crawlers need stable public HTML; GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended govern training or grounding opt-outs, not classic Search rankings. Align those layers and assistants can quote a West Bay clinic or Hamad Port logistics page without inventing the wrong language version.

Why bilingual Qatari sites lose the machine audience

Retail groups in Msheireb, clinics in Al Sadd, and contractors bidding near Hamad Port already know that residents switch between Arabic and English in the same session. National digital programmes reinforce that habit: Hukoomi and other MCIT-backed services present bilingual interfaces as the default public experience under Qatar Digital Agenda 2030. Private sites that copy the look but not the crawl architecture still fail when a model or search assistant needs a stable URL.

AI systems do not negotiate cookie banners or remember which language a visitor selected last week. They fetch a URL, parse HTML, and decide whether the page is Arabic, English, or a confused mix. Google’s multi-regional guidance states that Googlebot usually crawls without `Accept-Language` and that automatic redirects based on inferred language can hide alternate versions. The same pattern hurts GPTBot, ClaudeBot, OAI-SearchBot, Claude-SearchBot, PerplexityBot, and Applebot: if the only Arabic body lives behind a JavaScript toggle on an English URL, many crawlers record the English string and never store the Arabic passage that Qatari users actually asked for.

Qatar-specific pressure is higher than for a monolingual European brochure site. Commercial Registration (CR) text, Qatar Chamber profiles, Google Business listings, and WhatsApp welcome messages often sit in Arabic, while procurement PDFs and career pages sit in English. When those assets lack matching public HTML counterparts, retrieval systems invent answers from third-party directories. The rest of this pillar turns that failure mode into a build checklist: crawler map, markup, URL design, robots policy, rendering, locality signals, and operations.

The AI crawler map a Doha publisher must actually name

Marketing teams in Qatar still treat “AI bots” as one bucket. Vendors do not. OpenAI separates GPTBot from OAI-SearchBot and ChatGPT-User. Anthropic separates ClaudeBot from Claude-SearchBot. Google publishes Google-Extended as a training and grounding control token that does not change Search rankings. Apple separates Applebot discovery from Applebot-Extended training opt-out.

For a Lusail real-estate site or a West Bay law firm, decide training opt-in and citation opt-in separately. Blanket bans on every agent whose name contains “Bot” often delete AI citations while Google Search stays intact. Verify user-agent tokens against published IP ranges or reverse DNS. Qatar’s Personal Data Privacy Law (PDPL) and sector rules from QCB or QFC still require private portals to stay closed even when marketing pages stay open to search-oriented crawlers.

Training-oriented agents and what they collect

GPTBot and ClaudeBot gather public web text that may contribute to model development. Google-Extended and Applebot-Extended are control tokens rather than separate HTTP user agents in many cases: Google states that Google-Extended has no distinct request user-agent string and that crawling still uses existing Google agents while the robots token governs training and grounding use; Apple states that Applebot-Extended does not crawl pages and only decides how Applebot-crawled data may train foundation models. Blocking those tokens is how a Doha publisher opts out of generative training or Gemini grounding pathways without necessarily leaving Search.

A hospital group near Hamad Medical City often blocks training agents on clinical education pages that contain sensitive phrasing while still allowing OAI-SearchBot and Claude-SearchBot on location and appointment overview pages. A construction supplier near Mesaieed may allow training agents on product catalogues so assistants can learn Qatari stock-keeping language. Spoofed agents exist in logs, so treat robots.txt as declared policy and published IP or reverse-DNS checks as verification. The choice remains commercial and legal, not a ranking cheat code.

Search and citation agents that decide if Qatar appears in answers

OAI-SearchBot, Claude-SearchBot, PerplexityBot, Googlebot, Bingbot, and Applebot feed discovery surfaces that cite or summarise pages. Blocking them while “optimising for AI” is self-defeating. These agents need the same crawlable Arabic and English HTML that classic SEO already demands: unique URLs, 200 responses, internal links, and sitemaps. If Arabic lives only as a React state, citation quality collapses even when robots.txt is wide open.

ChatGPT-User and Claude-User represent live user-triggered fetches. They matter when a procurement officer pastes a `.qa` URL into a chat. Those agents should reach the same public marketing HTML; they should not unlock authenticated PDPL-sensitive routes. Rate limiting and WAF rules can still throttle abuse without a full robots ban.

Control tokens that do not replace Googlebot or Applebot

Google-Extended does not alter Google Search rankings. Applebot-Extended rules are not ranking signals for Apple Search. Publishers who Disallow Google-Extended still need Googlebot allowed if they want Search, Discover, and related Google surfaces. Publishers who Disallow Applebot-Extended can remain eligible for Spotlight and Safari suggestions via Applebot, subject to other indexing rules such as `noindex` or `nosnippet`. Apple’s `nosnippet` meta further limits use of page text as AI answer context while titles may still appear—useful for regulated financial copy in Qatar Financial Centre pages that must stay findable but not freely remixed.

Table. AI and search agents relevant to bilingual Qatari sites (policy snapshot for operators)

HTML, RTL, and language signals machines can parse

Qatari sites fail crawlers less often on DNS than on markup. A page that looks Arabic in Chrome can still serialize as an English DOM if the language pack loads after hydration. Google determines language from visible content algorithms, not from the HTML `lang` attribute or from hreflang, yet operators must still make each URL’s body unambiguously Arabic or English.

Set `<html lang="ar" dir="rtl">` on Arabic documents and `<html lang="en" dir="ltr">` on English documents. MDN treats `dir` as the semantic directionality signal; Google’s internationalisation tips still advise markup on the `html` element rather than CSS alone. Avoid side-by-side bilingual walls on primary service pages: publish full `/ar/` and `/en/` URLs for a Doha clinic, then link them with a visible switcher so assistants cite one clean language passage.

Document language, direction, and character encoding

Serve UTF-8 everywhere. Arabic text mis-decoded as Latin-1 becomes noise for embeddings. Declare charset in HTTP headers and in `<meta charset="utf-8">`. Keep Eastern Arabic–Indic digits consistent within a locale if the brand standard requires them, and document the choice so editors do not mix forms inside one H1. Titles, meta descriptions, Open Graph tags, and JSON-LD `inLanguage` should match the page language. Schema that claims `ar` while the body is English teaches models the wrong association for a Pearl Qatar address block.

Mixed strings—Latin brand names inside Arabic paragraphs—need careful Unicode bidirectional handling so telephone numbers and CR digits do not reverse. Screen readers used by residents and by enterprise accessibility audits will punish missing language tags even when Googlebot guesses the script. Crawlers that render JavaScript still prefer a first HTML response that already contains the main article text; if Arabic arrives only after an XHR to `/api/i18n/ar`, logs may show 200 while the stored body is empty. Server-side rendering or static generation for both locales keeps GPTBot and ClaudeBot from timing out on empty shells. Keep CSS and JS required for layout allowlisted for Applebot and Googlebot; Apple notes that blocked resources can prevent proper rendering.

Navigation and in-content language switching

Place a crawlable `<a href>` to the alternate language in the header, not only a button that fires `localStorage.setItem('lang','ar')`. Google recommends letting users switch languages with hyperlinks and discourages automatic redirects that hide versions from crawlers. The same links help AI agents discover the Arabic twin after landing on English. Annotate the switcher with clear accessible names (“العربية” / “English”) without stuffing keywords into the URL path beyond the locale segment.

Media, PDFs, and assets that fragment the bilingual story

Many Qatari firms publish Arabic HTML and English PDFs only, or the reverse. Crawlers index PDFs, but assistants often prefer HTML passages with clear headings. Mirror critical facts—working hours in West Bay, Hamad International Airport pickup rules, QAR pricing notes—in HTML for both languages. Provide `hreflang` or at least descriptive filenames for language-specific PDFs. Alt text should match page language; English alt on an Arabic page confuses multimodal pipelines.

URL architecture that keeps Arabic and English crawlable

Google recommends distinct URLs per language version rather than cookie- or browser-driven swaps. For `.qa` sites, path prefixes (`example.qa/ar/…`, `example.qa/en/…`) and subdomains (`ar.example.qa`, `en.example.qa`) dominate. Path prefixes usually win for mid-size Doha companies because one certificate, one Search Console property, and one CDN cover both languages.

Do not rely on `?lang=ar` as the only differentiator. Never serve different languages at the same URL based on IP geolocation alone: Google states it does not vary crawler source to find every variant, so a USA-origin crawl may only ever see English. Map West Bay naming consistently across Arabic and English pages, keep each URL as a 200 with unique content, and list both languages in sitemaps so internal links do not orphan Arabic service clusters.

Path prefixes versus subdomains on .qa hosts

Path prefixes (`/ar/`, `/en/`) keep authority consolidated on the ccTLD that already signals Qatar. Subdomains help when Arabic marketing runs on a headless CMS and English careers run on separate hiring software, but they split cookie scopes and each host needs its own robots.txt review. Avoid `www` versus non-`www` splits that duplicate both languages. Prefer HTTPS everywhere; mixed content on RTL stylesheets breaks rendering for bots that execute CSS. If a legacy ERP forces query parameters, canonicalize carefully and still expose clean paths for public marketing pages.

Soft 404s, language homepages, and thin translation shells

Machine-translated shells with fifty words of English boilerplate and a promise of “full Arabic soon” waste crawl budget. Publish a language version when it is complete enough to answer a user question: services, contact, and proof. If Arabic launches later, keep English live and add Arabic URLs only when ready—do not reserve empty `/ar/` paths that return soft 404 templates. Return true 404 or 410 for retired locales so assistants stop citing dead links.

English-first CMS workflows in Qatar agencies often leave Arabic pages reachable only from a language switcher. Add contextual links inside Arabic service clusters so a page about web development links to related Arabic case studies. If the Arabic page names “منطقة الخليج الغربي” and the English page says “West Bay,” keep both phrases on their respective pages and add a Doha zone reference both retrieval systems can ground. Hreflang annotations can label the pair for Search, but crawlability still depends on each URL existing as indexable HTML.

Cross-links, breadcrumbs, and sitemap pairs

Breadcrumbs should stay inside one language. An English breadcrumb that jumps into Arabic mid-trail confuses both users and chunking pipelines. XML sitemaps need lastmod dates that update when either translation changes. If English pricing in QAR changes, bump the English URL and decide whether Arabic must change the same day; stale Arabic prices create hallucination risk when models merge both versions.

robots.txt policies that separate training from citation

Place `robots.txt` at the site root on each host. Declare sitemaps with absolute HTTPS URLs. Write explicit groups for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Google-Extended, Applebot, Applebot-Extended, PerplexityBot, and Googlebot instead of hoping a single `*` rule expresses intent. Anthropic notes that IP blocking without a readable robots.txt can prevent opt-out from working; Google and Apple similarly expect robots directives for automated crawls.

A citation-forward policy for a public marketing site in Qatar often looks like: allow Googlebot, Bingbot, Applebot, OAI-SearchBot, Claude-SearchBot, and PerplexityBot on `/ar/` and `/en/` marketing paths; Disallow all of them on `/app/`, `/wp-admin/`, and authenticated portals; optionally Disallow GPTBot, ClaudeBot, Google-Extended, and Applebot-Extended if counsel wants training opt-out. A research university in Education City may reverse the training decision and allow GPTBot while still locking student portals.

Test after deploy. Fetch `https://www.example.qa/robots.txt` with curl, then sample logs for seven days. Confirm Google-Extended Disallow does not coincide with an accidental Googlebot Disallow. Confirm Applebot remains allowed if Spotlight visibility matters. Re-check after CDN or WAF changes; some edge configs inject bot challenges that return 403 to AI crawlers even when robots.txt allows them—assistants then cite competitors who answer with 200.

Keep crawl-delay only where a vendor documents support (Anthropic mentions non-standard Crawl-delay for its bots). Do not assume Google honors crawl-delay. Prefer server-level rate controls. Document the policy in an internal runbook so a new marketing hire in Doha does not “block AI” during a campaign and silently delete citations.

Content rendering: SSR, hydration, and language detection traps

Client-side internationalisation libraries popular in Next.js and Nuxt stacks often default to browser locale. That helps a resident opening the site on an Arabic phone. It harms crawlers that arrive without Accept-Language and without cookies. Force the URL locale to drive the rendered language. If the path is `/ar/contact`, the HTML must be Arabic even if the crawler fingerprint looks like a US data centre.

Server-side render the main content. Hydrate interactions afterward. For bilingual menus, render both language switcher links in the initial HTML. Lazy-load below-the-fold media, not the H1. If using edge middleware to redirect `/` based on geo, send Google and AI bots to an `x-default` chooser page or a stable default rather than a silent 302 into one language forever—Google warns that automatic language redirects can prevent discovery of all versions.

Monitor Core Web Vitals on both locales. Arabic fonts (for example, carefully licensed Naskh or Kufi families) can inflate LCP if subsetting is wrong; slow LCP correlates with incomplete renders in time-limited crawlers. Prefetch locale-specific font files. Ensure `dir="rtl"` flips layout without clipping phone numbers in sticky headers common on Qatari service sites.

Personalisation platforms that swap testimonials by city must not swap the entire language for bots. Serve a consistent public version. Save geo-personalisation for authenticated or explicitly consented experiences that remain non-indexed.

Structured locality signals that confirm the page is Qatari

AI answers about “best clinic in Doha” favour pages that carry verifiable local entities: zone names, PO Box or building references where published, `+974` phone numbers, QAR currency, CR numbers on about pages, and Organisation schema with `addressCountry: QA`. Bilingual sites should duplicate those facts in both languages with consistent NAP (name, address, phone). Mismatched phone formats between Arabic and English pages teach models that the business is uncertain.

Google Business Profile, Qatar Chamber directories, and Hukoomi-linked service descriptions should use the same trade name transliteration. If Arabic legal name and English trading name differ, state both on the about page. Link to maps embeds that resolve to the correct West Bay tower rather than a generic “Doha” pin. These signals do not replace crawlable bilingual HTML; they reinforce it when assistants triangulate sources.

For regulated sectors, publish clear scope statements in both languages: QCB-licensed banking copy, QFC-licensed legal services, Ministry of Public Health clinic licensing notes. Assistants reduce hallucinated licence claims when the licensed entity string appears in static HTML. Keep disclaimers crawlable, not only in images.

Operational checklist for teams shipping from Doha

Treat bilingual AI readiness as a release gate, not a blog task. Before launch: verify both locales return 200; confirm `lang`/`dir`; confirm language switcher links; confirm robots groups; confirm sitemaps; confirm no IP-based language cloak for Googlebot user agents; confirm PDFs mirrored or clearly labelled; confirm logging captures AI user agents. After launch: quarterly diff of robots.txt against vendor docs; sample assistant answers for branded queries in Arabic and English; fix whichever language is missing from citations first.

Agencies building for Qatari clients should add an “AI crawl preview” step: curl the Arabic and English URLs with a simple bot-like user agent, save HTML, and check that the main article text exists without executing JavaScript. If the saved file lacks the service description, GPTBot will lack it too. Pair that with Search Console language reports and server log dashboards.

Budget translation properly. Thin machine translation without human legal and industry review creates confident wrong answers about Qatari labour rules or clinic procedures. Human-edited Arabic marketing prose plus accurate English technical prose beats automated parity that invents CR numbers.

Owners who only measure Google organic traffic will under-invest in OAI-SearchBot and Claude-SearchBot readiness. Add referral and log-based monitors for those agents. When a competitor in The Pearl appears in ChatGPT answers for “interior design Qatar,” compare crawl access, language URL design, and on-page locality—not only classic backlink counts.

Frequently asked questions

Does blocking GPTBot hide a Qatar site from ChatGPT answers?

Not automatically. OpenAI treats GPTBot as a training-oriented crawler and OAI-SearchBot as the search indexing crawler; ChatGPT-User handles some user-triggered fetches. Disallowing GPTBot can reduce future training use of the site’s text without removing eligibility for search-oriented surfaces that rely on OAI-SearchBot. A Doha publisher that wants citations should keep OAI-SearchBot allowed on public Arabic and English marketing URLs. Always re-read OpenAI’s current bot documentation before changing production robots.txt.

Should Arabic and English share one URL with a language toggle?

No for primary marketing pages. Google recommends distinct URLs per language and warns that cookie- or browser-based language swaps can hide variants from crawlers that lack Accept-Language. AI crawlers inherit the same failure mode when Arabic HTML never appears at a stable address. Use `/ar/` and `/en/` paths or clear subdomains, then link them with a visible switcher. Keep toggles as progressive enhancement, not as the only storage for translated copy.

Does Google-Extended blocking hurt google.com.qa rankings?

Google states that Google-Extended does not impact inclusion in Google Search and is not used as a ranking signal. It controls whether content may be used for training future Gemini models and for certain grounding pathways in Gemini Apps and Vertex AI. A West Bay retailer can Disallow Google-Extended for training or grounding preference while still allowing Googlebot for Search. Do not confuse Google-Extended with GoogleOther, which Google describes as a generic research crawler without a specific consumer product mapping.

How should RTL markup be set for AI and accessibility?

Put `lang="ar"` and `dir="rtl"` on the Arabic document root and matching English attributes on English pages. Google may detect language from content, but browsers, assistive tech, and many NLP preprocessors still read those attributes. Prefer HTML `dir` over CSS-only direction flips. Ensure the first HTML response already contains Arabic body text so non-rendering or partially rendering crawlers store the correct script.

What robots.txt mistake most often kills AI citations in Qatar?

Blanket bans on every agent whose name contains “Bot,” including OAI-SearchBot, Claude-SearchBot, PerplexityBot, and sometimes Googlebot by accident. Teams react to training-data headlines and delete citation pathways. Write per-agent rules, allow search-oriented crawlers on public pages, and restrict only training tokens or private paths as counsel requires. Validate with log lines after deploy, not only with a robots.txt editor screenshot.

Do .qa domains make bilingual sites more readable to AI crawlers?

A `.qa` ccTLD is a strong Qatar geo signal for users and for Google’s locale understanding, but it does not fix missing Arabic HTML or blocked search bots. Crawlers still need distinct language URLs, clean markup, and allowed fetch access. Pair the ccTLD with bilingual architecture and accurate NAP data. Treat the domain as locality confirmation, not as a substitute for crawlable translations.

How often should a Lusail company re-audit AI crawler access?

Re-read vendor bot documentation whenever robots.txt changes, after major CDN or WAF releases, and at least quarterly. New tokens appear (search versus training splits already forced one industry-wide rewrite). Sample assistant answers for branded Arabic and English queries each quarter and compare HTML curl snapshots. If citations vanish after a “security hardening” project, inspect 403 rates for AI user agents before commissioning new content.