Arabic Content Chunking: Why RTL Text Breaks Retrieval

Arabic content on Qatari bilingual sites often looks correct in the browser and still fails inside retrieval pipelines. The Unicode Bidirectional Algorithm, HTML `dir="rtl"`, invisible BiDi control characters, and English-tuned chunkers interact badly with mixed Arabic–English strings common on .qa pages for Doha services, Lusail projects, and West Bay clinics. Fixed 500-character splits sever morphology, ignore Arabic question marks and full stops, and embed directional noise into vectors. AI crawlers and RAG systems then retrieve the wrong passage or none. Fixes start with normalisation, script-aware sentence chunking, identical query-time cleaning, and HTML that declares direction without relying on CSS alone.

What breaks when Qatari sites feed Arabic into English pipelines

Most retrieval stacks used by marketing teams, chatbot vendors, and internal knowledge bases in Qatar were first tuned on English documentation. They split on spaces and Latin punctuation, count tokens with byte-pair models that fracture Arabic stems, and assume left-to-right reading order when stitching snippets for a model context window. A services page that interleaves Arabic policy copy with English product names, CR numbers, and +974 phone strings is a stress test those defaults fail.

The failure is easy to miss because browsers implement the Unicode Bidirectional Algorithm and honour `dir` attributes, so editors in Doha see fluent Arabic. The crawler or PDF extractor may store a different logical order, leave U+202A–U+202E marks in the text, or flatten an RTL table into LTR cells. The chunker then cuts at character offset 500 in the middle of a word that carries a proclitic و. The embedding model, often multilingual but still uneven on Arabic morphology, places that fragment far from the query vector for the same topic written without diacritics. Retrieval returns a neighbouring English boilerplate block from the bilingual template. The generator answers from the wrong evidence or abstains.

Qatari sites amplify the problem because bilingual publishing is normal, not optional. Hukoomi-facing SMEs, clinic sites in West Bay, and developer microsites for Lusail towers routinely ship `/ar/` and `/en/` trees, RTL layouts, and mixed inline spans. Entity strings such as zone–street–building addresses and Latin brand names sit inside Arabic paragraphs. Any pipeline that treats “language = Arabic” as a binary flag without handling mixed direction will mis-chunk the passages customers actually need—pricing notes, PDPL notices, appointment rules—while cleanly indexing the English marketing slogans.

Unicode bidirectional algorithm and mixed Arabic–English strings

Unicode Standard Annex #9 defines the Bidirectional Algorithm that decides how left-to-right and right-to-left characters display in one paragraph. Strong Arabic letters and strong Latin letters receive embedding levels; neutrals such as spaces and punctuation resolve from neighbouring strong types. That is why an Arabic sentence mentioning Zone 66 or a CR number can still render correctly when the paragraph base direction is right. Trouble starts when stored text diverges from what editors see in Doha: Word paste, PDF export, and CMS tools inject invisible directional marks that survive NFC and flip mid-chunk behaviour for retrieval pipelines. Qatari bilingual pages multiply the risk because Latin brands, phone numbers, and English legal abbreviations sit inside Arabic clauses on almost every commercial template.

Logical order versus visual order in extracted text

HTML and most modern text APIs store characters in logical order—the order of typing and of backing store—while the BiDi algorithm produces visual order for display. Scrapers that re-implement “what I see” by reversing Arabic runs without a proper BiDi library create corrupted logical strings. PDF extractors are frequent offenders: Arabic runs reverse, diacritics detach, and digits migrate. Arabic-first PDF RAG toolkits explicitly reshape BiDi and normalise orthography before chunking because reversed letter order destroys embedding neighbourhoods. Qatari firms that index scanned Arabic contracts or municipality circulars without that step build a vector index of gibberish that still looks vaguely Arabic to a casual console print.

Copy-paste from Word and CMS editors often inserts left-to-right embedding (U+202A), right-to-left embedding (U+202B), pop directional formatting (U+202C), and overrides U+202D–U+202E. Those marks are invisible in many editors yet flip mid-chunk behaviour. Mixed Qatari copy also fills paragraphs with neutrals: date slashes, plus signs in +974 numbers, and Latin abbreviations such as QFC and MOCI. Without a correct base direction, plain-text dumps place punctuation on the wrong side even when the live site looked fine.

Invisible control characters that survive NFC

Canonical composition (NFC) does not remove BiDi controls. Production advice from RTL chunking references is to strip U+202A through U+202E (and often neighbouring format controls) before splitting. Zero-width joiners and non-joiners used in Arabic shaping can also confuse tokenisers when duplicated by CMS bugs. A practical ingest rule for .qa content stores two fields: `text_display` preserved for citations, and `text_index` cleaned for embedding. Cleaning must run on queries too. If the index dropped harakat and BiDi marks but the live chatbot sends raw user Arabic with tashkeel, recall collapses for reasons that look like “model quality” and are actually asymmetric preprocessing.

Digits, Latin brands, and neutrals inside Arabic paragraphs

Doha marketing pages routinely embed `West Bay`, `Lusail`, metro station names, and price tags in QAR inside Arabic sentences. Neutrals around those islands resolve differently if the base direction is wrong. When chunkers split on spaces alone, they may isolate `QAR` or a truncated Latin brand on one side of a boundary while the Arabic predicate lands in the next chunk. Dense retrieval then needs both chunks to answer a single question about price or location. Overlap helps, but script-aware sentence boundaries help more. Keep Latin entity spans attached to the Arabic clause that governs them whenever sentence packing allows.

HTML dir=rtl and what crawlers actually see

MDN documents `dir` as a global attribute with values `rtl`, `ltr`, and `auto`. Arabic pages for Qatari audiences should set direction on the root or main content container because directionality is semantic, not only visual. CSS can mirror layout, but scrapers that read text without stylesheets need markup. On bilingual .qa sites the Arabic template should declare `dir="rtl"` and `lang="ar"`; the English twin should declare `dir="ltr"` and `lang="en"`. Entire articles left on `dir="auto"` misclassify when a Latin brand node appears first. Crawlers vary: some keep attributes, others concatenate text nodes and lose table column order from RTL tables common on service comparison pages in Doha. Treating direction as a design polish instead of a content contract is how AI-facing extracts silently degrade after a theme redesign.

Root language and direction per locale template

A .qa WordPress or Next.js site should bind `lang` and `dir` at the document level for each locale, then again on components that intentionally flip—such as an English quotation inside Arabic policy text. Avoid a single HTML shell that stays `dir="ltr"` while Arabic CSS alone mirrors the layout. Extractors that ignore stylesheets will read Arabic bytes with an LTR base direction and misplace punctuation in plain-text exports. Hreflang and locale URLs help humans and search engines choose the right document; they do not repair missing `dir` on the Arabic document itself.

W3C internationalisation guidance prefers markup over CSS-only direction so text remains correct when stylesheets are absent. RAG loaders that keep only textContent create exactly that absence. Tables marked RTL reverse column order visually; naïve CSV dumps scramble headers relative to cells. Forms that mix Arabic labels with English validation messages produce sibling nodes with opposite strong directions. Join them without separators and later plain-text viewers look random. Prefer accessibility-tree or ordered extraction over OCR of page screenshots when a headless browser feeds the chunker.

Inline spans, mirrors, and CMS paste hazards

CMS paste from Microsoft Word into Arabic fields is a leading source of BiDi controls on Gulf sites. Editors in agencies serving Lusail developers often paste bilingual press releases into block editors that preserve unknown characters. Establish a paste filter that strips format controls and normalises Arabic presentation forms. Inline `dir="ltr"` on Latin product names inside RTL paragraphs is legitimate markup; Unicode override characters are not a substitute. Design systems should provide a “Latin island” component so authors do not improvise with invisible marks.

Tables, PDFs, and downloadable Arabic circulars

Qatari corporate sites host PDF circulars, price lists, and Arabic HR policies. HTML tables marked RTL still need cell-level language tags when mixed. PDF pipelines need Arabic-capable extraction: reshape reversed runs, normalise Alef forms, strip tatweel, then sentence-chunk. Indexing only the English PDF twin leaves Arabic-only user questions without evidence. If legal insists on PDF as source of truth, the retrieval corpus must include the Arabic file with a validated extractor, not a screenshot OCR trained primarily on Latin scripts.

Why fixed-size chunking fails Arabic morphology

Arabic writes many function words as attached prefixes and suffixes. Conjunction و, definite article ال, and object pronouns bind to hosts. A character-count chunker that cuts at 500 characters—common in tutorial RAG stacks—slices through those attachments. Research comparing Arabic RAG chunking strategies found sentence-aware chunking scoring higher on average (reported overall around 74.8) than fixed-size, recursive, and semantic chunking on several Arabic datasets. Semantic chunking, despite the name, underperformed in that study on morphologically complex text, suggesting embedding-based merges are not a free upgrade when the embedding space itself is noisy for Arabic orthographic variants.

Fixed-size windows also ignore Arabic punctuation. Splitting only on `.` `?` `!` misses Arabic question mark U+061F and Arabic full stop U+06D4, plus comma U+060C. English-centric sentence segmenters therefore produce monster spans or mid-clause breaks on properly punctuated Arabic from Qatari publishers. Recursive splitters that cascade through Markdown headings help structured blogs, but many Arabic service pages are long plain sections without dense heading trees. Sentence packing with overlap—on the order of 10–20% of chunk size in Arabic RAG toolkit defaults—keeps boundary context without resorting to huge windows that blow token budgets.

Morphology interacts with normalisation policy. If one chunk retains harakat and the next was stripped, embeddings of “the same” sentence diverge. If Alef with hamza and plain Alef both appear across the CMS history of a Doha clinic site, unnormalised indexes fragment recall. The fix is a documented normalisation profile applied uniformly, not ad hoc search-and-replace by interns. Teams that inherited an English help-centre chunker for a new Arabic knowledge base in Qatar should assume the defaults are wrong until proven otherwise on local pages.

Tokenization, embeddings, and AI crawler failure modes

English-centric tokenisers split Arabic into more pieces per meaning unit, raising cost and truncating context windows. Multilingual embedding models help when teams validate them on Arabic samples, yet they still punish inconsistent orthography across a CMS history. Exact identifiers that matter in Qatar—CR numbers, QFC numbers, zone codes—need hybrid retrieval: dense vectors for paraphrase questions plus sparse matching on normalised keys. AI crawlers that summarise .qa sites inherit extraction bugs; reversed Arabic or CSS-only RTL bodies leave English marketing in the evidence set while Arabic policies never enter the index that answer engines read. Clinics in West Bay and developers in Lusail see the failure as “the bot ignores Arabic FAQs” when the root cause sits one layer below the model.

Orthographic variants that split the vector space

Hamza seats on Alef (أ إ آ) versus bare ا, ta marbuta versus ha, yeh versus alef maqsura, and optional tashkeel create lookalike strings. Guides on Arabic normalisation for search recommend NFKC for index keys to fold presentation forms from PDFs, then language-specific folds for Alef and tatweel, while keeping a display copy untouched. Aggressive “search key” profiles maximise recall but must not replace public page copy. Apply the aggressive profile inside the indexer only. Query strings from users in Doha who type without hamza should hit the same key space as carefully typed editorial Arabic.

Teams often cite multilingual-e5-large and BGE-M3 class models for Arabic RAG. Those models still fail when the corpus mixes diacritized quotations with undiacritized web copy without a shared cleaning step. Dense-only stacks also miss decree-like numbers sitting inside Arabic paragraphs. Bilingual sites that hide Arabic in JavaScript tabs may leave English-only HTML for non-rendering fetchers. Qatari firms that want AI citation should ship Arabic primary content as server-rendered HTML with correct `lang` and `dir`, not only inside client widgets.

Hybrid retrieval for Qatari identifiers

Questions about “commercial registration”, clinic licence numbers, or Lusail unit codes often include digits and Latin abbreviations inside Arabic syntax. Pure embedding search underweights exact tokens. Production Arabic RAG write-ups recommend hybrid dense + sparse retrieval so decree-like strings survive. For public websites, that means the HTML should keep identifiers in selectable text, not only in images of certificates. Schema and plain-text legal pages reinforce the same strings for both classical SEO and AI extraction.

Crawler rendering gaps on bilingual .qa templates

Some AI bots execute limited JavaScript; others fetch raw HTML. If the Arabic body is injected after a geo or cookie gate, the crawler may index an English shell. Prefer locale-specific URLs with complete Arabic HTML. Avoid infinite scroll that never places Arabic paragraphs in the initial response. Monitor server logs for GPTBot, ClaudeBot, PerplexityBot, and Google-Extended according to the firm’s crawler policy, and verify that allowed bots receive the same RTL HTML humans see. Chunking quality cannot fix content the fetcher never stored.

Practical fixes for bilingual Qatari websites

Start with publishing standards. Every Arabic template sets `lang="ar"` and `dir="rtl"`. English twin sets `lang="en"` and `dir="ltr"`. Paste filters strip BiDi controls. Authors use components for Latin islands. PDFs intended for retrieval pass through an Arabic-capable extractor before entering any vector store. Public pages that matter for AI answers—pricing, policies, location NAP, service scope for West Bay versus Lusail—exist as HTML, not only as downloadable decks.

For RAG and on-site assistants, implement an ingest function that (1) extracts text with direction metadata when possible, (2) strips U+202A–U+202E and related format noise, (3) applies a frozen normalisation profile, (4) segments on Arabic and Latin sentence punctuation, (5) packs sentences into chunks with overlap under a token or character budget, (6) stores offsets into the cleaned text for citation highlighting, and (7) embeds with a multilingual model validated on Arabic samples from the actual site, not only on open benchmarks. Run the identical normalisation on user queries. Add a sparse index over normalised text for CR-like tokens.

Editorial workflow should sample retrieval monthly. Take ten real Arabic questions from clinic reception, sales, or support in Doha. Check whether the top chunk is the intended paragraph. Failures usually trace to a bad split, missing Arabic HTML, or normalisation drift after a CMS plugin update—not to “the LLM being weak at Arabic” as a vague diagnosis.

Table. Failure modes and fixes for Arabic/RTL retrieval on Qatari bilingual sites

Testing retrieval quality on live .qa content

Build a gold set from real pages: Arabic About text, bilingual contact blocks with West Bay or Lusail addresses, one PDPL-style privacy section, and one PDF circular if the business uses them. For each item write three questions: Arabic only, English only, and mixed (“كم سعر … in Lusail”). Record the expected paragraph. After each chunker or normaliser change, measure hit rate at rank 1 and rank 3. Track token cost per page to catch tokeniser regressions. When hit rate drops after a CMS update, diff extracted text for new BiDi controls before fine-tuning any model.

Human review remains necessary. Native Arabic readers on the content team should judge whether a cited chunk is complete enough to answer without the neighbouring chunk. If answers always need two chunks, increase overlap or raise sentence pack size. If English boilerplate outranks Arabic evidence, boost language-matched chunks or index locales in separate collections with a language router. Qatari firms that operate one shared index for `ar` and `en` without routing force the retriever to compare apples to oranges in the same top-k list.

Document the normalisation profile in the engineering wiki and freeze versions. Unversioned “cleanup scripts” edited during outages are how Doha teams create silent recall regressions. Treat chunking configuration with the same change control as payment gateway settings: small, reviewed, reversible.

For multi-brand groups that publish separate sites for a West Bay clinic and a Lusail project office, keep separate corpora or clear metadata filters so a question about one location cannot retrieve the other brand’s Arabic cancellation rules. Shared vector collections without tenant or locale filters are a frequent cause of confident wrong answers in Gulf deployments. Log the chunk ID returned with each assistant reply so editors can open the exact paragraph and judge completeness within minutes, not after a customer complaint reaches management. When hit rates stall despite clean normalisation, sample token counts per Arabic paragraph: if costs sit far above English twins for similar meaning, the embedding model or tokeniser choice—not the CMS copy—needs replacement on the next release cycle in Qatar. Re-run the gold-set battery after every model upgrade before promoting the assistant to the public .qa site.

Frequently asked questions

Why does Arabic look fine on the website but fail in a chatbot?

Browsers apply the Unicode Bidirectional Algorithm and CSS, so editors see correct RTL layout. Many chatbots ingest stripped text, PDF extracts, or DOM dumps that drop `dir`, reverse runs, or keep invisible BiDi controls. Chunkers tuned on English then cut Arabic morphology incorrectly. The public page can be valid while the retrieval corpus is corrupted. Fix extraction and normalisation before swapping the generator model.

Is HTML dir=rtl enough for AI crawlers?

`dir="rtl"` with `lang="ar"` is necessary semantic markup and helps any consumer that reads attributes, but crawlers that take textContent only still need clean logical strings without override characters. Direction markup does not repair reversed PDF text or fixed-size chunk cuts. Use correct HTML and an ingest pipeline that understands Arabic punctuation and BiDi controls together.

Should Qatari sites translate Arabic to English before indexing?

Modern Arabic RAG practice indexes native Arabic with multilingual embeddings rather than translating first. Translation adds latency, cost, and error—especially on legal and medical copy common to Doha providers. Validate that the chosen embedding model tokenises Arabic efficiently and pair it with normalisation plus hybrid sparse retrieval for identifiers. Keep English pages as a parallel corpus for English questions, not as a replacement for Arabic evidence.

Which chunking strategy works better for Arabic?

A 2025 comparative study on Arabic RAG components reported sentence-aware chunking outperforming fixed-size, recursive, and semantic strategies on average across evaluated datasets. Fixed-size splits are a weak default for morphologically rich Arabic. Qatari teams should segment on Arabic and Latin sentence punctuation, pack sentences under a budget with overlap, and verify on their own West Bay or Lusail site content rather than trusting a single global chunk length.

How do mixed Arabic–English NAP lines affect retrieval?

Addresses that mix zone numbers, Latin district names, and Arabic descriptors create BiDi neutrals and weak types inside paragraphs. If chunk boundaries separate the Arabic verb from the Latin location island, dense retrieval may miss the full fact. Keep NAP in dedicated structured fields and short bilingual blocks, and ensure sentence packing does not isolate `Zone` codes from their Arabic context when those strings appear in prose.

What should be stripped before embedding Arabic from a CMS?

Strip bidirectional formatting characters in the U+202A–U+202E range and decide explicitly whether to remove tashkeel and tatweel for index keys. Fold Alef variants when recall matters more than preserving typed orthography in the key. Never apply irreversible aggressive folds to the only stored copy of public content. Run the same cleaning on queries so users in Qatar who omit hamza still match the index.