Why AI models underperform in Gulf Arabic, and what to do about it

The gap between benchmark Arabic and real Arabic

An open model reports strong Arabic performance. You deploy it against real customer messages and it struggles noticeably. This is not a measurement error — the benchmark and the traffic are different languages in every way that matters computationally.

Arabic benchmarks are overwhelmingly built on Modern Standard Arabic: news text, formal writing, translated corpora. MSA is what people read. It is not what people write to a support channel, dictate to an assistant, or type into a form.

What arrives in production is dialect — Khaleeji in the Gulf, with heavy code-switching into English, inconsistent orthography, Arabizi transliteration, and none of the diacritics that disambiguate meaning. A model tuned on newspapers meets none of that in training.

Diglossia is the underlying problem

Arabic is a diglossic language: a formal register used in writing and broadcast coexists with regional spoken varieties that differ substantially in vocabulary, morphology and syntax. A Gulf speaker and a Maghrebi speaker share MSA but may struggle with each other's dialect.

For a model, this means Arabic is not one language with variation but several related languages with wildly unequal training data. MSA has abundant text. Gulf Arabic has comparatively little in machine-readable form, and what exists is scattered across social platforms with licensing that makes it awkward to use.

This is a data-availability problem before it is a modelling problem, which is also why it is tractable.

Tokenisation quietly taxes you

Arabic is morphologically rich. A single word can encode what English spreads across five: preposition, conjunction, subject, verb and object clitic can all attach to one stem.

Tokenisers trained predominantly on English fragment Arabic words into many subword pieces. The same sentence costs materially more tokens in Arabic than in English — commonly two to three times as many.

That has three consequences people underestimate. Your effective context window is smaller. Your per-request cost is higher for identical content. And more of the model's capacity is spent reassembling words rather than reasoning about meaning.

Where OCR fails

A great deal of Arabic business content arrives as scans: contracts, government forms, invoices, handwritten notes.

Arabic script is cursive and connected, with letter forms that change depending on position in the word. Diacritics are optional and inconsistently present. Documents mix right-to-left Arabic with left-to-right English and Western numerals on the same line, and layout detection frequently gets the reading order wrong even when character recognition succeeds.

Generic OCR tuned on Latin script degrades badly on all of this. If your retrieval pipeline depends on scanned Arabic documents, OCR quality — not the language model — is usually the limiting factor on answer quality, and it is worth measuring separately before blaming the model.

What actually improves performance

In rough order of return on effort:

• Fix ingestion first. If documents are being OCR'd badly, nothing downstream can recover. Measure character and word error rate on a real sample of your own documents before touching the model.
• Retrieve rather than rely on parametric knowledge. Arabic-language facts are more sparsely represented in base models than English ones. Grounding in your own documents sidesteps that entirely.
• Fine-tune on dialect data. This is where the largest quality gain lives. A few thousand well-constructed Khaleeji examples move dialect handling substantially, because the base model has seen so little of it.
• Generate the data you cannot license. Where a Gulf-dialect corpus does not exist in usable form, generating one with privacy and utility checks is frequently faster and cleaner than trying to acquire one.
• Evaluate in the target dialect. An evaluation set in MSA will report improvements you will not observe in production. Build the held-out set from real dialect traffic.

Treat Arabic as a first language, not a translation layer

The common architecture — translate Arabic input to English, process in English, translate back — is appealing and lossy. Dialect nuance, register and named entities degrade at each hop, and errors compound in ways that are hard to diagnose because the failure appears at a different stage from where it originated.

Working natively in Arabic is more work up front and produces a system that behaves consistently. For an organisation whose customers write in Arabic, that consistency is the product.