Falcon-H1-Arabic: new hybrid architecture for Arabic AI | Keryc
Building robust language models for Arabic is a constant iteration race. Today TII introduces Falcon-H1-Arabic, a family of models that brings architectural innovations and a data-and-finetuning pipeline designed for modern Arabic challenges: very long context, dialects, and technical reasoning.
What Falcon-H1-Arabic brings
Falcon-H1-Arabic ships in three sizes (3B, 7B, 34B) and promises notable improvements over the state of the art in its class. What's the main novelty? A hybrid architecture that combines State Space Models and block attention, context windows up to 256K tokens, and a post-training process focused on actually using that extended context.
This isn't just about bigger numbers. For real applications — legal analysis of hundreds of pages, summarizing long medical records, or holding multi-routine conversations with memory — these improvements mean fewer context cuts and more coherent answers. Want fewer annoying context breaks when you feed the model a long document? This is the kind of change that helps.
Hybrid architecture: Mamba (SSM) + Transformer
The technical base is the Falcon-H1 design: inside each block both SSM (here called Mamba) and attention run in parallel. Their representations are concatenated before the block's output projection.
Mamba provides linear scalability for extremely long sequences.
The result: linear-time scalability for long contexts and relational accuracy where attention still matters. For Arabic, with its rich morphology and flexible word order, this combo improves coherence and reasoning in long texts.
Context window and "lost in the middle"
They expanded the window from 32K in the previous Falcon-Arabic to 128K in the 3B and 256K in the 7B and 34B. But to make that more than a label, the post-training addresses the known "lost in the middle" problem: the model learns to use the whole context window effectively, not just the trailing pieces.
Parameters
Context Window
Architecture
Ideal uses
3B
128K
Hybrid
Fast agents, high QPS, light analytics
7B
256K
Hybrid
Production assistants, reasoning, enterprise chat
34B
256K
Hybrid
Long-document analysis, research, critical tasks
Data and pre-training
They rebuilt the data pipeline to reflect Arabic's complexity. Instead of simple heuristic filters, they applied deep linguistic analysis to clean spelling, morphology, diacritics, and syntactic patterns. The goal: a more coherent, stylistically consistent corpus.
Dialectal coverage was a priority. Modern Standard Arabic coexists with dialects like Egyptian, Levantine, Gulf, and Maghrebi. They expanded dialectal sources to avoid bias toward only MSA. They also preserved multilingual strengths by training on about 300 billion tokens in a balanced mix of Arabic, English, and multilingual content, keeping power in code, STEM, and cross-lingual reasoning.
Post-training: SFT and DPO
After pre-training comes a phase focused on instructiveness and preferences:
Supervised Fine-Tuning (SFT): the model is exposed to high-quality Arabic instructions, long-context examples, and structured reasoning tasks. This teaches the model to follow directives and stay coherent across extended sequences.
Direct Preference Optimization (DPO): refines alignment and preference consistency. DPO helps balance reasoning over long context with general linguistic competence, reducing unwanted effects like drift or overuse of context.
During both phases they control catastrophic forgetting with a curriculum that protects basic capabilities while improving long-range behavior.
Important: the architecture alone doesn't guarantee effective use of context. The post-training pipeline is key for the model to take advantage of 128K/256K windows.
Results and benchmarks (technical summary)
On the Open Arabic LLM Leaderboard (OALL) Falcon-H1-Arabic achieves state-of-the-art results at all evaluated scales. Evaluations used vLLM as backend (differences with the leaderboard's Accelerate implementation are usually under a point, with faster runtimes).
Highlights:
3B: ~62% on OALL. In 3LM (STEM) it reaches ~82% native and ~73% synthetic. AraDice around 50% on dialects. Excellent cost-performance for edge and high-demand systems.
7B: 71.7% on OALL, outperforming models in the ~10B class. 3LM: ~92% native and ~85% synthetic. AraDice in the mid-50s; ArabCulture ~80%.
34B: ~75% on OALL, outperforming even 70B systems like Llama-3.3-70B on many metrics. 3LM: ~96% native and ~94% synthetic. AraDice ~53. These percentages aren't just numbers: they mean fewer errors on long answers, better handling of internal references in documents, and less need to split texts for analysis.
Use cases and deployment recommendations
3B: ideal for fast agents, on-device apps, or pipelines with high QPS where latency and cost matter.
7B: a versatile production model: assistants, enterprise chatbots, document understanding, and generation.
34B: pick this for high-risk contexts where precision and long-range reasoning are critical: legal, medical, academic research, and large-scale enterprise automation.
Before deploying to production, run task-specific evaluations and add guardrails: filters, human verification, and bias tests.
Limitations and safe practices
Falcon-H1-Arabic improves many areas, but known limitations remain:
It can reflect biases present in training data.
It can generate inaccurate information or "hallucinate" facts.
Performance on extreme context cases can degrade.
Recommendation: don't use outputs as sole authority in medical, legal, or financial decisions. Evaluate by task, add monitoring, and include human review when the application requires it.
Final reflection
Falcon-H1-Arabic represents a tangible step forward for Arabic processing: a hybrid architecture, usable ultra-long context, and a data pipeline tuned for the language's complexity. If you work with long content, dialects, or technical reasoning tasks in Arabic, these models offer real production options. Interested in testing them for your use case? Think about what context window you need, what budget you have, and how you'll validate results in production.
