When you think of powerful language models, do you picture a single monster trained on the entire universe of data? FlexOlmo—and now FlexMoRE—show another path: models built as pieces you assemble, letting institutions with sensitive data contribute without sharing that data.
What FlexMoRE offers
FlexMoRE was born inside the Danish Foundation Models (DFM) project as a practical adaptation of the FlexOlmo architecture. The core idea of FlexOlmo is clear and technical: instead of passing every token through a monolithic model, a router sends that token only to a subset of experts that specialize. At inference time, only the selected experts run, not the whole model.
The challenge DFM faced was scale: in FlexOlmo each expert is often the size of a full model. That works if you have few experts, but as more groups (hospitals, universities, companies) contribute, the system becomes too big to run on ordinary machines.
FlexMoRE changes one key assumption: not all experts must be the same size. It keeps some experts at full size, but replaces most with compact versions called low-rank adapters. Those adapters approximate what a large expert learned using far fewer parameters. The parameter that defines how much each adapter is reduced is called rank.
In practice, DFM and the team found the ideal rank depends on the task: tasks that need deep reasoning require higher ranks, while factual-knowledge tasks can make do with lower ranks. Using that distinction, FlexMoRE matches or beats a FlexOlmo-style setup while using less than a third of the parameters in its best configuration. That means it can run on consumer-grade hardware and broaden who can participate in open models.
Separating the training of components and joining them at inference lets you share capabilities without sharing data. That opens doors for national initiatives subject to GDPR and similar regulations.
How it works technically (a summary you can picture mentally)
- Base architecture:
Mixture-of-Expertswith arouterthat decides which experts handle each token. - Expert differentiation: some at full size; most as
low-rank adapters(factorizations that reduce large matrices into two or more smaller matrices, lowering parameters and memory). - Choosing
rank: higherrankfor reasoning tasks; lower for factual-knowledge tasks. It's a design choice that balances accuracy versus size and latency. - Distributed training: each institution can train its local expert privately on sensitive data and only share the resulting weights (or an adapter), not raw data. At inference, those weights coexist in a combined assembly.
- Practical result: lower memory and parameter demand, competitive performance, and potential for federated or distributed deployments.
If you like numbers, the key finding is: well-chosen FlexMoRE configurations outperform a baseline with full-size experts while using under 33% of the parameters.
Why this matters for national projects and sensitive data
Do you have data you can’t share because of regulations or intellectual property? Think of hospitals with medical records, legal teams with confidential files, or universities with unique corpora. FlexMoRE lets those entities train useful components without handing over data.
Concrete benefits:
- Compliance: makes it easier for initiatives to align with GDPR and the AI Act by not moving sensitive data.
- Accessibility: lowers hardware requirements to run multimodal modular models, enabling local use in mid-sized organizations.
- Diversification: prevents capability concentration in a few commercial labs, promoting models that reflect underrepresented languages and domains.
Limitations and practical considerations
It’s not a magic bullet. A few things to keep in mind:
- Rank selection is critical: reducing too much can hurt reasoning or factual fidelity. You need to evaluate per task.
- Latency vs memory: although memory is saved, coordinating many experts and routing can add latency. You must optimize the
routerand where weights sit in memory. - Evaluation and governance: a common evaluation framework is necessary to measure each expert’s contribution and ensure no biases or unwanted behaviors creep in.
- Interoperability: open standards for packaging/adapting experts will make it easier for different participants to integrate.
Related projects and where this is headed
The work doesn’t stop at FlexMoRE. Ai2 and other groups are pushing modular complements at different stages:
- EMO: lets experts discover their own specializations during pretraining, instead of fixing categories up front.
- BAR: applies modularity to post-training stages, separating pipelines for instructions, reasoning, tool calls and safety. That way, improving one capability won’t break the others.
Together, these pieces strengthen the idea of separate training and joint inference as an efficient, practical paradigm. It’s a realistic alternative to centralizing massive models.
In the end, if you’re part of an institution with valuable but non-shareable data, FlexMoRE offers a technical and political route to contribute to the model ecosystem without losing control over your data. Don’t you think that’s a tangible step toward democratizing AI?