Can a GPU cluster produce more without changing its hardware? An experiment by Dharma AI shows that it can. The difference didn’t come from installing more powerful accelerators or running different workloads, but from changing the order in which allocation decisions are made.
When a constraint-aware allocator was compared with a FIFO scheduler, utilization improved by up to 33 percentage points, while priority-weighted output value increased by up to 105.1%. The tests used the same hardware and the same workloads.
The Problem Isn’t Filling the GPUs, but Deciding Who Uses Them
Saying “keep the GPUs busy” sounds simple, but it isn’t an instruction a system can execute directly. The scheduler must decide which GPU runs each job, when it runs, and for how long.
The difficulty increases because four types of workloads compete for the same resource:
- Training, which usually needs continuous blocks of GPUs for hours or days.
- Real-time inference, whose demand changes with traffic.
- Batch inference, which also requires relatively stable allocations.
- Quantization, which can consume GPU hours while other jobs wait.
Training, batch inference, and quantization need a rigid allocation model: a contiguous block of GPUs that cannot be interrupted. Real-time inference works in the opposite way. It can grow during a traffic spike and shrink overnight.
Do you see the conflict? If a system permanently reserves maximum capacity for inference, it wastes resources during low-demand hours. If it assigns everything according to arrival order, it may leave priority jobs or tasks requiring a specific number of GPUs without enough space.
What Changes Compared with FIFO
A FIFO scheduler places jobs according to when they arrive. Real-time inference receives a fixed reservation, while the remaining tasks wait their turn, without considering priority or the shape future allocations will need to take.
That approach can work when there is plenty of capacity. But when workloads compete, order stops being a detail and becomes a capacity decision.
A job that arrives first may occupy a block that another, higher-priority job will need later. The result is paradoxical: some GPUs remain reserved or unused while valuable jobs cannot find a compatible configuration.
The new allocator analyzes all pending jobs across a 24-hour horizon. This lets it decide what should run now without closing off options for tasks that still need to enter the plan.
Utilization doesn’t depend only on how many GPUs exist. It also depends on how the decisions determining who can use them are ordered.
Up to 33 More Percentage Points of Utilization
Dharma AI evaluated the system across seven scenarios. In five tests designed to create high competition, utilization rose from a range of 52% to 85% with FIFO to one of 72% to 88% with the new allocator.
The most striking case was the training-intensive scenario, with eight GPUs and 16 jobs. Utilization rose from 53.6% to 87%, an increase of 33.4 percentage points. Priority-weighted value increased by 105.1%, more than doubling.
These were the reported results:
| Scenario | Utilization | Value | Value increase | Latency |
|---|---|---|---|---|
| Mixed control | 51.6% → 72.4% | 7,093 → 10,980 | +54.8% | 1 ms |
| Real-time competition | 75.0% → 80.2% | 3,233 → 4,029 | +24.6% | 1 ms |
| Training-intensive | 53.6% → 87.0% | 8,553 → 17,545 | +105.1% | 2 ms |
| Large mixed | 76.8% → 82.7% | 13,977 → 20,101 | +43.8% | 2 ms |
| Oversubscribed | 85.4% → 87.5% | 4,311 → 5,760 | +33.6% | 1 ms |
| Scale test | 44.9% → 44.9% | 44,233 → 51,248 | +15.9% | 15 ms |
| Uniform priority | 76.8% → 87.5% | 25,219 → 31,052 | +23.1% | 2 ms |
Utilization improved in six of the seven scenarios and stayed exactly the same in the scale test. Value, however, increased in every scenario.
More Utilization Doesn’t Always Mean More Value
The test with 64 GPUs and 30 jobs offers an important lesson. FIFO and the allocator ended with the same utilization, 44.9%, and completed the same number of jobs: 27 out of 30.
However, the allocator produced 15.9% more priority-weighted value. In other words, monitoring dashboards showed identical occupancy, but the cluster delivered different results.
This matters because an occupied GPU isn’t necessarily producing the most valuable work. A system can maximize occupancy and still allocate its resources poorly.
The uniform-priority scenario is also revealing. Even when all jobs had the same priority, utilization rose from 76.8% to 87.5% and value increased by 23.1%. The improvement doesn’t depend only on sorting by importance. It also comes from planning allocations over time.
How the Allocator Works
The system incorporates five constraints to ensure that allocations are valid:
- Each GPU can serve at most one job at any given moment.
- Each task must respect its demand range and preserve jobs that are already running.
- Batch workloads occupy contiguous GPU blocks, with sizes based on powers of two.
- Real-time tasks have a limit on how many GPUs they can change between two consecutive time points.
- A job that has already started cannot be interrupted.
The objective combines two elements. Assigning a GPU to a batch task generates a reward based on its priority and the remaining time. Failing to meet a real-time task’s demand generates a penalty.
The penalty for unmet demand is five to ten times greater than the reward for assigning a GPU to a batch job with equivalent priority. This difference protects latency commitments without relying on a static reservation throughout the day.
During a period of low demand, a GPU can temporarily move to a batch task. If traffic grows again, the model accounts for the cost of reclaiming that capacity and limits how much an application can change between time points.
A 24-Hour Horizon, but Only One Committed Decision
The scheduler calculates a plan for the next 24 hours, although it confirms only the allocation for the current moment. It then runs the process again every 30 or 60 minutes with updated information.
This design avoids a problem known as the end-of-the-world effect. An optimizer that looks only toward the end of its horizon can make decisions that seem good but cause a disaster immediately after that cutoff.
Here, the future plan helps make a better decision now, rather than blocking every later decision. Each new run preserves what is actually working and reorganizes the rest according to current demand.
The architecture offers two modes:
- Fast mode: runs the heuristic and returns an allocation on the critical path.
- Full mode: uses that allocation as a starting point so a formal model can try to improve it, making it ideal for periodic reviews.
The heuristic takes between 1 and 2 milliseconds in the competitive scenarios. In the test with 64 GPUs and 30 jobs, it took 15 milliseconds—a low enough latency to run whenever a new request arrives.
Forecasts Are Also Part of the Scheduler
No allocator can plan well if it doesn’t know how long each job will run or how much demand real-time inference will have. This information isn’t certain: it’s predictive.
That’s why Dharma AI uses specialized estimators. Training, for example, depends on factors such as full model fine-tuning, LoRA, SFT, DPO, RLHF, RLVR, or CPT. Model size alone isn’t enough to estimate duration or the number of GPUs required accurately.
The training forecaster uses 22 features and identifies ten specific training variants. Quantization has a separate model based on the number of parameters and the algorithm used, such as bitsandbytes, AWQ, or GPTQ.
For real-time inference, the system doesn’t estimate each job in isolation. It builds a weekly demand profile from hourly traffic and converts it into GPU requirements. This demand-based profile replaces the fixed reservation based on peak demand.
What happens if the forecast is wrong? The answer is frequent reoptimization. The system doesn’t try to get the entire day right in a single run. It updates its decisions before errors accumulate.
The Lesson for Any Infrastructure Team
This experiment shows that optimizing GPUs doesn’t necessarily mean buying more GPUs. Sometimes the resource is already installed but is being wasted through rigid reservations, fragmented allocations, or decisions made without considering the full picture.
The idea applies beyond AI clusters. When multiple workloads with different priorities and shapes compete for a limited resource, the order of decisions can have as much impact as the total amount of available capacity.
In this case, the hardware didn’t change. The workloads didn’t change either. What changed was the discipline used to assign each GPU, taking into account the present, the near future, priority, and the cluster’s physical constraints.
That turns orchestration into more than an operational task. It becomes a strategic lever: it can transform reserved or poorly distributed capacity into more completed jobs and more value produced.