OlmoEarth is a platform that takes Earth-observation models from experiment to production at continental and planetary scales. Sounds like science fiction? It's not: it's already used for fire-risk maps, deforestation monitoring and food security, and it's designed for teams that can't maintain a large ML infrastructure on their own.
Qué es la OlmoEarth Platform
The OlmoEarth models are foundational Earth-observation models, pretrained on about 10 terabytes of multimodal satellite data. They are open, useful models, but the real value comes when you integrate them into an operational workflow: labeling, fine-tuning, evaluation and—above all—mass inference.
Ai2 leveraged experience running platforms like Skylight and EarthRanger to design infrastructure that answers real problems: cost, latency, geographic consistency and result verification. In short, having a good model isn't enough; you need to make it work every day for partners with limited resources.
Arquitectura y diseño de ejecución
OlmoEarth splits each job into three stages, matched to the most efficient hardware type:
- Data acquisition and preprocessing (CPU, high I/O): find, reproject, align and normalize images; write them in formats optimized for fast loading.
- Inference (GPU): run the model forward pass and dump minimally processed outputs to storage.
- Postprocessing (CPU): stitch windows into coherent maps, apply masks and export to
Zarr,GeoTIFForGeoJSON.
This separation avoids letting GPUs do work that CPUs handle better. To keep GPUs busy, multiprocessing continuously feeds each GPU and results flow to blob storage.
OlmoEarth Run: particiones y ventanas
The execution engine, OlmoEarth Run, divides a region into partitions sized for individual instances and then into windows that can be processed independently. Why windows? Because each window is a single model pass; if one piece fails, the rest keeps running.
In a real example, a continental run to generate a fire-risk map for North America used up to 19,600 CPUs and 994 GPUs in parallel, with 168 GB/s of throughput. That turned about 4,737 hours of serial compute into 30.5 wall-clock hours—a 155× speedup.
But watch out: fan-out has limits. Cloud quotas, budget and model size all constrain how much parallelism you can deploy. Choosing resolution, model size and cache policies is a product-and-cost decision.
Índices, datos y acceso eficiente
One classic geospatial bottleneck is simply getting and preparing imagery. OlmoEarth solves this with its own metadata index that updates when new scenes appear. For datasets in AWS Open Data, they receive SNS notifications; when there's no stream, they poll upstream indexes every few minutes.
The index stores metadata and pointers to all pixel locations. At runtime, the platform performs windowed reads over cloud-optimized formats like COG or Zarr, bringing only the bytes you need instead of downloading whole scenes. That reduces I/O and speeds up preprocessing.
Good practices for publishing Earth-observation data: notify new imagery, store in main clouds without odd limits, and use cloud-optimized formats that allow ranged reads.
The index also powers annotation tools: you can serve mosaics of Sentinel-1, Sentinel-2, Landsat or NISAR through the same windowed-read system without a separate ingestion pipeline.
Tolerancia a fallos y orquestación
At planetary scale, failures are the norm, not the exception. OlmoEarth treats each task as reentrant and idempotent: each task spins up a VM with a Docker runner that executes, returns results and shuts down. If something fails, the system retries, falls back to alternate providers if available, or marks fatal errors when appropriate.
There's also monitoring that detects stuck runners and restarts tasks. This, along with retries and fallback, turns cloud fragility into operational resilience.
Tradeoffs: precisión, costo y velocidad
Every large distributed system has design choices that affect results and budget:
- Resolution vs data volume: more detail = more data = higher cost.
- Model size vs GPU time: larger models are often more accurate but use more GPU per window.
- Scene cache vs storage: keeping copies speeds repeated runs but consumes space.
Tuning these parameters depends on your use case and budget. Do you need a daily fire map or a detailed quarterly assessment for research? That answer defines the configuration.
Desafíos técnicos clave y soluciones aplicadas
- Heterogeneous sources: different providers, projections and channels require robust reprojection and alignment.
- I/O-dominated runtime: the platform performs windowed reads and optimizes prep so GPUs don't sit idle waiting for data.
- Public API limits: to avoid hammering external STAC APIs, they maintain a local index updated by streams or polling.
- Seamless assembly: partitions overlap and when the final raster is built overlaps are reconciled to avoid seams.
Technically, solutions like ranged reads in COG/Zarr, in-house indexing, idempotent runners and automated monitoring are what lower production risk.
Hoja de ruta y mejoras futuras
Among the priorities they've published are:
- Automated runs: schedule inferences or trigger them when new imagery arrives.
- Change detection and alerts: turn rasters into actionable alerts for users.
- Agentic tools: lower the technical barrier for users without advanced ML teams.
- Faster models: research architectures that reduce GPU time per window.
- New modalities: integrate weather data (ERA-5) and additional sensors.
- Scale embeddings: precompute global embeddings to speed many use cases and reduce the need to run the full model.
- Run anywhere: multi-cloud support and deployments in partner accounts.
Reflexión final
OlmoEarth isn't just a model: it's systems engineering designed to turn geospatial models into operational products that organizations with limited resources can use. Do you work in conservation, disaster response or food security? This helps close the gap between research and action.
If you're starting with geospatial projects, take two practical notes: index your metadata and think about I/O before GPU. Most time and money in these systems is spent moving and preparing data, not on inference itself.
Original source
https://huggingface.co/blog/allenai/olmoearth-infrastructure