PRX built its corpus by mixing public and internal data, rewriting descriptions with a VLM and turning the result into the stream that fed PRX's pretraining model. Why so much emphasis on quantity and consistency instead of polishing every single image? Because at this stage the model learns visual coverage and variety, not a refined style.
Summary of the pipeline
At a high level the recipe was simple but powerful: gather lots of sources, standardize descriptions with a visual language model (VLM), index and explore with Lance for curation, and finally transform everything to MDS for distributed training.
The goal of pretraining is breadth: you want the model to learn objects, composition, lighting and the huge diversity of real images. Filter too much for aesthetics here and you impoverish the distribution and lose concepts that are hard to recover later. The "pretty" stuff comes afterwards, in fine-tuning and alignment on small, carefully curated sets.
Data and formats: why use Lance to build and MDS to train
Lance is a columnar format designed for exploring and doing feature engineering at scale: predicate pushdown, scalar indexes and vector search. Ideal for tables of hundreds of millions or billions of rows.
MDS (Mosaic Data Shards) is rigid but efficient for streaming in distributed training: ~128 MB shards, weighted mixing of streams, resumable and deterministic.
The strategy was to use Lance to build, filter and explore, then convert the finalized data to MDS for the trainer. That combo gives flexibility during curation and low operational cost during training.
Text and latents: precompute vs compute on-the-fly
They used to precompute text latents with T5Gemma and store them in MDS as bytes. For PRX 7B they switched to Qwen3-VL and decided to compute latents inside the training loop. Why?
Throughput cost: running the encoder inline cost only about 3–4% of throughput (roughly one extra day on a 30-day run) because the denoiser dominates compute at the 7B scale.
Benefits: much smaller shards (they can fit on a shared SSD in the SLURM cluster) and the freedom to swap the text encoder without rewriting terabytes.
Practical lesson: if the text encoder is light compared to the vision model, computing latents in real time pays off for flexibility and saves I/O and storage.
Images and compression: JPEG quality 92 validated
They stored all images as JPEG at quality 92 after measuring the impact. The reason: most sources are already JPEG-compressed; PNG would be 3–10x larger with no perceptual gain.
Results over 100 images (higher PSNR and lower LPIPS are better):
Resolución
PSNR 1x (dB)
LPIPS 1x
PSNR 10x (dB)
LPIPS 10x
1–2 MP
48.7
0.004
45.4
0.008
0.25–0.5 MP
45.1
0.005
42.2
0.010
They also trained two identical PRX models (1024px), one with PNG and one with JPEG 92. In metrics and appearance the generations were practically indistinguishable:
Modelo entrenado en
Detection rate
Mean est. JPEG quality
Median est. JPEG quality
JPEG (92)
12.0%
39.6
34.0
PNG
10.8%
42.1
45.0
Conclusion: for large-scale pretraining, high-quality JPEG is enough. For artifact-sensitive tasks (e.g., shadow models for Photoroom) they still use PNG.
Captioning: why long captions matter and how they generated them
They found that long, faithful captions noticeably improve generations. A small diffusion model experiment showed that using captions generated by Qwen2.5-VL-7B (long and detailed) reduced FID, CMMD and DINO-MMD compared to short captions.
Benchmark at ~100k steps comparing captioners:
Captioner
FID ↓
CMMD ↓
DINO-MMD ↓
Qwen2.5-VL-7B-Captioner-Relaxed
13.95
0.306
0.185
Qwen3-VL-8B (choice)
10.98
0.351
0.182
Qwen3.5-9B
10.51
0.278
0.162
Qwen2.5-VL-7B (reference)
15.86
0.393
0.285
Gemini 1.5 Flash (reference)
13.46
0.316
0.234
Pipeline tuning and throughput mattered: Qwen3-VL-8B ran ~20 img/s per H200 under vLLM and was chosen for quality, speed and stable support. Qwen3.5-9B gave better metrics but only ~6.5 img/s and required unstable builds at the time.
The system prompt for the captioner asked for a 100–200 word paragraph, descriptive, no interpretations or metaphors, including position, materials, colors, literal transcription of any visible text and marked overlays.
Indexing and exploration with Lance
Putting everything in Lance enabled text search and nearest-neighbor with embeddings, and made it possible to build a UI to browse the corpus. That revealed many screenshots, infographics and poor captions.
Fragmentation was a critical point. Initially they wrote 100k rows per fragment and ended up with thousands of small fragments that slowed queries. Compacting to ~1M rows per fragment (for tables with hundreds of millions of rows) kept the fragment count manageable and sped up queries.
Practical rule: keep the number of fragments low (order of hundreds for billion-row tables) and compact periodically. Lance supports distributed compaction.
Filtering, deduplication and skip-lists
Text filtering: they classified captions with Qwen3-8B in text mode to tag samples as visual, text, or nsfw. It's fast (~200 captions/s per GPU) and cheap because it leverages existing work.
Skip-lists: instead of rewriting shards, they added sidecar files with indexes to skip. The loader merges skip-lists and omits those items during training. Flexible for opt-outs or trying ablations without duplicating data.
Practical limit: if the percentage of skipped samples grows a lot (estimated ~10%), it's worth rewriting the dataset.
Deduplication: they used DCT-based perceptual hashes (thumbnail, 2D DCT, threshold low-frequency coefficients) and considered two fingerprints identical if Hamming distance was zero. This catches re-encodes and resizes; concept-level balancing requires deduplication in embedding space.
Aggregate results: dedup removed a few percent, text filtering a few more percent, and NSFW a small fraction.
Bucketing and streaming for diffusion
Buckets by tier: 512, 1024, 2048, 4096 px. An image goes into the largest tier without upscaling >4/3 (512 floor ≈ 384^2). Anything below the floor is discarded.
Buckets by aspect ratio: AR in [0.5, 2.0], 13 shapes per tier (patch-aligned, multiples of 16) to keep a constant token budget per image.
Resize with Lanczos and encode JPEG Q92. MDS organized in a tree by resolution and AR for efficient streaming.
Mosaic Streaming provides: deterministic, resumable reordering, weighted mixing of streams and mid-epoch elasticity without reprocessing data.
Practical recommendations and lessons
Prioritize coverage and diversity in pretraining; save polish for fine-tuning.
Use Lance to explore and MDS to train if you need robust streaming at scale.
Computing text latents on-the-fly is a good option if the encoder doesn't dominate compute; it gives you flexibility to swap text models.
Keep few, large fragments in Lance; compact periodically.
Prefer long, faithful captions: they turn visual noise into controllable attributes.
Implement skip-lists to exclude examples without rewriting shards; rewrite only if skips grow a lot.
The piece that documents PRX is technical, practical and full of details you can apply to any large image project. If you're building a big corpus, these decisions are exactly the ones that save you weeks of I/O, storage and engineering headaches.
