SkyPilot and Hugging Face: zero-egress for AI workloads

Running training or serving where GPUs are free without moving your data between clouds sounds like magic — but now it’s practical. Hugging Face and SkyPilot integrated the store: hf option to mount buckets and Hub repos with an hf:// URL and your HF_TOKEN. The result? Your models and datasets stay on the Hub, and SkyPilot runs the compute on whatever cloud or cluster has capacity, reading directly from the same bucket without charging egress for reads.

What this integration brings

• Mount any Hugging Face Bucket (read-write) or a model/dataset/Space repo (read-only) with a single hf:// URL and your HF_TOKEN.

• Run on any GPU across 20+ clouds, Kubernetes, Slurm, or on-prem: SkyPilot chooses where to run the job based on available capacity.

• No egress charges for reading. Hugging Face Storage doesn’t charge egress or CDN fees for reads, so your GPUs can load models and datasets without being billed for those bytes.

• Storage uses Xet and chunk-level deduplication. Incremental checkpoints and model variants transfer only the chunks that changed.

• Collaborative and open source: support came via joint work, including fixes to hf-mount for unprivileged containers.

How it works technically (practical summary)

• The hf:// scheme appears in the file_mounts section of a SkyPilot spec. You can use mode: MOUNT or mode: COPY.

• MOUNT uses hf-mount (FUSE). The repo or bucket appears as a local path. Reads are lazy: when your code does read(), only the bytes actually touched are downloaded.

• MOUNT_CACHED and MOUNT behave the same for store: hf: there’s on-disk cache, so repeated reads are local.

• COPY downloads everything up front using huggingface_hub, useful if you need the full copy before starting.

• Authentication: put HF_TOKEN in the environment and pass --secret HF_TOKEN to the run. The same token works on AWS, GCP, Azure, Nebius, Lambda, or your Kubernetes — no per-cloud keys needed.

Why it matters (the problem it solves)

Before, feeding GPUs spread across providers meant keeping copies of data in each cloud or paying egress every time a GPU in another network pulled an object. That makes teams "pin" their runs to the provider that holds the data, leaving capacity idle elsewhere.

With Hugging Face Storage and SkyPilot you read from the same bucket without paying egress for reads. When is that especially useful?

• When you iterate over datasets for many epochs.
• When you deploy inference serving the same model across multiple clouds.
• When you have frequent, large checkpoints to share.

Keep in mind that writing to the bucket still incurs egress from your compute cloud (outbound to the external service), so reads are the big win here.

Xet and deduplication: a detail that saves a lot

Hugging Face Buckets use Xet, which splits files into content chunks (~64 KB) and stores each chunk only once. Practical benefits?

• Incremental checkpoints upload only the modified chunks.
• Variants of the same model share common chunks, reducing duplicate transfers.
• When appending to Parquet tables you only transfer the new rows. In one test, adding 10k rows to a 100k-row table moved 10 MB instead of 106 MB.

Re-uploading a blob already in the bucket was much faster: first upload 24 s, re-upload 8 s in the test, because only hashes and new chunks were transferred.

Practical benchmark they ran (technical summary)

They fine-tuned Qwen/Qwen3.5-4B on the HuggingFaceH4/Multilingual-Thinking dataset with TRL SFTTrainer, mounting the model in read mode and writing checkpoints to a Bucket.

• Training started ready in ~30 s because reads are lazy and from_pretrained only touches what’s necessary (initial read up to ~500 MB/s).

• Checkpoints of 8.43 GB written to the bucket at sustained speeds:

  • AWS (us-east-2), GPU L40S: ~168 MB/s
  • GCP (us-central1), GPU L4: ~123 MB/s
  • Lambda (us-west-3), H100: ~112 MB/s

The same YAML ran on AWS, GCP and Lambda by changing only --infra, and SkyPilot placed each job where GPUs were free — all reading/writing to the same bucket.

Example spec (simplified):

resources:
  accelerators: H100:1
file_mounts:
  /base-model:
    source: hf://Qwen/Qwen3.5-4B
    store: hf
    mode: MOUNT
  /checkpoints:
    source: hf://buckets/my-org/qwen-sft
    store: hf
    mode: MOUNT
run: |
  python train.py --model /base-model --output_dir /checkpoints

Advantages and considerations for engineering teams

• Advantages:

  • You avoid egress for reads, cutting costs in distributed workloads.
  • No need to migrate data if you already use the Hub: most teams keep models and datasets there.
  • Automatic deduplication with Xet saves bandwidth and storage.
  • Multi-cloud and on-prem integration with a single spec.

• Considerations:

  • Writes from the compute cloud still incur that cloud’s egress.
  • MOUNT requires a base image with glibc >= 2.34 and /dev/fuse available.
  • Fixes were applied for unprivileged containers, but check your cluster’s permissions and limits.

How to get started quickly

1. Install SkyPilot with the Hugging Face plugin:

pip install "skypilot[huggingface]"

2. Authenticate your Hugging Face CLI:

hf auth login
# or: export HF_TOKEN=<your-token>

3. Add an hf:// mount to your spec and launch the job. Pass the token with --secret HF_TOKEN so SkyPilot can use it on the destination cloud.

Deployment notes: MOUNT needs glibc 2.34+ and /dev/fuse. COPY doesn’t have that requirement because it downloads everything up front.

Final reflection

This integration removes one of the most tangible frictions of running AI at multi-cloud scale: the cost and complexity of moving data. You no longer have to choose between keeping redundant copies or paying egress every time you spin up GPUs on another provider. The result for your team? More flexibility to use available capacity and less time wasted on transfers and surprise bills. Sound useful to you?

Original source

https://huggingface.co/blog/skypilot-hf-storage