Hugging Face in Foundry: open models on Managed Compute | Keryc
Microsoft introduces an operational layer so Hugging Face open models behave like enterprise services. Can you imagine taking a model published in the community and deploying it on managed GPUs, with traceability, automatic patches and the same API you use for commercial models? That’s what Foundry Managed Compute brings.
What is Foundry Managed Compute
Foundry is a platform to build and operate agentic applications. It starts with a clear promise: the widest model selection on any cloud. Microsoft models, OpenAI, Anthropic, Meta, Mistral, DeepSeek and Hugging Face — all reachable from a single endpoint and the same SDKs in Python, C#, JavaScript and Java.
Managed Compute is the third deployment option in Foundry: a GPU-managed PaaS designed for open-source models and your own weights. You choose by model characteristics: parameter count, context length, optimized for latency or throughput. Microsoft handles the GPU layout underneath — whether one or multiple accelerators — and keeps the runtime up to date without you having to redeploy.
You’ll find the same developer experience: a unified endpoint, the same SDKs and the same authentication. All on a single bill.
Runtimes and architecture: what's running under the hood
Foundry pairs models with optimized runtimes and community-built engines that have been tuned for production. The main ones are:
vLLM - the default engine for high-throughput LLMs, optimized for GPU workloads. Thanks to Hugging Face contributions, any Transformers model usually runs on vLLM without extra integrations.
SGLang - designed for structured and multi-modal output, useful when you need JSON, regex or grammars in generation, and for agents that call tools.
TEI (Text Embeddings Inference) - the hot path for embeddings and reranking, with kernels compiled per accelerator family.
llama.cpp - a CPU or small-GPU route with quantized GGUF models; ideal for low-cost deployments or regions without GPUs.
TensorRT-LLM and NIM - NVIDIA optimizations for latency and throughput on specific model families.
hf-serve - Hugging Face’s multi-model server for architectures outside the fast LLM/embedding path, for example vision and audio.
Each model in the Hugging Face Collection in Foundry goes through a publication pipeline: selection, license review, security scanning, packaging into signed images, uploading weights to secure Azure storage and validating API and performance.
Why choosing the right runtime matters
The runtime affects latency, memory consumption and feature compatibility (for example, streaming or output formats). Foundry selects the most suitable engine for each model and provides pre-tuned templates so you don’t have to tweak timeouts, concurrency or probes by hand.
Workflow: from community to enterprise service
The deployment process is clear and repeatable:
You explore the Hugging Face Collection inside the Foundry Model Catalog.
You select a deployment template — optimized for latency or throughput, accelerator family and context.
You configure the instance count to scale throughput.
You deploy from the portal, CLI, SDK or REST.
You call the model from the unified endpoint using the OpenAI-compatible API.
An example template for qwen3-32b shows how options come predefined:
Template
Runtime
Accelerator
Context
qwen3-32b - nvidia-a100 - 40k
vLLM
1 x A100 80 GB
40K
qwen3-32b - nvidia-h100 - 40k
vLLM
1 x H100 80 GB
40K
qwen3-32b - nvidia-2xa100 - 128k
vLLM
2 x A100 80 GB
128K
qwen3-32b - nvidia-2xh100 - 128k
vLLM
2 x H100 80 GB
128K
Each template brings preconfigured settings: runtime tuning, parsers for tool-calls, scoring paths, health probes and concurrency. When you script the deploy you reference the template and Foundry orchestrates the rest.
Here’s a minimal example using the Azure SDK to create a deployment (adapted for readability):
And then you call the deployment via the unified endpoint using the OpenAI-compatible client:
from openai import OpenAI
api_key = client.accounts.list_keys(RESOURCE_GROUP, ACCOUNT_NAME).key1
endpoint = f'https://{ACCOUNT_NAME}.services.ai.azure.com/openai/v1'
openai_client = OpenAI(base_url=endpoint, api_key=api_key)
completion = openai_client.chat.completions.create(
model=deployment.name,
messages=[{'role': 'user', 'content': 'What is the capital of France?'}],
)
print(completion.choices[0].message)
Practical benefits for developers and companies
Full operationalization: you discover, validate licenses, patch CVEs and publish signed runtimes without building the whole pipeline yourself.
Mix-and-match models without friction: you can combine open and frontier models in the same agent without separate integration paths.
Security and governance: weights in SafeTensors, scans for trust_remote_code and Data Zones options for data residency and sovereignty.
Billing and performance options: pay-per-token to get started quickly, provisioned throughput for predictable production and Managed Compute for GPU control and hourly cost management.
Observability and control: metrics in Azure Monitor, agent traceability, continuous evaluations and a prompt optimizer that closes the quality loop.
Technical considerations and limitations
Not all Hugging Face models will be in the Collection. Microsoft selects and curates models based on community signals and enterprise demand.
Models that require executing remote code at load time are remediated or excluded. If your model needs trust_remote_code, the security flow must be reviewed.
Costs and latency depend on the chosen template. Sometimes paying for accelerators hourly and right-sizing GPUs is more efficient than per-token billing, especially for steady or latency-sensitive loads.
This is preview. Accelerator coverage, BYOW and other improvements are on the roadmap.
Final thoughts
If you work with open-weight models and have hit operational friction — licenses, sizing, runtime, patching and enterprise endpoints — Foundry Managed Compute shows up as a layer that lowers that barrier. It’s not just serving weights: it integrates the Hugging Face community with governance, observability and the ergonomics of an enterprise platform. Want to try community-ready models but with enterprise controls? This integration is exactly that bridge.
