Inkling arrives as a reality check: an open multimodal LLM built to understand images, audio and text at the same time, with a 1M-token context window and a design focused on reasoning. Sound like science fiction? It's practical engineering for apps where you need to combine voice, vision and text without breaking everything into separate pieces.
What Inkling is and why it matters
Inkling is a decoder-only multimodal model released by Thinking Machines and available on Hugging Face. Technically it's presented as a Mixture-of-Experts model with about 975 billion parameters total, but only 41 billion active per step thanks to MoE sparsity. In the release literature they describe it as the first open model near the 1T-parameter scale with native support for image, audio and text, trained on 45 trillion multimodal tokens.
Why does this matter for you? Because it opens the door to applications that reason over a document, its associated audio and photos or video without a complex pipeline. Think of an assistant that summarizes recorded meetings, annotates slides and answers questions about numbers shown on screen. That’s already possible with modern tools and models in this family.
Key architecture (broken down but usable)
I'll get to the point: Inkling mixes several ideas you already know, but it puts them together in a practical way.
Decoder-only and autoregressive reasoning
It's a causal autoregressive model, which means it generates text token by token in the classic LLM mode. Ideal for agents, chatbots and conditioned generation in sequential tasks.
Multimodal, simple and efficient
Instead of heavy separate encoders, Inkling uses relatively compact towers for vision and audio that produce unified embeddings. Images are transformed with a hierarchical patchifier via MLPs and audio goes through a discretization into mel spectrogram bins. Both inputs are added to the text representation and fed into the decoder.
Mixture-of-Experts (MoE) with shared experts
- 256 total experts.
- Top-k selection: 6 experts are routed per token plus 2 shared experts always active.
- Result: 975B params total, 41B active per inference. It's a practical way to scale without multiplying weight-read latency.
Relative attention and hybrid attention
Inkling uses relative attention instead of RoPE. Each layer produces a quarter-sized projection R that incorporates the distance between key and query and alters the attention logits.
It also alternates global-attention layers with sliding-window layers in a 5:1 pattern (five sliding-window layers for every one global layer). The final layer is global to enrich the representation.
Short convolution (SConv)
There is a short 1D convolution that reads the local window of W tokens. The idea is to offload local representation from attention and the MoE layers, freeing them to capture longer-range dependencies.
MTP - token-predictive layers for speculative inference
MTP adds layers that predict multiple tokens at once and act as drafters during speculative inference. With use_mtp=True you get acceleration in generation while keeping exactly the same output, with a small VRAM cost for the drafter.
Sizes, formats and practical deployment
- BF16 checkpoint: requires ~2 TB of VRAM.
- NVFP4 checkpoint: requires ~600 GB of VRAM and is the target for Blackwell GPUs.
- Quantized versions in
ggufand 1-bit (by Unsloth) allow huge VRAM reductions, up to 95% less, with reasonable retention of accuracy.
Day-0 launch support: transformers, SGLang, vLLM, llama.cpp. That means you can:
- Try it in the cloud via Inference Providers or Hugging Face router.
- Run quantized locally with
llama.cppand GGUF if you have limited hardware. - Serve in production with
vLLMorSGLangon multinode clusters.
Minimal example with transformers any-to-any pipeline:
from transformers import pipeline
pipe = pipeline("any-to-any", model="thinkingmachines/Inkling")
output = pipe(messages, max_new_tokens=2000, reasoning_effort="medium")
print(output[0]["generated_text"])
And if you want more control: AutoModelForMultimodalLM + AutoProcessor and the reasoning_effort argument to tune cost and depth of "thought."
Cluster and production use
- For fast single-machine deployments with 8 GPUs they recommend
SGLangwith--tp-size 8. - For robust serving use
vllm servewith--tensor-parallel-sizeand possibly SLURM for multi-node setups. - For local experiments or demos:
llama.cppwith GGUF 1-bit and the included app lets you try agents and tool integrations at very low cost.
Practical tip: if you work with very long contexts, control --max-model-len or reserve memory for the KV cache using --mem-fraction-static in SGLang.
Evaluation and reasoning examples
Thinking Machines published a suite called "vibe eval" to assess multimodal reasoning. Quick summary:
- Image reasoning: good OCR ability and chain-of-thought style processing OCR -> characterize -> evaluate -> answer. Open questions consumed more tokens than multiple-choice ones.
- Audio: it transcribes first, then reasons. In GlobeAudio and BigBenchAudio tests it showed robustness for understanding and answering in multiple languages.
- Academic benchmarks: competitive results on high-level math and reasoning tests (AIME, HMMT, IMOAnswerBench) and strong metrics on some agent and coding benchmarks.
What does that mean day-to-day? If you’re building a product that needs to extract tables from an image, cross-check what was said in a recording and generate timestamped summaries, Inkling is a serious candidate to speed up that work.
Post-training and agents
Thinking Machines publishes post-training tools like tinker and examples with OpenEnv for RL agentic environments. They also propose using Inkling as a teacher for distillation, useful if you want to transfer its document understanding to a smaller model deployable at the edge.
There are also ready integrations with Pi agent so you can use Inference Providers endpoints as a backend and test agentic flows with real tools.
Risks, limits and recommendations
- Size and cost: full checkpoints are enormous. For most teams the realistic strategy is hosted inference, NVFP4 on modern GPUs or quantized models with
llama.cpp. - Video: the model accepts a temporal dimension, but there's no exhaustive out-of-the-box evaluation for video. Useful for fine-tuning, but don’t expect miracles immediately.
- Prompting matters: saving tokens and adjusting
reasoning_effortchanges costs and results significantly. For hard reasoning tryhighormaxonly when necessary.
If you want to try something quickly from Maracaibo or a limited laptop, my practical advice: use the Hugging Face router with the :auto or :cheapest suffix to experiment without breaking the bank, and when the project justifies it plan a deployment in vLLM or SGLang.
Final reflection
Inkling isn’t just another big model: it’s a practical bet on unified multimodality and efficient inference via MoE and MTP. If you work on products that mix image, audio and text, this release lowers the technical friction to create experiences closer to how humans process information.
It’s not the final word, but it’s a tool that lets you build the next generation of multimodal applications without fragmenting the whole pipeline.