Tencent Hunyuan Hy4-preview: 770B MoE and 1M Context Guide

Released on August 28, 2026, Tencent open‑sourced Hunyuan Hy4‑preview, a Mixture‑of‑Experts large language model with 770 B total parameters and only 49 B activated parameters per token. It supports a 1 M‑token maximum context window and is explicitly positioned as a productivity‑oriented open‑weight model. The release marks the completion of a 127‑day heavy‑weight retraining cycle. Rather than focusing purely on raw parameter scale, this iteration delivers tangible improvements for code workflows, office automation and long‑document analysis. Arena benchmark data ranks Hy4‑preview around #5 among mainstream models, competing directly against closed‑source heavy‑weight offerings. For open‑source developers, Hy4‑preview provides rare hands‑on access to a 700B‑class MoE model in its pre‑release preview stage.

What Exactly Is Hy4‑preview

MoE‑based Instead of Dense‑weight Architecture

Hy4‑preview adopts a Mixture‑of‑Experts (MoE) design rather than conventional dense transformer architecture.

• Total parameters: 770 B
• Activated parameters per token: 49 B

Dense models activate every weight parameter for every input token, which creates massive compute overhead. MoE routing logic dynamically selects only a subset of expert sub‑networks for each token. This preserves large‑model capacity while drastically lowering inference compute requirements.

Hy4‑preview follows a “large‑total‑size, small‑activation‑size” design philosophy. Out of its massive expert pool, each token routes to top‑8 routed experts plus one persistent shared expert. Only 49 B parameters are activated at runtime. This mechanism brings down inference costs significantly, making the huge‑capacity model feasible for real‑world deployment.

Built for Productivity, Not Pure Conversation‑focused Chat

Official documentation frames Hy4‑preview as a productivity‑first model, not a general‑purpose chat‑optimized checkpoint. Its core strength targets three high‑value practical domains:

1. Code‑centric workloads: Terminal Bench 2.1 reaches 85.4 points, and Deep‑SWE scores hit 64.3 points, showing strong competence for software engineering tasks.
2. Office‑oriented scenarios: Document processing, data analysis and automated workflow generation.
3. Long‑context research workloads: The 1 M‑token context window enables multi‑thousand‑page document parsing, academic paper comprehension and large‑codebase understanding.

This model prioritizes practical task completion over casual conversational fluency.

Deep Dive Into Core Architecture & Capability

Dual‑track MoE Design: 770B Total / 49B Activated

Hy4‑preview combines shared experts and routed experts inside its 78‑layer main transformer stack.

Key architectural logic breakdown:

1. Layer 1 uses a standard dense FFN layer without MoE routing, which stabilizes initial feature extraction.
2. Layers 2 through 78 adopt MoE design. Each layer holds 256 routed experts and one always‑active shared expert.
3. For each incoming token, the router picks top‑8 most relevant routed experts. Together with the fixed shared expert, 9 experts participate in computation.
4. Integrated MTP multi‑token prediction module accelerates generation throughput by forecasting multiple future tokens in one forward pass.
5. Sparse gated attention plus cross‑layer index cache optimizes memory usage under the 1 M long‑context setting.

Benchmark Performance: Competing Against Top‑tier Closed‑source Models

Public benchmark results demonstrate Hy4‑preview’s competitive positioning against leading proprietary and open‑source models.

• Terminal Bench 2.1: 85.4 points, matching the performance of Claude Opus 5 on code‑understanding and code‑generation tasks.
• Deep‑SWE: 64.3 points, showing major improvement inherited and upgraded from earlier Hy‑3 series baseline of 28.0 points. It handles practical software‑engineering assignments including bug repair, code refactoring and test‑case creation.
• Internal expert human evaluation: 163 internal annotators deliver an average score of 2.99 / 4.00, outperforming GLM‑5.3 and Kimi K3.

Arena AI WebDev ranking situates Hy4‑preview around position #5, sitting between GLM‑5.3‑Flash and Grok‑4.6 (High). It ranks alongside well‑known heavy‑weight models such as GPT‑5.6 Sol‑X and Claude Fable 5.

Two critical observations about benchmark metrics:

1. Terminal Bench results confirm it can compete with top closed‑source models for coding‑related work.
2. Benchmark scores reflect synthetic test‑set performance. Real‑world production behaviour may diverge from lab‑based numbers.

Preview‑release Mechanism: Tencent’s Acceleration‑through‑feedback Strategy

Hy4‑preview represents a preview‑stage release rather than a final stable version. Tencent adopts a real‑world feedback loop to drive iterative model improvement.

1. Practical‑loop iteration: The preview weights are exposed to developers. Real‑world usage logs, bug reports and failure cases feed back into model fine‑tuning cycles. This moves model evolution out of isolated lab environments.
2. Internal product‑suite testbeds: Tencent leverages its existing product matrix as validation playgrounds for Hy4‑preview before formal release. Products including WorkBuddy, Hunyuan Assistant and Hunyuan Agent run internal validation covering office workflows, dialogue reasoning, multi‑turn interaction and long‑context stability.

This product‑matrix‑as‑training‑ground approach lets the model encounter real‑world edge cases, instead of relying purely on static benchmark datasets.

Getting Started With Hy4‑preview

The model weights are publicly available on Hugging Face. Developers can pull and load weights with standard transformers tooling. Sample loading snippet

from transformers import AutoModelForCausalLM, AutoTokenizer

model_name = "tencent/Hy4-preview"
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    torch_dtype="bfloat16",
    device_map="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Environment requirements

• Python 3.10+
• PyTorch 2.0+ with BF16 support
• Multi‑GPU deployment required for 770 B MoE weight loading
• Storage footprint: approximately 1.42 TB for BF16‑format weights

Suitable application scenarios

• Code generation & comprehension
• Long‑document analysis, academic material processing
• Enterprise‑grade document workflow and data analysis
• Software‑engineering agent workloads with million‑token context inputs

Teams running multi‑model agent pipelines can streamline mixed‑model traffic via an API gateway such as 4sapi, which unifies routing across MoE models and dense‑weight alternatives.

Realistic Assessment: Boundaries & Risks of the Preview Release

What “preview” actually means

Hy4‑preview is a pre‑release checkpoint, not production‑ready final weights. This brings several caveats:

1. Non‑stable state: Model behaviour, API‑compatible interfaces and routing logic may still change in future iterations.
2. Potential residual bugs: Certain input patterns can trigger unexpected outputs. Quality will improve in subsequent official releases.
3. Incomplete documentation: Some advanced features lack comprehensive developer guidance.

Recommendation for practitioners: Use it for research, validation and internal prototype building. Direct mission‑critical production deployment is not advised at this preview stage.

Gap between benchmark scores and real‑world performance

Impressive benchmark figures do not guarantee equivalent real‑world outcomes.

• Terminal Bench and Deep‑SWE measure performance on standardized test cases. Complex private code repositories, messy business‑logic inputs may produce different results.
• Human evaluation from internal annotators reflects controlled testing conditions. Production‑scale noisy user inputs remain unvalidated.

Developers must conduct domain‑specific validation before trusting Hy4‑preview for business workflows.

Ecosystem challenges for open‑source MoE at this scale

Large‑scale open‑source MoE models bring notable engineering barriers:

1. High hardware barriers: 770 B MoE demands multi‑GPU high‑memory hardware, raising entry barriers for individual hobbyist developers.
2. Inference‑ecosystem maturity: Optimized inference frameworks, quantization strategies and fine‑tuning toolchains for 700B‑class MoE are still evolving.
3. Community adoption uncertainty: Whether the open‑source community will build rich tooling, adapters and derivative fine‑tunes remains to be seen.

Conclusion

Hy4‑preview’s greatest value lies not merely in its 770 B parameter count. In 2026, raw parameter volume is no longer the decisive competitive advantage. Its real significance is bringing a massive‑scale MoE checkpoint into open‑source hands in preview status.

The model demonstrates solid productivity‑oriented capabilities: competitive code‑task benchmark scores, a 1 M‑token ultra‑long context window, plus a feedback‑driven iteration mechanism. For individual developers, it offers rare hands‑on access to a 700B‑class MoE prototype. For enterprises, it provides an early preview for evaluating domestic large‑model applicability. For the broader open‑source community, it serves as a practical real‑world case study for large‑MoE model iteration.

As Hy4‑preview evolves toward its stable formal release, the accumulated engineering experience from Tencent’s model development process may prove as valuable as the model weights themselves. While preview‑stage limitations exist, Hy4‑preview expands the landscape of open‑source high‑capacity MoE alternatives for productivity‑focused AI applications.

Learn more: https://4sapi.com