
Izeybek
Add a review FollowOverview
-
Founded Date August 25, 1991
-
Sectors Security Guard
-
Posted Jobs 0
-
Viewed 6
Company Description
GitHub – Deepseek-ai/DeepSeek-V3
We present DeepSeek-V3, a strong Mixture-of-Experts (MoE) language model with 671B overall specifications with 37B triggered for each token. To accomplish efficient inference and cost-effective training, DeepSeek-V3 adopts Multi-head Latent Attention (MLA) and DeepSeekMoE architectures, which were completely confirmed in DeepSeek-V2. Furthermore, DeepSeek-V3 pioneers an auxiliary-loss-free technique for load balancing and sets a multi-token forecast training objective for more powerful efficiency. We pre-train DeepSeek-V3 on 14.8 trillion varied and premium tokens, followed by Supervised Fine-Tuning and Reinforcement Learning stages to completely harness its abilities. Comprehensive assessments expose that DeepSeek-V3 outshines other open-source designs and attains efficiency comparable to leading closed-source designs. Despite its excellent performance, DeepSeek-V3 needs only 2.788 M H800 GPU hours for its full training. In addition, its training process is remarkably steady. Throughout the entire training procedure, we did not experience any irrecoverable loss spikes or carry out any rollbacks.
2. Model Summary
Architecture: Innovative Load Balancing Strategy and Training Objective
– On top of the effective architecture of DeepSeek-V2, we leader an auxiliary-loss-free technique for load balancing, which decreases the performance deterioration that develops from encouraging load balancing.
– We investigate a Multi-Token Prediction (MTP) goal and prove it beneficial to model efficiency. It can likewise be utilized for speculative decoding for reasoning velocity.
Pre-Training: Towards Ultimate Training Efficiency
– We create an FP8 combined precision training framework and, for the very first time, validate the expediency and efficiency of FP8 training on an incredibly large-scale model.
– Through co-design of algorithms, frameworks, and hardware, we conquer the communication traffic jam in cross-node MoE training, almost achieving complete computation-communication overlap.
This substantially enhances our training efficiency and reduces the training expenses, enabling us to even more scale up the design size without extra overhead.
– At an economical expense of just 2.664 M H800 GPU hours, we complete the pre-training of DeepSeek-V3 on 14.8 T tokens, producing the presently greatest open-source base model. The subsequent training stages after pre-training need just 0.1 M GPU hours.
Post-Training: Knowledge Distillation from DeepSeek-R1
– We present an innovative approach to distill reasoning abilities from the long-Chain-of-Thought (CoT) design, specifically from among the DeepSeek R1 series designs, into basic LLMs, particularly DeepSeek-V3. Our pipeline elegantly integrates the verification and reflection patterns of R1 into DeepSeek-V3 and notably enhances its thinking performance. Meanwhile, we also preserve a control over the output design and length of DeepSeek-V3.
3. Model Downloads
The total size of DeepSeek-V3 models on Hugging Face is 685B, which consists of 671B of the Main Model weights and 14B of the Multi-Token Prediction (MTP) Module weights. **
To ensure optimum performance and flexibility, we have partnered with open-source communities and hardware vendors to supply multiple methods to run the design locally. For step-by-step assistance, take a look at Section 6: How_to Run_Locally.
For designers aiming to dive deeper, we suggest exploring README_WEIGHTS. md for details on the Main Model weights and the Multi-Token Prediction (MTP) Modules. Please note that MTP assistance is currently under active advancement within the community, and we welcome your contributions and feedback.
4. Evaluation Results
Base Model
Standard Benchmarks
Best outcomes are displayed in bold. Scores with a gap not going beyond 0.3 are considered to be at the very same level. DeepSeek-V3 accomplishes the finest performance on the majority of standards, especially on math and code jobs. For more evaluation information, please examine our paper.
Context Window
Evaluation results on the Needle In A Haystack (NIAH) tests. DeepSeek-V3 performs well across all context window lengths up to 128K.
Chat Model
Standard Benchmarks (Models larger than 67B)
All designs are assessed in a configuration that restricts the output length to 8K. Benchmarks consisting of fewer than 1000 samples are checked numerous times utilizing varying temperature level settings to derive robust last outcomes. DeepSeek-V3 stands as the best-performing open-source design, and also shows competitive efficiency against frontier closed-source designs.
Open Ended Generation Evaluation
English open-ended discussion assessments. For AlpacaEval 2.0, we use the length-controlled win rate as the metric.
5. Chat Website & API Platform
You can chat with DeepSeek-V3 on DeepSeek’s main site: chat.deepseek.com
We also supply OpenAI-Compatible API at DeepSeek Platform: platform.deepseek.com
6. How to Run Locally
DeepSeek-V3 can be deployed locally using the following hardware and open-source community software application:
DeepSeek-Infer Demo: We offer an easy and light-weight demonstration for FP8 and BF16 reasoning.
SGLang: Fully support the DeepSeek-V3 design in both BF16 and FP8 reasoning modes, with Multi-Token Prediction coming soon.
LMDeploy: Enables effective FP8 and BF16 inference for local and cloud deployment.
TensorRT-LLM: Currently supports BF16 inference and INT4/8 quantization, with FP8 assistance coming quickly.
vLLM: Support DeepSeek-V3 model with FP8 and BF16 modes for tensor parallelism and pipeline parallelism.
AMD GPU: Enables running the DeepSeek-V3 model on AMD GPUs via SGLang in both BF16 and FP8 modes.
Huawei Ascend NPU: Supports running DeepSeek-V3 on Huawei Ascend devices.
Since FP8 training is natively adopted in our structure, we only FP8 weights. If you need BF16 weights for experimentation, you can use the offered conversion script to carry out the improvement.
Here is an example of converting FP8 weights to BF16:
Hugging Face’s Transformers has not been directly supported yet. **
6.1 Inference with DeepSeek-Infer Demo (example only)
System Requirements
Note
Linux with Python 3.10 only. Mac and Windows are not supported.
Dependencies:
Model Weights & Demo Code Preparation
First, clone our DeepSeek-V3 GitHub repository:
Navigate to the reasoning folder and set up dependences listed in requirements.txt. Easiest method is to utilize a bundle supervisor like conda or uv to produce a brand-new virtual environment and install the dependencies.
Download the design weights from Hugging Face, and put them into/ path/to/DeepSeek-V 3 folder.
Model Weights Conversion
Convert Hugging Face design weights to a specific format:
Run
Then you can talk with DeepSeek-V3:
Or batch reasoning on a provided file:
6.2 Inference with SGLang (suggested)
SGLang presently supports MLA optimizations, DP Attention, FP8 (W8A8), FP8 KV Cache, and Torch Compile, providing advanced latency and throughput performance among open-source structures.
Notably, SGLang v0.4.1 completely supports running DeepSeek-V3 on both NVIDIA and AMD GPUs, making it an extremely versatile and robust solution.
SGLang also supports multi-node tensor parallelism, allowing you to run this design on several network-connected machines.
Multi-Token Prediction (MTP) remains in development, and development can be tracked in the optimization strategy.
Here are the launch guidelines from the SGLang group: https://github.com/sgl-project/sglang/tree/main/benchmark/deepseek_v3
6.3 Inference with LMDeploy (recommended)
LMDeploy, a versatile and high-performance inference and serving structure tailored for big language designs, now supports DeepSeek-V3. It uses both offline pipeline processing and online deployment capabilities, flawlessly integrating with PyTorch-based workflows.
For thorough detailed directions on running DeepSeek-V3 with LMDeploy, please describe here: InternLM/lmdeploy # 2960
6.4 Inference with TRT-LLM (recommended)
TensorRT-LLM now supports the DeepSeek-V3 model, offering accuracy alternatives such as BF16 and INT4/INT8 weight-only. Support for FP8 is currently in development and will be released quickly. You can access the custom branch of TRTLLM specifically for DeepSeek-V3 support through the following link to experience the brand-new features directly: https://github.com/NVIDIA/TensorRT-LLM/tree/deepseek/examples/deepseek_v3.
6.5 Inference with vLLM (recommended)
vLLM v0.6.6 supports DeepSeek-V3 reasoning for FP8 and BF16 modes on both NVIDIA and AMD GPUs. Aside from standard methods, vLLM provides pipeline parallelism permitting you to run this model on several devices connected by networks. For comprehensive assistance, please refer to the vLLM instructions. Please do not hesitate to follow the improvement strategy too.
6.6 Recommended Inference Functionality with AMD GPUs
In collaboration with the AMD group, we have achieved Day-One assistance for AMD GPUs using SGLang, with complete compatibility for both FP8 and BF16 precision. For detailed guidance, please refer to the SGLang guidelines.
6.7 Recommended Inference Functionality with Huawei Ascend NPUs
The MindIE structure from the Huawei Ascend community has effectively adapted the BF16 variation of DeepSeek-V3. For step-by-step assistance on Ascend NPUs, please follow the guidelines here.
7. License
This code repository is accredited under the MIT License. The usage of DeepSeek-V3 Base/Chat designs goes through the Model License. DeepSeek-V3 series (consisting of Base and Chat) supports commercial use.