The Meta Llama 3.1 collection of multilingual large language models (LLMs) is a collection of pretrained and instruction-tuned generative models available in 8B, 70B, and 405B sizes, with the Llama 3.1-450B being the largest in the series. These models are optimized for multilingual dialogue use cases and outperform many available open-source and closed chat models on common industry benchmarks.
Llama 3.1-450B is intended for commercial and research use in multiple languages, especially for assistant-like chat and various natural language generation tasks. The model also supports the ability to leverage its outputs to improve other models, including synthetic data generation and distillation.
Model Architecture
Llama 3.1 is an auto-regressive language model using an optimized transformer architecture. The tuned versions employ supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.
Training Data
Llama 3.1 was pretrained on approximately 15 trillion tokens of data from publicly available sources, with fine-tuning data including publicly available instruction datasets and over 25 million synthetically generated examples. The training data has a knowledge cutoff of December 2023.
Supported Languages
The model supports 8 languages: English, German, French, Italian, Portuguese, Hindi, Spanish, and Thai.
Key Features
- Auto-regressive model: Optimized transformer architecture.
- 405B parameters: Largest model size in the Llama 3.1 series.
- Multilingual support: Handles text and code in eight languages.
- Instruction-tuned: Aligned with human preferences for helpfulness and safety.
- Tool integrations: Supports multiple formats and longer context windows.
- Synthetic data generation: Useful for various research and commercial purposes.
- Extensive training data: Trained on 15 trillion tokens, with fine-tuning on 25M synthetic examples.