Meta developed and released the Meta Llama 3 family of large language models (LLMs), a collection of pretrained and instruction-tuned generative text models in 8 and 70B sizes. The Llama 3 instruction-tuned models are optimized for dialogue use cases and outperform many of the available open-source chat models on common industry benchmarks.
Variations
Llama 3 comes in two sizes — 8B and 70B parameters — in pre-trained and instruction-tuned variants.
Input
Models input text only.
Output
Models generate text and code only.
Model Architecture
Llama 3 is an auto-regressive language model that uses an optimized transformer architecture. The tuned versions use supervised fine-tuning (SFT) and reinforcement learning with human feedback (RLHF) to align with human preferences for helpfulness and safety.
Intended Use Cases
Llama 3 is intended for commercial and research use in English. Instruction-tuned models are intended for assistant-like chat, whereas pretrained models can be adapted for a variety of natural language generation tasks.
Key Features
- Model Size: Llama 3 is available in two sizes, 8B and 70B parameters, optimized for dialogue and instruction-tuned use cases.
- Architecture: Auto-regressive language model with an optimized transformer architecture, fine-tuned using supervised learning and reinforcement learning with human feedback (RLHF).
- Input & Output: Processes text as input and generates text and code as output.
- Grouped-Query Attention (GQA): Enhances inference scalability.
- Training Data: Pretrained on 15+ trillion tokens from publicly available data sources.
- Training Context Length: Supports 8k token context length.
- Benchmarks: Outperforms many open-source chat models on industry benchmarks, including high scores on MMLU (82.0), GSM-8K (93.0), and HumanEval (81.7).
- Safety: Extensive red-teaming and adversarial testing to improve model safety, refusal handling, and false prompt refusals.
- Sustainability: Offset 100% of the carbon emissions from the 6.4M GPU hours required to train the 70B model.
- Release Date: Released on April 18, 2024.
- License: Custom commercial license.
- Instruction-Tuned: Optimized for assistant-like chat applications.