Provider: BigScience Workshop (Hugging Face & collaborators)
License: BigScience RAIL License (Responsible AI License)
Access: Open weights (available for commercial and research use with conditions)
Architecture: Transformer decoder
Parameters: 176 billion
Languages: 46 natural languages, 13 programming languages
🔍 Overview
BLOOM (BigScience Large Open-science Open-access Multilingual Language Model) is a 176B-parameter autoregressive language model trained on a large, multilingual corpus. It is the result of a one-year-long collaborative research workshop involving over 1,000 researchers from around the world.
It was designed to be transparent, inclusive, and responsible, offering a multilingual alternative to closed-source LLMs.
Key features:
- Multilingual: Supports text generation in 46 languages and 13 programming languages
- Transparent & Reproducible: Training details and datasets are publicly documented
- Open Science Milestone: First multilingual LLM released at this scale with full transparency
⚙️ Technical Details
- Architecture: Transformer-based decoder-only model
- Parameters: 176B
- Context Length: 2,048 tokens
- Training Corpus: ROOTS dataset (341 billion tokens)
- Hardware: Trained on 384 A100 80GB GPUs for 117 days
🚀 Deployment
- Hugging Face Repo: bigscience/bloom
- Inference Tools: Hugging Face Transformers, text-generation-inference, 🤗 Accelerate
- Variants: BLOOMZ (instruction-tuned), mBLOOM (multilingual tuning), smaller BLOOM checkpoints
- Fine-Tuning: Supports standard and parameter-efficient tuning methods (LoRA, QLoRA)