The Stack Code Dataset

The Stack
Code Dataset

The Stack is a large-scale open-source code dataset built by the BigCode project, containing 546 million code files, totaling 6TB of data, covering 30+ programming languages. All source code comes from permissive repositories on GitHub and serves as the foundational dataset for training code generation models like StarCoder.

Get the Dataset Now

546 million files 30+ programming languages 6TB of data GitHub open-source code

📁

546M

Number of Code Files

💻

30+

Programming Languages

💾

6TB

Total Data

🐙

GitHub

Data Source

Dataset Highlights

Large-scale, multilingual, high-quality open-source code corpus

📚

Massive Code Corpus

Contains 546 million loosely licensed source code files, making it one of the largest open-source code datasets available, providing ample data foundation for code model training.

🌐

Language Diversity

Covers 30+ mainstream programming languages, including Python, JavaScript, Java, C++, Go, Rust, TypeScript, etc., meeting the needs of multilingual code modeling.

📜

License Filtering

All code has undergone license verification, containing only source code with loose licenses such as Apache, MIT, BSD, ensuring compliant use.

🔄

Approximate Deduplication

Utilizes advanced approximate deduplication algorithms to remove redundant and duplicate code patterns, improving dataset quality and training efficiency.

🛡️

PII Removal

Detects and removes personally identifiable information (PII), including email addresses, IP addresses, and other sensitive information, protecting privacy and security.

🚪

Opt-Out Support

Developers can use the Am I In The Stack tool to check and opt-out, fully respecting the wishes of code authors.

Applicable Scenarios

From code generation to vulnerability detection, empowering various AI programming applications

🤖

Code Generation

Train models like StarCoder to achieve code completion and automatic generation features

🔍

Code Search

Build semantic code search and retrieval systems to enhance development efficiency

🐛

Defect Detection

Train models to identify bugs and security vulnerabilities in code, improving code quality

🔀

Code Translation

Develop a code translation system across programming languages to achieve automatic conversion between languages

code programming GitHub open-source BigCode

Quick Start with The Stack

Quickly access The Stack code dataset via API

Python

import requests
url = "https://api.acedata.cloud/datasets/the-stack"
headers = {
    "Authorization": "Bearer YOUR_API_TOKEN",
    "Content-Type": "application/json"
}
params = {
    "language": "python",
    "limit": 10
}
response = requests.get(url, headers=headers, params=params)
# Parse the response
data = response.json()
for file in data.get("files", []):
    print(f"Path: {file['path']}")
    print(f"Language: {file['language']}")
    print(f"License: {file['license']}")
    print(f"Size: {file['size']} bytes")
    print("---")

3 Steps to Quick Start

Start using The Stack code dataset in just a few minutes

Register an Account

Get API Key

Create an API key in the console for authentication to access The Stack dataset interface.

Call Dataset API

Use the API key to call the dataset interface, filtering and retrieving code data by programming language, license, and other criteria.

Get Dataset

Start Exploring The Stack Code Dataset

546 million code files, 30+ programming languages, 6TB of massive data. Whether you are training code generation models, building code search engines, or researching software engineering, The Stack is the ideal data foundation.