How arthurheitmann/arctic_shift works
Project Type & Purpose
• Arctic Shift is an open-source data-processing and serving pipeline for large Reddit data dumps.
• It combines an ETL (Extract-Transform-Load) component (Python scripts + schemas) with a data-serving layer (API/submodule + web interfaces/download tool).
File Structure Analysis
• Top‐level directories:
• schemas/ – TypeScript and JSON schema definitions for each data slice (RC, RS, subreddits, etc.)
• scripts/ – Python scripts to read raw .zst/.jsonl files, apply transformations via schemas (fileStreams.py, processFiles.py, utils.py)
• api/ – a Git submodule providing a Node/TypeScript API server and search/download endpoints
• Config/metadata files: README.md, download_links.md, file_content_explanations.md, .gitmodules
README Insights
• Describes three main access methods:
• Large dumps (managed via download_links.md + GitHub releases)
• API (in api/ submodule)
• Web interface & external download tool
• Highlights Python 3.10 + zstandard for processing
Main Components
a. Data Ingest / ETL Layer
• Raw Data Store: source dump files (.zst, .zst_blocks, .jsonl)
• Python Processing Scripts (scripts/processFiles.py, fileStreams.py)
• Schema Validator/Generator (schemas/… TS + JSON files)
• Processed Output Store: normalized .jsonl or database-ready files
b. Data Serving Layer
• API Server (api/ submodule; Node.js/TS)
• Web UI & Download Tool (external “download-tool” service + manual search UI)
c. Metadata & Documentation
• download_links.md & file_content_explanations.md (documentation nodes)
• GitHub releases (for large dumps distribution)
d. External Clients & Services
• Researchers/moderators consume API or download dumps
• External download tool (hosted at arctic-shift.photon-reddit.com)
• Search UI (hosted at arctic-shift.photon-reddit.com/search)
Architectural Patterns & Principles
• Modular ETL pipeline: separates file I/O, schema validation, transformation logic
• Schema-first design: TypeScript + JSON schemas define the data contracts
• Submodule isolation: api/ kept as Git submodule to decouple service implementation
• Code-generation potential: TS schemas can generate types for API & scripts
Technologies & Frameworks
• Python 3.10 + zstandard library for data processing
• TypeScript + JSON schema for data modeling
• Node.js (Express or similar) for API server (api/ submodule)
• Git submodules for versioned API code
• Flat-file storage of dumps (.zst, .jsonl) and processed outputs