# arthurheitmann/arctic_shift architecture

> Architecture diagram of the GitHub repository arthurheitmann/arctic_shift, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arthurheitmann/arctic_shift
- Repository: https://github.com/arthurheitmann/arctic_shift

## Overview

##### Project Type & Purpose
- Arctic Shift is an open-source data-processing and serving pipeline for large Reddit data dumps.
- It combines an ETL (Extract-Transform-Load) component (Python scripts + schemas) with a data-serving layer (API/submodule + web interfaces/download tool).

##### File Structure Analysis
- Top‐level directories:
- schemas/ – TypeScript and JSON schema definitions for each data slice (RC, RS, subreddits, etc.)
- scripts/ – Python scripts to read raw .zst/.jsonl files, apply transformations via schemas (fileStreams.py, processFiles.py, utils.py)
- api/ – a Git submodule providing a Node/TypeScript API server and search/download endpoints
- Config/metadata files: README.md, download_links.md, file_content_explanations.md, .gitmodules

##### README Insights
- Describes three main access methods:
- Large dumps (managed via download_links.md + GitHub releases)
- API (in api/ submodule)
- Web interface & external download tool
- Highlights Python 3.10 + zstandard for processing

##### Main Components
a. Data Ingest / ETL Layer
- Raw Data Store: source dump files (.zst, .zst_blocks, .jsonl)
- Python Processing Scripts (scripts/processFiles.py, fileStreams.py)
- Schema Validator/Generator (schemas/… TS + JSON files)
- Processed Output Store: normalized .jsonl or database-ready files
b. Data Serving Layer
- API Server (api/ submodule; Node.js/TS)
- Web UI & Download Tool (external “download-tool” service + manual search UI)
c. Metadata & Documentation
- download_links.md & file_content_explanations.md (documentation nodes)
- GitHub releases (for large dumps distribution)
d. External Clients & Services
- Researchers/moderators consume API or download dumps
- External download tool (hosted at arctic-shift.photon-reddit.com)
- Search UI (hosted at arctic-shift.photon-reddit.com/search)

##### Architectural Patterns & Principles
- Modular ETL pipeline: separates file I/O, schema validation, transformation logic
- Schema-first design: TypeScript + JSON schemas define the data contracts
- Submodule isolation: api/ kept as Git submodule to decouple service implementation
- Code-generation potential: TS schemas can generate types for API & scripts

##### Technologies & Frameworks
- Python 3.10 + zstandard library for data processing
- TypeScript + JSON schema for data modeling
- Node.js (Express or similar) for API server (api/ submodule)
- Git submodules for versioned API code
- Flat-file storage of dumps (.zst, .jsonl) and processed outputs

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% ETL Layer
    subgraph "ETL Layer"
        direction TB
        RawDataStore["Raw Data Store (.zst/.jsonl)"]:::dataStore
        subgraph "Python ETL Scripts"
            direction TB
            ProcessFiles["processFiles.py"]:::compute
            FileStreams["fileStreams.py"]:::compute
            Utils["utils.py"]:::compute
            ZstBlocks["zst_blocks_format/"]:::compute
        end
        subgraph "Schema Library"
            direction TB
            SchemaDefs["Schema Definitions"]:::schema
            SchemaReadme["schemas/README.md"]:::doc
        end
        ProcessedStore["Processed Output Store"]:::dataStore
    end

    %% Service Layer
    subgraph "Service Layer"
        direction TB
        APIServer["API Server (Node.js/TypeScript)"]:::compute
    end

    %% Interfaces Layer
    subgraph "Interfaces & Clients"
        direction TB
        WebUI["Search UI (arctic-shift.photon-reddit.com/search)"]:::external
        DownloadTool["External Download Tool"]:::external
        EndUsers["Researchers / Moderators"]:::external
    end

    %% Documentation & Config
    subgraph "Documentation & Config"
        direction TB
        README["README.md"]:::doc
        DownloadLinks["download_links.md"]:::doc
        FileContentDocs["file_content_explanations.md"]:::doc
        GitModules[".gitmodules"]:::doc
    end

    %% Data Flows
    RawDataStore -->|read & decompress| ProcessFiles
    ProcessFiles -->|validate/transform| SchemaDefs
    SchemaDefs -->|schemas| ProcessFiles
    ProcessFiles -->|write output| ProcessedStore
    ProcessedStore -->|serve data| APIServer
    APIServer -->|endpoint| WebUI
    APIServer -->|endpoint| DownloadTool
    WebUI -->|access| EndUsers
    DownloadTool -->|access| EndUsers

    %% Click Events
    click ProcessFiles "https://github.com/arthurheitmann/arctic_shift/blob/master/scripts/processFiles.py"
    click FileStreams "https://github.com/arthurheitmann/arctic_shift/blob/master/scripts/fileStreams.py"
    click Utils "https://github.com/arthurheitmann/arctic_shift/blob/master/scripts/utils.py"
    click ZstBlocks "https://github.com/arthurheitmann/arctic_shift/tree/master/scripts/zst_blocks_format/"
    click SchemaDefs "https://github.com/arthurheitmann/arctic_shift/tree/master/schemas/"
    click SchemaReadme "https://github.com/arthurheitmann/arctic_shift/blob/master/schemas/README.md"
    click APIServer "https://github.com/arthurheitmann/arctic_shift/tree/master/api/"
    click README "https://github.com/arthurheitmann/arctic_shift/blob/master/README.md"
    click DownloadLinks "https://github.com/arthurheitmann/arctic_shift/blob/master/download_links.md"
    click FileContentDocs "https://github.com/arthurheitmann/arctic_shift/blob/master/file_content_explanations.md"
    click GitModules "https://github.com/arthurheitmann/arctic_shift/blob/master/.gitmodules"

    %% Styles
    classDef dataStore fill:#fddfae,stroke:#e38a0d;
    classDef compute fill:#cfe2f3,stroke:#5b9bd5;
    classDef schema fill:#f4cccc,stroke:#cc0000;
    classDef doc fill:#d9d9d9,stroke:#828282;
    classDef external fill:#d9ead3,stroke:#82b366;
```
