# aliyan-navaid/parallelpixels architecture

> Architecture diagram of the GitHub repository aliyan-navaid/parallelpixels, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aliyan-navaid/parallelpixels
- Repository: https://github.com/aliyan-navaid/parallelpixels

## Overview

##### Project Type and Purpose
- This is a CLI-based, concurrent image-processing pipeline written in C (C11) with POSIX Threads.
- Purpose: Monitor an input directory, chunk images, apply filters in parallel, reconstruct images, and write them out—displaying real‐time stats and supporting graceful shutdown.

##### File Structure
- Top-level:
- main.c, image.py (utility/script)
- CMakeLists.txt, clean.sh
- pipeline/… broken into chunking, filter, reconstruction sub-dirs
- shared/… generic data structures, thread pool, image I/O, hash, macros
- vendors/xxHash
- Each pipeline stage has its own `include/` and `src/` folders.
- Shared folder supplies thread-safe queues, dynamic arrays, dictionaries, image routines, threading primitives.
- Build: CMake + make producing `ppxl` executable.

##### Extract Architecture from README
- Stages: Watcher → Chunker → Filter → Reconstruction → Output
- Communication via thread-safe queues: `name_queue`, `chunker_filtering_queue`, `filtering_reconstruction_queue`.
- Auxiliary threads: statistics display, user-input listener (for shutdown).
- Graceful shutdown on signals or user input.

a. Main Components
- Input Directory (filesystem)
- Watcher Thread (directory_monitor)
- Name Queue (image_queue)
- Chunker Thread Pool (image_chunker + file_tracker)
- Chunker→Filter Queue (shared queue)
- Filter Thread Pool (chunk_threader + filter)
- Filter→Reconstruction Queue
- Reconstruction Thread
- Output Directory (filesystem)
- Stats & Shutdown Threads (thread_pool, atomic counters)
- Shared Utilities (th pool, darray/dlist/dict)
- External Libraries (stb_image, stb_image_write, xxHash)

b. Relationships & Data Flow

c. Architectural Patterns & Principles
- Pipeline / Producer–Consumer model
- Thread-safe queues as bounded buffers
- Thread pools per stage maximize CPU usage
- Separation of concerns: each stage in its own module
- Graceful teardown via broadcast shutdown flags (stdatomic.h + pthreads)

d. Key Technologies & Frameworks
- Language: C11, POSIX Threads (pthread)
- Build System: CMake
- Image I/O: stb_image.h, stb_image_write.h (or libpng/jpeg)
- Utilities: uthash, xxHash, custom dynamic array/dlist/dict in shared/

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Filesystem
    subgraph "Filesystem" 
        direction TB
        InputDir["Input Directory"]:::external
        OutputDir["Output Directory"]:::external
    end

    %% Pipeline Stages
    subgraph "Watcher Stage" 
        direction TB
        Watcher["Watcher Thread"]:::stage
    end

    subgraph "Chunker Stage" 
        direction TB
        NameQueue["File Name Queue"]:::queue
        ChunkerPool["Chunker Thread Pool"]:::stage
    end

    subgraph "Filtering Stage" 
        direction TB
        CFQueue["Chunker→Filter Queue"]:::queue
        FilterPool["Filter Thread Pool"]:::stage
    end

    subgraph "Reconstruction Stage" 
        direction TB
        FRQueue["Filter→Reconstruction Queue"]:::queue
        ReconThread["Reconstruction Thread"]:::stage
    end

    %% Auxiliary
    subgraph "Auxiliary Threads"
        direction TB
        Stats["Statistics Thread"]:::aux
        Shutdown["Shutdown Listener Thread"]:::aux
    end

    %% Shared Utilities
    subgraph "Shared Utilities" 
        direction TB
        ThreadPool["Thread Pool Impl"]:::util
        ImageIO["Image I/O"]:::util
        DArray["Dynamic Arrays"]:::util
        DList["Linked Lists"]:::util
        Dict["Hash Tables"]:::util
        Macros["Common Macros"]:::util
    end

    %% External Libraries
    subgraph "External Libraries"
        direction TB
        STB1["stb_image"]:::util
        STB2["stb_image_write"]:::util
        XXH["xxHash"]:::util
    end

    %% Entry Point & Scripts
    subgraph "CLI & Build"
        direction TB
        Main["main.c"]:::util
        CMake["CMakeLists.txt"]:::util
        Clean["clean.sh"]:::util
        ImagePy["image.py"]:::util
    end

    %% Connections
    InputDir -->|monitors| Watcher
    Watcher -->|pushes names| NameQueue
    NameQueue -->|dispatch| ChunkerPool
    ChunkerPool -->|chunks| CFQueue
    CFQueue -->|consumes| FilterPool
    FilterPool -->|processes| FRQueue
    FRQueue -->|assembles| ReconThread
    ReconThread -->|writes| OutputDir

    %% Auxiliary control
    Stats -->|reads counters| Stats
    Shutdown -->|signals| Watcher
    Shutdown -->|signals| ChunkerPool
    Shutdown -->|signals| FilterPool
    Shutdown -->|signals| ReconThread

    %% Utilities usage
    ChunkerPool --> ThreadPool
    FilterPool --> ThreadPool
    Stats --> ThreadPool
    Main --> ThreadPool
    ChunkerPool --> ImageIO
    ReconThread --> ImageIO

    %% External libs usage
    ImageIO --> STB1
    ImageIO --> STB2
    Dict --> XXH

    %% Click Events
    click Watcher "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/src/directory_monitor.c"
    click Watcher "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/include/directory_monitor.h"
    click NameQueue "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/src/image_queue.c"
    click NameQueue "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/include/image_queue.h"
    click ChunkerPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/thread_pool.c"
    click ChunkerPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/thread_pool.h"
    click ChunkerPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/src/image_chunker.c"
    click ChunkerPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/include/image_chunker.h"
    click ChunkerPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/src/file_tracker.c"
    click ChunkerPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/include/file_tracker.h"
    click CFQueue "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/src/image_queue.c"
    click CFQueue "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/include/image_queue.h"
    click FilterPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/thread_pool.c"
    click FilterPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/thread_pool.h"
    click FilterPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/filter/src/chunk_threader.c"
    click FilterPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/filter/include/chunk_threader.h"
    click FilterPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/filter/src/filter.c"
    click FilterPool "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/filter/include/filter.h"
    click FRQueue "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/src/image_queue.c"
    click FRQueue "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/chunking/include/image_queue.h"
    click ReconThread "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/reconstruction/reconstruction.c"
    click ReconThread "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/reconstruction/reconstruction.h"
    click ReconThread "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/reconstruction/image_unchunk.c"
    click ReconThread "https://github.com/aliyan-navaid/parallelpixels/blob/main/pipeline/reconstruction/image_unchunk.h"
    click ImageIO "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/image.c"
    click ImageIO "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/image.h"
    click DArray "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/darray.c"
    click DArray "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/darray.h"
    click DList "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/dlist.c"
    click DList "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/dlist.h"
    click Dict "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/dict.c"
    click Dict "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/dict.h"
    click Dict "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/uthash.h"
    click Macros "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/macros.h"
    click STB1 "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/stb_image.h"
    click STB2 "https://github.com/aliyan-navaid/parallelpixels/blob/main/shared/stb_image_write.h"
    click XXH "https://github.com/aliyan-navaid/parallelpixels/tree/main/vendors/xxHash"
    click Main "https://github.com/aliyan-navaid/parallelpixels/blob/main/main.c"
    click CMake "https://github.com/aliyan-navaid/parallelpixels/blob/main/CMakeLists.txt"
    click Clean "https://github.com/aliyan-navaid/parallelpixels/blob/main/clean.sh"
    click ImagePy "https://github.com/aliyan-navaid/parallelpixels/blob/main/image.py"

    classDef stage fill:#cfe2f3,stroke:#333,stroke-width:1px
    classDef queue fill:#d9ead3,stroke:#333,stroke-width:1px
    classDef aux fill:#f9cb9c,stroke:#333,stroke-width:1px
    classDef util fill:#d9d2e9,stroke:#333,stroke-width:1px
    classDef external fill:#f4cccc,stroke:#333,stroke-width:1px
```
