# kartikmandar/ripple architecture

> Architecture diagram of the GitHub repository kartikmandar/ripple, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kartikmandar/ripple
- Repository: https://github.com/kartikmandar/ripple

## Overview

##### Project Type and Purpose
- RIPPLe is a modular, sequential data‐processing pipeline (a Python “full‐stack” ETL+ML application) designed to fetch astronomical data from an LSST Butler repository, preprocess images, run inference through DeepLense models, and emit results.

##### File‐Structure Analysis
- Top‐level “ripple/” directory contains the core source.
- data_access/: LSST Butler client, coordinate conversion, caching, TAP service monitoring
- preprocessing/: image cutouts, cleaning, normalization, feature extraction
- models/: model loading, training, evaluation, inference interfaces
- pipeline/: orchestration logic, builder/factory/registry, executor, stages, steps
- utils/: logging, shared helpers
- main.py serves as the CLI/entry point, driven by a YAML config (examples in pipeline_configs/).
- Tests, manual_tests, pipeline_configs, notebooks illustrate usage and verification.

##### README Insights
- Configuration is YAML‐driven, enabling customization of data repo, preprocessing params, model specs, and pipeline execution.

1. CLI/Orchestrator Layer
– ripple/main.py
– pipeline_registry, pipeline_builder, pipeline_orchestrator
2. Configuration Manager
– config_handler (butler_repo), pipeline/config.py, preprocessing/config.py, models/config.py
3. Data Access Service
– butler_client, data_fetcher, rsp_tap_client, cache_manager, service_monitor
– External dependency: LSST Butler Repository (on-disk or networked)
4. Preprocessing Engine
– preprocessor, data_cleaner, image_normalizer, feature_extractor, cutout_creator
5. Model Interface
– model_factory, model_loader, model_trainer/evaluator, inference via PyTorch (DeepLense)
– External dependency: Pretrained DeepLense .pth model file
6. Pipeline Core
– pipeline_executor, pipeline_stage, pipeline_step, pipeline_orchestrator
– pipeline_stages: data_source, ingestion, preprocessing, model
7. Output Handler
– CSV writer, catalog exporter, logging
8. Utilities & Logging
– logger, utils

- CLI → Configuration Manager: loads YAML, instantiates pipeline
- CLI → Pipeline Orchestrator → Pipeline Executor
- Pipeline Executor → Data Source Stage → Data Access Service → LSST Butler Repo
- Data → Ingestion Stage → Preprocessing Stage → Preprocessing Engine
- Tensors → Model Stage → Model Interface → DeepLense Model → Predictions
- Predictions → Output Handler → Files/DB/CSV
- Throughout: Cache Manager and Service Monitor mediate data fetch reliability

c. Architectural Patterns & Principles
- Modular pipeline with builder, registry, and factory patterns
- Separation of concerns: data access, preprocessing, modeling, orchestration
- Config‐driven extensibility: new pipeline_stages or model types can be registered

d. Key Technologies & Frameworks
- Python 3.x, PyTorch for ML inference
- LSST Science Pipelines Butler API
- YAML configs, PyYAML
- Standard Python logging

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% CLI/Orchestrator Layer
    subgraph "CLI/Orchestrator Layer"
        CLI["CLI\n(ripple/main.py)"]:::core
        Reg["Pipeline Registry\n(pipeline_registry)"]:::core
        Builder["Pipeline Builder\n(pipeline_builder)"]:::core
        Orch["Pipeline Orchestrator\n(pipeline_orchestrator)"]:::core
    end

    %% Configuration Manager
    subgraph "Configuration Manager"
        ConfButler["Butler Config\n(config_handler.py)"]:::config
        ConfPipe["Pipeline Config\n(pipeline/config.py)"]:::config
        ConfPrep["Preprocessing Config\n(preprocessing/config.py)"]:::config
        ConfModel["Model Config\n(models/config.py)"]:::config
    end

    %% Pipeline Core
    subgraph "Pipeline Core"
        Exec["Pipeline Executor\n(pipeline_executor.py)"]:::core
        Stage["Pipeline Stage\n(pipeline_stage.py)"]:::core
        Step["Pipeline Step\n(pipeline_step.py)"]:::core
        CorePipe["Pipeline\n(pipeline.py)"]:::core
        DSStage["Data Source Stage\n(data_source_stage.py)"]:::core
        IngestStage["Ingestion Stage\n(ingestion_stage.py)"]:::core
        PrepStage["Preprocessing Stage\n(preprocessing_stage.py)"]:::core
        ModelStage["Model Stage\n(model_stage.py)"]:::core
    end

    %% Data Access Service
    subgraph "Data Access Service"
        ButlerClient["Butler Client\n(butler_client.py)"]:::core
        Fetcher["Data Fetcher\n(data_fetcher.py)"]:::core
        TAP["RSP TAP Client\n(rsp_tap_client.py)"]:::core
        Cache["Cache Manager\n(cache_manager.py)"]:::core
        Monitor["Service Monitor\n(service_monitor.py)"]:::core
    end

    %% Preprocessing Engine
    subgraph "Preprocessing Engine"
        Preproc["Preprocessor\n(preprocessor.py)"]:::core
        Cleaner["Data Cleaner\n(data_cleaner.py)"]:::core
        Norm["Image Normalizer\n(image_normalizer.py)"]:::core
        Feature["Feature Extractor\n(feature_extractor.py)"]:::core
        Cutout["Cutout Creator\n(cutout_creator.py)"]:::core
    end

    %% Model Interface
    subgraph "Model Interface"
        Factory["Model Factory\n(model_factory.py)"]:::core
        Base["Base Model\n(base_model.py)"]:::core
        Trainer["Model Trainer\n(model_trainer.py)"]:::core
        Evaluator["Model Evaluator\n(model_evaluator.py)"]:::core
        RegistryModel["Model Registry\n(model_registry.py)"]:::core
    end

    %% Output Handler
    subgraph "Output Handler"
        CSV["CSV Writer"]:::core
        Exporter["Catalog Exporter"]:::core
        LogOut["Output Logging"]:::core
    end

    %% Utilities & Logging
    Utils["Logger & Utils\n(utils/logger.py)"]:::utils

    %% External Dependencies
    LSST["LSST Butler Repository"]:::external
    DeepLense["DeepLense Model File"]:::external

    %% Flow Connections
    CLI -->|loads YAML| ConfButler
    CLI -->|loads YAML| ConfPipe
    CLI -->|loads YAML| ConfPrep
    CLI -->|loads YAML| ConfModel

    CLI --> Orch
    Orch --> Exec
    Exec --> DSStage
    DSStage --> ButlerClient
    ButlerClient --> LSST
    DSStage --> Fetcher
    DSStage --> TAP
    TAP --> LSST
    Fetcher --> Cache
    Fetcher --> Monitor
    DSStage --> IngestStage
    IngestStage --> PrepStage
    PrepStage --> Preproc
    Preproc --> Cleaner
    Cleaner --> Norm
    Norm --> Feature
    Feature --> Cutout
    Cutout --> ModelStage
    ModelStage --> Factory
    Factory --> Base
    Base --> Trainer
    Trainer --> Evaluator
    Evaluator --> RegistryModel
    RegistryModel --> DeepLense
    ModelStage -->|tensors| ModelStage
    ModelStage -->|predictions| Exporter
    Exporter --> CSV
    CSV --> LogOut
    LogOut --> Utils

    %% Click Events
    click CLI "https://github.com/kartikmandar/ripple/blob/main/ripple/main.py"
    click Reg "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/pipeline_registry.py"
    click Builder "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/pipeline_builder.py"
    click Orch "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/pipeline_orchestrator.py"
    click ConfButler "https://github.com/kartikmandar/ripple/blob/main/ripple/butler_repo/config_handler.py"
    click ConfPipe "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/config.py"
    click ConfPrep "https://github.com/kartikmandar/ripple/blob/main/ripple/preprocessing/config.py"
    click ConfModel "https://github.com/kartikmandar/ripple/blob/main/ripple/models/config.py"
    click ButlerClient "https://github.com/kartikmandar/ripple/blob/main/ripple/data_access/butler_client.py"
    click Fetcher "https://github.com/kartikmandar/ripple/blob/main/ripple/data_access/data_fetcher.py"
    click TAP "https://github.com/kartikmandar/ripple/blob/main/ripple/data_access/rsp_tap_client.py"
    click Cache "https://github.com/kartikmandar/ripple/blob/main/ripple/data_access/cache_manager.py"
    click Monitor "https://github.com/kartikmandar/ripple/blob/main/ripple/data_access/service_monitor.py"
    click Preproc "https://github.com/kartikmandar/ripple/blob/main/ripple/preprocessing/preprocessor.py"
    click Cleaner "https://github.com/kartikmandar/ripple/blob/main/ripple/preprocessing/data_cleaner.py"
    click Norm "https://github.com/kartikmandar/ripple/blob/main/ripple/preprocessing/image_normalizer.py"
    click Feature "https://github.com/kartikmandar/ripple/blob/main/ripple/preprocessing/feature_extractor.py"
    click Cutout "https://github.com/kartikmandar/ripple/blob/main/ripple/preprocessing/cutout_creator.py"
    click Factory "https://github.com/kartikmandar/ripple/blob/main/ripple/models/model_factory.py"
    click Base "https://github.com/kartikmandar/ripple/blob/main/ripple/models/base_model.py"
    click Trainer "https://github.com/kartikmandar/ripple/blob/main/ripple/models/model_trainer.py"
    click Evaluator "https://github.com/kartikmandar/ripple/blob/main/ripple/models/model_evaluator.py"
    click RegistryModel "https://github.com/kartikmandar/ripple/blob/main/ripple/models/model_registry.py"
    click Exec "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/pipeline_executor.py"
    click Stage "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/pipeline_stage.py"
    click Step "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/pipeline_step.py"
    click CorePipe "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/pipeline.py"
    click DSStage "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/stages/data_source_stage.py"
    click IngestStage "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/stages/ingestion_stage.py"
    click PrepStage "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/stages/preprocessing_stage.py"
    click ModelStage "https://github.com/kartikmandar/ripple/blob/main/ripple/pipeline/stages/model_stage.py"
    click Utils "https://github.com/kartikmandar/ripple/blob/main/ripple/utils/logger.py"

    %% Styles
    classDef core fill:#AED6F1,stroke:#2471A3,stroke-width:1px
    classDef config fill:#F9E79F,stroke:#B7950B,stroke-width:1px
    classDef external fill:#A9DFBF,stroke:#196F3D,stroke-width:1px
    classDef utils fill:#D5D8DC,stroke:#566573,stroke-width:1px
```
