How kartikmandar/ripple works
Project Type and Purpose
• RIPPLe is a modular, sequential data‐processing pipeline (a Python “full‐stack” ETL+ML application) designed to fetch astronomical data from an LSST Butler repository, preprocess images, run inference through DeepLense models, and emit results.
File‐Structure Analysis
• Top‐level “ripple/” directory contains the core source.
• data_access/: LSST Butler client, coordinate conversion, caching, TAP service monitoring
• preprocessing/: image cutouts, cleaning, normalization, feature extraction
• models/: model loading, training, evaluation, inference interfaces
• pipeline/: orchestration logic, builder/factory/registry, executor, stages, steps
• utils/: logging, shared helpers
• main.py serves as the CLI/entry point, driven by a YAML config (examples in pipeline_configs/).
• Tests, manual_tests, pipeline_configs, notebooks illustrate usage and verification.
README Insights
• Configuration is YAML‐driven, enabling customization of data repo, preprocessing params, model specs, and pipeline execution.
1. CLI/Orchestrator Layer
– ripple/main.py
– pipeline_registry, pipeline_builder, pipeline_orchestrator
2. Configuration Manager
– config_handler (butler_repo), pipeline/config.py, preprocessing/config.py, models/config.py
3. Data Access Service
– butler_client, data_fetcher, rsp_tap_client, cache_manager, service_monitor
– External dependency: LSST Butler Repository (on-disk or networked)
4. Preprocessing Engine
– preprocessor, data_cleaner, image_normalizer, feature_extractor, cutout_creator
5. Model Interface
– model_factory, model_loader, model_trainer/evaluator, inference via PyTorch (DeepLense)
– External dependency: Pretrained DeepLense .pth model file
6. Pipeline Core
– pipeline_executor, pipeline_stage, pipeline_step, pipeline_orchestrator
– pipeline_stages: data_source, ingestion, preprocessing, model
7. Output Handler
– CSV writer, catalog exporter, logging
8. Utilities & Logging
– logger, utils
• CLI → Configuration Manager: loads YAML, instantiates pipeline
• CLI → Pipeline Orchestrator → Pipeline Executor
• Pipeline Executor → Data Source Stage → Data Access Service → LSST Butler Repo
• Data → Ingestion Stage → Preprocessing Stage → Preprocessing Engine
• Tensors → Model Stage → Model Interface → DeepLense Model → Predictions
• Predictions → Output Handler → Files/DB/CSV
• Throughout: Cache Manager and Service Monitor mediate data fetch reliability
c. Architectural Patterns & Principles
• Modular pipeline with builder, registry, and factory patterns
• Separation of concerns: data access, preprocessing, modeling, orchestration
• Config‐driven extensibility: new pipeline_stages or model types can be registered
d. Key Technologies & Frameworks
• Python 3.x, PyTorch for ML inference
• LSST Science Pipelines Butler API
• YAML configs, PyYAML
• Standard Python logging