# tirthankar4/elemental-maps-using-class-data architecture

> Architecture diagram of the GitHub repository tirthankar4/elemental-maps-using-class-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tirthankar4/elemental-maps-using-class-data
- Repository: https://github.com/tirthankar4/elemental-maps-using-class-data

## Overview

##### Project Type and Purpose
- This is a batch-oriented scientific data-processing pipeline (not a web service or full-stack app).
- It automates analysis of Chandrayaan-2 CLASS L1 XRF data, extracting solar-flare windows, performing spectral corrections, and producing sub-pixel elemental ratio maps.

##### Top-Level Components
A. Data Ingestion & Flare Sorting
– Inputs: CLASS L1 files, flare catalogs (Hinode/XRT), SPICE kernels
– Scripts: `fetch-flares.py`, `fetch-flares_hxrt.py`
B. Spectral Aggregation & Background Generation
– Scripts: `addClassSpectra.py` (aggregates 8s spectra), `gen_bg.py` (nighttime background)
C. Spectral Correction & Peak Detection
– Libraries: HEASoft / PyXSPEC, OGIP ARF/RMF
– Scripts: `arfCorrection2.py`, `responseCorrectedSpecrtum*.py`, `catalogue.py` (GMM peak detection)
D. Ratio Calculation & Mapping
– Scripts: `fluxRatios/*` (ratio computation: Mg/Si, Al/Si, …), `colorMap.py` (projection, plotting)
E. Sub-Pixel Resolution Enhancement
– Scripts: `SubpixelResolution/drizzle.py`, `interlacing_*.py`, `processSubsets.py`

##### External Services & Infrastructure
- HPC Cluster / SLURM scheduler
– Job scripts: `fluxRatios/SLURM_JOB.sh`, `fluxRatios/python_job.sh`
- Geo/Spatial Libraries
– GeoPandas, Cartopy (for geospatial reprojection)
- Visualization
– Matplotlib, Blender (3D), plus a future web-based 2D/3D viewer

##### Data Stores & Artifacts
- Raw Data Store: CLASS L1 repository (5 TB)
- Intermediate FITS files (8s spectra aggregates, background files)
- Final Products: GeoTIFFs or image tiles of elemental ratios (4.125 km resolution)
- Metadata: logs, `catalogue` outputs, notebooks (mareDetection subset)

##### Relationships & Data Flow
Ingestion → Aggregation → Correction → Mapping → Subpixel
- Bidirectional or feedback loops if any (e.g., parameter tuning between Correction ↔ Catalogue)
- Python scripts under “Compute Layer” and external libraries under “Dependency Layer.”

##### Architectural Patterns & Design Principles
- Pipeline (staged, batch) pattern: each stage transforms and enriches data
- Modular scripts: loosely coupled, can run independently or on HPC
- Extensibility: new elements or machine-learning modules can slot into Correction or Mapping stages
- Reproducibility: SLURM jobs encapsulate environment and resource specs

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Pipeline Container
    subgraph "Data Pipeline"
        direction TB

        %% Stage 1: Data Ingestion & Flare Sorting
        subgraph "1. Data Ingestion & Flare Sorting"
            direction TB
            A1["fetch-flares.py"]:::ingestion
            A2["fetch-flares_hxrt.py"]:::ingestion
        end

        %% Stage 2: Spectral Aggregation & Background Generation
        subgraph "2. Spectral Aggregation & Background Generation"
            direction TB
            B1["addClassSpectra.py"]:::aggregation
            B2["gen_bg.py"]:::aggregation
            B3["addNight.py"]:::aggregation
            B4["addNight2.py"]:::aggregation
            B5["class_bg_generator_FINAL.py"]:::aggregation
        end

        %% Stage 3: Spectral Correction & Peak Detection
        subgraph "3. Spectral Correction & Peak Detection"
            direction TB
            C1["arfCorrection2.py"]:::correction
            C2["arf_rmf_bg_correction_pyspextools.py"]:::correction
            C3["responseCorrectedSpecrtum.py"]:::correction
            C4["responseCorrectedSpecrtum_new.py"]:::correction
            C5["catalogue.py"]:::correction
        end

        %% Stage 4: Ratio Calculation & Mapping
        subgraph "4. Ratio Calculation & Mapping"
            direction TB
            D1["element_csv.py"]:::mapping
            D2["colorMap.py"]:::mapping
            D3["highlandDetection.py"]:::mapping
            D4["mareDetection.py"]:::mapping
            D5["parseOutput.py"]:::mapping
            D6["parseOutputandPlot.py"]:::mapping
            D7["parseOutputandPlot_mare_highland.py"]:::mapping
            D8["plotSubpixel.py"]:::mapping
            D9["mare.ipynb"]:::mapping
            D10["mare.py"]:::mapping
            D11["mareDetection.py"]:::mapping
            D12["subsets_mare_highlandDetection.py"]:::mapping
        end

        %% Stage 5: Sub-Pixel Resolution Enhancement
        subgraph "5. Sub-Pixel Resolution Enhancement"
            direction TB
            E1["drizzle.py"]:::subpixel
            E2["interlacing_v1.py"]:::subpixel
            E3["interlacing_v1_2.py"]:::subpixel
            E4["interlacing_v2_with_Gaussian.py"]:::subpixel
            E5["interlacing_v2_with_Gaussian_2.py"]:::subpixel
            E6["interlacing_v3_gauss_r2_drizz.py"]:::subpixel
            E7["processSubsets.py"]:::subpixel
        end

    end

    %% Data Flow Arrows
    A1 -->|feeds| B1
    A2 -->|feeds| B1
    B1 -->|generates| C1
    C5 -->|outputs peaks| D1
    D8 -->|feeds mapping| E1

    %% Data Stores
    Raw["CLASS L1 Repository"]:::datastore
    Intermediate["Intermediate FITS Files"]:::datastore
    Final["GeoTIFF / Image Tiles"]:::datastore
    Metadata["Logs & Catalogue Outputs"]:::datastore

    Raw -->|raw data| A1
    B2 -->|bg files| Intermediate
    C5 -->|catalogue| Metadata
    D2 -->|final products| Final

    %% External Services & Infrastructure
    subgraph "Dependency Layer"
        direction TB
        GeoLibs["GeoPandas & Cartopy"]:::dependency
        PyXSPEC["HEASoft / PyXSPEC"]:::dependency
    end

    subgraph "HPC Cluster / SLURM"
        direction TB
        SLURM1["SLURM_JOB.sh"]:::hpc
        SLURM2["python_job.sh"]:::hpc
    end

    GeoLibs -->|"used in"| D2
    PyXSPEC -->|"used in"| C1
    SLURM1 -.->|orchestrates| A1
    SLURM2 -.->|orchestrates| D1

    %% Click Events
    click A1 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/ClassFilesDuringFlares/fetch-flares.py"
    click A2 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/ClassFilesDuringFlares/fetch-flares_hxrt.py"
    click B1 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/addClassSpectra.py"
    click B2 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/gen_bg.py"
    click B3 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/addNight.py"
    click B4 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/addNight2.py"
    click B5 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/class_bg_generator_FINAL.py"
    click C1 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/arfCorrection2.py"
    click C2 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/arf_rmf_bg_correction_pyspextools.py"
    click C3 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/responseCorrectedSpecrtum.py"
    click C4 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/responseCorrectedSpecrtum_new.py"
    click C5 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/catalogue.py"
    click D1 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/element_csv.py"
    click D2 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/colorMap.py"
    click D3 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/highlandDetection.py"
    click D4 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/mareDetection.py"
    click D5 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/parseOutput.py"
    click D6 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/parseOutputandPlot.py"
    click D7 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/parseOutputandPlot_mare_highland.py"
    click D8 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/plotSubpixel.py"
    click D9 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/mareDetection/subset_with_mare/mare.ipynb"
    click D10 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/mareDetection/subset_with_mare/mare.py"
    click D11 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/mareDetection/subset_with_mare/mareDetection.py"
    click D12 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/mareDetection/subset_with_mare/subsets_mare_highlandDetection.py"
    click E1 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/SubpixelResolution/drizzle.py"
    click E2 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/SubpixelResolution/interlacing_v1.py"
    click E3 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/SubpixelResolution/interlacing_v1_2.py"
    click E4 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/SubpixelResolution/interlacing_v2_with_Gaussian.py"
    click E5 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/SubpixelResolution/interlacing_v2_with_Gaussian_2.py"
    click E6 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/SubpixelResolution/interlacing_v3_gauss_r2_drizz.py"
    click E7 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/SubpixelResolution/processSubsets.py"
    click SLURM1 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/SLURM_JOB.sh"
    click SLURM2 "https://github.com/tirthankar4/elemental-maps-using-class-data/blob/main/fluxRatios/python_job.sh"

    %% Styles
    classDef ingestion fill:#B3CDE3,stroke:#005B96,color:#000
    classDef aggregation fill:#FCAE91,stroke:#E6550D,color:#000
    classDef correction fill:#C1E1C1,stroke:#31A354,color:#000
    classDef mapping fill:#BC80BD,stroke:#7B4173,color:#000
    classDef subpixel fill:#80B1D3,stroke:#1F78B4,color:#000
    classDef dependency fill:#FDB462,stroke:#D95F02,color:#000
    classDef hpc fill:#8DD3C7,stroke:#008837,color:#000
    classDef datastore fill:#FFFFB3,stroke:#9C9C00,color:#000
```
