# sebastianvp/estudio_cintilaciones_peru architecture

> Architecture diagram of the GitHub repository sebastianvp/estudio_cintilaciones_peru, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sebastianvp/estudio_cintilaciones_peru
- Repository: https://github.com/sebastianvp/estudio_cintilaciones_peru

## Overview

##### Project type and purpose
- This is a research-grade full-pipeline ML system for forecasting ionospheric scintillation (S4 index) over Peru.
- It integrates data ingestion, preprocessing, model training (SVM, Random Forest, LSTM), and result visualization, all orchestrated via Python scripts and Jupyter notebooks.

##### File structure
- Top‐level scripts for data acquisition and preprocessing:
- descargar_DATOS*.py & descomprimir_DATOS*.py – fetch and unzip OmniWeb, ROJ and LISN raw files
- generar_dataset*.py – assemble cleaned CSV/Excel datasets
- DATA & Diciembre directories hold raw and intermediate data files (.s4, Excel, JSON)
- PLOTEOS & PLOTEOS_GRUPOS folders contain plotting routines (ploteo_diario.py, ploteo_integrado4.py)
- Notebooks (DESCARGA_DATOS_LISN.ipynb, TESIS_…ipynb) combine exploratory analysis, model development, and visualization
- README.md describes the high‐level scientific workflow and ML techniques

##### README for additional insights
- Three external sources (NASA OmniWeb, ROJ radars/ionosondes, LISN network) feed data
- Core ML models: LSTM for regression, SVM/Random Forest for classification of S4 levels
- Preprocessing steps: normalization, outlier removal, temporal aggregation

a. Main components
- External Data Sources: NASA OmniWeb Service, ROJ Instruments, LISN Sensors
- Ingestion Layer: Python scripts (descargar_*, descomprimir_*)
- Raw Data Store: DATA/, Diciembre/ directories (.s4, Excel)
- Preprocessing Layer: generar_dataset*.py
- Feature Store: unified CSV/Excel (output_combined_satellites_data.csv)
- Model Training & Validation: notebooks and scripts implementing SVM, RF, LSTM
- Model Artifacts: serialized model files (pickles, HDF5)
- Forecasting Engine: the trained models generating predictions in real time or batch
- Visualization & Reporting: plotting scripts and IPython notebooks; final reports

b. Relationships and interactions
- Raw Data Store → Preprocessing Layer → Feature Store
- Feature Store → Model Training → Model Artifacts
- Model Artifacts → Forecasting Engine → Predictions
- Predictions → Visualization & Reporting

c. Architectural patterns & design principles
- Modular pipeline: each stage (ingest, preprocess, train, predict) is isolated in its own directory or script
- Data-centric: raw files are versioned and stored separately; intermediate and final datasets are explicit CSV/Excel artifacts
- ML best practices: separation of feature engineering, training, validation, and inference

d. Key technologies/frameworks
- Python 3 with Anaconda on Ubuntu
- Pandas, NumPy for data wrangling
- Scikit-learn (SVM, Random Forest)
- TensorFlow/Keras or PyTorch for LSTM
- Jupyter Notebooks for exploratory data analysis and model development
- Matplotlib/Seaborn (or similar) for plotting

##### Detail is key
- External dependencies (NASA, ROJ, LISN) and local artifacts (DATA/, PLOTEOS/).
- This level of granularity ensures stakeholders understand both data flow and processing logic.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data Sources
    subgraph "External Data Sources"
        direction TB
        NASA["NASA OmniWeb Service"]:::external
        ROJ["ROJ Instruments"]:::external
        LISN["LISN Sensors"]:::external
    end

    %% Ingestion Layer
    subgraph "Ingestion Layer"
        direction TB
        Ingest1["descargar_DATOS.py"]:::processing
        click Ingest1 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/descargar_DATOS.py"
        Ingest2["descargar_DATOS_ANUAL.py"]:::processing
        click Ingest2 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/descargar_DATOS_ANUAL.py"
        Ingest3["descomprimir_DATOS.py"]:::processing
        click Ingest3 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/descomprimir_DATOS.py"
        Ingest4["descomprimir_DATOS_ANUAL.py"]:::processing
        click Ingest4 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/descomprimir_DATOS_ANUAL.py"
        Ingest5["DESCARGA_DATOS_LISN.ipynb"]:::processing
        click Ingest5 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/DESCARGA_DATOS_LISN.ipynb"
    end

    %% Raw Data Store
    subgraph "Raw Data Store"
        direction TB
        Raw1["DATA/DATA_S4/"]:::datastore
        click Raw1 "https://github.com/sebastianvp/estudio_cintilaciones_peru/tree/main/DATA/DATA_S4/"
        Raw2["Diciembre/2024_december_scint_data_ljic/data/*.s4"]:::datastore
        click Raw2 "https://github.com/sebastianvp/estudio_cintilaciones_peru/tree/main/Diciembre/2024_december_scint_data_ljic/data/"
        Raw3["ljic_241231.s4"]:::datastore
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        direction TB
        Pre1["generar_dataset.py"]:::processing
        click Pre1 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/generar_dataset.py"
        Pre2["generar_dataset2.py"]:::processing
        click Pre2 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/generar_dataset2.py"
        Pre3["generar_dataset3.py"]:::processing
        click Pre3 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/generar_dataset3.py"
        Pre4["generar_DATASETM.py"]:::processing
        click Pre4 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/generar_DATASETM.py"
        Pre5["generar_DATASET_TOTAL.py"]:::processing
        click Pre5 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/DATA/DATA_S4/generar_DATASET_TOTAL.py"
    end

    %% Feature Store
    subgraph "Feature Store"
        direction TB
        Features["output_combined_satellites_data.csv"]:::datastore
    end

    %% Model Training & Validation
    subgraph "Model Training & Validation"
        direction TB
        Train1["TESIS_MAESTRIA_CINTILACIONES_2025.ipynb"]:::ml
        click Train1 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/TESIS_MAESTRIA_CINTILACIONES_2025.ipynb"
        Train2["TESIS_PAPERI_NOTAS.ipynb"]:::ml
        click Train2 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/TESIS_PAPERI_NOTAS.ipynb"
        Train3["TEORIA_SEMINARIO_TESIS_I.ipynb"]:::ml
        click Train3 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/TEORIA_SEMINARIO_TESIS_I.ipynb"
    end

    %% Model Artifacts
    ModelArtifacts["Serialized Model Files\n(pickles, HDF5)"]:::ml

    %% Forecasting Engine
    Forecast["Forecasting Engine"]:::ml

    %% Visualization & Reporting
    subgraph "Visualization & Reporting"
        direction TB
        Plot1["ploteo_diario.py"]:::viz
        click Plot1 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/ploteo_diario.py"
        Plot2["ploteo_integrado4.py"]:::viz
        click Plot2 "https://github.com/sebastianvp/estudio_cintilaciones_peru/blob/main/ploteo_integrado4.py"
        PlotGroup1["PLOTEOS/"]:::viz
        click PlotGroup1 "https://github.com/sebastianvp/estudio_cintilaciones_peru/tree/main/PLOTEOS/"
        PlotGroup2["PLOTEOS_GRUPOS/"]:::viz
        click PlotGroup2 "https://github.com/sebastianvp/estudio_cintilaciones_peru/tree/main/PLOTEOS_GRUPOS/"
    end

    %% Documentation
    subgraph "Documentation"
        direction TB
        Readme["README.md"]:::docs
        Doc1["DOCUMENTOS/lisn_data_formats_V2-3-1.pdf"]:::docs
        Doc2["DOCUMENTOS/ljic_250301_s4.pdf"]:::docs
    end

    %% Connections
    NASA --> Ingest1
    ROJ --> Ingest1
    LISN --> Ingest5
    Ingest1 --> Ingest3
    Ingest2 --> Ingest4
    Ingest3 --> Raw1
    Ingest4 --> Raw2
    Ingest5 --> Raw2
    Raw1 --> Pre1
    Raw1 --> Pre2
    Raw2 --> Pre3
    Raw3 --> Pre4
    Pre1 --> Features
    Pre2 --> Features
    Pre3 --> Features
    Pre4 --> Features
    Pre5 --> Features
    Features --> Train1
    Features --> Train2
    Features --> Train3
    Train1 --> ModelArtifacts
    Train2 --> ModelArtifacts
    Train3 --> ModelArtifacts
    ModelArtifacts --> Forecast
    Forecast --> Plot1
    Forecast --> Plot2
    Plot1 --> PlotGroup1
    Plot2 --> PlotGroup2
    Readme --> Ingest1
    Readme --> Pre1
    Readme --> Train1

    %% Styles
    classDef external fill:#ddeeff,stroke:#6699cc,color:#333;
    classDef processing fill:#e0f7e9,stroke:#34a853,color:#070;
    classDef datastore fill:#e8f0fe,stroke:#4285f4,color:#1a73e8;
    classDef ml fill:#fff4e5,stroke:#fbbc04,color:#c66900;
    classDef viz fill:#f3e5f5,stroke:#ab47bc,color:#6a1b9a;
    classDef docs fill:#f0f0f0,stroke:#999999,color:#333333;
```
