How sebastianvp/estudio_cintilaciones_peru works
Project type and purpose
• This is a research-grade full-pipeline ML system for forecasting ionospheric scintillation (S4 index) over Peru.
• It integrates data ingestion, preprocessing, model training (SVM, Random Forest, LSTM), and result visualization, all orchestrated via Python scripts and Jupyter notebooks.
File structure
• Top‐level scripts for data acquisition and preprocessing:
• descargar_DATOS*.py & descomprimir_DATOS*.py – fetch and unzip OmniWeb, ROJ and LISN raw files
• generar_dataset*.py – assemble cleaned CSV/Excel datasets
• DATA & Diciembre directories hold raw and intermediate data files (.s4, Excel, JSON)
• PLOTEOS & PLOTEOS_GRUPOS folders contain plotting routines (ploteo_diario.py, ploteo_integrado4.py)
• Notebooks (DESCARGA_DATOS_LISN.ipynb, TESIS_…ipynb) combine exploratory analysis, model development, and visualization
• README.md describes the high‐level scientific workflow and ML techniques
README for additional insights
• Three external sources (NASA OmniWeb, ROJ radars/ionosondes, LISN network) feed data
• Core ML models: LSTM for regression, SVM/Random Forest for classification of S4 levels
• Preprocessing steps: normalization, outlier removal, temporal aggregation
a. Main components
• External Data Sources: NASA OmniWeb Service, ROJ Instruments, LISN Sensors
• Ingestion Layer: Python scripts (descargar_*, descomprimir_*)
• Raw Data Store: DATA/, Diciembre/ directories (.s4, Excel)
• Preprocessing Layer: generar_dataset*.py
• Feature Store: unified CSV/Excel (output_combined_satellites_data.csv)
• Model Training & Validation: notebooks and scripts implementing SVM, RF, LSTM
• Model Artifacts: serialized model files (pickles, HDF5)
• Forecasting Engine: the trained models generating predictions in real time or batch
• Visualization & Reporting: plotting scripts and IPython notebooks; final reports
b. Relationships and interactions
• Raw Data Store → Preprocessing Layer → Feature Store
• Feature Store → Model Training → Model Artifacts
• Model Artifacts → Forecasting Engine → Predictions
• Predictions → Visualization & Reporting
c. Architectural patterns & design principles
• Modular pipeline: each stage (ingest, preprocess, train, predict) is isolated in its own directory or script
• Data-centric: raw files are versioned and stored separately; intermediate and final datasets are explicit CSV/Excel artifacts
• ML best practices: separation of feature engineering, training, validation, and inference
d. Key technologies/frameworks
• Python 3 with Anaconda on Ubuntu
• Pandas, NumPy for data wrangling
• Scikit-learn (SVM, Random Forest)
• TensorFlow/Keras or PyTorch for LSTM
• Jupyter Notebooks for exploratory data analysis and model development
• Matplotlib/Seaborn (or similar) for plotting
Detail is key
• External dependencies (NASA, ROJ, LISN) and local artifacts (DATA/, PLOTEOS/).
• This level of granularity ensures stakeholders understand both data flow and processing logic.