How junyaok/ml_pyrite works
Project Type & Purpose
• This project supports geochemical research by distinguishing between different pyrite origins using trace element data.
Analyzing the File Structure
• Top-level Directories/Files: The project contains a LICENSE, README.md, a “code” directory with several Jupyter Notebooks, and a “data” directory with CSV files.
• The “code” directory holds four main notebooks:
• RF_XG.ipynb: Trains Random Forest/XGBoost models using a balanced dataset sampled from Training.csv.
• Ternary.ipynb: Uses predictions from RF_XG.ipynb to generate ternary plots for visual analysis.
• Cluster.ipynb: Conducts hierarchical clustering on the power-transformed trace element data.
• PCA.ipynb: Performs principal component analysis on the same power-transformed dataset.
• The “data” directory includes several CSV files for training, testing, and supplementary datasets.
• a. Main Components to Represent:
• Data Sources: Represent the different CSV files (Training.csv, OSL_Test.csv, TE_Data.csv, Xiao_Cui.csv) as a “Data Storage” or “Data Layer”.
• Data Preprocessing & Transformation: Implicit in each notebook’s pipeline where raw data may undergo power transformations.
• Data Visualization: Show processes using Ternary.ipynb and the generation of charts (ternary plots).
• Dimensionality Reduction and Pattern Discovery: Represent the PCA.ipynb and Cluster.ipynb components as performing data reduction and clustering analysis.
• External Libraries & Environment: Note key technologies (Python 3.10.4, Jupyter Notebook/JupyterLab, and libraries such as pandas, numpy, sklearn, xgboost, matplotlib, seaborn, and plotly).
• b. Relationships and Interactions:
• Data flows from the data layer (CSV files) into the notebooks.
• RF_XG.ipynb consumes training data, builds the classification models, and produces model prediction outputs.
• PCA.ipynb and Cluster.ipynb take raw or transformed trace element data to derive insights on element behavior.
• The notebooks might share intermediate processed data and even use common pre-processing steps.
• c. Architectural Patterns & Design Principles:
• Modular design: Each notebook is a self-contained analysis component that collectively forms a processing pipeline.
• Separation of Concerns: Data ingestion, model training, dimensionality reduction, clustering analysis, and visualization are all handled separately.
• Reproducibility & Dependency Management: The dedicated requirements section in the README and explicit use of Jupyter notebooks emphasize an environment where reproducible research is paramount.
• d. Relevant Technologies & Frameworks:
• Python (with version 3.10.4)
• Jupyter Notebook/JupyterLab for execution and interactive analysis.
• Scientific libraries (numpy, pandas, scipy) and specialized ones (sklearn, xgboost, plotly, matplotlib, seaborn) for machine learning and visualizations.