# izavits/msc_datascience architecture

> Architecture diagram of the GitHub repository izavits/msc_datascience, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/izavits/msc_datascience
- Repository: https://github.com/izavits/msc_datascience

## Overview

##### Project type and purpose
- This is an educational Data Science “example gallery” repository for a Big Data Mining class. It contains Jupyter notebooks and a few standalone Python scripts illustrating common data-science workflows: EDA, preprocessing, classification, clustering, frequent-itemset mining, outlier detection, text mining, etc.

##### File structure
- Top level:
- README.md, requirements.txt, LICENSE, .gitignore
- helper_funcs.py and several standalone .py scripts (e.g., apriori_algorithm.py)
- A directory named data/ with CSV, JSON, DB files
- Dozens of .ipynb notebooks, each demonstrating a particular technique
- No “frontend” vs. “backend” split: it’s a single monolithic codebase. The notebooks import helper_funcs.py, read from data/, plot results, etc.

##### README for additional insights
- Confirms the library dependencies and setup steps (virtualenv + Jupyter kernel)
- Enumerates the categories of examples: preprocessing, classification, clustering, itemsets, outliers, text

## Diagram (Mermaid)

```mermaid
graph TD
    %% Environment & Dependencies
    subgraph "Environment & Dependencies"
        ENV_REQ["requirements.txt"]:::env
        ENV_GIT[".gitignore"]:::env
        ENV_LIC["LICENSE"]:::env
        ENV_README["README.md"]:::env
    end

    %% Data Storage
    subgraph "Data Storage"
        DATA_DIR["data/"]:::storage
    end

    %% Utility Module
    subgraph "Utility Module"
        UTIL_HELP["helper_funcs.py"]:::module
        UTIL_API["apriori_algorithm.py"]:::module
    end

    %% Notebook Workspace
    subgraph "Notebook Workspace"
        subgraph "Exploratory Data Analysis"
            NB_EDA1["EDA_example.ipynb"]:::notebook
            NB_EDA2["EDA_example_with_sweetviz.ipynb"]:::notebook
            NB_EDA3["correlation_example.ipynb"]:::notebook
            NB_EDA4["data_playground.ipynb"]:::notebook
        end
        subgraph "Preprocessing & Feature Engineering"
            NB_PRE1["binning_example.ipynb"]:::notebook
            NB_PRE2["discretization_with_k_means.ipynb"]:::notebook
            NB_PRE3["normalization.ipynb"]:::notebook
            NB_PRE4["ordinal-encoding.ipynb"]:::notebook
            NB_PRE5["feature_selection.ipynb"]:::notebook
            NB_PRE6["pca_examples.ipynb"]:::notebook
        end
        subgraph "Classification & Recommendation"
            NB_CLASS1["decision_trees.ipynb"]:::notebook
            NB_CLASS2["knn_example.ipynb"]:::notebook
            NB_CLASS3["svm_example.ipynb"]:::notebook
            NB_CLASS4["bayesian_net_example.ipynb"]:::notebook
            NB_CLASS5["movie_recommendation.ipynb"]:::notebook
            NB_CLASS6["text_nb_classification.ipynb"]:::notebook
            NB_CLASS7["text_zip_classification.ipynb"]:::notebook
        end
        subgraph "Clustering"
            NB_CL1["clustering_exampes.ipynb"]:::notebook
            NB_CL2["clustering_silhouette_coeff.ipynb"]:::notebook
        end
        subgraph "Frequent Itemset Mining"
            NB_FI1["apriori_example.ipynb"]:::notebook
        end
        subgraph "Outlier Detection"
            NB_OUT1["outliers.ipynb"]:::notebook
        end
        subgraph "Text Mining & NLP"
            NB_TX1["language_models.ipynb"]:::notebook
            NB_TX2["text_topic_modeling.ipynb"]:::notebook
            NB_TX3["word2vec.ipynb"]:::notebook
        end
    end

    %% Visualization/Reporting
    subgraph "Visualization/Reporting Layer"
        VIS["SWEETVIZ_REPORT.html"]:::output
    end

    %% Connections
    ENV_REQ -->|"provides environment"| NB_EDA1
    ENV_REQ -->|"provides environment"| NB_PRE1
    ENV_REQ -->|"provides environment"| NB_CLASS1
    ENV_REQ -->|"provides environment"| NB_CL1
    ENV_REQ -->|"provides environment"| NB_FI1
    ENV_REQ -->|"provides environment"| NB_OUT1
    ENV_REQ -->|"provides environment"| NB_TX1

    DATA_DIR -->|"raw data"| NB_EDA1
    DATA_DIR -->|"raw data"| NB_PRE1
    DATA_DIR -->|"raw data"| NB_CLASS1

    UTIL_HELP -->|"shared routines"| NB_EDA1
    UTIL_HELP -->|"shared routines"| NB_PRE1
    UTIL_HELP -->|"shared routines"| NB_CLASS1
    UTIL_HELP -->|"shared routines"| NB_CL1
    UTIL_HELP -->|"shared routines"| NB_OUT1
    UTIL_HELP -->|"shared routines"| NB_TX1

    UTIL_API -->|"apriori functions"| NB_FI1

    NB_EDA4 -->|"cleaned data"| NB_PRE1
    NB_PRE6 -->|"features"| NB_CLASS1
    NB_CLASS7 -->|"predictions"| NB_CL1
    NB_FI1 -->|"itemsets"| NB_CLASS1
    NB_OUT1 -->|"outliers info"| NB_CLASS1
    NB_TX3 -->|"embeddings"| NB_CLASS1

    NB_CLASS1 -->|"results"| VIS
    NB_CL2 -->|"clusters"| VIS
    NB_FI1 -->|"patterns"| VIS
    NB_OUT1 -->|"anomalies"| VIS
    NB_TX2 -->|"topics"| VIS

    %% Click Events
    click ENV_REQ "https://github.com/izavits/msc_datascience/blob/master/requirements.txt"
    click ENV_GIT "https://github.com/izavits/msc_datascience/blob/master/.gitignore"
    click ENV_LIC "https://github.com/izavits/msc_datascience/tree/master/LICENSE"
    click ENV_README "https://github.com/izavits/msc_datascience/blob/master/README.md"
    click DATA_DIR "https://github.com/izavits/msc_datascience/tree/master/data/"
    click UTIL_HELP "https://github.com/izavits/msc_datascience/blob/master/helper_funcs.py"
    click UTIL_API "https://github.com/izavits/msc_datascience/blob/master/apriori_algorithm.py"
    click NB_EDA1 "https://github.com/izavits/msc_datascience/blob/master/EDA_example.ipynb"
    click NB_EDA2 "https://github.com/izavits/msc_datascience/blob/master/EDA_example_with_sweetviz.ipynb"
    click NB_EDA3 "https://github.com/izavits/msc_datascience/blob/master/correlation_example.ipynb"
    click NB_EDA4 "https://github.com/izavits/msc_datascience/blob/master/data_playground.ipynb"
    click NB_PRE1 "https://github.com/izavits/msc_datascience/blob/master/binning_example.ipynb"
    click NB_PRE2 "https://github.com/izavits/msc_datascience/blob/master/discretization_with_k_means.ipynb"
    click NB_PRE3 "https://github.com/izavits/msc_datascience/blob/master/normalization.ipynb"
    click NB_PRE4 "https://github.com/izavits/msc_datascience/blob/master/ordinal-encoding.ipynb"
    click NB_PRE5 "https://github.com/izavits/msc_datascience/blob/master/feature_selection.ipynb"
    click NB_PRE6 "https://github.com/izavits/msc_datascience/blob/master/pca_examples.ipynb"
    click NB_CLASS1 "https://github.com/izavits/msc_datascience/blob/master/decision_trees.ipynb"
    click NB_CLASS2 "https://github.com/izavits/msc_datascience/blob/master/knn_example.ipynb"
    click NB_CLASS3 "https://github.com/izavits/msc_datascience/blob/master/svm_example.ipynb"
    click NB_CLASS4 "https://github.com/izavits/msc_datascience/blob/master/bayesian_net_example.ipynb"
    click NB_CLASS5 "https://github.com/izavits/msc_datascience/blob/master/movie_recommendation.ipynb"
    click NB_CLASS6 "https://github.com/izavits/msc_datascience/blob/master/text_nb_classification.ipynb"
    click NB_CLASS7 "https://github.com/izavits/msc_datascience/blob/master/text_zip_classification.ipynb"
    click NB_CL1 "https://github.com/izavits/msc_datascience/blob/master/clustering_exampes.ipynb"
    click NB_CL2 "https://github.com/izavits/msc_datascience/blob/master/clustering_silhouette_coeff.ipynb"
    click NB_FI1 "https://github.com/izavits/msc_datascience/blob/master/apriori_example.ipynb"
    click NB_OUT1 "https://github.com/izavits/msc_datascience/blob/master/outliers.ipynb"
    click NB_TX1 "https://github.com/izavits/msc_datascience/blob/master/language_models.ipynb"
    click NB_TX2 "https://github.com/izavits/msc_datascience/blob/master/text_topic_modeling.ipynb"
    click NB_TX3 "https://github.com/izavits/msc_datascience/blob/master/word2vec.ipynb"
    click VIS "https://github.com/izavits/msc_datascience/blob/master/SWEETVIZ_REPORT.html"

    %% Styles
    classDef env fill:#D6EAF8,stroke:#1F618D,color:#1F618D;
    classDef storage fill:#D4EFDF,stroke:#196F3D,color:#196F3D;
    classDef module fill:#EAFAF1,stroke:#28B463,color:#28B463;
    classDef notebook fill:#FCF3CF,stroke:#B7950B,color:#B7950B;
    classDef output fill:#FDEDEC,stroke:#C0392B,color:#C0392B
```
