# aycignl/peak architecture

> Architecture diagram of the GitHub repository aycignl/peak, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aycignl/peak
- Repository: https://github.com/aycignl/peak

## Overview

##### Project Type and Purpose
- PEAK is an offline ML/data-processing pipeline for “Explainable Privacy Assistant.”
- It ingests labeled image data, auto-tags images, extracts topics, trains a classifier, and generates human-readable explanations.
- Not a web service: focus is on batch processing stages, data stores, and occasional external APIs (e.g. Clarifai).

##### File Structure
Top-level components:
- LICENSE.md, README.md
- classify_images.ipynb (Stage 4)
- generate_tags.py (Stage 2)
- extract_topics.ipynb (Stage 3)
- generate_explanations.ipynb (Stage 5)
- dataset_dir/ (raw labeled CSVs pointing to images)
- pickle_dir/ (intermediate model inputs & outputs as .pickle)
Key observations:
- Data lives in filesystem directories (dataset_dir, pickle_dir).
- Processing happens in Jupyter notebooks and one Python script.
- External tag-generation tool (e.g. Clarifai) may be invoked in generate_tags.py.

##### Extract Insights from README
- Five sequential stages, each building on the prior stage’s outputs.
- The README’s PNG (`peak_system.png`) shows a high-level flow. We will expand on it by adding data stores, tech stack details, and control flow.

##### Main Components & Their Interactions

A. Data Ingestion (Stage 1)
- Component: “Raw Image Dataset”
- Inputs: dataset_dir/df_train.csv, df_test.csv (plus image files)
- Data Store: “Raw CSV + JPG Store”

B. Tag Generation (Stage 2)
- Component: `generate_tags.py`
- Action: Calls Clarifai API or local tagger → produces plain-language tags
- Output Store: `dataset_dir/train_df_shapley.csv` & `test_df_shapley.csv`

C. Topic Extraction (Stage 3)
- Component: `extract_topics.ipynb`
- Action: Uses NLP/topic-modeling library (e.g. gensim or sklearn LDA)
- Inputs: tagged CSVs
- Output Store: intermediate topic distributions (could be pickled or saved in CSV)

D. Image Classification (Stage 4)
- Component: `classify_images.ipynb`
- Action: Trains tree-based classifier (e.g. RandomForest, XGBoost)
- Computes Shapley values per topic contribution
- Inputs: topic distributions + labels
- Output Store: pickle_dir/*.pickle (train_input.pickle, train_label.pickle, test_*.pickle)

E. Explanation Generation (Stage 5)
- Component: `generate_explanations.ipynb`
- Action: Defines explanation categories (Dominant, Opposing, Collaborative, Weak) based on Shapley contributions
- Output: final human-readable texts & visuals (could be notebooks or exported files)

F. (Optional) User Interaction / Reporting
- Component: “Report Generator” or “Notebook Outputs”
- Shows how the user reviews explanations.

##### Architectural Patterns & Design Principles
- Linear data-pipeline / ETL style workflow.
- Separation of concerns: each notebook/script handles one logical stage.
- Data versioning via file directories (dataset_dir vs. pickle_dir).

##### Relevant Technologies & Libraries
- Python 3.x, Jupyter Notebooks
- Clarifai (external API) or any image tagger
- NLP/topic modeling: gensim, scikit-learn LDA
- ML classification: scikit-learn, XGBoost, tree-based models
- SHAP (for Shapley value computation)
- pandas (for CSV handling), pickle (for serializing intermediate data)

##### Details
- Don’t omit the distinction between raw CSVs (df_train.csv) and tagged CSVs (train_df_shapley.csv).
- That classify_images outputs both model artifacts and Shapley contributions (to pickle_dir).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Store for Raw Images
    RawCSV[(Raw CSV + JPG Store)]:::datastore
    click RawCSV "https://github.com/aycignl/peak/tree/main/dataset_dir/"

    %% External Service
    Clarifai(("Clarifai API")):::external

    %% Stage 2: Tag Generation
    GenerateTags["generate_tags.py"]:::process
    click GenerateTags "https://github.com/aycignl/peak/blob/main/generate_tags.py"

    %% Tagged CSV Outputs
    TrainTaggedCSV[(train_df_shapley.csv)]:::datastore
    click TrainTaggedCSV "https://github.com/aycignl/peak/blob/main/dataset_dir/train_df_shapley.csv"
    TestTaggedCSV[(test_df_shapley.csv)]:::datastore
    click TestTaggedCSV "https://github.com/aycignl/peak/blob/main/dataset_dir/test_df_shapley.csv"

    %% Stage 3: Topic Extraction
    ExtractTopics["extract_topics.ipynb"]:::process
    click ExtractTopics "https://github.com/aycignl/peak/blob/main/extract_topics.ipynb"

    %% Topic Distributions Store
    TopicStore[(Topic Distribution Store)]:::datastore

    %% Stage 4: Image Classification
    Classify["classify_images.ipynb"]:::process
    click Classify "https://github.com/aycignl/peak/blob/main/classify_images.ipynb"

    %% Pickle Store Outputs
    TrainInput[(train_input.pickle)]:::datastore
    click TrainInput "https://github.com/aycignl/peak/blob/main/pickle_dir/train_input.pickle"
    TrainLabel[(train_label.pickle)]:::datastore
    click TrainLabel "https://github.com/aycignl/peak/blob/main/pickle_dir/train_label.pickle"
    TestInput[(test_input.pickle)]:::datastore
    click TestInput "https://github.com/aycignl/peak/blob/main/pickle_dir/test_input.pickle"
    TestLabel[(test_label.pickle)]:::datastore
    click TestLabel "https://github.com/aycignl/peak/blob/main/pickle_dir/test_label.pickle"

    %% Stage 5: Explanation Generation
    Explain["generate_explanations.ipynb"]:::process
    click Explain "https://github.com/aycignl/peak/blob/main/generate_explanations.ipynb"

    %% Final Explanations Output
    ExplanationOutputs["Final Explanations & Visuals"]:::datastore

    %% Data Flow
    RawCSV --> GenerateTags
    Clarifai --> GenerateTags
    GenerateTags --> TrainTaggedCSV
    GenerateTags --> TestTaggedCSV
    TrainTaggedCSV --> ExtractTopics
    TestTaggedCSV --> ExtractTopics
    ExtractTopics --> TopicStore
    TopicStore --> Classify
    Classify --> TrainInput
    Classify --> TrainLabel
    Classify --> TestInput
    Classify --> TestLabel
    TrainInput --> Explain
    TrainLabel --> Explain
    TestInput --> Explain
    TestLabel --> Explain
    Explain --> ExplanationOutputs

    %% Styles
    classDef process fill:#9f6,stroke:#333,stroke-width:1px
    classDef datastore fill:#6cf,stroke:#333,stroke-width:1px
    classDef external fill:#f96,stroke:#333,stroke-width:1px
```
