# bigheadmax/extraction-features-avec-spark architecture

> Architecture diagram of the GitHub repository bigheadmax/extraction-features-avec-spark, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bigheadmax/extraction-features-avec-spark
- Repository: https://github.com/bigheadmax/extraction-features-avec-spark

## Overview

##### Main Components
- Data Source & Storage:
- The Fruits-360 dataset that comprises thousands of images.
- A dedicated S3 bucket which holds the raw images, intermediate logs, Spark job outputs (e.g., the CSV file with PCA features), and even the notebooks (using JupyterHub persistence).
- Processing Engines:
- Local environment for testing (represented by “01 - Traitements (local).ipynb”) that runs Spark jobs in a local (WSL) environment.
- AWS EMR cluster for scalable production processing (captured by “02 - Traitements (AWS-EMR).ipynb”), with a clear distinction between the Master (Driver) and Worker nodes.
- Spark Framework:
- That the notebooks (Jupyter Notebooks) are the entry points for submitting Spark jobs.
- That the Spark framework enables both local and distributed (cloud) processing.
- AWS Infrastructure:
- The EMR cluster configuration: Include the cluster’s master node (driver) and the worker nodes.
- Bootstrap actions for dependency installation on the EMR nodes.
- The AWS CLI configuration and S3 interactions (bucket creation, file sync, S3 access policies).
- Networking elements such as VPC, security groups, and SSH tunneling (for accessing JupyterHub running on the EMR master node).

##### Relationships and Data Flows
- Data Flow:
- Two-step processing:
○ Feature extraction using transfer learning by a pretrained model.
○ Dimensionality reduction via PCA.
- Control Flow:
- Differentiation between the local execution for testing and cloud-based execution through the notebooks.
- External Interactions:
- Demonstrate that AWS CLI and IAM roles manage access and connectivity between EMR and S3.
- How JupyterHub is accessed via an SSH tunnel with tools like foxyproxy.

##### Architectural Patterns / Design Principles
- Separation of Concerns:
- Divide processing tasks clearly between the feature extraction and PCA reduction stages.
- Isolate local testing from cloud production by showing two distinct execution environments.
- Scalability:
- That the production environment (AWS EMR) is designed for high performance with multiple worker nodes.
- That bootstrap actions and logging configurations ensure that dependency management and fault-tracing are in place.
- Cloud Best Practices:
- Components like IAM roles, security groups, and S3 bucket policies to indicate secure, reproducible deployments.

##### Relevant Technologies, Frameworks, and Tools
- Apache Spark (both local and on AWS EMR) for distributed data processing.
- AWS EMR for scalable cloud-based Spark processing.
- S3 for managing input images, job logs, and output results.
- Jupyter Notebook/JupyterHub for interacting with the Spark applications.
- Transfer learning frameworks (e.g., using Keras or TensorFlow if applicable) for image feature extraction.
- PCA as a statistical technique for dimensionality reduction.
- AWS CLI and associated AWS services (IAM, EC2, VPC) for cluster management and file storage.
- WSL (Windows Subsystem for Linux) for local Spark testing on Windows.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Storage
    S3["S3 Bucket"]:::storage

    %% Project Documentation
    DOC["Project Documentation"]:::doc

    %% Local Environment Subgraph
    subgraph "Local Environment"
        L_NB["Local Test Environment"]:::local
        L_FE["Feature Extraction (Transfer Learning)"]:::process
        L_PCA["PCA Dimensionality Reduction"]:::process
    end

    %% AWS EMR Production Environment Subgraph
    subgraph "AWS EMR Production Environment"
        EMR_Master["AWS EMR Master (Driver)"]:::emr
        EMR_Worker["AWS EMR Worker(s)"]:::emr
        EMR_FE["Feature Extraction (Transfer Learning)"]:::process
        EMR_PCA["PCA Dimensionality Reduction"]:::process
        JH["JupyterHub Interface"]:::jupyter
        AWS_INF["AWS Infrastructure (CLI, IAM, VPC, Security Groups)"]:::awsinfra
    end

    %% Client using SSH tunnel for JupyterHub access
    Client["Client Machine (SSH with foxyproxy)"]:::client

    %% Data Flow for Local Environment
    S3 -->|"ReadImages"| L_NB
    L_NB -->|"RunSparkJob"| L_FE
    L_FE -->|"ProcessData"| L_PCA
    L_PCA -->|"SaveCSV"| S3

    %% Data Flow for AWS EMR Production Environment
    S3 -->|"ReadImages"| EMR_Master
    EMR_Master -->|"DistributeJob"| EMR_Worker
    EMR_Master -->|"RunSparkJob"| EMR_FE
    EMR_FE -->|"ProcessData"| EMR_PCA
    EMR_PCA -->|"SaveCSV"| S3

    %% AWS Infrastructure interactions
    AWS_INF -->|"Manages"| EMR_Master
    JH -->|"Access"| EMR_Master
    Client -->|"SSH_Tunnel"| JH

    %% Control Flow
    L_NB -->|"POCValidated"| EMR_Master

    %% Documentation links to both environments
    DOC --- L_NB
    DOC --- EMR_Master

    %% Click Events
    click L_NB "https://github.com/bigheadmax/extraction-features-avec-spark/blob/main/01 - Traitements (local).ipynb"
    click EMR_Master "https://github.com/bigheadmax/extraction-features-avec-spark/blob/main/02 - Traitements (AWS-EMR).ipynb"
    click DOC "https://github.com/bigheadmax/extraction-features-avec-spark/blob/main/readme.md"

    %% Styles
    classDef storage fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef local fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef emr fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef process fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef awsinfra fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef jupyter fill:#d1ecf1,stroke:#0c5460,stroke-width:2px;
    classDef client fill:#f5c6cb,stroke:#721c24,stroke-width:2px;
    classDef doc fill:#fefefe,stroke:#6c757d,stroke-width:2px;
```
