How bigheadmax/extraction-features-avec-spark works
Main Components
• Data Source & Storage:
• The Fruits-360 dataset that comprises thousands of images.
• A dedicated S3 bucket which holds the raw images, intermediate logs, Spark job outputs (e.g., the CSV file with PCA features), and even the notebooks (using JupyterHub persistence).
• Processing Engines:
• Local environment for testing (represented by “01 - Traitements (local).ipynb”) that runs Spark jobs in a local (WSL) environment.
• AWS EMR cluster for scalable production processing (captured by “02 - Traitements (AWS-EMR).ipynb”), with a clear distinction between the Master (Driver) and Worker nodes.
• Spark Framework:
• That the notebooks (Jupyter Notebooks) are the entry points for submitting Spark jobs.
• That the Spark framework enables both local and distributed (cloud) processing.
• AWS Infrastructure:
• The EMR cluster configuration: Include the cluster’s master node (driver) and the worker nodes.
• Bootstrap actions for dependency installation on the EMR nodes.
• The AWS CLI configuration and S3 interactions (bucket creation, file sync, S3 access policies).
• Networking elements such as VPC, security groups, and SSH tunneling (for accessing JupyterHub running on the EMR master node).
Relationships and Data Flows
• Data Flow:
• Two-step processing:
○ Feature extraction using transfer learning by a pretrained model.
○ Dimensionality reduction via PCA.
• Control Flow:
• Differentiation between the local execution for testing and cloud-based execution through the notebooks.
• External Interactions:
• Demonstrate that AWS CLI and IAM roles manage access and connectivity between EMR and S3.
• How JupyterHub is accessed via an SSH tunnel with tools like foxyproxy.
Architectural Patterns / Design Principles
• Separation of Concerns:
• Divide processing tasks clearly between the feature extraction and PCA reduction stages.
• Isolate local testing from cloud production by showing two distinct execution environments.
• Scalability:
• That the production environment (AWS EMR) is designed for high performance with multiple worker nodes.
• That bootstrap actions and logging configurations ensure that dependency management and fault-tracing are in place.
• Cloud Best Practices:
• Components like IAM roles, security groups, and S3 bucket policies to indicate secure, reproducible deployments.
Relevant Technologies, Frameworks, and Tools
• Apache Spark (both local and on AWS EMR) for distributed data processing.
• AWS EMR for scalable cloud-based Spark processing.
• S3 for managing input images, job logs, and output results.
• Jupyter Notebook/JupyterHub for interacting with the Spark applications.
• Transfer learning frameworks (e.g., using Keras or TensorFlow if applicable) for image feature extraction.
• PCA as a statistical technique for dimensionality reduction.
• AWS CLI and associated AWS services (IAM, EC2, VPC) for cluster management and file storage.
• WSL (Windows Subsystem for Linux) for local Spark testing on Windows.