How anshuman1823/pca_and_lloyds works
Project type and purpose
• This is a standalone Python data-analysis pipeline.
• It implements two core algorithms from scratch: PCA on MNIST and Lloyd’s k-means clustering on 2D data.
File structure
• Top-level scripts/modules:
– main_file.py: orchestrator that imports and calls all functions.
– PCA.py: PCA implementation (covariance computation, eigen decomposition, reconstruction).
– Lloyds.py: Lloyd’s k-means algorithm (assignment, update, convergence).
– Helpers.py: plotting and visualization utilities.
– datasets/: raw input data (train.csv, test.csv, MNIST parquet files).
– output_images/: directory for generated charts and reconstructed image grids.
• Configuration: No external build or deployment scripts—pure Python scripts executed locally.
• Artifacts: Report.pdf, output image files.
Extract architecture insights from the README
• Data preprocessing: flattening MNIST images to 784-dimensional vectors, subsampling 1000 images (100 per digit).
• Two pipeline branches:
– PCA branch: variance explained → dimensionality selection → reconstruction.
– Clustering branch: Lloyd’s algorithm runs for k={2,3,4,5}, random initializations and Voronoi plots.
• Downstream tasks: classification recommendation, algorithm limitations.
1. Data Sources
• datasets/train.csv, datasets/test.csv
• mnist_parquet/train-00000… .parquet, test-00000… .parquet
2. Data Loader & Preprocessor (in main_file.py + Helpers.py)
• Reads CSV/parquet, flattens images, subsamples data
3. PCA Module (PCA.py)
• Covariance matrix → eigenvalues/vectors → dimensionality reduction → image reconstruction
4. Clustering Module (Lloyds.py)
• Lloyd’s algorithm loop → cluster assignments → center updates → convergence
5. Visualization & Reporting (Helpers.py + main_file.py)
6. Outputs
• output_images/ (PNGs of each visualization)
• Report.pdf (final summary)
– Data Sources → Data Loader & Preprocessor
– Preprocessor → PCA Module
– Preprocessor → Clustering Module
– PCA Module → Visualization & Reporting
– Clustering Module → Visualization & Reporting
– Visualization & Reporting → Outputs (output_images folder + PDF)
c. Architectural patterns & design principles
• Modularity: each algorithm lives in its own module (separation of concerns).
• Reusability: Helpers.py contains common plotting functions used by both PCA and clustering branches.
• Single entrypoint: main_file.py orchestrates the end-to-end flow.
d. Key technologies & libraries
• Python 3.x
• NumPy (covariance, eigen decomposition, vector operations)
• Pandas (CSV/parquet I/O, DataFrame handling)
• Matplotlib / Seaborn (visualizations)
• Pyarrow or fastparquet (for parquet reading, if used)
• Standard library (os, argparse, etc.)