How janitabose13/personalised_skincare_recommendation_system works
1. This repository is primarily an offline data pipeline for a personalized skincare recommendation system. The visible codebase is focused on collecting skincare data, cleaning it, and preparing a unified dataset for later recommendation logic.
2. The ingestion layer consists of three scraper scripts in Code/: nykaa_full_scraper.py, incidecoder_scraper.py, and ewg_scraper.py. These pull data from Nykaa, INCIDecoder, and EWG respectively, producing raw CSV extracts in data/raw/.
3. data/raw/ is the repository’s source-data boundary. It contains the original datasets for:
- nykaa_products.csv
- nykaa_reviews.csv
- incidecoder_ingredients.csv
- ewg_ingredients.csv
This suggests the system is designed to fuse product catalog data, reviews, ingredient metadata, and ingredient safety information.
4. The first transformation stage appears to be Code/00_clean_nykaa.py. Based on the naming and dataset layout, this script likely standardizes and cleans Nykaa product/review data before any cross-source integration happens.
5. The next stage is Code/01_data_preparation.py, which likely performs feature engineering and dataset merging across the three sources. This is the main preparation boundary where raw product, ingredient, and safety data become a model-ready dataset.
6. data/processed/ is the curated output layer. It is meant to hold cleaned, transformed, and possibly joined datasets that downstream recommendation or analysis code can consume without needing to touch the raw scrapes.
7. The central domain model is ingredient-centric skincare analysis. The repository is set up to support recommendations based on product attributes, ingredient composition, ingredient safety, and user-review signals rather than only product popularity.
8. The architecture is batch-oriented rather than service-oriented. There is no visible API, UI, or database layer in the file tree; execution appears to be driven by standalone Python scripts run in sequence to populate the data folders.
9. Python is the main runtime, with data-processing dependencies implied by the README such as pandas and NumPy, and likely scikit-learn for later recommendation work. Those libraries matter architecturally because they indicate a tabular-ML pipeline rather than a real-time transactional system.
10. The key external boundaries are the three source websites and the local filesystem. Data flows one way: scrape external sources → land in data/raw/ → clean and prepare in Code/ → write curated artifacts to data/processed/ for future modeling or recommendation generation.
11. From an architecture-diagram perspective, the important subsystems are: source scrapers, cleaning/preparation scripts, raw data storage, processed data storage, and the future recommendation engine implied by the README but not present in the visible tree.
Main entry points: 01_data_preparation.py Prepare data · nykaa_full_scraper.py Nykaa scraper · 00_clean_nykaa.py Clean Nykaa · incidecoder_scraper.py INCIDecoder scraper
Components 14
Pipeline code
Standalone Python scripts that scrape, clean, and prepare skincare data.
- Nykaa scraper · Collects product and review data from Nykaa.
Code/nykaa_full_scraper.py - INCIDecoder scraper · Extracts ingredient data from INCIDecoder.
Code/incidecoder_scraper.py - EWG scraper · Extracts ingredient safety data from EWG.
Code/ewg_scraper.py - Clean Nykaa · Standardizes and cleans scraped Nykaa data before integration.
Code/00_clean_nykaa.py - Prepare data · Merges and engineers features across sources into a unified dataset.
Code/01_data_preparation.py
Raw data
Source-data boundary containing original scraped CSV extracts.
- Nykaa products · Raw product catalog extracted from Nykaa.
data/raw/nykaa_products.csv - Nykaa reviews · Raw customer review data extracted from Nykaa.
data/raw/nykaa_reviews.csv - INCIDecoder ingredients · Raw ingredient metadata extracted from INCIDecoder.
data/raw/incidecoder_ingredients.csv - EWG ingredients · Raw ingredient safety data extracted from EWG.
data/raw/ewg_ingredients.csv
Processed data
Curated outputs produced by the pipeline for downstream use.
- Curated datasets · Cleaned, transformed, and joined datasets ready for downstream consumption.
data/processed
External sources
Upstream websites scraped by the ingestion layer.
- Nykaa · Source website for products and reviews.
- INCIDecoder · Source website for ingredient information.
- EWG · Source website for ingredient safety information.
Downstream modeling
Implied recommendation layer referenced by the repository purpose.
- Recommender · Implied downstream recommendation logic using the prepared skincare dataset.
Connections 14
- Nykaa → Nykaa scraper: scrape · Pull product and review data.
- INCIDecoder → INCIDecoder scraper: scrape · Pull ingredient metadata.
- EWG → EWG scraper: scrape · Pull ingredient safety data.
- Nykaa scraper → Nykaa products: land · Write raw product extracts.
- Nykaa scraper → Nykaa reviews: land · Write raw review extracts.
- INCIDecoder scraper → INCIDecoder ingredients: land · Write raw ingredient extracts.
- EWG scraper → EWG ingredients: land · Write raw safety extracts.
- Nykaa products → Clean Nykaa: input · Clean scraped product catalog data.
- Nykaa reviews → Clean Nykaa: input · Clean scraped review data.
- Clean Nykaa → Prepare data: cleaned nykaa · Provide standardized Nykaa tables to the integration stage.
- INCIDecoder ingredients → Prepare data: ingredient data · Provide ingredient metadata for joining and feature creation.
- EWG ingredients → Prepare data: safety data · Provide safety annotations for joining and feature creation.
- Prepare data → Curated datasets: write · Persist curated datasets for downstream consumption.
- Curated datasets → Recommender: feeds · Supply model-ready data to the implied recommendation layer.