How vigneshwarr3/car_price_predictor works
Project Type and Purpose
• This is a full-stack ML application that:
• Scrapes and stores raw data
• Cleans and analyzes that data
• Trains multiple regression models and picks the best one
• Exposes predictions via a FastAPI backend
• Presents a user-friendly Streamlit frontend
• Is containerized with Docker and deployed on a private server
File Structure
• Top-level ML components (Codes/ notebooks):
• scrapeCarData.ipynb → data collection (Selenium + BeautifulSoup)
• clean.ipynb → data preprocessing (Pandas)
• EDA.ipynb → exploratory analysis (Matplotlib/Seaborn)
• testing_models.ipynb → model evaluation (scikit-learn)
• Data folder: raw and intermediate files (all_urls.txt, car_data.zip, columns.txt, etc.)
• API layer: api.py (FastAPI with key/token auth)
• Frontend: Home.py plus pages/1_EDA.py, 2_Model Comparison.py, 3_Predict Price.py (Streamlit multipage)
• Deployment: Dockerfile, requirements.txt
README for Additional Insights
• The README confirms the ML pipeline stages, tech stack (Selenium, BeautifulSoup, Pandas, scikit-learn, FastAPI, Streamlit, Docker), and deployment target (private server).
Main Components & Interactions
a. Data Acquisition
• External Service: Carvana website
• Scraper Service: Selenium + BeautifulSoup
• Raw Data Store: local files (data/*.txt, data/car_data.zip)
b. Data Processing
• Cleaning Pipeline: Pandas (Codes/clean.ipynb)
• Clean Data Store: processed CSV/Parquet (you may bundle under “Processed Data”)
c. Model Training & Selection
• EDA & Feature Engineering (Codes/EDA.ipynb)
• Model Evaluation (testing_models.ipynb)
• Model Artifact: serialized best model (pickle/joblib)
d. Backend API
• FastAPI app (api.py)
• Authentication layer (API key / token)
• Endpoints:
– GET /vehicles → fetch cleaned data
– POST /predict → return price prediction
• Containerization: Dockerfile
e. Frontend Interface
• Streamlit app (Home.py + pages/*.py)
• Pages for EDA, Model Comparison, Price Prediction
• Consumes FastAPI endpoints
Architectural Patterns & Design Principles
• Separation of concerns: scraper, ETL, model training, API, UI all decoupled
• Pipeline pattern for ML: data ingestion → preprocessing → training → serving
• Micro-service style for prediction API
• Containerization to ensure environment consistency
Essential Elements
• Names & labels for each component (e.g., “Model Trainer (scikit-learn)”)