How nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit works
Project Type and Purpose
• This is a full-stack, open-source data preprocessing toolkit (“AutoEDA”) that automates CSV loading, cleaning, feature engineering, and exposes these capabilities via a REST API and a React/Vite UI.
File-Structure Analysis
• Top-level:
• autoeda/ → core Python library (feature scaling, outlier detection, etc.)
• backend/ → FastAPI service (app.py, router, services, config, uploaded_files)
• frontend/ → React + Vite UI (components, assets, config)
• notebooks/ → exploratory Jupyter notebooks (reference EDA workflows)
• unit_tests/ → Python tests for outlier detection, etc.
• Deployment: docker-compose.yml at root and in backend/ orchestrate “frontend” and “backend” containers.
Key Architectural Patterns
• Layered architecture in backend:
• Router layer (FastAPI endpoints)
• Service/runner layer (invokes autoeda modules)
• Utility & config modules (database.js, .env)
• Persistence: local file storage (uploaded_files, output/) or external DB via database.js
• Frontend: component-based React UI with a theme/provider layer, and API integration layer (utils.js).
a. User Clients (Browser)
b. Frontend Container (React/Vite)
• UI components: FileUpload, Dashboard, Charts
• API client (utils.js)
c. API Gateway / Reverse Proxy (e.g., Vite dev server or Docker network)
d. Backend Container (FastAPI service)
• Routers (file_router.py)
• Services (runner.py)
• autoeda library modules
• Config & Database connector (database.js)
• File storage (uploaded_files/, output/)
e. Persistence Layer
• Flat files (CSV, JSON summaries) or optional DB
f. External Services
• Docker and Docker Compose for orchestration
• (Optional) Cloud storage or ML model endpoints
Relationships and Interactions
1. Browser → POST /upload → API Gateway → file_router
2. file_router → runner service → autoeda modules → data cleaning & feature engineering
3. runner → writes results to output/ and/or DB → returns JSON response
4. Browser ← GET /results → receives processed data → renders in Dashboard
5. Browser → GET /download → exports CSV/JSON
• Deployment flow: docker-compose spins up two containers on a shared network; environment variables configured via .env
Important Design Principles
• Separation of concerns: UI vs. API vs. data processing modules
• Modularity & extensibility: autoeda library plugged into backend services
• Stateless API: each request is independent; intermediate files used for persistence
• Dockerized micro-services: frontend and backend isolated
Tools, Frameworks & Libraries
• Frontend: React, Vite, Tailwind (via tailwind.config.js), eslint, PostCSS
• Backend: Python 3.x, FastAPI, Pandas, Uvicorn
• DevOps: Docker, Docker Compose
• Testing: pytest or unittest (unit_tests/outlier_test.py)
Final Checklist for the Engineer
• Major components listed above
• API endpoints (e.g., POST /upload, GET /results)
• Docker Compose network linking “frontend” and “backend” containers
• Frontend (blue), Backend (green), Storage (orange), External/Docker (gray)