# nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit architecture

> Architecture diagram of the GitHub repository nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit
- Repository: https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit

## Overview

##### Project Type and Purpose
- This is a full-stack, open-source data preprocessing toolkit (“AutoEDA”) that automates CSV loading, cleaning, feature engineering, and exposes these capabilities via a REST API and a React/Vite UI.

##### File-Structure Analysis
- Top-level:
- autoeda/ → core Python library (feature scaling, outlier detection, etc.)
- backend/ → FastAPI service (app.py, router, services, config, uploaded_files)
- frontend/ → React + Vite UI (components, assets, config)
- notebooks/ → exploratory Jupyter notebooks (reference EDA workflows)
- unit_tests/ → Python tests for outlier detection, etc.
- Deployment: docker-compose.yml at root and in backend/ orchestrate “frontend” and “backend” containers.

##### Key Architectural Patterns
- Layered architecture in backend:
- Router layer (FastAPI endpoints)
- Service/runner layer (invokes autoeda modules)
- Utility & config modules (database.js, .env)
- Persistence: local file storage (uploaded_files, output/) or external DB via database.js
- Frontend: component-based React UI with a theme/provider layer, and API integration layer (utils.js).

a. User Clients (Browser)
b. Frontend Container (React/Vite)
- UI components: FileUpload, Dashboard, Charts
- API client (utils.js)
c. API Gateway / Reverse Proxy (e.g., Vite dev server or Docker network)
d. Backend Container (FastAPI service)
- Routers (file_router.py)
- Services (runner.py)
- autoeda library modules
- Config & Database connector (database.js)
- File storage (uploaded_files/, output/)
e. Persistence Layer
- Flat files (CSV, JSON summaries) or optional DB
f. External Services
- Docker and Docker Compose for orchestration
- (Optional) Cloud storage or ML model endpoints

##### Relationships and Interactions
1. Browser → POST /upload → API Gateway → file_router
2. file_router → runner service → autoeda modules → data cleaning & feature engineering
3. runner → writes results to output/ and/or DB → returns JSON response
4. Browser ← GET /results → receives processed data → renders in Dashboard
5. Browser → GET /download → exports CSV/JSON
- Deployment flow: docker-compose spins up two containers on a shared network; environment variables configured via .env

##### Important Design Principles
- Separation of concerns: UI vs. API vs. data processing modules
- Modularity & extensibility: autoeda library plugged into backend services
- Stateless API: each request is independent; intermediate files used for persistence
- Dockerized micro-services: frontend and backend isolated

##### Tools, Frameworks & Libraries
- Frontend: React, Vite, Tailwind (via tailwind.config.js), eslint, PostCSS
- Backend: Python 3.x, FastAPI, Pandas, Uvicorn
- DevOps: Docker, Docker Compose
- Testing: pytest or unittest (unit_tests/outlier_test.py)

##### Final Checklist for the Engineer
- Major components listed above
- API endpoints (e.g., POST /upload, GET /results)
- Docker Compose network linking “frontend” and “backend” containers
- Frontend (blue), Backend (green), Storage (orange), External/Docker (gray)

## Diagram (Mermaid)

```mermaid
flowchart TD
    Browser["Browser (User)"]:::external

    subgraph "Docker Compose Network"
        direction TB

        subgraph "Frontend Container"
            direction TB
            FrontendContainer["React/Vite App"]:::frontend
            FileUpload["FileUpload"]:::frontend
            Dashboard["Dashboard"]:::frontend
            ThemeProvider["Theme/Provider"]:::frontend
            Utils["API Client (utils.js)"]:::frontend
        end

        subgraph "Backend Container"
            direction TB
            BackendContainer["FastAPI Service"]:::backend
            App["app.py"]:::backend
            Router["file_router.py"]:::backend
            Runner["runner.py"]:::backend
            AutoEDA["AutoEDA Library (feat_scaling.py, outliers.py)"]:::backend
            Config["database.js"]:::backend
            Env[".env"]:::backend
        end
    end

    subgraph Persistence
        direction TB
        Uploaded["Uploaded Files Storage"]:::storage
        Output["Processed Output Storage"]:::storage
    end

    DockerCompose["Docker Compose"]:::external

    %% Flows
    Browser --> FrontendContainer
    FrontendContainer --> FileUpload
    FrontendContainer --> Dashboard
    FrontendContainer --> ThemeProvider
    FrontendContainer --> Utils

    FileUpload -->|"POST /upload"| App
    Dashboard -->|"GET /results"| App
    Dashboard -->|"GET /download"| App

    App --> Router
    Router --> Runner
    Runner --> AutoEDA
    AutoEDA --> Runner
    Runner --> Uploaded
    Runner --> Output
    Router --> App

    App --> Utils
    Utils --> Dashboard
    Dashboard --> Browser

    DockerCompose --> FrontendContainer
    DockerCompose --> BackendContainer

    %% Click events
    click FrontendContainer "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/tree/main/frontend/"
    click FileUpload "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/frontend/src/components/FileUpload.jsx"
    click Dashboard "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/frontend/src/components/Dashboard.jsx"
    click ThemeProvider "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/frontend/src/components/theme-provider.jsx"
    click Utils "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/frontend/src/lib/utils.js"
    click BackendContainer "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/tree/main/backend/"
    click App "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/backend/app.py"
    click Router "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/backend/router/file_router.py"
    click Runner "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/backend/services/runner.py"
    click Config "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/backend/config/database.js"
    click Env "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/backend/.env"
    click Uploaded "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/tree/main/backend/uploaded_files/"
    click Output "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/tree/main/backend/output/"
    click AutoEDA "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/autoeda/feat_scaling.py"
    click DockerCompose "https://github.com/nidhi-satyapriya/autoeda-automated-data-preprocessing-toolkit/blob/main/docker-compose.yml"

    %% Styles
    classDef frontend fill:#add8e6,stroke:#333,stroke-width:1px
    classDef backend fill:#90ee90,stroke:#333,stroke-width:1px
    classDef storage fill:#ffcc99,stroke:#333,stroke-width:1px
    classDef external fill:#d3d3d3,stroke:#333,stroke-width:1px
```
