# zzhenjie01/dsa3101-ay2425s2-team6 architecture

> Architecture diagram of the GitHub repository zzhenjie01/dsa3101-ay2425s2-team6, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/zzhenjie01/dsa3101-ay2425s2-team6
- Repository: https://github.com/zzhenjie01/dsa3101-ay2425s2-team6

## Overview

##### Project type and purpose
- This is a full-stack web application for automating ESG (Environmental, Social, Governance) data extraction, evaluation and display, supplemented by offline data-processing pipelines and machine-learning model services (ESG extraction & LSTM stock forecast).

##### Main architectural components
a. Frontend (React/Vite/Tailwind)
b. Backend API (Node.js + Express)
c. Data stores
- PostgreSQL (relational)
- MongoDB (user profiles & click data)
- Elasticsearch (ESG report & external ESG data index)
d. Machine‐learning services
- ESG data extraction model (FastAPI/Uvicorn at port 8000)
- Stock price forecasting model (standalone script/pipeline)
e. Data pipelines (Python/Spark scripts)
- PDF → JSON → CSV → Elasticsearch
- NewsAPI & Google Drive scrapers
- Fact‐checking pipeline
f. Container infrastructure (Docker Compose)
- MongoDB, PostgreSQL, Elasticsearch, Spark master/worker
- .env configs to wire up credentials & ports

##### Relationships and data flow
- User → Browser → React App (port 5173)
- React App calls Backend API (port 5000) for:
- Auth (JWT generation; uses MongoDB)
- Company & ESG data queries (reads Elasticsearch & PostgreSQL)
- Click tracking & user preferences (writes MongoDB)
- React App embeds a Chatbot UI:
- Frontend sends user messages → Backend/chatbot storage
- Backend or Frontend calls ESG extraction model endpoint (port 8000)
- Model queries Elasticsearch for raw text, extracts metrics, returns results
- Offline data pipelines ingest and process ESG reports & external data:
- Run periodically or on demand, connect to Elasticsearch and local folders
- Populate/refresh ES indexes used by Backend & Model services
- Stock forecast model ingests historical stock CSV → trains LSTM → outputs forecasts stored in PostgreSQL

##### Architectural patterns & design principles
- Separation of concerns:
- UI layer (React)
- API layer (Express)
- Data & ML services (Python microservices)
- Offline ETL pipelines
- Microservices style: Each model & pipeline is its own process with a clear API or script interface
- Containerization: All online services & data stores run under Docker Compose for reproducible environments
- Eventual consistency: Data pipelines update ES & PG asynchronously from the app

## Diagram (Mermaid)

```mermaid
graph TB
    %% User & Client
    subgraph "User/Client"
        User("User"):::user
        Browser("Browser"):::frontend
    end
    User --> Browser
    Browser -->|"HTTP (5173)"| ReactApp

    %% Docker Compose Network
    subgraph "Docker Compose Network"
        direction TB
        ReactApp("React/Vite (5173)"):::frontend
        ExpressAPI("Node/Express API (5000; JWT Auth)"):::api
        ESGModel("ESG Extraction Model (FastAPI/Uvicorn 8000)"):::ml
        Postgres("PostgreSQL (5678)"):::datastore
        MongoDB("MongoDB (27017)"):::datastore
        Elastic("Elasticsearch (9200)"):::datastore
        Spark("Spark Master/Worker (7077/9090)"):::pipeline
    end

    %% Data flows within Docker
    ReactApp -->|"REST (5000)"| ExpressAPI
    ExpressAPI -->|"JDBC (5678)"| Postgres
    ExpressAPI -->|"MongoDriver (27017)"| MongoDB
    ExpressAPI -->|"HTTP (9200)"| Elastic
    ExpressAPI -->|"REST (8000)"| ESGModel
    ESGModel -->|"HTTP (9200)"| Elastic

    %% Stock Forecast outside container
    StockModel("Stock Forecast Model (LSTM)"):::ml
    StockModel -->|"SQL Insert"| Postgres

    %% Offline Pipelines
    subgraph "Offline Pipelines"
        direction TB
        GDrive("GoogleDrive Scraper"):::pipeline
        PDFtoJSON("PDF→JSON"):::pipeline
        JSONtoCSV("JSON→CSV"):::pipeline
        CSVtoES("CSV→Elasticsearch"):::pipeline
        NewsScrape1("NewsAPI Scraper"):::pipeline
        FactCheck("Fact-Checking Pipeline"):::pipeline

        GDrive --> PDFtoJSON
        PDFtoJSON --> JSONtoCSV
        JSONtoCSV --> CSVtoES
        NewsScrape1 --> CSVtoES
        FactCheck --> Elastic
        CSVtoES --> Elastic
    end

    %% Click Events
    click ReactApp "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/blob/main/frontend/src/main.jsx"
    click ExpressAPI "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/blob/main/backend/src/server.js"
    click Postgres "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/blob/main/backend/src/api/models/postgresDB.js"
    click MongoDB "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/blob/main/backend/src/mongoDB.js"
    click Elastic "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/blob/main/backend/src/esgScoresHelper.js"
    click ESGModel "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/blob/main/models/esg-data-extraction-model/model-code/esg_extraction_model_endpoint.py"
    click StockModel "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/blob/main/models/stock-price-forecast-model/stocks_prediction_lstm.py"
    click PDFtoJSON "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/tree/main/data-pipelines/esg_pdf_to_json/"
    click JSONtoCSV "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/tree/main/data-pipelines/esg_json_to_csv/"
    click CSVtoES "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/tree/main/data-pipelines/esg_csv_to_elasticsearch/"
    click NewsScrape1 "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/tree/main/data-pipelines/newsapi_web_to_csv/"
    click GDrive "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/tree/main/data-pipelines/esg_gdrive_to_local/"
    click FactCheck "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/tree/main/data-pipelines/fact_check_esg_report_against_ext/"
    click Docker "https://github.com/zzhenjie01/dsa3101-ay2425s2-team6/blob/main/docker/docker-compose.yaml"

    %% Styles
    classDef user fill:#D6EAFE,stroke:#333,stroke-width:1px
    classDef frontend fill:#E0F7FF,stroke:#333,stroke-width:1px
    classDef api fill:#DFFFE0,stroke:#333,stroke-width:1px
    classDef datastore fill:#FFE0B2,stroke:#333,stroke-width:1px
    classDef ml fill:#F3E5F5,stroke:#333,stroke-width:1px
    classDef pipeline fill:#FFF9C4,stroke:#333,stroke-width:1px
    classDef docker fill:#ECECEC,stroke:#333,stroke-width:1px
```
