How zzhenjie01/dsa3101-ay2425s2-team6 works
Project type and purpose
• This is a full-stack web application for automating ESG (Environmental, Social, Governance) data extraction, evaluation and display, supplemented by offline data-processing pipelines and machine-learning model services (ESG extraction & LSTM stock forecast).
Main architectural components
a. Frontend (React/Vite/Tailwind)
b. Backend API (Node.js + Express)
c. Data stores
• PostgreSQL (relational)
• MongoDB (user profiles & click data)
• Elasticsearch (ESG report & external ESG data index)
d. Machine‐learning services
• ESG data extraction model (FastAPI/Uvicorn at port 8000)
• Stock price forecasting model (standalone script/pipeline)
e. Data pipelines (Python/Spark scripts)
• PDF → JSON → CSV → Elasticsearch
• NewsAPI & Google Drive scrapers
• Fact‐checking pipeline
f. Container infrastructure (Docker Compose)
• MongoDB, PostgreSQL, Elasticsearch, Spark master/worker
• .env configs to wire up credentials & ports
Relationships and data flow
• User → Browser → React App (port 5173)
• React App calls Backend API (port 5000) for:
• Auth (JWT generation; uses MongoDB)
• Company & ESG data queries (reads Elasticsearch & PostgreSQL)
• Click tracking & user preferences (writes MongoDB)
• React App embeds a Chatbot UI:
• Frontend sends user messages → Backend/chatbot storage
• Backend or Frontend calls ESG extraction model endpoint (port 8000)
• Model queries Elasticsearch for raw text, extracts metrics, returns results
• Offline data pipelines ingest and process ESG reports & external data:
• Run periodically or on demand, connect to Elasticsearch and local folders
• Populate/refresh ES indexes used by Backend & Model services
• Stock forecast model ingests historical stock CSV → trains LSTM → outputs forecasts stored in PostgreSQL
Architectural patterns & design principles
• Separation of concerns:
• UI layer (React)
• API layer (Express)
• Data & ML services (Python microservices)
• Offline ETL pipelines
• Microservices style: Each model & pipeline is its own process with a clear API or script interface
• Containerization: All online services & data stores run under Docker Compose for reproducible environments
• Eventual consistency: Data pipelines update ES & PG asynchronously from the app