# ralo93/open-discourse architecture

> Architecture diagram of the GitHub repository ralo93/open-discourse, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ralo93/open-discourse
- Repository: https://github.com/ralo93/open-discourse

## Overview

##### Project type and purpose
- This is a full-stack, data-driven web platform called “Open Discourse” that:
- Uses a Python ETL pipeline to fetch, clean and load parliamentary speech data into a PostgreSQL database
- Exposes the database through a lightweight Node.js proxy service
- Serves a Next.js/React frontend for full-text search and exploration of that data
- Leverages Docker for local development and production

##### File structure
- Top-level directories:
- database: Dockerized Postgres, schema files, migration scripts (TypeScript)
- python: Data‐gathering & processing pipeline (multiple stages under od_lib)
- proxy: Node.js proxy (API layer) protecting direct DB access
- frontend: Next.js/React UI, components, pages, Chakra UI for styling
- Other artifacts:
- docker-compose.yml: orchestrates database, proxy and frontend containers
- DOCKER_SETUP.md: instructions for Docker deployment
- CI/CD workflows in .github/workflows (build, deploy, codeql)

##### README for additional insights
- The README confirms the four main pieces: database, python, proxy and frontend
- Instructions on local setup, Docker orchestration, database seeding (yarn db:update), data generation (sh build.sh)
- Links to per-module READMEs for deeper detail on each subproject

##### Detailed component breakdown (Don’t overthink—break it all out!)
- Python ETL:
- od_lib modules (01_preprocessing → 07_database)
- build.sh orchestrator
- Database:
- Dockerfile, Dockerfile.prod
- Schema scripts in database/src/model
- Migration TS scripts (dbUpdate.ts, teardown.ts)
- Proxy:
- index.ts entry point
- Configured via .env.local
- Frontend:
- Pages (index.tsx)
- Components tree (custom-inputs, result-table, react-table, etc.)
- next.config, tsconfig.json

##### Final advice
- Be very granular—show each container as a separate node
- Versioning or branch (dev vs. prod) distinctions if relevant

## Diagram (Mermaid)

```mermaid
flowchart TD
    External["External Data Sources"]:::external
    DC["Docker Compose"]:::orchestration

    subgraph "Service Containers"
        direction TB
        ETL["Python ETL Service"]:::compute
        DB["PostgreSQL Database Service"]:::database
        Proxy["Node.js Proxy"]:::compute
        Frontend["Next.js/React Frontend"]:::compute
    end

    External -->|"Ingest"| ETL
    DC --> ETL
    DC --> DB
    DC --> Proxy
    DC --> Frontend

    ETL -->|"Writes Data"| DB
    Frontend -->|"API Calls"| Proxy
    Proxy -->|"Queries"| DB

    click ETL "https://github.com/ralo93/open-discourse/blob/main/python/build.sh"
    click ETL "https://github.com/ralo93/open-discourse/tree/main/python/src/od_lib"
    click ETL "https://github.com/ralo93/open-discourse/blob/main/python/src/od_lib/definitions/path_definitions.py"
    click ETL "https://github.com/ralo93/open-discourse/tree/main/python/src/od_lib/helper_functions"
    click ETL "https://github.com/ralo93/open-discourse/blob/main/python/src/od_lib/topic_modelling/upload_data_cubes.py"
    click ETL "https://github.com/ralo93/open-discourse/blob/main/python/requirements.txt"
    click ETL "https://github.com/ralo93/open-discourse/blob/main/python/setup.sh"
    click ETL "https://github.com/ralo93/open-discourse/blob/main/python/setup.py"

    click DB "https://github.com/ralo93/open-discourse/tree/main/database/Dockerfile"
    click DB "https://github.com/ralo93/open-discourse/blob/main/database/Dockerfile.prod"
    click DB "https://github.com/ralo93/open-discourse/blob/main/database/.env.local"
    click DB "https://github.com/ralo93/open-discourse/tree/main/database/src/model"
    click DB "https://github.com/ralo93/open-discourse/blob/main/database/src/model/dbUpdate.ts"

    click Proxy "https://github.com/ralo93/open-discourse/blob/main/proxy/src/index.ts"
    click Proxy "https://github.com/ralo93/open-discourse/tree/main/proxy/Dockerfile"
    click Proxy "https://github.com/ralo93/open-discourse/blob/main/proxy/Dockerfile.prod"
    click Proxy "https://github.com/ralo93/open-discourse/blob/main/proxy/.env"
    click Proxy "https://github.com/ralo93/open-discourse/blob/main/proxy/package.json"
    click Proxy "https://github.com/ralo93/open-discourse/blob/main/proxy/tsconfig.json"

    click Frontend "https://github.com/ralo93/open-discourse/tree/main/frontend/Dockerfile"
    click Frontend "https://github.com/ralo93/open-discourse/blob/main/frontend/Dockerfile.prod"
    click Frontend "https://github.com/ralo93/open-discourse/blob/main/frontend/.env.development"
    click Frontend "https://github.com/ralo93/open-discourse/blob/main/frontend/src/pages/index.tsx"
    click Frontend "https://github.com/ralo93/open-discourse/tree/main/frontend/src/components"
    click Frontend "https://github.com/ralo93/open-discourse/blob/main/frontend/src/templates/base-template.tsx"
    click Frontend "https://github.com/ralo93/open-discourse/blob/main/frontend/next-env.d.ts"
    click Frontend "https://github.com/ralo93/open-discourse/blob/main/frontend/tsconfig.json"
    click Frontend "https://github.com/ralo93/open-discourse/blob/main/frontend/package.json"

    click DC "https://github.com/ralo93/open-discourse/blob/main/docker-compose.yml"
    click DC "https://github.com/ralo93/open-discourse/blob/main/DOCKER_SETUP.md"
    click DC "https://github.com/ralo93/open-discourse/blob/main/distribute_docker_images.sh"
    click DC "https://github.com/ralo93/open-discourse/blob/main/.github/workflows/deploy.yml"
    click DC "https://github.com/ralo93/open-discourse/blob/main/.github/workflows/codeql-analysis.yml"

    classDef compute fill:#cdeefd,stroke:#036b81
    classDef database fill:#fff2cc,stroke:#bf9000
    classDef external fill:#d5e8d4,stroke:#82b366
    classDef orchestration stroke-dasharray: 5 5,stroke:#666
```
