How ralo93/open-discourse works
Project type and purpose
• This is a full-stack, data-driven web platform called “Open Discourse” that:
• Uses a Python ETL pipeline to fetch, clean and load parliamentary speech data into a PostgreSQL database
• Exposes the database through a lightweight Node.js proxy service
• Serves a Next.js/React frontend for full-text search and exploration of that data
• Leverages Docker for local development and production
File structure
• Top-level directories:
• database: Dockerized Postgres, schema files, migration scripts (TypeScript)
• python: Data‐gathering & processing pipeline (multiple stages under od_lib)
• proxy: Node.js proxy (API layer) protecting direct DB access
• frontend: Next.js/React UI, components, pages, Chakra UI for styling
• Other artifacts:
• docker-compose.yml: orchestrates database, proxy and frontend containers
• DOCKER_SETUP.md: instructions for Docker deployment
• CI/CD workflows in .github/workflows (build, deploy, codeql)
README for additional insights
• The README confirms the four main pieces: database, python, proxy and frontend
• Instructions on local setup, Docker orchestration, database seeding (yarn db:update), data generation (sh build.sh)
• Links to per-module READMEs for deeper detail on each subproject
Detailed component breakdown (Don’t overthink—break it all out!)
• Python ETL:
• od_lib modules (01_preprocessing → 07_database)
• build.sh orchestrator
• Database:
• Dockerfile, Dockerfile.prod
• Schema scripts in database/src/model
• Migration TS scripts (dbUpdate.ts, teardown.ts)
• Proxy:
• index.ts entry point
• Configured via .env.local
• Frontend:
• Pages (index.tsx)
• Components tree (custom-inputs, result-table, react-table, etc.)
• next.config, tsconfig.json
Final advice
• Be very granular—show each container as a separate node
• Versioning or branch (dev vs. prod) distinctions if relevant