How digitaldemocracy2030/polimoney works
Project Type & Purpose
• This is a full-stack web application (“Polimoney”) that visualizes Japanese political funding data.
• It comprises:
• A Next.js/TypeScript frontend (in app/, components/) for UI and static site generation (SSG/SSR).
• A FastAPI backend (in backend/) exposing REST endpoints for auth, funds data, health checks, and profile management, backed by a relational database with Alembic migrations.
• A set of Python and TypeScript “tools” for PDF → JSON extraction, pipeline tasks, converters, and election-specific data generators.
File Structure Analysis & Architectural Clues
• Top-level separation:
• app/, components/, public/, data/: frontend and static assets/data.
• backend/: Python FastAPI service + DB schema/migrations + tests.
• tools/, tools_election/: data ingestion, PDF processing (pdf2image + LangChain), custom election pipelines, scrapers.
• scripts/: shell orchestration (e.g., create-json-for-web.sh).
• Configuration: Dockerfiles (.devcontainer/, backend/Dockerfile), docker-compose.yml, GitHub workflows for CI (linting, tests).
• Patterns:
• Frontend uses Next.js app directory, dynamic routing ([politicianId], [year]) for pages.
• Backend follows an MVC-like module separation: routers → schemas → models → core utilities → dependencies.
• Data pipeline is decoupled: source PDFs → Python tools → JSON in public/reports → data/converter.ts → normalized JSON for frontend.
Key README Insights
• Tech stack: Next.js/TypeScript frontend; Python (pdf2image, LangChain) for OCR and data tooling; FastAPI + Alembic + PostgreSQL in backend.
• Usage:
• npm run dev to launch the frontend.
• ./scripts/create-json-for-web.sh to generate public/reports/*.json.
• Python tools live under tools/ and tools_election/ with their own tests.
Diagramming the System
• Frontend (Next.js app + React components)
• Backend Service (FastAPI)
• Database (PostgreSQL)
• Data Ingestion Pipeline
– PDF Downloader & Parser (Python scripts & scrapers)
– OCR & LangChain Processor
– JSON Converter (TypeScript)
• Static Asset Store (public/reports JSON)
• External Services
– Google Cloud Vision API (OCR)
– 3rd-party LLM or PDF sources
• CI/CD & Dev Environments
– Docker / Docker-Compose
– GitHub Actions
1. PDF Sources → Downloader scripts (tools/downloader → Python OCR) → raw JSON.
2. raw JSON → data/converter.ts → Web-ready JSON in public/reports.
3. Frontend fetches static JSON and/or calls Backend REST API.
4. Backend to Database via ORM (SQLAlchemy) and Alembic migrations.
5. Users ↔ Frontend (UI interactions, dynamic routing) → Backend (auth, data APIs).
6. CI/CD pipelines build Docker images, run linters/tests, deploy.
c. Architectural Patterns & Design Principles:
• Separation of concerns: UI layer, API layer, data layer, tooling layer.
• SSG/SSR for public pages, fallback to client-side data fetch.
• Modular API (routers + schemas + dependencies).
• Infrastructure as code: Docker, Compose, Devcontainer.
• CI enforcement: lint-go, lint-python, Next.js checks.
• Next.js, React, TypeScript
• FastAPI, SQLAlchemy, Alembic, Uvicorn, pytest
• Python (pdf2image, LangChain)
• Node.js Tools: tsx, data converter
• Docker, Docker-Compose, GitHub Actions