# digitaldemocracy2030/polimoney architecture

> Architecture diagram of the GitHub repository digitaldemocracy2030/polimoney, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/digitaldemocracy2030/polimoney
- Repository: https://github.com/digitaldemocracy2030/polimoney

## Overview

##### Project Type & Purpose
- This is a full-stack web application (“Polimoney”) that visualizes Japanese political funding data.
- It comprises:
- A Next.js/TypeScript frontend (in `app/`, `components/`) for UI and static site generation (SSG/SSR).
- A FastAPI backend (in `backend/`) exposing REST endpoints for auth, funds data, health checks, and profile management, backed by a relational database with Alembic migrations.
- A set of Python and TypeScript “tools” for PDF → JSON extraction, pipeline tasks, converters, and election-specific data generators.

##### File Structure Analysis & Architectural Clues
- Top-level separation:
- `app/`, `components/`, `public/`, `data/`: frontend and static assets/data.
- `backend/`: Python FastAPI service + DB schema/migrations + tests.
- `tools/`, `tools_election/`: data ingestion, PDF processing (pdf2image + LangChain), custom election pipelines, scrapers.
- `scripts/`: shell orchestration (e.g., `create-json-for-web.sh`).
- Configuration: Dockerfiles (`.devcontainer/`, `backend/Dockerfile`), `docker-compose.yml`, GitHub workflows for CI (linting, tests).
- Patterns:
- Frontend uses Next.js app directory, dynamic routing (`[politicianId]`, `[year]`) for pages.
- Backend follows an MVC-like module separation: routers → schemas → models → core utilities → dependencies.
- Data pipeline is decoupled: source PDFs → Python tools → JSON in `public/reports` → `data/converter.ts` → normalized JSON for frontend.

##### Key README Insights
- Tech stack: Next.js/TypeScript frontend; Python (pdf2image, LangChain) for OCR and data tooling; FastAPI + Alembic + PostgreSQL in backend.
- Usage:
- `npm run dev` to launch the frontend.
- `./scripts/create-json-for-web.sh` to generate `public/reports/*.json`.
- Python tools live under `tools/` and `tools_election/` with their own tests.

##### Diagramming the System
- Frontend (Next.js app + React components)
- Backend Service (FastAPI)
- Database (PostgreSQL)
- Data Ingestion Pipeline
– PDF Downloader & Parser (Python scripts & scrapers)
– OCR & LangChain Processor
– JSON Converter (TypeScript)
- Static Asset Store (public/reports JSON)
- External Services
– Google Cloud Vision API (OCR)
– 3rd-party LLM or PDF sources
- CI/CD & Dev Environments
– Docker / Docker-Compose
– GitHub Actions

1. PDF Sources → Downloader scripts (`tools/downloader` → Python OCR) → raw JSON.
2. raw JSON → `data/converter.ts` → Web-ready JSON in `public/reports`.
3. Frontend fetches static JSON and/or calls Backend REST API.
4. Backend to Database via ORM (SQLAlchemy) and Alembic migrations.
5. Users ↔ Frontend (UI interactions, dynamic routing) → Backend (auth, data APIs).
6. CI/CD pipelines build Docker images, run linters/tests, deploy.

c. Architectural Patterns & Design Principles:
- Separation of concerns: UI layer, API layer, data layer, tooling layer.
- SSG/SSR for public pages, fallback to client-side data fetch.
- Modular API (routers + schemas + dependencies).
- Infrastructure as code: Docker, Compose, Devcontainer.
- CI enforcement: lint-go, lint-python, Next.js checks.

- Next.js, React, TypeScript
- FastAPI, SQLAlchemy, Alembic, Uvicorn, pytest
- Python (pdf2image, LangChain)
- Node.js Tools: tsx, data converter
- Docker, Docker-Compose, GitHub Actions

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Services
    subgraph "External Services" 
        direction TB
        OCR_API["Google Cloud Vision API"]:::external
        PDF_SOURCES["3rd-party PDF Sources"]:::external
        LLM_SERVICE["3rd-party LLM"]:::external
    end

    %% Data Ingestion Pipeline
    subgraph "Data Ingestion Pipeline" 
        direction TB
        DL_SCRIPTS["Downloader Scripts (tools/downloader)"]:::pipeline
        click DL_SCRIPTS "https://github.com/digitaldemocracy2030/polimoney/tree/main/tools/downloader/"
        OCR_TOOL["OCR & LangChain Processor"]:::pipeline
        click OCR_TOOL "https://github.com/digitaldemocracy2030/polimoney/blob/main/tools/pdf_to_images.py"
        click OCR_TOOL "https://github.com/digitaldemocracy2030/polimoney/tree/main/tools/analyzer/"
        PREPROCESS["JSON Merge & Preprocess"]:::pipeline
        click PREPROCESS "https://github.com/digitaldemocracy2030/polimoney/blob/main/tools/preprocess.py"
        click PREPROCESS "https://github.com/digitaldemocracy2030/polimoney/blob/main/tools/merge_jsons.py"
        ELECTION_GEN["Election Data Generators"]:::pipeline
        click ELECTION_GEN "https://github.com/digitaldemocracy2030/polimoney/tree/main/tools_election/"
    end

    %% JSON Conversion
    subgraph "JSON Conversion" 
        direction TB
        STANDARD_CONV["Converter (data/converter.ts)"]:::pipeline
        click STANDARD_CONV "https://github.com/digitaldemocracy2030/polimoney/blob/main/data/converter.ts"
        UNIFORM_CONV["Uniformed Converter (data/uniformed/converter.ts)"]:::pipeline
        click UNIFORM_CONV "https://github.com/digitaldemocracy2030/polimoney/blob/main/data/uniformed/converter.ts"
    end

    %% Static Store
    STATIC_STORE["Static JSON Store\n(public/reports/*.json)"]:::frontend
    click STATIC_STORE "https://github.com/digitaldemocracy2030/polimoney/tree/main/public/reports/"

    %% Backend Service
    subgraph "Backend Service (FastAPI)" 
        direction TB
        MAIN_APP["app/main.py"]:::backend
        click MAIN_APP "https://github.com/digitaldemocracy2030/polimoney/blob/main/backend/app/main.py"
        Routers["Routers (backend/app/routers/)"]:::backend
        click Routers "https://github.com/digitaldemocracy2030/polimoney/tree/main/backend/app/routers/"
        Schemas["Schemas (backend/app/schemas/)"]:::backend
        click Schemas "https://github.com/digitaldemocracy2030/polimoney/tree/main/backend/app/schemas/"
        Models["Models (backend/app/models/)"]:::backend
        click Models "https://github.com/digitaldemocracy2030/polimoney/tree/main/backend/app/models/"
        CoreUtils["Core Utils & Security\n(backend/app/core/)"]:::backend
        click CoreUtils "https://github.com/digitaldemocracy2030/polimoney/tree/main/backend/app/core/"
        Dependencies["Dependencies (backend/app/dependencies/)"]:::backend
        click Dependencies "https://github.com/digitaldemocracy2030/polimoney/tree/main/backend/app/dependencies/"
        ConfigBE["Config & .env (backend/app/config.py)"]:::backend
        click ConfigBE "https://github.com/digitaldemocracy2030/polimoney/blob/main/backend/app/config.py"
        DB_FLOW["Alembic Migrations"]:::backend
        click DB_FLOW "https://github.com/digitaldemocracy2030/polimoney/blob/main/backend/alembic.ini"
        click DB_FLOW "https://github.com/digitaldemocracy2030/polimoney/blob/main/backend/alembic/env.py"
        click DB_FLOW "https://github.com/digitaldemocracy2030/polimoney/blob/main/backend/alembic/versions/214ef336386f_initial_migration.py"
    end

    %% Database
    DB["PostgreSQL Database"]:::database

    %% Frontend Layer
    subgraph "Frontend Layer (Next.js/TypeScript)" 
        direction TB
        AppDir["app/ directory"]:::frontend
        click AppDir "https://github.com/digitaldemocracy2030/polimoney/tree/main/app/"
        Layout["app/layout.tsx"]:::frontend
        click Layout "https://github.com/digitaldemocracy2030/polimoney/blob/main/app/layout.tsx"
        Page["app/page.tsx & dynamic routes"]:::frontend
        UIComponents["UI Components (components/)"]:::frontend
        click UIComponents "https://github.com/digitaldemocracy2030/polimoney/tree/main/components/"
        ConfigFE["next.config.ts & tsconfig.json"]:::frontend
        click ConfigFE "https://github.com/digitaldemocracy2030/polimoney/blob/main/next.config.ts"
    end

    %% Scripts & Assets
    SCRIPTS["Scripts & OG Generator"]:::pipeline
    click SCRIPTS "https://github.com/digitaldemocracy2030/polimoney/blob/main/scripts/create-json-for-web.sh"
    click SCRIPTS "https://github.com/digitaldemocracy2030/polimoney/blob/main/tools/generate-og-images.js"

    %% CI/CD & Dev Environments
    subgraph "CI/CD & Infra" 
        direction TB
        GH_ACTIONS["GitHub Actions (.github/workflows/)"]:::infra
        click GH_ACTIONS "https://github.com/digitaldemocracy2030/polimoney/tree/main/.github/workflows/"
        DEVCONTAINER["Devcontainer (.devcontainer/)"]:::infra
        click DEVCONTAINER "https://github.com/digitaldemocracy2030/polimoney/tree/main/.devcontainer/"
        DOCKER_COMPOSE["docker-compose.yml"]:::infra
        click DOCKER_COMPOSE "https://github.com/digitaldemocracy2030/polimoney/blob/main/docker-compose.yml"
        BE_DOCKER["Backend Dockerfile"]:::infra
        click BE_DOCKER "https://github.com/digitaldemocracy2030/polimoney/tree/main/backend/Dockerfile"
        BE_COMPOSE["Backend docker-compose.yml"]:::infra
        click BE_COMPOSE "https://github.com/digitaldemocracy2030/polimoney/blob/main/backend/docker-compose.yml"
    end

    %% Relationships
    PDF_SOURCES -->|provides PDFs| DL_SCRIPTS
    DL_SCRIPTS -->|raw JSON| OCR_TOOL
    OCR_API -->|OCR request| OCR_TOOL
    OCR_TOOL -->|processed JSON| PREPROCESS
    PREPROCESS -->|merged JSON| STANDARD_CONV
    PREPROCESS -->|merged JSON| UNIFORM_CONV
    STANDARD_CONV -->|web JSON| STATIC_STORE
    UNIFORM_CONV -->|uniformed JSON| STATIC_STORE
    SCRIPTS -->|trigger conversion| STANDARD_CONV
    SCRIPTS -->|trigger conversion| UNIFORM_CONV

    STATIC_STORE -->|static fetch| AppDir
    AppDir -->|uses| UIComponents
    AppDir -->|layout| Layout
    AppDir -->|routes| Page
    UIComponents -->|import| Layout
    UIComponents -->|import| Page

    AppDir -->|API calls| MAIN_APP
    MAIN_APP -->|calls routers| Routers
    Routers -->|uses schemas| Schemas
    Routers -->|query| Models
    Routers -->|validate| Dependencies
    MAIN_APP -->|config| ConfigBE
    MAIN_APP -->|migrations| DB_FLOW
    Models -->|ORM| DB
    DB_FLOW -->|migrate| DB

    USERS["Users"]:::external
    USERS -->|interact| AppDir
    USERS -->|interact| MAIN_APP

    LLM_SERVICE -->|LLM calls| OCR_TOOL

    %% CI/CD & Infra links
    GH_ACTIONS -->|build/test| DEVCONTAINER
    GH_ACTIONS -->|deploy| DOCKER_COMPOSE
    DEVCONTAINER -->|defines| BE_DOCKER
    DOCKER_COMPOSE -->|orchestrates| BE_DOCKER

    %% Styles
    classDef frontend fill:#bbdefb,stroke:#1e88e5,color:#0d47a1;
    classDef backend fill:#c8e6c9,stroke:#43a047,color:#1b5e20;
    classDef pipeline fill:#ffe0b2,stroke:#fb8c00,color:#e65100;
    classDef database fill:#eeeeee,stroke:#bdbdbd,color:#424242;
    classDef infra fill:#e1bee7,stroke:#8e24aa,color:#4a148c;
    classDef external fill:#f5f5f5,stroke:#9e9e9e,color:#424242;
```
