# etalab/transport-site architecture

> Architecture diagram of the GitHub repository etalab/transport-site, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-04). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/etalab/transport-site
- Repository: https://github.com/etalab/transport-site

## Overview

##### Main System Components

a. Core Applications (Umbrella Structure):
- transport: Main web application
- datagouvfr: Integration with data.gouv.fr
- shared: Common functionality
- unlock: Proxy service for real-time data

b. External Services:
- PostgreSQL Database with PostGIS
- S3-compatible Storage (Scaleway)
- OAuth Provider (data.gouv.fr)
- External Validation Tools
- Monitoring Services (updown.io)

##### Key Flows to Highlight

a. Data Flow:
- User Authentication via OAuth
- Dataset Import/Validation Pipeline
- Real-time Data Proxy
- Geospatial Data Processing

b. Integration Points:
- data.gouv.fr API Integration
- External Validation Tools
- Monitoring Services
- Email Services

##### Important Details

- Domain routing (transport.data.gouv.fr, proxy.transport.data.gouv.fr)
- Background Job Processing
- Validation Pipeline
- Caching Strategy
- Security Boundaries

##### Specific Components to Detail

a. Transport App:
- Controllers
- LiveView Components
- Job Processors
- API Endpoints

b. Unlock App:
- Proxy Configuration
- Caching Mechanism
- Request Handling

c. DataGouvFr App:
- API Client
- Authentication
- Data Synchronization

## Diagram (Mermaid)

```mermaid
graph TB
    %% Client Layer
    subgraph Clients
        Browser["Web Browser"]
        APIConsumer["API Consumer"]
        ProxyClient["Proxy Client"]
    end

    %% Application Layer
    subgraph "Application Layer"
        WebServer["Phoenix Web Server"]:::web
        click WebServer "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/transport_web/"

        subgraph "Web Components"
            Controllers["Controllers"]:::web
            click Controllers "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/transport_web/controllers/"
            LiveView["LiveView Components"]:::web
            click LiveView "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/transport_web/live/"
            APIControllers["API Controllers"]:::web
            click APIControllers "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/transport_web/api/controllers/"
            Templates["Templates"]:::web
            click Templates "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/transport_web/templates/"
        end

        subgraph "Core Apps"
            Transport["Transport App"]:::core
            click Transport "https://github.com/etalab/transport-site/tree/master/apps/transport/"
            DataGouvFr["DataGouvFr App"]:::core
            click DataGouvFr "https://github.com/etalab/transport-site/tree/master/apps/datagouvfr/"
            Shared["Shared App"]:::core
            click Shared "https://github.com/etalab/transport-site/tree/master/apps/shared/"
            Unlock["Unlock App"]:::core
            click Unlock "https://github.com/etalab/transport-site/tree/master/apps/unlock/"
        end

        subgraph "Authentication"
            OAuth["OAuth Implementation"]:::auth
            click OAuth "https://github.com/etalab/transport-site/blob/master/apps/datagouvfr/lib/datagouvfr/authentication.ex"
            Session["Session Management"]:::auth
            click Session "https://github.com/etalab/transport-site/blob/master/apps/transport/lib/transport_web/session.ex"
        end
    end

    %% Service Layer
    subgraph "Service Layer"
        subgraph "Data Processing"
            Validators["Validators"]:::processing
            click Validators "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/validators/"
            FileProcessing["File Processing"]:::processing
            click FileProcessing "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/S3/"
            Geospatial["Geospatial Processing"]:::processing
            click Geospatial "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/db/geo_data/"
        end

        subgraph "Background Processing"
            JobDefs["Job Definitions"]:::jobs
            click JobDefs "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/jobs/"
            JobProc["Job Processing"]:::jobs
            click JobProc "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/transport/jobs/"
        end

        subgraph "Notification System"
            Mailer["Mailer"]:::notification
            click Mailer "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/mailer/"
            Notifications["Notifications"]:::notification
            click Notifications "https://github.com/etalab/transport-site/blob/master/apps/transport/lib/db/notification.ex"
        end

        subgraph "Caching System"
            Cache["Cache Implementation"]:::cache
            click Cache "https://github.com/etalab/transport-site/blob/master/apps/transport/lib/transport/cache.ex"
            CacheConfig["Cache Configuration"]:::cache
            click CacheConfig "https://github.com/etalab/transport-site/blob/master/apps/shared/lib/req_custom_cache.ex"
        end
    end

    %% Data Layer
    subgraph "Data Layer"
        PostgreSQL[("PostgreSQL + PostGIS")]:::database
        S3["S3 Storage"]:::storage
        Schema["Schema Definitions"]:::database
        click Schema "https://github.com/etalab/transport-site/tree/master/apps/transport/lib/db/"
        Migrations["Migrations"]:::database
        click Migrations "https://github.com/etalab/transport-site/tree/master/apps/transport/priv/repo/migrations/"
    end

    %% External Services
    subgraph "External Services"
        DataGouvFR["data.gouv.fr"]:::external
        ValidationTools["External Validation Tools"]:::external
        Monitoring["Monitoring (updown.io)"]:::external
        EmailService["Email Service"]:::external
    end

    %% Relationships
    Browser --> WebServer
    APIConsumer --> APIControllers
    ProxyClient --> Unlock

    WebServer --> Controllers
    WebServer --> LiveView
    WebServer --> APIControllers
    WebServer --> Templates

    Controllers --> Transport
    LiveView --> Transport
    APIControllers --> Transport

    Transport --> OAuth
    Transport --> Session
    Transport --> Validators
    Transport --> FileProcessing
    Transport --> Geospatial
    Transport --> JobDefs
    Transport --> Cache

    DataGouvFr --> DataGouvFR
    Unlock --> Cache
    Shared --> Transport
    Shared --> Unlock
    Shared --> DataGouvFr

    JobDefs --> JobProc
    JobProc --> Notifications
    Notifications --> Mailer
    Mailer --> EmailService

    Transport --> PostgreSQL
    Transport --> S3
    Schema --> PostgreSQL
    Migrations --> PostgreSQL

    Validators --> ValidationTools
    Transport --> Monitoring

    %% Styles
    classDef web fill:#3498db
    classDef core fill:#2ecc71
    classDef auth fill:#9b59b6
    classDef processing fill:#e67e22
    classDef jobs fill:#f1c40f
    classDef notification fill:#1abc9c
    classDef cache fill:#34495e
    classDef database fill:#8e44ad
    classDef storage fill:#2c3e50
    classDef external fill:#e74c3c
```
