# pulibrary/dpul-collections architecture

> Architecture diagram of the GitHub repository pulibrary/dpul-collections, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pulibrary/dpul-collections
- Repository: https://github.com/pulibrary/dpul-collections

## Overview

##### Main Components

a. Web Layer:
- Phoenix LiveView application (primary user interface)
- Controllers and Live Views for different pages
- Multilingual support (via gettext)

b. Indexing Pipeline:
- Broadway-based pipeline with three main stages:
1. Hydration Consumer
2. Transformation Consumer
3. Indexing Consumer
- Database Producers
- Cache system for each stage

c. External Services:
- Figgy (source system for collections)
- Solr (search index)
- PostgreSQL database
- Grafana (monitoring)
- Loki (logging)

##### Component Relationships

- User requests → Phoenix LiveView → Solr → Response
- Figgy → Indexing Pipeline (Hydration → Transformation → Indexing) → Solr
- All components → Logging/Monitoring (Grafana/Loki)

##### Architectural Patterns to Highlight

- Pipeline Pattern (for indexing)
- MVC/LiveView Pattern (for web interface)
- Caching Strategy (for pipeline stages)
- Microservices Integration (Figgy, Solr)

##### Specific Implementation Details

- Blue rectangles: Web components (Phoenix controllers, LiveView)
- Green rectangles: Pipeline components
- Yellow diamonds: External services (Figgy, Solr)
- Purple rectangles: Infrastructure (Grafana, Loki)
- Orange rectangles: Databases/Storage

##### Elements

a. Main Flow:
- [Web Browser] → [Phoenix LiveView] → [Solr]
- ↑
- [Figgy] → [Hydration] → [Transformation] → [Indexing]
- ↓             ↓                  ↓
- [Cache Layer] → [PostgreSQL]

b. Monitoring Flow:
- [All Components] → [Telemetry] → [Grafana/Loki]

##### Additional Details

- Cache version management
- Deployment paths (staging/production)
- Authentication flows
- Error handling paths
- Language/locale handling

##### Technical Stack Callouts

- Elixir/Phoenix
- Broadway for pipeline
- PostgreSQL
- Solr
- Docker containerization
- Nomad for orchestration

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Web_Layer
        Browser["Web Browser"]
        LiveView["Phoenix LiveView"]:::web
        Controllers["Web Controllers"]:::web
        Layouts["Layout Components"]:::web
        Gettext["Gettext"]:::web
    end

    subgraph Pipeline
        direction TB
        HydrationConsumer["Hydration Consumer"]:::pipeline
        TransformationConsumer["Transformation Consumer"]:::pipeline
        IndexingConsumer["Indexing Consumer"]:::pipeline
        DatabaseProducer["Database Producer"]:::pipeline
        HydrationCache["Hydration Cache"]:::cache
        TransformationCache["Transformation Cache"]:::cache
    end

    subgraph External_Services
        Figgy["Figgy API"]:::external
        Solr["Solr Search"]:::external
        PostgreSQL["PostgreSQL"]:::database
    end

    subgraph Monitoring
        Grafana["Grafana"]:::monitoring
        Loki["Loki"]:::monitoring
        Telemetry["Telemetry"]:::monitoring
    end

    %% Web Layer Connections
    Browser --> LiveView
    LiveView --> Controllers
    LiveView --> Layouts
    LiveView --> Gettext
    LiveView --> Solr

    %% Pipeline Connections
    Figgy --> HydrationConsumer
    HydrationConsumer --> HydrationCache
    HydrationConsumer --> TransformationConsumer
    TransformationConsumer --> TransformationCache
    TransformationConsumer --> IndexingConsumer
    IndexingConsumer --> Solr
    DatabaseProducer --> HydrationConsumer
    PostgreSQL --> DatabaseProducer

    %% Monitoring Connections
    LiveView --> Telemetry
    Pipeline --> Telemetry
    Telemetry --> Grafana
    Telemetry --> Loki

    %% Click Events
    click LiveView "https://github.com/pulibrary/dpul-collections/tree/main/lib/dpul_collections_web/live/"
    click Controllers "https://github.com/pulibrary/dpul-collections/tree/main/lib/dpul_collections_web/controllers/"
    click Layouts "https://github.com/pulibrary/dpul-collections/tree/main/lib/dpul_collections_web/components/layouts/"
    click Gettext "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections_web/gettext.ex"
    click HydrationConsumer "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections/indexing_pipeline/figgy/hydration_consumer.ex"
    click TransformationConsumer "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections/indexing_pipeline/figgy/transformation_consumer.ex"
    click IndexingConsumer "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections/indexing_pipeline/figgy/indexing_consumer.ex"
    click DatabaseProducer "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections/indexing_pipeline/database_producer.ex"
    click Figgy "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections/figgy_repo.ex"
    click Solr "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections/solr.ex"
    click PostgreSQL "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections/repo.ex"
    click Grafana "https://github.com/pulibrary/dpul-collections/blob/main/config/deploy/grafana.hcl"
    click Loki "https://github.com/pulibrary/dpul-collections/blob/main/config/deploy/loki.hcl"
    click Telemetry "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections_web/telemetry.ex"
    click HydrationCache "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections/indexing_pipeline/figgy/hydration_cache_entry.ex"
    click TransformationCache "https://github.com/pulibrary/dpul-collections/blob/main/lib/dpul_collections/indexing_pipeline/figgy/transformation_cache_entry.ex"

    %% Styling
    classDef web fill:#2196F3,stroke:#fff,stroke-width:2px,color:#fff
    classDef pipeline fill:#4CAF50,stroke:#fff,stroke-width:2px,color:#fff
    classDef external fill:#FFC107,stroke:#fff,stroke-width:2px,color:#000
    classDef monitoring fill:#9C27B0,stroke:#fff,stroke-width:2px,color:#fff
    classDef database fill:#FF9800,stroke:#fff,stroke-width:2px,color:#fff
    classDef cache fill:#E91E63,stroke:#fff,stroke-width:2px,color:#fff
```
