How pulibrary/dpul-collections works
Main Components
a. Web Layer:
• Phoenix LiveView application (primary user interface)
• Controllers and Live Views for different pages
• Multilingual support (via gettext)
b. Indexing Pipeline:
• Broadway-based pipeline with three main stages:
1. Hydration Consumer
2. Transformation Consumer
3. Indexing Consumer
• Database Producers
• Cache system for each stage
c. External Services:
• Figgy (source system for collections)
• Solr (search index)
• PostgreSQL database
• Grafana (monitoring)
• Loki (logging)
Component Relationships
• User requests → Phoenix LiveView → Solr → Response
• Figgy → Indexing Pipeline (Hydration → Transformation → Indexing) → Solr
• All components → Logging/Monitoring (Grafana/Loki)
Architectural Patterns to Highlight
• Pipeline Pattern (for indexing)
• MVC/LiveView Pattern (for web interface)
• Caching Strategy (for pipeline stages)
• Microservices Integration (Figgy, Solr)
Specific Implementation Details
• Blue rectangles: Web components (Phoenix controllers, LiveView)
• Green rectangles: Pipeline components
• Yellow diamonds: External services (Figgy, Solr)
• Purple rectangles: Infrastructure (Grafana, Loki)
• Orange rectangles: Databases/Storage
Elements
a. Main Flow:
• [Web Browser] → [Phoenix LiveView] → [Solr]
• ↑
• [Figgy] → [Hydration] → [Transformation] → [Indexing]
• ↓ ↓ ↓
• [Cache Layer] → [PostgreSQL]
b. Monitoring Flow:
• [All Components] → [Telemetry] → [Grafana/Loki]
Additional Details
• Cache version management
• Deployment paths (staging/production)
• Authentication flows
• Error handling paths
• Language/locale handling
Technical Stack Callouts
• Elixir/Phoenix
• Broadway for pipeline
• PostgreSQL
• Solr
• Docker containerization
• Nomad for orchestration