# mralwyn/legal-semantic-pipeline architecture

> Architecture diagram of the GitHub repository mralwyn/legal-semantic-pipeline, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mralwyn/legal-semantic-pipeline
- Repository: https://github.com/mralwyn/legal-semantic-pipeline

## Overview

##### Project Type and Purpose
- This is a full-stack, container-orchestrated semantic processing pipeline for legal documents. It combines:
- A FastAPI backend (REST endpoints for upload, query, model management, health, metrics)
- A minimal frontend UI (static templates, webm asset)
- A vector database (Qdrant) for embeddings storage
- MLOps components (MLflow for experiment tracking & model registry)
- Observability stack (Prometheus for metrics collection, Grafana for dashboards)
- Auxiliary services (OCR, drift detection, system monitoring)

##### File Structure
- Top-level:
- Dockerfile & docker-compose.yml orchestrate containers
- .github/workflows/ci-cd.yml runs CI/CD (build & test)
- grafana/, prometheus/ supply monitoring configs
- mlflow_db/ holds the SQLite backend
- app/ module:
- main.py boots the FastAPI app
- api/v1 defines REST routes (upload, query, models, health, metrics)
- services/ implement domain logic (chunking, OCR, vector store, drift detection, MLflow integration)
- core/ pipeline orchestration, model registry, settings
- middleware/ injects Prometheus metrics
- config/ chunking/model parameters
- models/schemas define Pydantic models
- static/ + templates/ implement the upload/query UI

##### README for Additional Insights
- Architecture Overview section explicitly lists the six main services.
- Quick start shows ports and URLs for each container.
- API Usage and Core API Endpoints map features to endpoints.
- MLOps and Monitoring describe cross-cutting integrations.

##### Detail
- Every key directory as a logical component: “OCR Service”, “Chunking Service”, etc.
- How the pipeline orchestrator in app/core/pipeline.py ties services together.
- That middleware emits Prometheus metrics at each request.
- Don’t forget CI/CD’s role in deploying new versions and running tests.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client
    Client["Web Browser"]:::external

    %% Docker Compose Orchestration
    DockerCompose["Docker-Compose Orchestration"]:::external

    %% CI/CD
    CICD["CI/CD Pipeline (GitHub Actions)"]:::external

    %% Frontend UI
    subgraph "Frontend UI"
        direction TB
        Static["Frontend Static Assets"]:::frontend
        Templates["Frontend Templates (UI)"]:::frontend
    end

    %% FastAPI Backend
    subgraph "FastAPI Backend"
        direction TB
        API_Routing["API Routing Layer"]:::compute
        Upload["Upload & Query Endpoints"]:::compute
        ModelsEP["Model Management Endpoints"]:::compute
        Health["Health Check Endpoint"]:::compute
        MetricsEP["Metrics Endpoint"]:::compute
        Feedback["Feedback Endpoint"]:::compute
        Middleware["Prometheus Metrics Middleware"]:::compute
        Schemas["Pydantic Schemas"]:::compute

        CorePipeline["Core Pipeline Orchestration"]:::compute
        ModelRegistryCore["Model Registry (Core)"]:::compute
        Settings["Application Settings & Config"]:::compute
        ConfigLoader["Core Configuration Loader"]:::compute

        subgraph "Service Layer"
            direction TB
            ChunkingService["Chunking & Text Processing Service"]:::compute
            OCRService["OCR Service"]:::compute
            VectorStoreService["Vector Store Service"]:::compute
            DriftDetectionService["Drift Detection Service"]:::compute
            SystemMonitorService["System Monitoring Service"]:::compute
            MLflowService["MLflow Service"]:::compute
            MetricsService["Internal Metrics Service"]:::compute
        end
    end

    %% Data Stores
    subgraph "Data Stores"
        direction TB
        Qdrant["Qdrant Vector DB<br/>port 6333"]:::data
        MLflowDB["MLflow SQLite Backend Store"]:::data
    end

    %% MLOps
    subgraph "MLOps"
        direction TB
        MLflowServer["MLflow Server<br/>port 5000"]:::compute
    end

    %% Observability
    subgraph "Observability"
        direction TB
        Prometheus["Prometheus Server<br/>port 9090"]:::obs
        Grafana["Grafana Server<br/>port 3000"]:::obs
    end

    %% Connections
    Client -->|HTTP 8000| Static
    Client -->|HTTP 8000| Templates
    Static -->|served by| DockerCompose
    Templates -->|served by| DockerCompose

    Client -->|HTTP 8000| API_Routing
    API_Routing --> Upload
    API_Routing --> ModelsEP
    API_Routing --> Health
    API_Routing --> MetricsEP
    API_Routing --> Feedback

    Upload --> CorePipeline
    CorePipeline --> ChunkingService
    CorePipeline --> OCRService
    ChunkingService --> VectorStoreService
    OCRService --> VectorStoreService
    VectorStoreService --> Qdrant
    CorePipeline --> MLflowService
    MLflowService --> MLflowServer
    MLflowServer --> MLflowDB

    CorePipeline --> DriftDetectionService
    CorePipeline --> SystemMonitorService

    API_Routing --> Schemas
    Schemas -->|validate| API_Routing

    FastAPI_Backend_Metrics["FastAPI /metrics"]:::compute
    MetricsEP --> FastAPI_Backend_Metrics
    FastAPI_Backend_Metrics --> Prometheus
    Middleware --> Prometheus

    Prometheus -->|datasource| Grafana
    Grafana -->|dashboards| Prometheus

    DockerCompose --> Client
    DockerCompose --> Static
    DockerCompose --> Templates
    DockerCompose --> FastAPI_Backend
    DockerCompose --> Qdrant
    DockerCompose --> MLflowServer
    DockerCompose --> Prometheus
    DockerCompose --> Grafana

    CICD --> DockerCompose

    %% Click Events
    click API_Routing "https://github.com/mralwyn/legal-semantic-pipeline/tree/master/app/api/v1/"
    click Upload "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/api/v1/upload_routes.py"
    click ModelsEP "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/api/v1/models.py"
    click Health "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/api/v1/health_routes.py"
    click MetricsEP "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/api/v1/metrics_endpoint.py"
    click Feedback "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/api/v1/feedback.py"
    click CorePipeline "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/core/pipeline.py"
    click ModelRegistryCore "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/core/model_registry.py"
    click Settings "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/core/settings.py"
    click ConfigLoader "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/core/config.py"
    click ChunkingService "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/services/chunking.py"
    click OCRService "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/services/ocr.py"
    click VectorStoreService "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/services/vector_store.py"
    click DriftDetectionService "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/services/drift_detection.py"
    click SystemMonitorService "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/services/system_monitor.py"
    click MLflowService "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/services/mlflow_service.py"
    click MetricsService "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/services/metrics.py"
    click Middleware "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/middleware/metrics_middleware.py"
    click Static "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/static/credarthaUXlogo-alwyn.webm"
    click Templates "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/app/templates/index.html"
    click DockerCompose "https://github.com/mralwyn/legal-semantic-pipeline/tree/master/Dockerfile"
    click CICD "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/.github/workflows/ci-cd.yml"
    click Prometheus "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/prometheus/prometheus.yml"
    click Grafana "https://github.com/mralwyn/legal-semantic-pipeline/tree/master/grafana/provisioning/dashboards/"
    click MLflowDB "https://github.com/mralwyn/legal-semantic-pipeline/blob/master/mlflow_db/mlflow.db"

    %% Styles
    classDef compute fill:#add8e6,stroke:#000,stroke-width:1px
    classDef data fill:#90ee90,stroke:#000,stroke-width:1px
    classDef obs fill:#ffd700,stroke:#000,stroke-width:1px
    classDef external fill:#d3d3d3,stroke:#000,stroke-width:1px
    classDef frontend fill:#f0e68c,stroke:#000,stroke-width:1px
```
