How mralwyn/legal-semantic-pipeline works
Project Type and Purpose
• This is a full-stack, container-orchestrated semantic processing pipeline for legal documents. It combines:
• A FastAPI backend (REST endpoints for upload, query, model management, health, metrics)
• A minimal frontend UI (static templates, webm asset)
• A vector database (Qdrant) for embeddings storage
• MLOps components (MLflow for experiment tracking & model registry)
• Observability stack (Prometheus for metrics collection, Grafana for dashboards)
• Auxiliary services (OCR, drift detection, system monitoring)
File Structure
• Top-level:
• Dockerfile & docker-compose.yml orchestrate containers
• .github/workflows/ci-cd.yml runs CI/CD (build & test)
• grafana/, prometheus/ supply monitoring configs
• mlflow_db/ holds the SQLite backend
• app/ module:
• main.py boots the FastAPI app
• api/v1 defines REST routes (upload, query, models, health, metrics)
• services/ implement domain logic (chunking, OCR, vector store, drift detection, MLflow integration)
• core/ pipeline orchestration, model registry, settings
• middleware/ injects Prometheus metrics
• config/ chunking/model parameters
• models/schemas define Pydantic models
• static/ + templates/ implement the upload/query UI
README for Additional Insights
• Architecture Overview section explicitly lists the six main services.
• Quick start shows ports and URLs for each container.
• API Usage and Core API Endpoints map features to endpoints.
• MLOps and Monitoring describe cross-cutting integrations.
Detail
• Every key directory as a logical component: “OCR Service”, “Chunking Service”, etc.
• How the pipeline orchestrator in app/core/pipeline.py ties services together.
• That middleware emits Prometheus metrics at each request.
• Don’t forget CI/CD’s role in deploying new versions and running tests.