# darpa-criticalmaas/uncharted-ta1 architecture

> Architecture diagram of the GitHub repository darpa-criticalmaas/uncharted-ta1, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darpa-criticalmaas/uncharted-ta1
- Repository: https://github.com/darpa-criticalmaas/uncharted-ta1

## Overview

##### Project type and purpose
- This is a data-processing platform composed of five independent but structurally identical “pipelines” (map segmentation, metadata extraction, point extraction, georeferencing, text extraction) plus two auxiliary microservices (cdr, cdr_writer).
- Each pipeline is a pip-installable Python package that (a) orchestrates reusable tasks, (b) can run as a CLI, and (c) can run as a REST server in Docker.
- The overall goal is to process geologic map images, extract features/text/geocoordinates, and integrate results with DARPA’s CriticalMAAS CDR system.

##### File structure
- Top-level directories
- pipelines/… – five pipeline packages, each with:
– run_pipeline.py (CLI entry)
– run_server.py (REST server entry)
– segmentation, metadata_extraction, etc. subfolders with “pipeline.py” (orchestration) and “deploy” (Docker build scripts)
- tasks/… – shared pip-installable library of individual task modules (common I/O, ML inference, GCP lookup, etc.)
- cdr/… – request_publisher, chaining_result_subscriber, server.py; handles submitting jobs and listening for CDR callbacks
- cdr_writer/… – result_writer, write_result_subscriber; posts final results back into the CDR
- util/… – shared utility functions (messaging adapters, ETL, logging, geo utilities)
- schema/… – Pydantic-style schema definitions and mappers for CDR messages
- deploy/… – docker-compose templates and example vars for bringing up the full stack
- CI/CD (.github/workflows) – builds and tests all containers

##### README for additional insights
- Confirms five pipelines, each built and deployed via Docker
- Describes Makefile targets and a docker-compose based full-stack deployment for integration testing with CDR

a. Main components
- Pipeline Services (five separate Dockerized microservices: Segmentation, Metadata, Point, GeoRef, Text Extraction)
- Task Library (shared code – common tasks imported by all pipelines)
- CDR Integration Services
– Request Publisher (submits requests into the pipeline mesh)
– Result Subscriber / Writer (consumes final results and writes back to CDR)
- External Dependencies
– DARPA CDR API (HTTP/message bus)
– External ML backends (e.g., PyTorch models, Google Vision OCR)
– Static reference data (data/…, certs/, schema/)
- Orchestration
– Docker Compose – spawns pipeline services, CDR mocks/adapters, message bus, data storage
- CI/CD (GitHub Actions) – builds Docker images, runs tests

b. Relationships and interactions
1. The Request Publisher service receives job definitions (from CDR or CLI), and pushes them over the message bus or HTTP into the appropriate pipeline service’s REST endpoint.
2. A pipeline service container (e.g., segmentation) receives the request via run_server.py, orchestrates a sequence of task modules (imported from tasks/common, tasks/segmentation, util, schema), runs inference or data transforms, and emits a result object.
3. Completed results are forwarded to the Result Subscriber (via messaging adapter or HTTP callback). The Result Writer then invokes the CDR API to persist the results.
4. Docker Compose wires up all services: pipelines, cdr, cdr_writer, message bus (if used), and mounts data volumes or secrets.

c. Architectural patterns and design principles
- Microservices per pipeline – isolates functionality, allows independent scaling.
- Shared library (tasks) – DRY, testable units of work.
- Adapter pattern – util/messaging for plugging in different message brokers (stub, real CDR).
- Containerization – each service has its own Dockerfile and build script.
- Message-driven orchestration – decoupled request/response via publisher/subscriber.
- Schema validation – Pydantic models in schema/ ensure strong typing of messages.

d. Key technologies
- Python 3.8+ with pip packaging (pyproject.toml, requirements.txt)
- Docker, Docker Compose, build scripts (build.sh, run.sh)
- PyTorch (in tasks/point_extraction, tasks/segmentation)
- Google Vision OCR adapter (in tasks/text_extraction/ocr)
- Pydantic (in schema/)
- GitHub Actions for CI/CD

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "External Systems"
        CDR_API["DARPA CDR API"]:::external
        OCR["Google Vision OCR"]:::external
        Torch["PyTorch Models"]:::external
    end

    subgraph "CDR Integration"
        RP["Request Publisher"]:::cdr
        RW["Result Writer"]:::cdr
    end

    subgraph "Pipeline Services"
        SEG["Segmentation Pipeline"]:::pipeline
        MET["Metadata Extraction Pipeline"]:::pipeline
        PT["Point Extraction Pipeline"]:::pipeline
        GEO["GeoReferencing Pipeline"]:::pipeline
        TXT["Text Extraction Pipeline"]:::pipeline
    end

    subgraph "Shared Libraries"
        TASKS["Tasks Library"]:::shared
        UTIL["Utility Library"]:::shared
        SCHEMA["Schema Definitions"]:::shared
    end

    DOCKER["Docker Compose Orchestrator"]:::infra
    CI["CI/CD Workflows"]:::infra

    CDR_API -->|"HTTP/JSON"| RP
    RP -->|"job request"| SEG
    RP -->|"job request"| MET
    RP -->|"job request"| PT
    RP -->|"job request"| GEO
    RP -->|"job request"| TXT

    SEG -->|"imports tasks"| TASKS
    MET -->|"imports tasks"| TASKS
    PT -->|"imports tasks"| TASKS
    GEO -->|"imports tasks"| TASKS
    TXT -->|"imports tasks"| TASKS

    SEG -->|"uses OCR"| OCR
    TXT -->|"uses OCR"| OCR
    MET -->|"uses Torch"| Torch
    PT -->|"uses Torch"| Torch
    GEO -->|"uses Torch"| Torch

    SEG -->|"results"| RW
    MET -->|"results"| RW
    PT -->|"results"| RW
    GEO -->|"results"| RW
    TXT -->|"results"| RW

    RW -->|"persist results"| CDR_API

    SEG -->|"imports util"| UTIL
    MET -->|"imports util"| UTIL
    PT -->|"imports util"| UTIL
    GEO -->|"imports util"| UTIL
    TXT -->|"imports util"| UTIL

    SEG -->|"imports schema"| SCHEMA
    MET -->|"imports schema"| SCHEMA
    PT -->|"imports schema"| SCHEMA
    GEO -->|"imports schema"| SCHEMA
    TXT -->|"imports schema"| SCHEMA

    DOCKER --> RP
    DOCKER --> RW
    DOCKER --> SEG
    DOCKER --> MET
    DOCKER --> PT
    DOCKER --> GEO
    DOCKER --> TXT
    DOCKER --> TASKS
    DOCKER --> UTIL
    DOCKER --> SCHEMA

    CI --> DOCKER

    click SEG "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/pipelines/segmentation"
    click MET "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/pipelines/metadata_extraction"
    click PT "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/pipelines/point_extraction"
    click GEO "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/pipelines/geo_referencing"
    click TXT "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/pipelines/text_extraction"
    click TASKS "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/tasks/"
    click RP "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/cdr/"
    click RW "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/cdr_writer/"
    click UTIL "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/util/"
    click SCHEMA "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/schema/"
    click DOCKER "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/deploy/"
    click CI "https://github.com/darpa-criticalmaas/uncharted-ta1/tree/main/.github/workflows/"

    classDef external fill:#d3d3dx,stroke:#555;
    classDef cdr fill:#ffa500,stroke:#b85c00;
    classDef pipeline fill:#add8e6,stroke:#0000ff;
    classDef shared fill:#98fb98,stroke:#008000;
    classDef infra fill:#f0f0f0,stroke:#888;
```
