How darpa-criticalmaas/uncharted-ta1 works
Project type and purpose
• This is a data-processing platform composed of five independent but structurally identical “pipelines” (map segmentation, metadata extraction, point extraction, georeferencing, text extraction) plus two auxiliary microservices (cdr, cdr_writer).
• Each pipeline is a pip-installable Python package that (a) orchestrates reusable tasks, (b) can run as a CLI, and (c) can run as a REST server in Docker.
• The overall goal is to process geologic map images, extract features/text/geocoordinates, and integrate results with DARPA’s CriticalMAAS CDR system.
File structure
• Top-level directories
• pipelines/… – five pipeline packages, each with:
– run_pipeline.py (CLI entry)
– run_server.py (REST server entry)
– segmentation, metadata_extraction, etc. subfolders with “pipeline.py” (orchestration) and “deploy” (Docker build scripts)
• tasks/… – shared pip-installable library of individual task modules (common I/O, ML inference, GCP lookup, etc.)
• cdr/… – request_publisher, chaining_result_subscriber, server.py; handles submitting jobs and listening for CDR callbacks
• cdr_writer/… – result_writer, write_result_subscriber; posts final results back into the CDR
• util/… – shared utility functions (messaging adapters, ETL, logging, geo utilities)
• schema/… – Pydantic-style schema definitions and mappers for CDR messages
• deploy/… – docker-compose templates and example vars for bringing up the full stack
• CI/CD (.github/workflows) – builds and tests all containers
README for additional insights
• Confirms five pipelines, each built and deployed via Docker
• Describes Makefile targets and a docker-compose based full-stack deployment for integration testing with CDR
a. Main components
• Pipeline Services (five separate Dockerized microservices: Segmentation, Metadata, Point, GeoRef, Text Extraction)
• Task Library (shared code – common tasks imported by all pipelines)
• CDR Integration Services
– Request Publisher (submits requests into the pipeline mesh)
– Result Subscriber / Writer (consumes final results and writes back to CDR)
• External Dependencies
– DARPA CDR API (HTTP/message bus)
– External ML backends (e.g., PyTorch models, Google Vision OCR)
– Static reference data (data/…, certs/, schema/)
• Orchestration
– Docker Compose – spawns pipeline services, CDR mocks/adapters, message bus, data storage
• CI/CD (GitHub Actions) – builds Docker images, runs tests
b. Relationships and interactions
1. The Request Publisher service receives job definitions (from CDR or CLI), and pushes them over the message bus or HTTP into the appropriate pipeline service’s REST endpoint.
2. A pipeline service container (e.g., segmentation) receives the request via run_server.py, orchestrates a sequence of task modules (imported from tasks/common, tasks/segmentation, util, schema), runs inference or data transforms, and emits a result object.
3. Completed results are forwarded to the Result Subscriber (via messaging adapter or HTTP callback). The Result Writer then invokes the CDR API to persist the results.
4. Docker Compose wires up all services: pipelines, cdr, cdr_writer, message bus (if used), and mounts data volumes or secrets.
c. Architectural patterns and design principles
• Microservices per pipeline – isolates functionality, allows independent scaling.
• Shared library (tasks) – DRY, testable units of work.
• Adapter pattern – util/messaging for plugging in different message brokers (stub, real CDR).
• Containerization – each service has its own Dockerfile and build script.
• Message-driven orchestration – decoupled request/response via publisher/subscriber.
• Schema validation – Pydantic models in schema/ ensure strong typing of messages.
d. Key technologies
• Python 3.8+ with pip packaging (pyproject.toml, requirements.txt)
• Docker, Docker Compose, build scripts (build.sh, run.sh)
• PyTorch (in tasks/point_extraction, tasks/segmentation)
• Google Vision OCR adapter (in tasks/text_extraction/ocr)
• Pydantic (in schema/)
• GitHub Actions for CI/CD