# weecology/doublecounting architecture

> Architecture diagram of the GitHub repository weecology/doublecounting, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/weecology/doublecounting
- Repository: https://github.com/weecology/doublecounting

## Overview

##### Project Type and Purpose
- This is a Python-based data-processing pipeline / command-line tool (not a web app) for airborne ecological object detection and de-duplication (“double counting”) in overlapping imagery.
- Its workflows include: building a structure-from-motion (SfM) model, running an object detector on images, aligning detections via homographies, removing duplicates, and visualizing results.

##### File Structure Analysis
- Top level:
- pipeline.py & pipeline.sh – the main orchestration script (CLI entry point and shell wrapper)
- create_image_pairs_for_annotation.py – utility to generate overlapping image pairs for manual annotation (LabelStudio)
- public/ – example outputs and images
- requirements.txt – Python dependencies
- tests/ – unit tests covering each sub-module
- scripts/ (Python package):
- stitching.py – core homography alignment and duplicate removal logic (align_and_delete)
- reconstruction.py – builds a SfM model using hloc & pycolmap
- labelstudio.py – helper to interface with LabelStudio for manual annotation
- reupload_to_dropbox.sh – uploads results to Dropbox
- Tests mirror scripts/ to validate predict, stitching, reconstruction, etc.

##### README Insights
- Describes three de-duplication strategies: “Stitch and Predict,” “Predict and Delete,” “Predict and Verify.”
- Emphasizes external dependencies: hloc, pycolmap, DeepForest, OpenCV, Matplotlib.
- Shows how each pipeline stage is invoked via Python API.
- Illustrates both 2D homography examples and optional 3D reconstruction outputs.

##### Core Components & Interactions (
a. Data Ingestion
- Input: Raw overlapping images (folder)
- Optional: Annotated image pairs from LabelStudio (for training/evaluation)
b. SfM Reconstruction Module
- scripts/reconstruction.py uses hloc & pycolmap → outputs matches.h5 and 3D point cloud
c. Object Detection Module
d. De-duplication Module
- scripts/stitching.py align_and_delete:
– Loads matches from SfM
– Computes homographies between image pairs
– Applies chosen strategy (stitch & predict, etc.)
– Outputs unique detections
e. Visualization & Output
- pipeline.py or user script
- Optionally re-uploads results via reupload_to_dropbox.sh
f. CLI Orchestration
- pipeline.sh calls pipeline.py or individual scripts in sequence
g. Testing
- pytest on tests/test_*.py ensures each component behaves as expected

##### Architectural Patterns & Design Principles
- Modular scripts with single responsibility: one script per major task
- File-based data passing ensures reproducibility and traceability
- Strategy pattern for duplicate removal (stitch-predict, predict-delete, predict-verify)
- Test-driven components allowing incremental validation

- Blue for input/data
- Green for core processing modules
- Orange for optional/third-party services
- Purple for output/visualization

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        RawImages[(Raw Images)]:::data
        DataIngestion["Image Pair Generator"]:::processing
        AnnotatedPairs[(Annotated Image Pairs)]:::data
        Requirements[(requirements.txt)]:::data
        Examples[(public/)]:::data
    end

    %% Processing Layer
    subgraph "Processing Layer"
        Reconstruction["SfM Reconstruction"]:::processing
        Predict["Object Detection"]:::processing
        Deduplication["De-duplication Module"]:::processing
    end

    %% Output Layer
    subgraph "Output Layer"
        Visualization["Visualization & Output"]:::output
        DropboxUpload("Dropbox Upload"):::external
    end

    %% External Services
    subgraph "External Services"
        LabelStudio("LabelStudio"):::external
    end

    %% CLI Orchestration
    subgraph "CLI Orchestration"
        PipelineSh["pipeline.sh"]:::cli
        PipelinePy["pipeline.py"]:::cli
    end

    %% Testing Suite
    subgraph "Testing Suite"
        Tests[(tests/)]:::data
    end

    %% Connections - Data to Processing
    RawImages -->|raw images| DataIngestion
    DataIngestion -->|image pairs| AnnotatedPairs
    RawImages -->|raw images| Reconstruction
    RawImages -->|raw images| Predict

    %% Processing Flow
    Reconstruction -->|matches.h5| Deduplication
    Predict -->|bbox predictions| Deduplication
    LabelStudio -->|annotated pairs| AnnotatedPairs
    Deduplication -->|unique detections| Visualization

    %% Output Flow
    Visualization -->|plots & overlays| DropboxUpload
    Examples --> Visualization

    %% CLI Flow
    PipelineSh --> PipelinePy
    PipelinePy --> DataIngestion
    PipelinePy --> Reconstruction
    PipelinePy --> Predict
    PipelinePy --> Deduplication
    PipelinePy --> Visualization
    Requirements --> PipelineSh
    Tests --> PipelineSh

    %% Click Events
    click DataIngestion "https://github.com/weecology/doublecounting/blob/main/create_image_pairs_for_annotation.py"
    click LabelStudio "https://github.com/weecology/doublecounting/blob/main/scripts/labelstudio.py"
    click Reconstruction "https://github.com/weecology/doublecounting/blob/main/scripts/reconstruction.py"
    click Predict "https://github.com/weecology/doublecounting/blob/main/scripts/predict.py"
    click Deduplication "https://github.com/weecology/doublecounting/blob/main/scripts/stitching.py"
    click DropboxUpload "https://github.com/weecology/doublecounting/blob/main/scripts/reupload_to_dropbox.sh"
    click PipelinePy "https://github.com/weecology/doublecounting/blob/main/pipeline.py"
    click PipelineSh "https://github.com/weecology/doublecounting/blob/main/pipeline.sh"
    click Examples "https://github.com/weecology/doublecounting/tree/main/public/"
    click Requirements "https://github.com/weecology/doublecounting/blob/main/requirements.txt"
    click Tests "https://github.com/weecology/doublecounting/tree/main/tests/"

    %% Styles
    classDef data fill:#cce5ff,stroke:#004085
    classDef processing fill:#d4edda,stroke:#155724
    classDef output fill:#f8d7da,stroke:#721c24
    classDef external fill:#fff3cd,stroke:#856404
    classDef cli fill:#e2e3e5,stroke:#6c757d
```
