# bgswaroop/scd-images architecture

> Architecture diagram of the GitHub repository bgswaroop/scd-images, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bgswaroop/scd-images
- Repository: https://github.com/bgswaroop/scd-images

## Overview

##### Main Components
- Data Preparation & Ingestion:
– From the README you see instructions to download the Dresden dataset and extract homogeneous patches to an LMDB database.
– Although a “data_modules” folder is mentioned in the README (with extract_and_save_homo_patches.py and train_test_split.py), notice here that jobscripts (like data_gen.sh) and various miscellaneous scripts also play a role in preparing, converting, and processing data.
- Model Pipelines:
– Two main network components are evident from the file tree: “signature_net” and “similarity_net”. These directories contain model definitions, data processing modules (e.g., data_rgb.py, data_fft.py, models.py), and flow control scripts (like sig_net_flow.py and sim_net_flow.py).
– There is also a “sota” folder that includes a “prnu” submodule, suggesting an integration of a state‐of‐the-art method as another processing / analysis component.
- Orchestration / Flow Control:
– At the root level you have several run_flow*.py scripts and jobscripts that seem dedicated to coordinating the experiments and training procedures.
– The miscellaneous directory (especially the “scd_delivery” subfolder) has its own modules and tests, which indicate library functions or services that are part of the experimental pipeline.
- Utilities and Evaluation Components:
– The “utils” directory contains everything from cost functions and evaluation metrics to logging, training utilities, and visualization helpers.
– Subdirectories under miscellaneous house visualization scripts and statistical analysis (like learning_curves.py, feature_maps.py) for result publication.
- Testing & Experimentation:
– The test scripts in “miscellaneous/scd_delivery/test” indicate unit testing for important functions such as computing the signature and similarity.

##### Relationships and Interactions
- Data flows start with raw dataset preparation (instructions in README, data generation jobscripts) then pass through a homogenization process to extract patches and save data in an LMDB store.
- The processed data then feeds into two major model training pipelines: one that builds the “signature_net” and another that builds the “similarity_net”. These may work in tandem or sequentially in the flow scripts (run_flow.py series) to perform the camera model identification.
- The “sota” module (with its prnu functions) may operate as an alternative branch or additional analysis module alongside the signature and similarity networks.
- Utilities are used by all modules either during training (e.g., torchsummary, training_utils) or evaluation (e.g., cost_functions, evaluation_metrics), acting as shared resources.
- Visualization scripts (in miscellaneous/visualizations_for_publication) are deployed after training/evaluation to generate plots, heatmaps, parameter analyses, and more.

##### Architectural Patterns / Design Principles
- Modular Separation: The clear separation into distinct directories (data handling, model definition, orchestration, utilities) allows the architecture to be visualized as independent, loosely coupled components.
- Flow-Based Orchestration: Run_flow scripts and jobscripts indicate a linear or directed flow – starting with data extraction, moving into training and evaluation stages, and finally into visualization and publication.
- Test-Driven / Experimentation Focused: The presence of unit tests in the “scd_delivery/test” folder ensures the core computational modules are validated.

##### Relevant Technologies
- Python as the core language – with libraries that include PyTorch (hinted by torchsummary and typical deep learning module conventions).
- LMDB for storing preprocessed image patches.
- Shell scripts (within jobscripts) for automating different flows, data generation, and possibly deployment operations.

1. Data Preparation & Preprocessing
2. Model Training (Signature and Similarity Networks)
3. Evaluation and Testing
4. Visualization and Results Documentation
- Annotations or call-outs to mark down important scripts or configuration files like “configure.py”, “run_flow.py”, and note if the flows are orchestrated by shell scripts in the jobscripts folder.
- Finally, ensure you capture even utility integrations (e.g., logging, evaluation metrics), by placing these elements either as shared libraries or side components that interact with the main pipeline nodes.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation & Ingestion
    subgraph "Data Preparation & Ingestion"
        DPG1["jobscripts/data_gen.sh"]:::data
        DPG2["jobscripts/run_data_gen_script.sh"]:::data
        DPG3["miscellaneous/prepare_image_and_patch_data.py"]:::data
        DPG4["miscellaneous/prepare_image_and_patch_data_2.py"]:::data
        DP_END["LMDB Database"]:::data
    end

    %% Orchestration / Flow Control
    subgraph "Orchestration / Flow Control"
        OR1["run_flow.py"]:::orch
        OR2["jobscripts/run_flow.sh"]:::orch
        OR3["jobscripts/run_flow_3_then_2.sh"]:::orch
    end

    %% Model Training Phase
    subgraph "Model Training"
        %% Signature Net Module
        subgraph "Signature Net Module"
            SN1["signature_net/data_fft.py"]:::model
            SN2["signature_net/data_rgb.py"]:::model
            SN3["signature_net/models.py"]:::model
            SN4["signature_net/sig_net_flow.py"]:::model
            SN5["signature_net/utils.py"]:::model
            SN6["miscellaneous/scd_delivery/main/scd_modules/signature_net.py"]:::model
        end
        %% Similarity Net Module
        subgraph "Similarity Net Module"
            SIM1["similarity_net/data.py"]:::model
            SIM2["similarity_net/models.py"]:::model
            SIM3["similarity_net/sim_net_flow.py"]:::model
            SIM4["miscellaneous/scd_delivery/main/scd_modules/similarity_net.py"]:::model
        end
        %% SOTA / PRNU Module
        subgraph "SOTA / PRNU Module"
            SOTA1["sota/__init__.py"]:::model
            SOTA2["sota/prnu/__init__.py"]:::model
            SOTA3["sota/prnu/functions.py"]:::model
            SOTA4["sota/prnu/run_flow_prnu.py"]:::model
        end
    end

    %% Utilities and Evaluation Components
    subgraph "Utilities and Evaluation Components"
        UT1["utils/cost_functions.py"]:::util
        UT2["utils/evaluation_metrics.py"]:::util
        UT3["utils/logging.py"]:::util
        UT4["utils/torchsummary.py"]:::util
        UT5["utils/training_utils.py"]:::util
        UT6["utils/visualization_utils.py"]:::util
    end

    %% Visualization & Publication
    subgraph "Visualization & Publication"
        VIS1["miscellaneous/visualizations_for_publication/cm_brand_classification.py"]:::vis
        VIS2["miscellaneous/visualizations_for_publication/cm_hierarchical_classification.py"]:::vis
        VIS3["miscellaneous/visualizations_for_publication/cm_model_level_classification.py"]:::vis
        VIS4["miscellaneous/visualizations_for_publication/determine_num_of_params.py"]:::vis
        VIS5["miscellaneous/visualizations_for_publication/feature_maps.py"]:::vis
        VIS6["miscellaneous/visualizations_for_publication/image_patches_heatmap.py"]:::vis
        VIS7["miscellaneous/visualizations_for_publication/learning_curves.py"]:::vis
        VIS8["miscellaneous/visualizations_for_publication/measure_patch_extraction_time.py"]:::vis
        VIS9["miscellaneous/visualizations_for_publication/patch_std_ditribution.py"]:::vis
        VIS10["miscellaneous/visualizations_for_publication/plot_accuracy_vs_num_patches.py"]:::vis
    end

    %% Testing & Experimentation
    subgraph "Testing & Experimentation"
        TST1["miscellaneous/scd_delivery/test/test_compute_signature.py"]:::test
        TST2["miscellaneous/scd_delivery/test/test_compute_similarity.py"]:::test
    end

    %% Connections between phases
    DPG4 ---|"patches"| OR1
    DP_END ---|"patches"| OR1
    OR1 ---|"flow_control"| SN1
    OR1 ---|"flow_control"| SIM1
    OR1 ---|"flow_control"| SOTA1
    SN6 ---|"training_data"| UT1
    SIM4 ---|"training_data"| UT1
    SOTA4 ---|"analysis"| UT1
    UT1 ---|"evaluation"| VIS1
    VIS1 ---|"validation"| TST1

    %% Additional connections to indicate utilities across model modules
    SN6 ---|"uses"| UT2
    SIM4 ---|"uses"| UT2
    SOTA4 ---|"uses"| UT2

    %% Styles
    classDef data fill:#f9c,stroke:#333,stroke-width:2px;
    classDef orch fill:#afa,stroke:#333,stroke-width:2px;
    classDef model fill:#ccf,stroke:#333,stroke-width:2px;
    classDef util fill:#ffc,stroke:#333,stroke-width:2px;
    classDef vis fill:#cfc,stroke:#333,stroke-width:2px;
    classDef test fill:#fcc,stroke:#333,stroke-width:2px;

    %% Click Events for Data Preparation
    click DPG1 "https://github.com/bgswaroop/scd-images/blob/master/jobscripts/data_gen.sh"
    click DPG2 "https://github.com/bgswaroop/scd-images/blob/master/jobscripts/run_data_gen_script.sh"
    click DPG3 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/prepare_image_and_patch_data.py"
    click DPG4 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/prepare_image_and_patch_data_2.py"

    %% Click Events for Orchestration / Flow Control
    click OR1 "https://github.com/bgswaroop/scd-images/blob/master/run_flow.py"
    click OR2 "https://github.com/bgswaroop/scd-images/blob/master/jobscripts/run_flow.sh"
    click OR3 "https://github.com/bgswaroop/scd-images/blob/master/jobscripts/run_flow_3_then_2.sh"

    %% Click Events for Signature Net Module
    click SN1 "https://github.com/bgswaroop/scd-images/blob/master/signature_net/data_fft.py"
    click SN2 "https://github.com/bgswaroop/scd-images/blob/master/signature_net/data_rgb.py"
    click SN3 "https://github.com/bgswaroop/scd-images/blob/master/signature_net/models.py"
    click SN4 "https://github.com/bgswaroop/scd-images/blob/master/signature_net/sig_net_flow.py"
    click SN5 "https://github.com/bgswaroop/scd-images/blob/master/signature_net/utils.py"
    click SN6 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/scd_delivery/main/scd_modules/signature_net.py"

    %% Click Events for Similarity Net Module
    click SIM1 "https://github.com/bgswaroop/scd-images/blob/master/similarity_net/data.py"
    click SIM2 "https://github.com/bgswaroop/scd-images/blob/master/similarity_net/models.py"
    click SIM3 "https://github.com/bgswaroop/scd-images/blob/master/similarity_net/sim_net_flow.py"
    click SIM4 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/scd_delivery/main/scd_modules/similarity_net.py"

    %% Click Events for SOTA / PRNU Module
    click SOTA1 "https://github.com/bgswaroop/scd-images/blob/master/sota/__init__.py"
    click SOTA2 "https://github.com/bgswaroop/scd-images/blob/master/sota/prnu/__init__.py"
    click SOTA3 "https://github.com/bgswaroop/scd-images/blob/master/sota/prnu/functions.py"
    click SOTA4 "https://github.com/bgswaroop/scd-images/blob/master/sota/prnu/run_flow_prnu.py"

    %% Click Events for Utilities and Evaluation Components
    click UT1 "https://github.com/bgswaroop/scd-images/blob/master/utils/cost_functions.py"
    click UT2 "https://github.com/bgswaroop/scd-images/blob/master/utils/evaluation_metrics.py"
    click UT3 "https://github.com/bgswaroop/scd-images/blob/master/utils/logging.py"
    click UT4 "https://github.com/bgswaroop/scd-images/blob/master/utils/torchsummary.py"
    click UT5 "https://github.com/bgswaroop/scd-images/blob/master/utils/training_utils.py"
    click UT6 "https://github.com/bgswaroop/scd-images/blob/master/utils/visualization_utils.py"

    %% Click Events for Visualization & Publication
    click VIS1 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/visualizations_for_publication/cm_brand_classification.py"
    click VIS2 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/visualizations_for_publication/cm_hierarchical_classification.py"
    click VIS3 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/visualizations_for_publication/cm_model_level_classification.py"
    click VIS4 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/visualizations_for_publication/determine_num_of_params.py"
    click VIS5 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/visualizations_for_publication/feature_maps.py"
    click VIS6 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/visualizations_for_publication/image_patches_heatmap.py"
    click VIS7 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/visualizations_for_publication/learning_curves.py"
    click VIS8 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/visualizations_for_publication/measure_patch_extraction_time.py"
    click VIS9 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/visualizations_for_publication/patch_std_ditribution.py"
    click VIS10 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/visualizations_for_publication/plot_accuracy_vs_num_patches.py"

    %% Click Events for Testing & Experimentation
    click TST1 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/scd_delivery/test/test_compute_signature.py"
    click TST2 "https://github.com/bgswaroop/scd-images/blob/master/miscellaneous/scd_delivery/test/test_compute_similarity.py"
```
