# larryjiang134/image_manipulation_detection architecture

> Architecture diagram of the GitHub repository larryjiang134/image_manipulation_detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/larryjiang134/image_manipulation_detection
- Repository: https://github.com/larryjiang134/image_manipulation_detection

## Overview

1. **What this repository is**
   This is a TensorFlow 1.x implementation of a **two-stream Faster R-CNN–style detector** for **image manipulation detection**. The core idea is to combine a normal RGB stream with a second “noise” stream derived from SRM-like filtering, then use region proposals and ROI classification/regression to localize manipulated regions.

2. **Top-level execution flow**
   The main pipeline is:
   **dataset creation** → **VOC-style training set on disk** → **training** → **checkpointed model** → **demo/inference**.  
   `main_create_training_set.py` prepares the dataset, `train.py` trains the detector, and `demo_two_stream.py` is the inference/demo entry point.

3. **Dataset and annotation boundary**
   The repository expects data in a **PASCAL VOC-style layout**, even for custom manipulation data.  
   `lib/datasets/` contains dataset abstractions for:
   - `pascal_voc.py` and `voc_eval.py` for VOC-style detection
   - `coco.py` for COCO-style support
   - `DIY_pascal_voc.py` for the custom manipulation dataset
   - `imdb.py`, `factory.py`, `roidb.py`, and `ds_utils.py` for dataset registration and region-of-interest database handling  
   This is the main boundary between raw labeled images and the training code.

4. **Training-data preparation**
   `main_create_training_set.py` is the offline preprocessing step that turns the source manipulation dataset into the VOC-like structure consumed by training.  
   The repo’s README indicates the generated dataset lives under `default/DIY_dataset/default`, which is the path the training pipeline expects.

5. **Model architecture**
   The detector lives in `lib/nets/`, especially `vgg16.py`, which defines the **two-stream network**.  
   One stream operates on RGB input; the other processes a noise-enhanced representation. The README notes that the noise stream begins with **SRM convolution layers**, and that the two streams can both contribute to bounding-box prediction.

6. **Backbone choices**
   The repo supports at least:
   - `vgg16.py` as the main model used in the paper implementation
   - `resnet_v1.py` as an alternate backbone implementation  
   `network.py` provides the shared base network abstraction used by these architectures.

7. **Region proposal and ROI logic**
   The object-detection mechanics are classic Faster R-CNN:
   - `layer_utils/anchor_target_layer.py` assigns training labels to anchors
   - `layer_utils/proposal_layer.py` converts RPN outputs into proposals
   - `layer_utils/proposal_target_layer.py` samples proposals for ROI training
   - `layer_utils/proposal_top_layer.py` is another proposal-related variant
   - `layer_utils/roi_data_layer.py` feeds minibatches of image/ROI data  
   This is the main boundary between dense convolutional features and region-level classification/regression.

8. **Custom image-forensics feature processing**
   The repo includes specialized feature operators for manipulation detection:
   - `layer_utils/noise_stream_SRM_layer.py` for the SRM-like noise stream
   - `lib/utils/compact_bilinear_pooling.py` for feature fusion
   - `lib/utils/sequential_fft/` for FFT-based operations, with C++/CUDA support  
   These components are important because they are not standard Faster R-CNN parts; they implement the paper-specific feature extraction and fusion logic.

9. **Compiled extensions and performance-sensitive code**
   Several parts are implemented as native extensions for speed:
   - `lib/utils/bbox.c` / `bbox.pyx`
   - `lib/utils/sequential_fft/sequential_batch_fft_*`
   - `data/coco/common/maskApi.c` and related COCO mask code
   - build artifacts under `data/coco/PythonAPI/build`  
   The repo is therefore a hybrid Python + C/C++/CUDA project, not pure Python.

10. **Detection utilities**
   Common detection support code lives in `lib/utils/`:
   - `bbox_transform.py` for box encoding/decoding
   - `blob.py` and `minibatch.py` for input tensor assembly
   - `nms_wrapper.py` and `py_cpu_nms.py` for non-maximum suppression
   - `timer.py` for runtime measurement
   - `test.py` and `test_cbp.py` for validation helpers  
   These form the glue between the dataset, network, and detection outputs.

11. **Training/runtime dependencies**
   The implementation targets **Python 3.6** and **TensorFlow 1.8.0**.  
   It also depends on a pretrained **VGG16 ImageNet checkpoint** for the RGB stream, while the noise stream is initially randomized unless you choose to pretrain it separately as the README suggests.

12. **Vendored external evaluation packages**
   The `data/coco/` tree contains the COCO API in Python, MATLAB, and Lua, plus some sample result files.  
   This looks like a vendored dependency for dataset evaluation and experimentation rather than the core manipulation-detection pipeline.

13. **Operational outputs**
   Training writes logs for **TensorBoard** under `/default`, and model weights/checkpoints under the dataset/model directory mentioned in the README.  
   In architecture terms, the outputs are: event logs, learned checkpoints, and detection results from the demo/inference path.

14. **Overall architecture summary**
   The repository is best understood as a **custom two-stream region-based detector** built on Faster R-CNN, with a strong separation between:
   **data preparation**, **dataset adapters**, **feature extraction/backbone**, **proposal/ROI detection layers**, and **specialized signal-processing utilities** for manipulation forensics.

Main entry points: [`lib/nets/vgg16.py`](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/nets/vgg16.py) (VGG16), [`lib/datasets/DIY_pascal_voc.py`](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/DIY_pascal_voc.py) (DIY VOC), [`main_create_training_set.py`](https://github.com/larryjiang134/image_manipulation_detection/blob/master/main_create_training_set.py) (Prep set), [`train.py`](https://github.com/larryjiang134/image_manipulation_detection/blob/master/train.py) (Train)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["Pipeline"]
  node_n1["Prep set<br/>script"]
  node_n2["Train<br/>script<br/>[train.py]"]
  node_n3["Demo<br/>script<br/>[demo_two_stream.py]"]
end

subgraph group_g2["Data"]
  node_n4["Dataset factory<br/>registry<br/>[factory.py]"]
  node_n5["IMDB base<br/>dataset base<br/>[imdb.py]"]
  node_n6["DIY VOC<br/>custom dataset<br/>[DIY_pascal_voc.py]"]
  node_n7["VOC eval<br/>evaluation<br/>[voc_eval.py]"]
  node_n8["ROIDB<br/>dataset prep<br/>[roidb.py]"]
  node_n9["Dataset utils<br/>[ds_utils.py]"]
end

subgraph group_g3["Detector"]
  node_n10["VGG16<br/>two-stream net<br/>[vgg16.py]"]
  node_n11["ResNet<br/>backbone<br/>[resnet_v1.py]"]
  node_n12["Network base<br/>[network.py]"]
  node_n13["Anchor targets<br/>rpn"]
  node_n14["Proposals<br/>rpn<br/>[proposal_layer.py]"]
  node_n15["ROI sampling"]
  node_n16["ROI data<br/>input feeder<br/>[roi_data_layer.py]"]
  node_n17["SRM noise<br/>feature layer"]
end

subgraph group_g4["Utils"]
  node_n18["CBP fusion"]
  node_n19["FFT ops<br/>native op"]
  node_n20["Box utils<br/>detection utils<br/>[bbox_transform.py]"]
  node_n21["Native bbox<br/>native ext<br/>[bbox.c]"]
  node_n22["COCO API<br/>vendored dep<br/>[coco.py]"]
end

node_n1 -->|"creates"| node_n6
node_n2 -->|"loads"| node_n4
node_n4 -->|"registers"| node_n6
node_n6 -->|"extends"| node_n5
node_n6 -->|"uses"| node_n8
node_n6 -->|"evaluates"| node_n7
node_n6 -->|"parses"| node_n9
node_n2 -->|"trains"| node_n10
node_n10 -->|"inherits"| node_n12
node_n11 -->|"inherits"| node_n12
node_n10 -->|"feeds"| node_n17
node_n10 -->|"fuses"| node_n18
node_n10 -->|"rpn step"| node_n13
node_n10 -->|"rpn step"| node_n14
node_n10 -->|"roi step"| node_n15
node_n16 -->|"feeds"| node_n10
node_n20 -->|"supports"| node_n10
node_n21 -->|"accelerates"| node_n20
node_n19 -->|"supports"| node_n10
node_n22 -.->|"reference"| node_n6
node_n3 -->|"loads"| node_n10
node_n1 -->|"prepares"| node_n16

click node_n1 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/main_create_training_set.py"
click node_n2 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/train.py"
click node_n3 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/demo_two_stream.py"
click node_n4 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/factory.py"
click node_n5 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/imdb.py"
click node_n6 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/DIY_pascal_voc.py"
click node_n7 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/voc_eval.py"
click node_n8 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/roidb.py"
click node_n9 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/ds_utils.py"
click node_n10 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/nets/vgg16.py"
click node_n11 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/nets/resnet_v1.py"
click node_n12 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/nets/network.py"
click node_n13 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/layer_utils/anchor_target_layer.py"
click node_n14 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/layer_utils/proposal_layer.py"
click node_n15 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/layer_utils/proposal_target_layer.py"
click node_n16 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/layer_utils/roi_data_layer.py"
click node_n17 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/layer_utils/noise_stream_SRM_layer.py"
click node_n18 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/utils/compact_bilinear_pooling.py"
click node_n19 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/utils/sequential_fft/sequential_batch_fft_ops.py"
click node_n20 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/utils/bbox_transform.py"
click node_n21 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/utils/bbox.c"
click node_n22 "https://github.com/larryjiang134/image_manipulation_detection/blob/master/data/coco/PythonAPI/pycocotools/coco.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2,node_n3 toneBlue
class node_n4,node_n5,node_n6,node_n7,node_n8,node_n9 toneAmber
class node_n10,node_n11,node_n12,node_n13,node_n14,node_n15,node_n16,node_n17 toneMint
class node_n18,node_n19,node_n20,node_n21,node_n22 toneRose
```

## Components

### Pipeline

Offline dataset creation, training, and demo/inference entry points.

- [Prep set](https://github.com/larryjiang134/image_manipulation_detection/blob/master/main_create_training_set.py): script · Builds the VOC-like training dataset from source manipulation images. · `main_create_training_set.py`
- [Train](https://github.com/larryjiang134/image_manipulation_detection/blob/master/train.py): script · Main training entry point for the detector. · `train.py`
- [Demo](https://github.com/larryjiang134/image_manipulation_detection/blob/master/demo_two_stream.py): script · Inference and demonstration entry point. · `demo_two_stream.py`

### Data

Dataset adapters and ROI database handling for VOC-style manipulation data.

- [Dataset factory](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/factory.py): registry · Registers and resolves dataset implementations. · `lib/datasets/factory.py`
- [IMDB base](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/imdb.py): dataset base · Common dataset abstraction used by detection datasets. · `lib/datasets/imdb.py`
- [DIY VOC](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/DIY_pascal_voc.py): custom dataset · Custom PASCAL VOC-style dataset for manipulation detection. · `lib/datasets/DIY_pascal_voc.py`
- [VOC eval](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/voc_eval.py): evaluation · VOC-style detection evaluation helpers. · `lib/datasets/voc_eval.py`
- [ROIDB](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/roidb.py): dataset prep · Builds and filters region-of-interest databases. · `lib/datasets/roidb.py`
- [Dataset utils](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/datasets/ds_utils.py): dataset utils · Shared dataset helpers and XML/annotation handling. · `lib/datasets/ds_utils.py`

### Detector

Two-stream Faster R-CNN backbone, proposal logic, and training data flow.

- [VGG16](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/nets/vgg16.py): two-stream net · Main two-stream detector with RGB and noise branches. · `lib/nets/vgg16.py`
- [ResNet](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/nets/resnet_v1.py): backbone · Alternate backbone implementation. · `lib/nets/resnet_v1.py`
- [Network base](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/nets/network.py): network base · Shared network abstraction for detector architectures. · `lib/nets/network.py`
- [Anchor targets](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/layer_utils/anchor_target_layer.py): rpn · Assigns training labels to anchors for the RPN. · `lib/layer_utils/anchor_target_layer.py`
- [Proposals](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/layer_utils/proposal_layer.py): rpn · Turns RPN outputs into region proposals. · `lib/layer_utils/proposal_layer.py`
- [ROI sampling](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/layer_utils/proposal_target_layer.py): roi · Samples proposals for classification and box regression training. · `lib/layer_utils/proposal_target_layer.py`
- [ROI data](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/layer_utils/roi_data_layer.py): input feeder · Feeds minibatches of image and ROI data into training. · `lib/layer_utils/roi_data_layer.py`
- [SRM noise](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/layer_utils/noise_stream_SRM_layer.py): feature layer · Extracts manipulation-oriented noise features for the second stream. · `lib/layer_utils/noise_stream_SRM_layer.py`

### Utils

Detection utilities and native performance-sensitive operators.

- [CBP fusion](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/utils/compact_bilinear_pooling.py): fusion · Compact bilinear pooling for feature fusion. · `lib/utils/compact_bilinear_pooling.py`
- [FFT ops](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/utils/sequential_fft/sequential_batch_fft_ops.py): native op · FFT-based feature processing and native kernels. · `lib/utils/sequential_fft/sequential_batch_fft_ops.py`
- [Box utils](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/utils/bbox_transform.py): detection utils · Bounding-box transforms, minibatch assembly, NMS, and timing utilities. · `lib/utils/bbox_transform.py`
- [Native bbox](https://github.com/larryjiang134/image_manipulation_detection/blob/master/lib/utils/bbox.c): native ext · Compiled bounding-box extension. · `lib/utils/bbox.c`
- [COCO API](https://github.com/larryjiang134/image_manipulation_detection/blob/master/data/coco/PythonAPI/pycocotools/coco.py): vendored dep · Vendored COCO evaluation and mask API package. · `data/coco/PythonAPI/pycocotools/coco.py`

## Connections

- Prep set → DIY VOC: creates · Preprocesses source data into the dataset layout expected by training.
- Train → Dataset factory: loads · Resolves dataset definitions for training.
- Dataset factory → DIY VOC: registers · Exposes the custom manipulation dataset through the registry.
- DIY VOC → IMDB base: extends · Implements the common dataset abstraction.
- DIY VOC → ROIDB: uses · Builds ROI databases for the detector.
- DIY VOC → VOC eval: evaluates · Supports VOC-style evaluation.
- DIY VOC → Dataset utils: parses · Uses shared dataset helpers for annotation handling.
- Train → VGG16: trains · Main training script optimizes the two-stream detector.
- VGG16 → Network base: inherits · Built on the shared network base class.
- ResNet → Network base: inherits · Alternative backbone built on the same base.
- VGG16 → SRM noise: feeds · Uses SRM-like noise processing for the second stream.
- VGG16 → CBP fusion: fuses · Combines stream features through compact bilinear pooling.
- VGG16 → Anchor targets: rpn step · Learns anchor assignments for proposal generation.
- VGG16 → Proposals: rpn step · Converts RPN predictions into proposals.
- VGG16 → ROI sampling: roi step · Samples proposals for ROI classification/regression.
- ROI data → VGG16: feeds · Supplies minibatches to the detector during training.
- Box utils → VGG16: supports · Provides box encoding, minibatching, and suppression utilities.
- Native bbox → Box utils: accelerates · Compiled extension backing box operations.
- FFT ops → VGG16: supports · Specialized FFT kernels used by the model pipeline.
- COCO API → DIY VOC: reference · Vendored external COCO tooling for evaluation and experimentation.
- Demo → VGG16: loads · Demo path runs the trained detector for inference.
- Prep set → ROI data: prepares · Produces the training-ready ROI data consumed by the feeder.
