How larryjiang134/image_manipulation_detection works
1. What this repository is
This is a TensorFlow 1.x implementation of a two-stream Faster R-CNN–style detector for image manipulation detection. The core idea is to combine a normal RGB stream with a second “noise” stream derived from SRM-like filtering, then use region proposals and ROI classification/regression to localize manipulated regions.
2. Top-level execution flow
The main pipeline is:
dataset creation → VOC-style training set on disk → training → checkpointed model → demo/inference.
main_create_training_set.py prepares the dataset, train.py trains the detector, and demo_two_stream.py is the inference/demo entry point.
3. Dataset and annotation boundary
The repository expects data in a PASCAL VOC-style layout, even for custom manipulation data.
lib/datasets/ contains dataset abstractions for:
- pascal_voc.py and voc_eval.py for VOC-style detection
- coco.py for COCO-style support
- DIY_pascal_voc.py for the custom manipulation dataset
- imdb.py, factory.py, roidb.py, and ds_utils.py for dataset registration and region-of-interest database handling
This is the main boundary between raw labeled images and the training code.
4. Training-data preparation
main_create_training_set.py is the offline preprocessing step that turns the source manipulation dataset into the VOC-like structure consumed by training.
The repo’s README indicates the generated dataset lives under default/DIY_dataset/default, which is the path the training pipeline expects.
5. Model architecture
The detector lives in lib/nets/, especially vgg16.py, which defines the two-stream network.
One stream operates on RGB input; the other processes a noise-enhanced representation. The README notes that the noise stream begins with SRM convolution layers, and that the two streams can both contribute to bounding-box prediction.
6. Backbone choices
The repo supports at least:
- vgg16.py as the main model used in the paper implementation
- resnet_v1.py as an alternate backbone implementation
network.py provides the shared base network abstraction used by these architectures.
7. Region proposal and ROI logic
The object-detection mechanics are classic Faster R-CNN:
- layer_utils/anchor_target_layer.py assigns training labels to anchors
- layer_utils/proposal_layer.py converts RPN outputs into proposals
- layer_utils/proposal_target_layer.py samples proposals for ROI training
- layer_utils/proposal_top_layer.py is another proposal-related variant
- layer_utils/roi_data_layer.py feeds minibatches of image/ROI data
This is the main boundary between dense convolutional features and region-level classification/regression.
8. Custom image-forensics feature processing
The repo includes specialized feature operators for manipulation detection:
- layer_utils/noise_stream_SRM_layer.py for the SRM-like noise stream
- lib/utils/compact_bilinear_pooling.py for feature fusion
- lib/utils/sequential_fft/ for FFT-based operations, with C++/CUDA support
These components are important because they are not standard Faster R-CNN parts; they implement the paper-specific feature extraction and fusion logic.
9. Compiled extensions and performance-sensitive code
Several parts are implemented as native extensions for speed:
- lib/utils/bbox.c / bbox.pyx
- lib/utils/sequential_fft/sequential_batch_fft_*
- data/coco/common/maskApi.c and related COCO mask code
- build artifacts under data/coco/PythonAPI/build
The repo is therefore a hybrid Python + C/C++/CUDA project, not pure Python.
10. Detection utilities
Common detection support code lives in lib/utils/:
- bbox_transform.py for box encoding/decoding
- blob.py and minibatch.py for input tensor assembly
- nms_wrapper.py and py_cpu_nms.py for non-maximum suppression
- timer.py for runtime measurement
- test.py and test_cbp.py for validation helpers
These form the glue between the dataset, network, and detection outputs.
11. Training/runtime dependencies
The implementation targets Python 3.6 and TensorFlow 1.8.0.
It also depends on a pretrained VGG16 ImageNet checkpoint for the RGB stream, while the noise stream is initially randomized unless you choose to pretrain it separately as the README suggests.
12. Vendored external evaluation packages
The data/coco/ tree contains the COCO API in Python, MATLAB, and Lua, plus some sample result files.
This looks like a vendored dependency for dataset evaluation and experimentation rather than the core manipulation-detection pipeline.
13. Operational outputs
Training writes logs for TensorBoard under /default, and model weights/checkpoints under the dataset/model directory mentioned in the README.
In architecture terms, the outputs are: event logs, learned checkpoints, and detection results from the demo/inference path.
14. Overall architecture summary
The repository is best understood as a custom two-stream region-based detector built on Faster R-CNN, with a strong separation between:
data preparation, dataset adapters, feature extraction/backbone, proposal/ROI detection layers, and specialized signal-processing utilities for manipulation forensics.
Main entry points: vgg16.py VGG16 · DIY_pascal_voc.py DIY VOC · main_create_training_set.py Prep set · train.py Train
Components 22
Pipeline
Offline dataset creation, training, and demo/inference entry points.
Data
Dataset adapters and ROI database handling for VOC-style manipulation data.
- Dataset factory · Registers and resolves dataset implementations.
lib/datasets/factory.py - IMDB base · Common dataset abstraction used by detection datasets.
lib/datasets/imdb.py - DIY VOC · Custom PASCAL VOC-style dataset for manipulation detection.
lib/datasets/DIY_pascal_voc.py - VOC eval · VOC-style detection evaluation helpers.
lib/datasets/voc_eval.py - ROIDB · Builds and filters region-of-interest databases.
lib/datasets/roidb.py - Dataset utils · Shared dataset helpers and XML/annotation handling.
lib/datasets/ds_utils.py
Detector
Two-stream Faster R-CNN backbone, proposal logic, and training data flow.
- VGG16 · Main two-stream detector with RGB and noise branches.
lib/nets/vgg16.py - ResNet · Alternate backbone implementation.
lib/nets/resnet_v1.py - Network base · Shared network abstraction for detector architectures.
lib/nets/network.py - Anchor targets · Assigns training labels to anchors for the RPN.
lib/layer_utils/anchor_target_layer.py - Proposals · Turns RPN outputs into region proposals.
lib/layer_utils/proposal_layer.py - ROI sampling · Samples proposals for classification and box regression training.
lib/layer_utils/proposal_target_layer.py - ROI data · Feeds minibatches of image and ROI data into training.
lib/layer_utils/roi_data_layer.py - SRM noise · Extracts manipulation-oriented noise features for the second stream.
lib/layer_utils/noise_stream_SRM_layer.py
Utils
Detection utilities and native performance-sensitive operators.
- CBP fusion · Compact bilinear pooling for feature fusion.
lib/utils/compact_bilinear_pooling.py - FFT ops · FFT-based feature processing and native kernels.
lib/utils/sequential_fft/sequential_batch_fft_ops.py - Box utils · Bounding-box transforms, minibatch assembly, NMS, and timing utilities.
lib/utils/bbox_transform.py - Native bbox · Compiled bounding-box extension.
lib/utils/bbox.c - COCO API · Vendored COCO evaluation and mask API package.
data/coco/PythonAPI/pycocotools/coco.py
Connections 22
- Prep set → DIY VOC: creates · Preprocesses source data into the dataset layout expected by training.
- Train → Dataset factory: loads · Resolves dataset definitions for training.
- Dataset factory → DIY VOC: registers · Exposes the custom manipulation dataset through the registry.
- DIY VOC → IMDB base: extends · Implements the common dataset abstraction.
- DIY VOC → ROIDB: uses · Builds ROI databases for the detector.
- DIY VOC → VOC eval: evaluates · Supports VOC-style evaluation.
- DIY VOC → Dataset utils: parses · Uses shared dataset helpers for annotation handling.
- Train → VGG16: trains · Main training script optimizes the two-stream detector.
- VGG16 → Network base: inherits · Built on the shared network base class.
- ResNet → Network base: inherits · Alternative backbone built on the same base.
- VGG16 → SRM noise: feeds · Uses SRM-like noise processing for the second stream.
- VGG16 → CBP fusion: fuses · Combines stream features through compact bilinear pooling.
- VGG16 → Anchor targets: rpn step · Learns anchor assignments for proposal generation.
- VGG16 → Proposals: rpn step · Converts RPN predictions into proposals.
- VGG16 → ROI sampling: roi step · Samples proposals for ROI classification/regression.
- ROI data → VGG16: feeds · Supplies minibatches to the detector during training.
- Box utils → VGG16: supports · Provides box encoding, minibatching, and suppression utilities.
- Native bbox → Box utils: accelerates · Compiled extension backing box operations.
- FFT ops → VGG16: supports · Specialized FFT kernels used by the model pipeline.
- COCO API → DIY VOC: reference · Vendored external COCO tooling for evaluation and experimentation.
- Demo → VGG16: loads · Demo path runs the trained detector for inference.
- Prep set → ROI data: prepares · Produces the training-ready ROI data consumed by the feeder.