# packtpublishing/advanced-web-scraping-with-python architecture

> Architecture diagram of the GitHub repository packtpublishing/advanced-web-scraping-with-python, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/packtpublishing/advanced-web-scraping-with-python
- Repository: https://github.com/packtpublishing/advanced-web-scraping-with-python

## Overview

##### Project type and purpose
- This is a desktop/video-processing application (not a web full-stack) with both GUI (gui.py) and CLI (backend/main.py), packaged in Docker and via conda/venv.
- Purpose: Remove hard-coded subtitles from video frames using AI inpainting algorithms (STTN, LAMA, ProPainter), with optional scene detection and customizable regions.

##### File structure
- Top-level:
- gui.py – desktop GUI entry point
- backend/ – core processing library and CLI
- docker/ – Dockerfile for containerized execution
- design/ – design papers & demos
- requirements.txt – Python dependencies
- test/ – sample videos
- backend/ organizes:
- config.py – runtime settings
- ffmpeg/ – platform-specific FFmpeg binaries for video I/O
- inpaint/ – inpainting algorithms:
– lama_inpaint.py (LAMA)
– sttn/ (STTN submodule)
– video/ (video inpainting pipeline: dataset, models, trainer, utilities)
– sttn_inpaint.py orchestrator
- scenedetect/ – video scene detection module (third-party integrated CLI)
- tools/ – helper scripts: train models, merge videos, common utilities
- models/ – pre-trained model files and manifests
- main.py – CLI entry point
- CI/CD (GitHub workflows) and Docker build for multiple GPU backends (CUDA, DirectML).

##### README for architecture cues
- Two execution modes: GUI and CLI
- Three inpainting strategies (configurable in config.py)
- Pre/post-processing: subtitle detection (scene or text region), frame extraction, inpainting, frame merging
- External dependencies: FFmpeg, PaddlePaddle GPU/CPU, PyTorch / torch_directml, ONNX runtime

a. Main components:
1. User Interface
- GUI Application (gui.py)
- CLI (backend/main.py)
2. Video I/O Layer
- FFmpeg binaries (platform-specific wrappers)
3. Pre-processing Services
- Scene Detection (backend/scenedetect)
- Subtitle/Text Region Detection (configurable STTN_SKIP_DETECTION or custom input)
4. Inpainting Core
- STTN Module (backend/inpaint/sttn)
- LAMA Module (backend/inpaint/lama_inpaint.py)
- ProPainter Module (backend/inpaint/video/model/propainter)
5. Model Storage
- Pre-trained model files (backend/models/*)
- Manifests for dynamic loading
6. Post-processing
- Video Frame Reassembly & Muxing (backend/tools/merge_video.py)
7. Training Tools (optional extension point)
- Trainer & configs (backend/tools/train)
8. Infrastructure
- Docker Container (docker/Dockerfile)
- CI/CD Pipelines (.github/workflows)

b. Relationships & Data Flow:
1. User → GUI/CLI
2. GUI/CLI → Video I/O (invokes FFmpeg to extract frames)
3. Extracted frames → Scene Detection → candidate segments
4. Frames + region info → Inpainting Core (choose STTN/LAMA/ProPainter based on config)
5. Inpainting → cleaned frames
6. Cleaned frames → Post-processing → FFmpeg muxe back to video
7. Output → User
8. Optional: Training Tools read datasets → train models → produce new model files

c. Architectural Patterns & Design Principles
- Pipeline pattern for sequential video processing (Extract → Detect → Inpaint → Reassemble)
- Plugin-style modules for inpainting algorithms (configured via config.py)
- Modular separation:
- UI vs. core processing
- Pre-processing vs. inpainting vs. post-processing
- Cross-platform binary management (ffmpeg folder)
- CI/CD for reproducible builds (Docker + GitHub Actions)

d. Key Technologies & Frameworks
- Python 3.12+
- FFmpeg (video I/O)
- PaddlePaddle GPU/CPU (STTN, LAMA via ONNX / PyTorch)
- PyTorch (ProPainter)
- ONNX Runtime DirectML (AMD/Intel GPU)
- GitHub Actions, Docker

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% UI Layer
    subgraph "User Interface"
        GUI["GUI Application"]:::ui
        CLI["CLI Application"]:::ui
    end

    %% Video I/O Layer
    subgraph "Video I/O Layer"
        FFmpeg["FFmpeg Binaries"]:::ext
    end

    %% Pre-processing Layer
    subgraph "Pre-processing Services"
        SceneDetect["Scene Detection Module"]:::core
    end

    %% Inpainting Pipeline
    subgraph "Inpainting Core"
        Orchestrator["Inpaint Orchestrator"]:::core
        subgraph "Algorithm Plugins"
            STTN["STTN Module"]:::core
            LAMA["LAMA Module"]:::core
            ProPainter["ProPainter Module"]:::core
        end
    end

    %% Model Storage
    Models[(Pre-trained Models & Manifests)]:::ext

    %% Post-processing Layer
    subgraph "Post-processing"
        Reassembler["Frame Reassembler & Muxing"]:::core
        Output["Output Video"]:::ext
    end

    %% Training Tools
    subgraph "Training Extension"
        Training["Training Tools & Configs"]:::infra
    end

    %% Infrastructure
    subgraph "Infrastructure & Deployment"
        Docker["Docker Container"]:::infra
        CICD["CI/CD Pipelines"]:::infra
    end

    %% Data Flow
    GUI -->|"start processing"| FFmpeg
    CLI -->|"start processing"| FFmpeg
    FFmpeg -->|"extract frames"| SceneDetect
    SceneDetect -->|"detect scenes/masks"| Orchestrator
    Orchestrator -->|"select algorithm"| STTN
    Orchestrator -->|"select algorithm"| LAMA
    Orchestrator -->|"select algorithm"| ProPainter
    STTN -->|"inpaint frames"| Reassembler
    LAMA -->|"inpaint frames"| Reassembler
    ProPainter -->|"inpaint frames"| Reassembler
    Models --> Orchestrator
    Reassembler -->|"mux video"| Output
    Training --> Models

    %% Infrastructure connections
    Docker --> GUI
    Docker --> CLI
    Docker --> FFmpeg
    Docker --> SceneDetect
    Docker --> Orchestrator
    CICD --> Docker

    %% Click Events
    click GUI "https://github.com/yaofanguk/video-subtitle-remover/blob/main/gui.py"
    click CLI "https://github.com/yaofanguk/video-subtitle-remover/blob/main/backend/main.py"
    click FFmpeg "https://github.com/yaofanguk/video-subtitle-remover/tree/main/backend/ffmpeg"
    click SceneDetect "https://github.com/yaofanguk/video-subtitle-remover/tree/main/backend/scenedetect"
    click Orchestrator "https://github.com/yaofanguk/video-subtitle-remover/blob/main/backend/inpaint/sttn_inpaint.py"
    click STTN "https://github.com/yaofanguk/video-subtitle-remover/tree/main/backend/inpaint/sttn"
    click LAMA "https://github.com/yaofanguk/video-subtitle-remover/blob/main/backend/inpaint/lama_inpaint.py"
    click ProPainter "https://github.com/yaofanguk/video-subtitle-remover/blob/main/backend/inpaint/video/model/propainter.py"
    click Models "https://github.com/yaofanguk/video-subtitle-remover/tree/main/backend/models"
    click Reassembler "https://github.com/yaofanguk/video-subtitle-remover/blob/main/backend/tools/merge_video.py"
    click Training "https://github.com/yaofanguk/video-subtitle-remover/tree/main/backend/tools/train"
    click Docker "https://github.com/yaofanguk/video-subtitle-remover/tree/main/docker/Dockerfile"
    click CICD "https://github.com/yaofanguk/video-subtitle-remover/blob/main/.github/workflows/*.yml"

    %% Styles
    classDef ui fill:#8FBC8F,stroke:#333,stroke-width:1px
    classDef core fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef ext fill:#FFD700,stroke:#333,stroke-width:1px
    classDef infra fill:#D3D3D3,stroke:#333,stroke-width:1px
```
