# facebook/threatexchange architecture

> Architecture diagram of the GitHub repository facebook/threatexchange, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/facebook/threatexchange
- Repository: https://github.com/facebook/threatexchange

## Overview

##### High-Level Architecture Components

a) Core Hashing Algorithms
- PDQ (Image hashing)
- TMK (Video hashing)
- vPDQ (Video hashing)

b) Platform Components
- Hasher-Matcher-Actioner (HMA)
- Open Media Match
- ThreatExchange API Integration

c) Client Libraries & Tools
- python-threatexchange
- API Reference implementations

##### Component Relationships

a) Data Flow
- Content → Hashing Algorithms → Hash Signatures
- Hash Signatures → Matching Systems → Match Results
- ThreatExchange API ↔ Local Systems

b) Integration Points
- How HMA connects with hashing algorithms
- Demonstrate API client integration points
- Database and storage connections

##### Detailed Component Breakdown

For each major component, include:

a) Hasher-Matcher-Actioner (HMA)
- Background Tasks (fetcher, build_index)
- Blueprints (curation, hashing, matching)
- Storage Layer (postgres)
- API Endpoints

b) Hashing Algorithms
- Input Processors
- Hash Generators
- Matching Engines
- Storage Formats

c) ThreatExchange Integration
- Authentication
- API Clients
- Data Exchange Formats

##### Implementation Details

Technical specifications:

a) Languages/Frameworks
- C++ (PDQ, TMK, vPDQ)
- Python (threatexchange, HMA)
- Various API clients (Java, Ruby, PHP, etc.)

b) Storage Systems
- PostgreSQL
- File system storage
- Cache systems

##### Deployment Context

Deployment options:

a) HMA Deployment Patterns
- Single instance
- Redundant instances
- Split roles

##### Integration Flows

Workflows:

a) Content Processing
- Content ingestion
- Hash generation
- Matching process
- Action execution

b) Threat Exchange
- Data synchronization
- Update mechanisms
- API communication

Creating separate views:
##### Data Flow Patterns

This will help stakeholders understand the system at different levels of abstraction while maintaining clarity and usefulness of the documentation.

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Content ["Content Processing Layer"]
        Image[Image Files]
        Video[Video Files]
    end

    subgraph Hashing ["Hashing Algorithms"]
        PDQ["PDQ Image Hashing"]:::core
        TMK["TMK Video Hashing"]:::core
        VPDQ["vPDQ Video Hashing"]:::core
    end

    subgraph Matching ["Matching Systems"]
        PDQMatch["PDQ Matcher"]:::matcher
        TMKMatch["TMK Matcher"]:::matcher
        VPDQMatch["vPDQ Matcher"]:::matcher
    end

    subgraph HMA ["Hasher-Matcher-Actioner"]
        Core["HMA Core"]:::hma
        BackgroundTasks["Background Tasks"]:::hma
        APIEndpoints["API Endpoints"]:::hma
        Storage["Storage Layer"]:::storage
    end

    subgraph Integration ["External Integration"]
        ThreatExchange["ThreatExchange API"]:::external
        subgraph APIClients ["API Clients"]
            Python["Python Client"]:::client
            Java["Java Client"]:::client
            PHP["PHP Client"]:::client
            Ruby["Ruby Client"]:::client
            JS["JavaScript Client"]:::client
        end
    end

    %% Relationships
    Image --> PDQ
    Video --> TMK
    Video --> VPDQ

    PDQ --> PDQMatch
    TMK --> TMKMatch
    VPDQ --> VPDQMatch

    PDQMatch --> Core
    TMKMatch --> Core
    VPDQMatch --> Core

    Core --> BackgroundTasks
    Core --> APIEndpoints
    Core --> Storage
    Core <--> ThreatExchange

    ThreatExchange --> Python
    ThreatExchange --> Java
    ThreatExchange --> PHP
    ThreatExchange --> Ruby
    ThreatExchange --> JS

    %% Click Events
    click PDQ "https://github.com/facebook/ThreatExchange/blob/main/pdq/cpp/pdq/cpp/hashing/pdqhashing.cpp"
    click TMK "https://github.com/facebook/ThreatExchange/blob/main/tmk/cpp/algo/tmkfv.cpp"
    click VPDQ "https://github.com/facebook/ThreatExchange/blob/main/vpdq/cpp/vpdq/cpp/hashing/filehasher.cpp"
    click Core "https://github.com/facebook/ThreatExchange/blob/main/hasher-matcher-actioner/src/OpenMediaMatch/app.py"
    click BackgroundTasks "https://github.com/facebook/ThreatExchange/tree/main/hasher-matcher-actioner/src/OpenMediaMatch/background_tasks/"
    click APIEndpoints "https://github.com/facebook/ThreatExchange/tree/main/hasher-matcher-actioner/src/OpenMediaMatch/blueprints/"
    click Storage "https://github.com/facebook/ThreatExchange/tree/main/hasher-matcher-actioner/src/OpenMediaMatch/storage/postgres/"
    click ThreatExchange "https://github.com/facebook/ThreatExchange/tree/main/python-threatexchange/threatexchange/exchanges/clients/fb_threatexchange/"
    click Python "https://github.com/facebook/ThreatExchange/tree/main/api-reference-examples/python/"
    click Java "https://github.com/facebook/ThreatExchange/tree/main/api-reference-examples/java/"
    click PHP "https://github.com/facebook/ThreatExchange/tree/main/api-reference-examples/php/"
    click Ruby "https://github.com/facebook/ThreatExchange/tree/main/api-reference-examples/ruby/"
    click JS "https://github.com/facebook/ThreatExchange/tree/main/api-reference-examples/js/"
    click PDQMatch "https://github.com/facebook/ThreatExchange/blob/main/python-threatexchange/threatexchange/signal_type/pdq/pdq_faiss_matcher.py"
    click TMKMatch "https://github.com/facebook/ThreatExchange/blob/main/tmk/cpp/bin/tmk-query.cpp"
    click VPDQMatch "https://github.com/facebook/ThreatExchange/blob/main/vpdq/cpp/vpdq/cpp/hashing/matchTwoHash.cpp"

    %% Styling
    classDef core fill:#2374ab
    classDef matcher fill:#ff7f50
    classDef hma fill:#41b883
    classDef storage fill:#666666
    classDef external fill:#34495e
    classDef client fill:#95a5a6

    %% Legend
    subgraph Legend
        CoreL["Core Algorithms"]:::core
        MatcherL["Matching Systems"]:::matcher
        HMAL["HMA Components"]:::hma
        StorageL["Storage Systems"]:::storage
        ExternalL["External Systems"]:::external
        ClientL["API Clients"]:::client
    end
```
