How facebook/threatexchange works
High-Level Architecture Components
a) Core Hashing Algorithms
• PDQ (Image hashing)
• TMK (Video hashing)
• vPDQ (Video hashing)
b) Platform Components
• Hasher-Matcher-Actioner (HMA)
• Open Media Match
• ThreatExchange API Integration
c) Client Libraries & Tools
• python-threatexchange
• API Reference implementations
Component Relationships
a) Data Flow
• Content → Hashing Algorithms → Hash Signatures
• Hash Signatures → Matching Systems → Match Results
• ThreatExchange API ↔ Local Systems
b) Integration Points
• How HMA connects with hashing algorithms
• Demonstrate API client integration points
• Database and storage connections
Detailed Component Breakdown
For each major component, include:
a) Hasher-Matcher-Actioner (HMA)
• Background Tasks (fetcher, build_index)
• Blueprints (curation, hashing, matching)
• Storage Layer (postgres)
• API Endpoints
b) Hashing Algorithms
• Input Processors
• Hash Generators
• Matching Engines
• Storage Formats
c) ThreatExchange Integration
• Authentication
• API Clients
• Data Exchange Formats
Implementation Details
Technical specifications:
a) Languages/Frameworks
• C++ (PDQ, TMK, vPDQ)
• Python (threatexchange, HMA)
• Various API clients (Java, Ruby, PHP, etc.)
b) Storage Systems
• PostgreSQL
• File system storage
• Cache systems
Deployment Context
Deployment options:
a) HMA Deployment Patterns
• Single instance
• Redundant instances
• Split roles
Integration Flows
Workflows:
a) Content Processing
• Content ingestion
• Hash generation
• Matching process
• Action execution
b) Threat Exchange
• Data synchronization
• Update mechanisms
• API communication
Creating separate views:
Data Flow Patterns
This will help stakeholders understand the system at different levels of abstraction while maintaining clarity and usefulness of the documentation.