How khanxmetu/interpol-red-dl works
Main Components
• api-poller:
– Purpose: Polls the Interpol Red Notices API at regular intervals.
– Key functions: Establishes HTTP sessions (using customized headers from config-data), applies configurable query strategies (e.g., simple default search and brute-force search), and sends retrieved notice data to a messaging queue via RabbitMQ (see api_poller/api_request_session.py and query_strategy.py).
– Technology: Python, Dockerized via its own Dockerfile.
• notice-saver:
– Purpose: Consumes notices from RabbitMQ, downloads associated images, and performs additional processing (“offense type classification”) before saving the notice details into a database.
– Key functions: Consuming messages (using rabbitmq_client.py), classification (via BasicOffenseTypeClassifier and placeholder GPTOffenseTypeClassifier), and repository management for persistence.
– Technology: Python, with Docker and configuration files for header spoofing.
• notice-dashboard:
– Purpose: Provides an interactive visualization and data analytics interface (built with Streamlit) for viewing the collected notices and associated statistics.
– Key functions: UI Components for notice details, lists, maps, and charts (files in ui_components and pages directories), and database access through repositories.
– Technology: Python/Streamlit, Dockerized.
• Supporting components (External/Infrastructure):
– RabbitMQ: Acts as the message broker between api-poller and notice-saver.
– Database: Although not explicitly shown in the file tree with a dedicated directory, notice-saver and notice-dashboard reference “db.py” and repository modules that imply a common datastore for persisted notices.
Relationships and Data Flow
• The api-poller periodically retrieves data from the Interpol API and pushes notice information to a RabbitMQ queue.
• The notice-saver reads from the RabbitMQ queue, processes the data (including image downloading and offense classification), then saves it to the database.
• The notice-dashboard connects to the same database to display up-to-date notices and related visual analytics.
• Optional flows include configuration settings (like header spoofing) and logging/exception handling between components.
Architectural Patterns and Design Principles
• Separation of Concerns: Each service has a dedicated role (polling, saving/processing, and displaying), ensuring loose coupling between components.
• Message-Driven Architecture: RabbitMQ acts as the mediator, decoupling the api-poller and notice-saver, which gives scalability and resilience to message queuing.
• Containerization: All services are isolated within Docker containers, making deployment consistent and environment-independent.
• Modularity and Testability: Notice the presence of tests for each module, suggesting an emphasis on clear module boundaries and unit testing.