How rodekruis/social-media-listening works
Project Overview
• This project is a dockerized Python-based data pipeline used to collect text messages from social media platforms (X and Telegram), translate them using Azure AI Translator, classify them with custom Huggingface models, and finally store the validated messages into Argilla.
• It supports both local execution (with Poetry) and deployment via Docker. Integration with external systems (such as Azure Translator, Argilla, and optionally Microsoft Azure SQL Server) is also a key aspect.
Analyzing the File Structure
• Top-Level Files and Folders:
• “Dockerfile”, “.dockerignore”, and GitHub workflows indicate containerization and CI/CD practices.
• “config” (with a config-template.yaml) and “credentials” are used to set up environment-specific settings and secrets.
• “pyproject.toml” manages Python dependencies via Poetry.
• “run_pipeline.py” likely serves as the entry script for the pipeline.
• The “sml” Folder:
• Contains the core pipeline modules (extract.py, transform.py, load.py, pipeline.py, etc.) for different stages of the ETL (Extract, Transform, Load) process.
• “message.py” and “topics.py” suggest specialized handling or enrichment of the text data.
• “secrets.py” indicates management for sensitive configuration elements.
• The “tests” Folder:
• Provides both unit and integration tests, confirming a modular and testable architecture.
Main Components
a. External Systems:
• Social media platforms (X and Telegram): the data sources.
• Azure AI Translator: used during the transformation phase.
• Custom classification models hosted on Huggingface for categorization of messages.
• Argilla and optionally Microsoft Azure SQL Server: destinations where the final data is stored.
b. Data Pipeline Components:
• Extract Module: Represented by sml/extract.py, this part handles API calls and data collection from the social media channels.
• Transform Module: Represented by sml/transform.py and related modules (such as sml/message.py), where language translation and text processing occur.
• Load Module: Represented by sml/load.py, responsible for inserting or storing the processed messages into Argilla (or SQL Server).
• Pipeline Orchestration: Indicated by sml/pipeline.py, which coordinates the execution flow between extraction, transformation, and loading.
c. Infrastructure & Configuration:
• Containerization (Dockerfile and associated configs) shows how the pipeline is deployed.
• Configuration and Secrets: The “config” and “credentials” directories store critical runtime parameters.
• CI/CD: GitHub Workflows mention automated build or update processes (e.g., update-image.yaml).
Relationships and Data Flow
• The extracted data then flows into the Transform module, where Azure AI Translator is called and custom classification models are used.
• After transformation, data flows to the Load module to be saved into Argilla (and/or SQL Server).
• Orchestration mechanisms (depicted via the pipeline module) manage the sequence of these transformations.
• Configuration and secrets are referenced at each stage to securely manage credentials and other parameters.
• Docker and deployment configuration dictate how external calls and integrations are packaged and executed.
Architectural Patterns
• ETL (Extract, Transform, Load): Clearly segmented into three stages.
• Modular design: Each stage (extract, transform, load) is encapsulated in separate Python modules, making the system testable and maintainable.
• External API Integration: Emphasis on interfacing with third-party services (social media APIs, translation services, machine learning models, and data storage).
• Containerized Deployment: Docker is used for both development and deployment, emphasizing consistency across environments.
Components’ Representation
• Data flow from external sources to the Extract Module.
• Sequential flow: Extract → Transform → Load.
• Side connections from the Transform stage to external services (like Azure Translator and Huggingface models).
• Interactions between pipeline modules and configuration management.