How meminee/dataengot7-branch works
Project’s primary components
• External Data Sources: Websites like Know Your Meme, Imgflip, and Wikidata.
• Scraping Services: Two separate scrapy-based projects located in the “scraping” folder (one for KYM and one for Imgflip). These scrapers use Redis as a job queue/cache and output data to databases.
• Data Ingestion: A landing zone implemented using MongoDB for raw meme data and using PostgreSQL to store relationships (such as parent–child links) between memes.
• Workflow Orchestration/ETL: Apache Airflow orchestrates the series of tasks (DAGs) found in the “dags” directory. These include tasks for scraping, cleansing (via PapermillOperator to run notebooks), transforming (using multiple extraction notebooks), and enhancing the data.
• Data Transformation & Enrichment: Jupyter notebooks (found in the “notebooks” folder) perform cleaning, extraction, and enrichment. These are automatically executed as part of the Airflow DAGs.
• Mappings & RDF Generation: The “mappings” directory holds various YAML, TTL, and shell scripts which are used to convert data into specific RDF formats. Also, there’s a “mapper.jar” indicating some Java-based processing.
• Data Presentation & Querying: Finally, the processed data is exposed via tools like Virtuoso (for SPARQL queries) and a Neo4j graph database (for visualizing meme relationships).
Sketch the Relationships and Data Flow
• To the Scraping Services:
• Data Ingestion:
• Workflow Orchestration:
► A large Airflow component that sits above the ingestion pipeline (drawing from “dags” and “Docker-compose” files). Within Airflow, show DAG tasks invoking Python, Docker, Papermill, and Bash operators.
• Data Transformation & Enrichment:
► A separate block labeled “Mappings & RDF Converter” driven by scripts and resources in the “mappings” directory.
• Final Presentation Components:
Architectural Patterns & Technologies
• Ingestion (Scraping and MongoDB/PostgreSQL),
• Processing (Airflow-managed ETL via notebooks and mappings),
• and Presentation (Neo4j/Virtuoso).
• Point out that containerization and orchestration via Docker (as seen in Dockerfile and docker-compose.yaml) are central to ensuring a consistent, deployable environment.
• That airflow operators (Python, Papermill, Docker, Bash) indicate a modular design allowing for extension and fault isolation.