# meminee/dataengot7-branch architecture

> Architecture diagram of the GitHub repository meminee/dataengot7-branch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/meminee/dataengot7-branch
- Repository: https://github.com/meminee/dataengot7-branch

## Overview

##### Project’s primary components
- External Data Sources: Websites like Know Your Meme, Imgflip, and Wikidata.
- Scraping Services: Two separate scrapy-based projects located in the “scraping” folder (one for KYM and one for Imgflip). These scrapers use Redis as a job queue/cache and output data to databases.
- Data Ingestion: A landing zone implemented using MongoDB for raw meme data and using PostgreSQL to store relationships (such as parent–child links) between memes.
- Workflow Orchestration/ETL: Apache Airflow orchestrates the series of tasks (DAGs) found in the “dags” directory. These include tasks for scraping, cleansing (via PapermillOperator to run notebooks), transforming (using multiple extraction notebooks), and enhancing the data.
- Data Transformation & Enrichment: Jupyter notebooks (found in the “notebooks” folder) perform cleaning, extraction, and enrichment. These are automatically executed as part of the Airflow DAGs.
- Mappings & RDF Generation: The “mappings” directory holds various YAML, TTL, and shell scripts which are used to convert data into specific RDF formats. Also, there’s a “mapper.jar” indicating some Java-based processing.
- Data Presentation & Querying: Finally, the processed data is exposed via tools like Virtuoso (for SPARQL queries) and a Neo4j graph database (for visualizing meme relationships).

##### Sketch the Relationships and Data Flow
- To the Scraping Services:
- Data Ingestion:
- Workflow Orchestration:
► A large Airflow component that sits above the ingestion pipeline (drawing from “dags” and “Docker-compose” files). Within Airflow, show DAG tasks invoking Python, Docker, Papermill, and Bash operators.
- Data Transformation & Enrichment:
► A separate block labeled “Mappings & RDF Converter” driven by scripts and resources in the “mappings” directory.
- Final Presentation Components:

##### Architectural Patterns & Technologies
- Ingestion (Scraping and MongoDB/PostgreSQL),
- Processing (Airflow-managed ETL via notebooks and mappings),
- and Presentation (Neo4j/Virtuoso).
- Point out that containerization and orchestration via Docker (as seen in Dockerfile and docker-compose.yaml) are central to ensuring a consistent, deployable environment.
- That airflow operators (Python, Papermill, Docker, Bash) indicate a modular design allowing for extension and fault isolation.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        A1["Know Your Meme"]:::external
        A2["Imgflip"]:::external
        A3["Wikidata"]:::external
    end

    %% Scraping Services
    subgraph "Scraping Services"
        S1["KYM Scraper (Scrapy)"]:::scraper
        S2["Imgflip Scraper (Scrapy)"]:::scraper
        R["Redis (JobQueue)"]:::service
    end

    %% Data Ingestion
    subgraph "Data Ingestion"
        DB1[("MongoDB: Raw Meme Data")]:::db
        DB2[("PostgreSQL: Relationships")]:::db
    end

    %% Workflow Orchestration / ETL
    subgraph "Workflow Orchestration / ETL"
        WF["Apache Airflow DAGs"]:::workflow
        DCS["Docker Setup"]:::docker
    end

    %% Data Transformation & Enrichment
    subgraph "Data Transformation & Enrichment"
        DT["Jupyter Notebooks (Cleaning,Extraction,Enrichment)"]:::transformation
    end

    %% Mappings & RDF Generation
    subgraph "Mappings & RDF Generation"
        MR["Mappings & RDF Converter"]:::mapping
    end

    %% Data Presentation & Querying
    subgraph "Data Presentation & Querying"
        P1["Virtuoso Query Layer"]:::presentation
        P2["Neo4j Graph DB"]:::presentation
    end

    %% Legend
    subgraph "Legend"
        L1["External: LightGreen"]:::external
        L2["Scraper: LightYellow"]:::scraper
        L3["Database: LightBlue"]:::db
        L4["Workflow: LightCoral"]:::workflow
        L5["Transformation: LightSkyBlue"]:::transformation
        L6["Mapping: LightGray"]:::mapping
        L7["Presentation: LightPink"]:::presentation
        L8["Docker: LightOrange"]:::docker
    end

    %% Connections: External Data Sources to Scrapers
    A1 -->|"scrapes"| S1
    A2 -->|"scrapes"| S2

    %% Scrapers use Redis
    S1 -- "uses" --> R
    S2 -- "uses" --> R

    %% Scrapers output to Data Ingestion
    S1 -->|"outputs"| DB1
    S2 -->|"outputs"| DB1
    S1 -->|"sends metadata"| DB2
    S2 -->|"sends metadata"| DB2

    %% Wikidata supports Enrichment via Airflow
    A3 -->|"enrichment"| WF

    %% Data Ingestion feeds into Workflow Orchestration
    DB1 -->|"ingested by"| WF
    DB2 -->|"ingested by"| WF

    %% Airflow triggers Data Transformation & Mappings
    WF -->|"executes notebooks"| DT
    WF -->|"runs mappings"| MR

    %% Docker Setup supports Airflow orchestration
    DCS -->|"orchestrates"| WF

    %% Transformation and Mappings supply data to Presentation Layer
    DT -->|"prepares data"| P1
    DT -->|"prepares data"| P2
    MR -->|"supplies RDF"| P1
    MR -->|"supplies RDF"| P2

    %% Click Events
    click S1 "https://github.com/meminee/dataengot7-branch/tree/main/scraping/kym-scrapper"
    click S2 "https://github.com/meminee/dataengot7-branch/tree/main/scraping/imgflip-scraper"
    click DB1 "https://github.com/meminee/dataengot7-branch/blob/main/dags/insert_mongo_data.py"
    click DB2 "https://github.com/meminee/dataengot7-branch/blob/main/dags/ingestion.py"
    click WF "https://github.com/meminee/dataengot7-branch/tree/main/dags"
    click DCS "https://github.com/meminee/dataengot7-branch/tree/main/Dockerfile"
    click DCS "https://github.com/meminee/dataengot7-branch/blob/main/docker-compose.yaml"
    click DT "https://github.com/meminee/dataengot7-branch/tree/main/notebooks"
    click MR "https://github.com/meminee/dataengot7-branch/tree/main/mappings"
    click MR "https://github.com/meminee/dataengot7-branch/blob/main/mappings/mapper.jar"

    %% Styles
    classDef external fill:#ccffcc,stroke:#006600,stroke-width:2px;
    classDef scraper fill:#ffffcc,stroke:#999900,stroke-width:2px;
    classDef service fill:#e0e0e0,stroke:#666666,stroke-width:2px;
    classDef db fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef workflow fill:#f8d7da,stroke:#721c24,stroke-width:2px;
    classDef transformation fill:#b3e5fc,stroke:#0277bd,stroke-width:2px;
    classDef mapping fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef presentation fill:#fce4ec,stroke:#880e4f,stroke-width:2px;
    classDef docker fill:#ffe0b2,stroke:#e65100,stroke-width:2px;
```
