# rnlkja/australia-social-media-analytics-on-the-cloud architecture

> Architecture diagram of the GitHub repository rnlkja/australia-social-media-analytics-on-the-cloud, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rnlkja/australia-social-media-analytics-on-the-cloud
- Repository: https://github.com/rnlkja/australia-social-media-analytics-on-the-cloud

## Overview

##### Main Components
- Backend Application – Represent the Flask API server (located in the “1_Flask_Backend” directory) that handles API requests, data ingestion, and integration with social media and spatial data.
- Data Harvesters – Within the Flask backend and the “harvester” subdirectory, highlight that there are dedicated modules for processing data from Mastodon and Twitter.
- Database Service – Include CouchDB (detailed in “3_CouchDB_database”). Clearly mark its purpose for storing data and serving queries. Note the two different access interfaces (Fauxton/Photon).
- Frontend Application – Based on ReactJS (found in “2_ReactJS_frontend”), which provides a user interface combining visualizations (graphs, maps, and content cards) with dynamic content, plus integration with static assets served via Nginx.
- Data Processing & Scraping – Represent the Python data processing pipelines and scripts in “4_Python_data_processing” where tasks like MapReduce operations, sentiment analysis, and data formatting are handled.
- Automation & Deployment – Indicate that Ansible (in “5_Ansible_IT_Automation”) is used to automate the deployment of the backend, database clusters, Docker swarm setup, and IT configurations.

##### Relationships and Interactions between Components
- From the Data Harvesters (integrated with the backend or as parallel pipelines) show connections to external social media APIs (Twitter, Mastodon) and then feeding into both the database and the data processing pipelines.
- That the Data Processing modules (in Python) can operate as offline jobs or scheduled tasks, processing the raw data and generating outputs for visualization.

##### Architectural Patterns and Design Principles
- Full-Stack Separation: Clearly separate the Frontend (ReactJS) from the Backend (Flask), emphasizing the client-server model.
- Microservices/Modular Design: Even if tightly coupled, the project divides responsibilities (harvester vs. API vs. data processing) into different directories/containers.
- Containerization: Identify that Docker is used (noted by Dockerfiles in several modules), meaning each service/component can be deployed independently.
- Automated Deployment: The presence of Ansible playbooks signals that the infrastructure is automated and follows DevOps best practices.

##### Relevant Technologies
- For the Backend: Python (with Flask) and associated utilities (like sentiment analysis scripts).
- For Harvesting: Python scripts specifically for Mastodon and Twitter data extraction.
- For the Frontend: ReactJS, Node.js ecosystem, Nginx configurations, Webpack, Tailwind CSS.
- For Database: CouchDB version 3.2.1.
- For Data Processing: Jupyter Notebooks, MPI, and MapReduce scripts written in JavaScript (for processing crime and income data) along with Python scripts.
- For IT Automation: Ansible along with shell scripts for Docker swarm and CouchDB cluster management.

##### Putting It All Together
– Below, indicate Data Harvesters that feed into the Backend.
– Show the Frontend connecting to the Backend to request and display processed visualizations.
– Finally, outline the IT Automation layer (Ansible) which automates the deployment and orchestration of all these containerized services.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        extTwitter["Twitter API"]:::external
        extMastodon["Mastodon API"]:::external
        extSUDO["SUDO Official Data"]:::external
    end

    %% Frontend Layer
    subgraph "Frontend Layer"
        FE["ReactJS Frontend"]:::frontend
    end

    %% Backend Layer
    subgraph "Backend Layer"
        BE["Flask API Backend"]:::backend
        DH["Data Harvesters"]:::backend
    end

    %% Database Service
    subgraph "Database Service"
        DB["CouchDB Database"]:::database
        DBAccess["CouchDB Access Code"]:::database
    end

    %% Data Processing Layer
    subgraph "Data Processing"
        DP["Data Processing Pipelines"]:::processing
    end

    %% Automation & Deployment
    subgraph "Automation & Deployment"
        AD["Ansible Deployment"]:::automation
    end

    %% Connections between External Sources and Backend Components
    extTwitter -->|"Ingests"| DH
    extMastodon -->|"Ingests"| DH
    extSUDO -->|"Feeds"| BE

    %% Internal Backend connections
    DH -->|"Integrates"| BE
    FE -->|"REST_API"| BE

    %% Backend to Database interactions
    BE -->|"Data_ReadWrite"| DB
    BE -->|"Uses"| DBAccess

    %% Data Processing flows
    DH -->|"RawData"| DP
    DP -->|"ProcessedData"| FE

    %% Automation orchestrates all components
    AD ---|"Manages"| BE
    AD ---|"Manages"| DB
    AD ---|"Manages"| DP
    AD ---|"Manages"| FE

    %% Styles
    classDef external fill:#f9e79f,stroke:#333,stroke-width:2px;
    classDef frontend fill:#aed6f1,stroke:#333,stroke-width:2px;
    classDef backend fill:#a9dfbf,stroke:#333,stroke-width:2px;
    classDef database fill:#fadbd8,stroke:#333,stroke-width:2px;
    classDef processing fill:#d6eaf8,stroke:#333,stroke-width:2px;
    classDef automation fill:#f5cba7,stroke:#333,stroke-width:2px;

    %% Click Events
    click BE "https://github.com/rnlkja/australia-social-media-analytics-on-the-cloud/tree/main/1_Flask_Backend/flask-backend"
    click DH "https://github.com/rnlkja/australia-social-media-analytics-on-the-cloud/tree/main/1_Flask_Backend/harvester"
    click DB "https://github.com/rnlkja/australia-social-media-analytics-on-the-cloud/tree/main/3_CouchDB_database"
    click DBAccess "https://github.com/rnlkja/australia-social-media-analytics-on-the-cloud/blob/main/1_Flask_Backend/flask-backend/database/CouchDB.py"
    click FE "https://github.com/rnlkja/australia-social-media-analytics-on-the-cloud/tree/main/2_ReactJS_frontend/frontend"
    click DP "https://github.com/rnlkja/australia-social-media-analytics-on-the-cloud/tree/main/4_Python_data_processing"
    click AD "https://github.com/rnlkja/australia-social-media-analytics-on-the-cloud/tree/main/5_Ansible_IT_Automation"
```
