How rnlkja/australia-social-media-analytics-on-the-cloud works
Main Components
• Backend Application – Represent the Flask API server (located in the “1_Flask_Backend” directory) that handles API requests, data ingestion, and integration with social media and spatial data.
• Data Harvesters – Within the Flask backend and the “harvester” subdirectory, highlight that there are dedicated modules for processing data from Mastodon and Twitter.
• Database Service – Include CouchDB (detailed in “3_CouchDB_database”). Clearly mark its purpose for storing data and serving queries. Note the two different access interfaces (Fauxton/Photon).
• Frontend Application – Based on ReactJS (found in “2_ReactJS_frontend”), which provides a user interface combining visualizations (graphs, maps, and content cards) with dynamic content, plus integration with static assets served via Nginx.
• Data Processing & Scraping – Represent the Python data processing pipelines and scripts in “4_Python_data_processing” where tasks like MapReduce operations, sentiment analysis, and data formatting are handled.
• Automation & Deployment – Indicate that Ansible (in “5_Ansible_IT_Automation”) is used to automate the deployment of the backend, database clusters, Docker swarm setup, and IT configurations.
Relationships and Interactions between Components
• From the Data Harvesters (integrated with the backend or as parallel pipelines) show connections to external social media APIs (Twitter, Mastodon) and then feeding into both the database and the data processing pipelines.
• That the Data Processing modules (in Python) can operate as offline jobs or scheduled tasks, processing the raw data and generating outputs for visualization.
Architectural Patterns and Design Principles
• Full-Stack Separation: Clearly separate the Frontend (ReactJS) from the Backend (Flask), emphasizing the client-server model.
• Microservices/Modular Design: Even if tightly coupled, the project divides responsibilities (harvester vs. API vs. data processing) into different directories/containers.
• Containerization: Identify that Docker is used (noted by Dockerfiles in several modules), meaning each service/component can be deployed independently.
• Automated Deployment: The presence of Ansible playbooks signals that the infrastructure is automated and follows DevOps best practices.
Relevant Technologies
• For the Backend: Python (with Flask) and associated utilities (like sentiment analysis scripts).
• For Harvesting: Python scripts specifically for Mastodon and Twitter data extraction.
• For the Frontend: ReactJS, Node.js ecosystem, Nginx configurations, Webpack, Tailwind CSS.
• For Database: CouchDB version 3.2.1.
• For Data Processing: Jupyter Notebooks, MPI, and MapReduce scripts written in JavaScript (for processing crime and income data) along with Python scripts.
• For IT Automation: Ansible along with shell scripts for Docker swarm and CouchDB cluster management.
Putting It All Together
– Below, indicate Data Harvesters that feed into the Backend.
– Show the Frontend connecting to the Backend to request and display processed visualizations.
– Finally, outline the IT Automation layer (Ansible) which automates the deployment and orchestration of all these containerized services.