# mikhailklassen/mining-the-social-web-3rd-edition architecture

> Architecture diagram of the GitHub repository mikhailklassen/mining-the-social-web-3rd-edition, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mikhailklassen/mining-the-social-web-3rd-edition
- Repository: https://github.com/mikhailklassen/mining-the-social-web-3rd-edition

## Overview

##### Project type and purpose
- This is an open-source, instructional data-science toolkit. It provides Jupyter notebooks that demonstrate how to mine various social media platforms (Twitter, Facebook, Instagram, LinkedIn, GitHub, etc.) plus text files, web pages, mailboxes.
- The repository also includes support for running in cloud (Binder) and local (Docker, direct Python install) environments.

##### File structure
- Top-level:
- binder/ (apt.txt, postBuild, requirements.txt): environment specification for Binder/Docker.
- notebooks/: one notebook per chapter, each implementing a standalone data-mining workflow.
- notebooks/resources/: sample data and visualization outputs for each chapter.
- images/: static images used in the README or book.
- Root README.md: instructions for quickstart (Binder badge), Docker (repo2docker), manual setup (pip + Jupyter), and optional MongoDB for Chapter 9.
- No persistent backend services except optional MongoDB.

##### README for additional insights
- Dependency management via binder/requirements.txt and top-level requirements.txt.
- Binder provides an automated Docker image build and launch of Jupyter; repo2docker underpins this.
- MongoDB is only needed for a subset of examples (Chapter 9).

##### Main components & their relationships
a. Developer / User Machine
- Git client pulls the repo
- Python + pip installs dependencies from requirements.txt
- Jupyter Notebook server runs notebooks/
b. Binder Service (Cloud)
- Reads binder/ files → builds Docker image → launches Jupyter environment
- No state persisted between sessions
c. Docker (Local via repo2docker)
- Builds same image locally if users prefer not to use Binder
d. Jupyter Notebooks
- Each chapter is a standalone pipeline:
1) To external API or load local data (resources/)
2) Fetch or parse data → transform via Python libraries (requests, pandas, networkx, etc.)
3) Produce visualizations (HTML, JSON, KML) into notebooks/resources/chXX-*/viz
4) (Optional) Store/query data in MongoDB for Chapter 9
e. External Data Sources
- Social media APIs (Twitter, Facebook, Instagram, LinkedIn, GitHub)
- Local files (text, mailboxes, pickled graphs, CSVs)
f. Optional MongoDB Instance
- For advanced examples in Chapter 9; persists example data sets for querying via PyMongo

##### Architectural patterns & design principles
- Modular “one-chapter=one-pipeline” pattern
- Separation of concerns:
- Environment definition (binder/)
- Code (notebooks/)
- Sample data & output (notebooks/resources/)
- Stateless compute environments (Binder, Docker) vs. stateful data store (MongoDB)

## Diagram (Mermaid)

```mermaid
graph TB
    %% Compute Environments
    subgraph "Compute Environments"
        Dev["Developer Machine"]:::compute
        Binder["Binder Service"]:::compute
        Docker["Docker Environment"]:::compute
    end

    %% Execution Environment
    Jupyter["Jupyter Notebook Server"]:::compute

    %% Pipelines
    subgraph "Jupyter Notebooks"
        NB1["Chapter 1 - Mining Twitter"]:::notebook
        NB2["Chapter 2 - Mining Facebook"]:::notebook
        NB3["Chapter 3 - Mining Instagram"]:::notebook
        NB4["Chapter 4 - Mining LinkedIn"]:::notebook
        NB5["Chapter 5 - Mining Text Files"]:::notebook
        NB6["Chapter 6 - Mining Web Pages"]:::notebook
        NB7["Chapter 7 - Mining Mailboxes"]:::notebook
        NB8["Chapter 8 - Mining GitHub"]:::notebook
        NB9["Chapter 9 - Twitter Cookbook"]:::notebook
        NA["_Appendix A - VM Experience"]:::notebook
        NB["_Appendix B - OAuth Primer"]:::notebook
        NC["_Appendix C - Python & Jupyter Tips"]:::notebook
    end

    %% Data Sources
    subgraph "Data Sources"
        APIs("External Social Media APIs"):::source
        LocalFiles("Local Files & Resources"):::source
    end

    %% Data Stores
    Resources["Sample Data & Viz (notebooks/resources/)"]:::store
    MongoDB["Optional MongoDB Instance"]:::store

    %% Static Resources
    Images["Static Images (images/)"]:::store

    %% Relationships
    Dev -->|clone repo| Jupyter
    Dev -->|pip install deps| Jupyter
    Binder -->|builds Docker image| Jupyter
    Docker -->|builds local image| Jupyter
    Jupyter -->|launch notebooks| NB1 & NB2 & NB3 & NB4 & NB5 & NB6 & NB7 & NB8 & NB9 & NA & NB & NC

    %% Pipelines to Data Sources and Stores
    NB1 & NB2 & NB3 & NB4 & NB5 & NB6 & NB7 & NB8 & NB9 & NA & NB & NC -->|HTTP REST API call| APIs
    NB5 & NB6 & NB7 -->|pandas read_*| LocalFiles
    NB9 -->|PyMongo driver| MongoDB
    NB1 & NB2 & NB3 & NB4 & NB5 & NB6 & NB7 & NB8 & NB9 & NA & NB & NC -->|write outputs| Resources

    %% Click Events
    click Dev "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/README.md"
    click Binder "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/binder/apt.txt"
    click Binder "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/tree/master/binder/postBuild"
    click Binder "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/binder/requirements.txt"
    click Jupyter "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/tree/master/notebooks/"
    click NB1 "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/Chapter 1 - Mining Twitter.ipynb"
    click NB2 "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/Chapter 2 - Mining Facebook.ipynb"
    click NB3 "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/Chapter 3 - Mining Instagram.ipynb"
    click NB4 "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/Chapter 4 - Mining LinkedIn.ipynb"
    click NB5 "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/Chapter 5 - Mining Text Files.ipynb"
    click NB6 "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/Chapter 6 - Mining Web Pages.ipynb"
    click NB7 "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/Chapter 7 - Mining Mailboxes.ipynb"
    click NB8 "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/Chapter 8 - Mining GitHub.ipynb"
    click NB9 "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/Chapter 9 - Twitter Cookbook.ipynb"
    click NA "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/_Appendix A - Virtual Machine Experience.ipynb"
    click NB "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/_Appendix B - OAuth Primer.ipynb"
    click NC "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/blob/master/notebooks/_Appendix C - Python & Jupyter Notebook Tips.ipynb"
    click Resources "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/tree/master/notebooks/resources/"
    click Images "https://github.com/mikhailklassen/mining-the-social-web-3rd-edition/tree/master/images/"

    %% Styles
    classDef compute fill:#ADD8E6,stroke:#000,color:#000
    classDef notebook fill:#D8BFD8,stroke:#000,color:#000
    classDef source fill:#90EE90,stroke:#000,color:#000
    classDef store fill:#FFA500,stroke:#000,color:#000
```
