How mikhailklassen/mining-the-social-web-3rd-edition works
Project type and purpose
• This is an open-source, instructional data-science toolkit. It provides Jupyter notebooks that demonstrate how to mine various social media platforms (Twitter, Facebook, Instagram, LinkedIn, GitHub, etc.) plus text files, web pages, mailboxes.
• The repository also includes support for running in cloud (Binder) and local (Docker, direct Python install) environments.
File structure
• Top-level:
• binder/ (apt.txt, postBuild, requirements.txt): environment specification for Binder/Docker.
• notebooks/: one notebook per chapter, each implementing a standalone data-mining workflow.
• notebooks/resources/: sample data and visualization outputs for each chapter.
• images/: static images used in the README or book.
• Root README.md: instructions for quickstart (Binder badge), Docker (repo2docker), manual setup (pip + Jupyter), and optional MongoDB for Chapter 9.
• No persistent backend services except optional MongoDB.
README for additional insights
• Dependency management via binder/requirements.txt and top-level requirements.txt.
• Binder provides an automated Docker image build and launch of Jupyter; repo2docker underpins this.
• MongoDB is only needed for a subset of examples (Chapter 9).
Main components & their relationships
a. Developer / User Machine
• Git client pulls the repo
• Python + pip installs dependencies from requirements.txt
• Jupyter Notebook server runs notebooks/
b. Binder Service (Cloud)
• Reads binder/ files → builds Docker image → launches Jupyter environment
• No state persisted between sessions
c. Docker (Local via repo2docker)
• Builds same image locally if users prefer not to use Binder
d. Jupyter Notebooks
• Each chapter is a standalone pipeline:
1) To external API or load local data (resources/)
2) Fetch or parse data → transform via Python libraries (requests, pandas, networkx, etc.)
3) Produce visualizations (HTML, JSON, KML) into notebooks/resources/chXX-*/viz
4) (Optional) Store/query data in MongoDB for Chapter 9
e. External Data Sources
• Social media APIs (Twitter, Facebook, Instagram, LinkedIn, GitHub)
• Local files (text, mailboxes, pickled graphs, CSVs)
f. Optional MongoDB Instance
• For advanced examples in Chapter 9; persists example data sets for querying via PyMongo
Architectural patterns & design principles
• Modular “one-chapter=one-pipeline” pattern
• Separation of concerns:
• Environment definition (binder/)
• Code (notebooks/)
• Sample data & output (notebooks/resources/)
• Stateless compute environments (Binder, Docker) vs. stateful data store (MongoDB)