How chinoogawa/fbht works
Project type and purpose
• Type: A command-line, Python-based data-collection and graph-analysis tool
• Purpose: Automate Facebook data extraction (via Selenium/Mechanize), parse and store it, then compute and render social graphs using NetworkX/Matplotlib/Pygraphviz, finally dumping results as DOT/PDF/HTML.
File structure
• Top-level scripts (main.py, mainFunc.py, mainLib.py): orchestrate workflows
• Modules:
– MyParser.py – raw HTML/text parsing
– community.py – community-detection routines
– database.py – sqlite access (fb_db.db)
– handlers.py – low-level HTTP/form handlers (Mechanize)
• Data & dumps:
– fb_db.db – SQLite storage
– dumps/… – DOT, PDF, HTML graph artifacts
– massive/, cellphones/, PRIVATE/ – sample data sets
• Config/deps: setup.py, dependencias.txt
• Logging: logs/error.log
Key external dependencies (from README)
• Selenium (drives browser emulation)
• Mechanize (HTTP form automation)
• NetworkX, Pygraphviz, Matplotlib, Numpy, Simplejson
• Gephi (post-processing graphs)
Main components and interactions
a. User/CLI Layer
• main.py – parses CLI args, triggers workflows in mainFunc.py
b. Scraping Layer
• handlers.py + Selenium/Mechanize
• Invokes browser sessions, fetches pages
c. Parsing Layer
• MyParser.py – extracts user nodes, edges, metadata
d. Storage Layer
• database.py – writes/reads SQLite (fb_db.db)
e. Analysis Layer
• community.py – runs community detection algorithms on stored graph data
f. Visualization Layer
• mainLib.py + NetworkX/Matplotlib/Pygraphviz – builds and exports graph visualizations (DOT → PDF/PNG)
• Optional: Gephi import for interactive exploration
g. Output Layer
• dumps/… – file sinks for graphs, references, logs
• massive/, cellphones/, PRIVATE/ as data input or example repositories
h. Logging & Config
• logs/error.log – captures runtime errors
• setup.py/dependencias.txt – defines environment and dependencies
Architectural patterns & principles
• Modular layering: clear separation of scraping, parsing, storage, analysis, visualization
• Single responsibility per module (e.g., community.py only cares about graph algorithms)