cle07/memoire-hn-tpot

Purpose and execution model

This is a research-analysis repository supporting a digital-humanities thesis on a Twitter/X subculture using the Community Archive dataset. It is notebook-led rather than an application: notebooks perform exploratory, text, embedding, network, and temporal analyses; scripts acquire source archives and produce graph exports.

Primary entry points:

• scripts/fetch_data.py

• scripts/export_gephi.py

• notebooks/embeddings_db.ipynb

• notebooks/network_analysis.ipynb

Data acquisition and local corpus

fetch_data.py retrieves archive releases from the Community Archive online API; releases can alternatively be sourced from the upstream GitHub release repository. Retrieved raw and preprocessed datasets are persisted under data/, organized by archive version/date and analytical format.

Key storage boundaries:

• data/2024-9-8

• data/2026

• data/parquet_files

• data/DATA_STRUCTURE.md

The archive data itself is intentionally excluded from Git, so reproducibility depends on fetching releases or restoring local datasets.

Analysis workspace

Jupyter notebooks are the main control plane. They consume local archive data and write visual outputs into a shared output area:

• notebooks/structure_exploration.ipynb

• notebooks/text_analysis.ipynb

• notebooks/temporal_comparison.ipynb

• notebooks/output

structure_exploration establishes dataset understanding; temporal_comparison compares releases spanning 2024, 2025, and June 2026. This creates a boundary between versioned source data and derived comparative findings.

Vector and topic-analysis pipeline

The embedding workflow generates text embeddings, indexes them in ChromaDB, and reuses indexed vectors for downstream PCA, UMAP, HDBSCAN, and BERTopic analysis. Its stated design avoids re-embedding during topic-model fitting.

Core artifacts:

• notebooks/embeddings_db.ipynb

• chroma_db/chroma.sqlite3

• notebooks/text_analysis.ipynb

ChromaDB is the material persistence layer for vector retrieval. Text-oriented analysis remains distinct: tweet text feeds topic, emotion, named-entity, lexical-field, and word-cloud work rather than relying on vectors.

Network-analysis and export boundary

Network analysis derives social graphs from archive relationships in the notebook environment, while the export script creates files intended for interactive visualization in Gephi:

• notebooks/network_analysis.ipynb

• scripts/export_gephi.py

• data/graphs

This separates computational network analysis from external graph inspection and presentation in Gephi.

Runtime and external dependencies

Python is the runtime, pinned to a project-specific version via .python-version. uv manages the virtual environment and locked dependencies:

• pyproject.toml

• uv.lock

• .python-version

Material external services/tools are the Community Archive API, the upstream Community Archive GitHub releases, ChromaDB for local vector storage, and Gephi for graph visualization. Generated figures are retained separately from notebooks in notebooks/output.