# cle07/memoire-hn-tpot architecture

> Architecture diagram of the GitHub repository cle07/memoire-hn-tpot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-08-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cle07/memoire-hn-tpot
- Repository: https://github.com/cle07/memoire-hn-tpot

## Overview

#### Purpose and execution model
This is a research-analysis repository supporting a digital-humanities thesis on a Twitter/X subculture using the Community Archive dataset. It is notebook-led rather than an application: notebooks perform exploratory, text, embedding, network, and temporal analyses; scripts acquire source archives and produce graph exports.

Primary entry points:
- `scripts/fetch_data.py`
- `scripts/export_gephi.py`
- `notebooks/embeddings_db.ipynb`
- `notebooks/network_analysis.ipynb`

#### Data acquisition and local corpus
`fetch_data.py` retrieves archive releases from the Community Archive online API; releases can alternatively be sourced from the upstream GitHub release repository. Retrieved raw and preprocessed datasets are persisted under `data/`, organized by archive version/date and analytical format.

Key storage boundaries:
- `data/2024-9-8`
- `data/2026`
- `data/parquet_files`
- `data/DATA_STRUCTURE.md`

The archive data itself is intentionally excluded from Git, so reproducibility depends on fetching releases or restoring local datasets.

#### Analysis workspace
Jupyter notebooks are the main control plane. They consume local archive data and write visual outputs into a shared output area:
- `notebooks/structure_exploration.ipynb`
- `notebooks/text_analysis.ipynb`
- `notebooks/temporal_comparison.ipynb`
- `notebooks/output`

`structure_exploration` establishes dataset understanding; `temporal_comparison` compares releases spanning 2024, 2025, and June 2026. This creates a boundary between versioned source data and derived comparative findings.

#### Vector and topic-analysis pipeline
The embedding workflow generates text embeddings, indexes them in ChromaDB, and reuses indexed vectors for downstream PCA, UMAP, HDBSCAN, and BERTopic analysis. Its stated design avoids re-embedding during topic-model fitting.

Core artifacts:
- `notebooks/embeddings_db.ipynb`
- `chroma_db/chroma.sqlite3`
- `notebooks/text_analysis.ipynb`

ChromaDB is the material persistence layer for vector retrieval. Text-oriented analysis remains distinct: tweet text feeds topic, emotion, named-entity, lexical-field, and word-cloud work rather than relying on vectors.

#### Network-analysis and export boundary
Network analysis derives social graphs from archive relationships in the notebook environment, while the export script creates files intended for interactive visualization in Gephi:
- `notebooks/network_analysis.ipynb`
- `scripts/export_gephi.py`
- `data/graphs`

This separates computational network analysis from external graph inspection and presentation in Gephi.

#### Runtime and external dependencies
Python is the runtime, pinned to a project-specific version via `.python-version`. `uv` manages the virtual environment and locked dependencies:
- `pyproject.toml`
- `uv.lock`
- `.python-version`

Material external services/tools are the Community Archive API, the upstream Community Archive GitHub releases, ChromaDB for local vector storage, and Gephi for graph visualization. Generated figures are retained separately from notebooks in `notebooks/output`.

Main entry points: [`data/parquet_files`](https://github.com/cle07/memoire-hn-tpot/tree/main/data/parquet_files) (Parquet datasets), [`scripts/fetch_data.py`](https://github.com/cle07/memoire-hn-tpot/blob/main/scripts/fetch_data.py) (Archive fetcher), [`notebooks/structure_exploration.ipynb`](https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/structure_exploration.ipynb) (Structure exploration), [`pyproject.toml`](https://github.com/cle07/memoire-hn-tpot/blob/main/pyproject.toml) (Python + uv)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Python environment"]
  node_python_env{{"Python + uv<br/>runtime<br/>[pyproject.toml]"}}
end

subgraph group_sources["Archive sources"]
  node_archive_api(("Community Archive API<br/>external API"))
  node_github_releases(("Upstream GitHub releases<br/>release source"))
  node_fetch_data["Archive fetcher<br/>Python script<br/>[fetch_data.py]"]
end

subgraph group_storage["Local data &amp; artifacts"]
  node_versioned_data[("Versioned archive data<br/>local corpus")]
  node_current_data[("2026 archive data<br/>local corpus")]
  node_parquet_data[("Parquet datasets<br/>analytical format")]
  node_data_structure["Data structure guide<br/>dataset documentation<br/>[DATA_STRUCTURE.md]"]
  node_chroma_db[("ChromaDB vectors<br/>vector store<br/>[chroma.sqlite3]")]
end

subgraph group_analysis["Notebook analysis"]
  node_structure_notebook["Structure exploration<br/>Jupyter notebook"]
  node_embedding_notebook["Embedding workflow<br/>Jupyter notebook"]
  node_text_notebook["Text &amp; topic analysis<br/>Jupyter notebook"]
  node_temporal_notebook["Temporal comparison<br/>Jupyter notebook"]
  node_figures["Analysis outputs<br/>generated figures"]
end

subgraph group_network["Network export"]
  node_network_notebook["Network analysis<br/>Jupyter notebook"]
  node_gephi_export["Gephi exporter<br/>Python script<br/>[export_gephi.py]"]
  node_graph_files["Graph exports<br/>visualization files"]
  node_gephi(("Gephi<br/>external visualization tool"))
end

node_python_env -->|"runs"| node_fetch_data
node_python_env -->|"runs"| node_structure_notebook
node_python_env -->|"runs"| node_embedding_notebook
node_archive_api -->|"archive releases"| node_fetch_data
node_github_releases -.->|"alternative source"| node_fetch_data
node_fetch_data -->|"persists"| node_versioned_data
node_fetch_data -->|"persists"| node_current_data
node_versioned_data -->|"preprocessed as"| node_parquet_data
node_data_structure -.->|"documents"| node_structure_notebook
node_parquet_data -->|"explores"| node_structure_notebook
node_parquet_data -->|"tweet text"| node_embedding_notebook
node_embedding_notebook -->|"indexes embeddings"| node_chroma_db
node_chroma_db -->|"retrieves vectors"| node_text_notebook
node_parquet_data -->|"text analysis"| node_text_notebook
node_versioned_data -->|"release history"| node_temporal_notebook
node_current_data -->|"June 2026 comparison"| node_temporal_notebook
node_structure_notebook -->|"writes visuals"| node_figures
node_text_notebook -->|"writes visuals"| node_figures
node_temporal_notebook -->|"writes findings"| node_figures
node_parquet_data -->|"relationships"| node_network_notebook
node_network_notebook -->|"graph data"| node_gephi_export
node_gephi_export -->|"writes"| node_graph_files
node_graph_files -->|"opens in"| node_gephi

click node_python_env "https://github.com/cle07/memoire-hn-tpot/blob/main/pyproject.toml"
click node_fetch_data "https://github.com/cle07/memoire-hn-tpot/blob/main/scripts/fetch_data.py"
click node_versioned_data "https://github.com/cle07/memoire-hn-tpot/tree/main/data/2024-9-8"
click node_current_data "https://github.com/cle07/memoire-hn-tpot/tree/main/data/2026"
click node_parquet_data "https://github.com/cle07/memoire-hn-tpot/tree/main/data/parquet_files"
click node_data_structure "https://github.com/cle07/memoire-hn-tpot/blob/main/data/DATA_STRUCTURE.md"
click node_structure_notebook "https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/structure_exploration.ipynb"
click node_embedding_notebook "https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/embeddings_db.ipynb"
click node_chroma_db "https://github.com/cle07/memoire-hn-tpot/blob/main/chroma_db/chroma.sqlite3"
click node_text_notebook "https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/text_analysis.ipynb"
click node_temporal_notebook "https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/temporal_comparison.ipynb"
click node_figures "https://github.com/cle07/memoire-hn-tpot/tree/main/notebooks/output"
click node_network_notebook "https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/network_analysis.ipynb"
click node_gephi_export "https://github.com/cle07/memoire-hn-tpot/blob/main/scripts/export_gephi.py"
click node_graph_files "https://github.com/cle07/memoire-hn-tpot/tree/main/data/graphs"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_python_env toneBlue
class node_archive_api,node_github_releases,node_fetch_data toneAmber
class node_versioned_data,node_current_data,node_parquet_data,node_data_structure,node_chroma_db toneMint
class node_structure_notebook,node_embedding_notebook,node_text_notebook,node_temporal_notebook,node_figures toneRose
class node_network_notebook,node_gephi_export,node_graph_files,node_gephi toneIndigo
```

## Components

### Python environment

- [Python + uv](https://github.com/cle07/memoire-hn-tpot/blob/main/pyproject.toml): runtime · Pinned Python environment and locked dependencies. · `pyproject.toml`

### Archive sources

- **Community Archive API**: external API
- **Upstream GitHub releases**: release source
- [Archive fetcher](https://github.com/cle07/memoire-hn-tpot/blob/main/scripts/fetch_data.py): Python script · Retrieves archive releases for local restoration. · `scripts/fetch_data.py`

### Local data & artifacts

- [Versioned archive data](https://github.com/cle07/memoire-hn-tpot/tree/main/data/2024-9-8): local corpus · Raw and preprocessed releases organized by version and date. · `data/2024-9-8`
- [2026 archive data](https://github.com/cle07/memoire-hn-tpot/tree/main/data/2026): local corpus · `data/2026`
- [Parquet datasets](https://github.com/cle07/memoire-hn-tpot/tree/main/data/parquet_files): analytical format · `data/parquet_files`
- [Data structure guide](https://github.com/cle07/memoire-hn-tpot/blob/main/data/DATA_STRUCTURE.md): dataset documentation · `data/DATA_STRUCTURE.md`
- [ChromaDB vectors](https://github.com/cle07/memoire-hn-tpot/blob/main/chroma_db/chroma.sqlite3): vector store · Persistent embedding retrieval layer. · `chroma_db/chroma.sqlite3`

### Notebook analysis

- [Structure exploration](https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/structure_exploration.ipynb): Jupyter notebook · Establishes dataset understanding. · `notebooks/structure_exploration.ipynb`
- [Embedding workflow](https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/embeddings_db.ipynb): Jupyter notebook · Generates and reuses embeddings without re-embedding for topic fitting. · `notebooks/embeddings_db.ipynb`
- [Text & topic analysis](https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/text_analysis.ipynb): Jupyter notebook · Runs topic, emotion, entity, lexical-field, and word-cloud analysis. · `notebooks/text_analysis.ipynb`
- [Temporal comparison](https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/temporal_comparison.ipynb): Jupyter notebook · Compares 2024, 2025, and June 2026 releases. · `notebooks/temporal_comparison.ipynb`
- [Analysis outputs](https://github.com/cle07/memoire-hn-tpot/tree/main/notebooks/output): generated figures · Shared visual output area. · `notebooks/output`

### Network export

- [Network analysis](https://github.com/cle07/memoire-hn-tpot/blob/main/notebooks/network_analysis.ipynb): Jupyter notebook · Derives social graphs from archive relationships. · `notebooks/network_analysis.ipynb`
- [Gephi exporter](https://github.com/cle07/memoire-hn-tpot/blob/main/scripts/export_gephi.py): Python script · Creates graph files for external inspection. · `scripts/export_gephi.py`
- [Graph exports](https://github.com/cle07/memoire-hn-tpot/tree/main/data/graphs): visualization files · `data/graphs`
- **Gephi**: external visualization tool

## Connections

- Python + uv → Archive fetcher: runs
- Python + uv → Structure exploration: runs
- Python + uv → Embedding workflow: runs
- Community Archive API → Archive fetcher: archive releases
- Upstream GitHub releases → Archive fetcher: alternative source
- Archive fetcher → Versioned archive data: persists
- Archive fetcher → 2026 archive data: persists
- Versioned archive data → Parquet datasets: preprocessed as
- Data structure guide → Structure exploration: documents
- Parquet datasets → Structure exploration: explores
- Parquet datasets → Embedding workflow: tweet text
- Embedding workflow → ChromaDB vectors: indexes embeddings
- ChromaDB vectors → Text & topic analysis: retrieves vectors
- Parquet datasets → Text & topic analysis: text analysis
- Versioned archive data → Temporal comparison: release history
- 2026 archive data → Temporal comparison: June 2026 comparison
- Structure exploration → Analysis outputs: writes visuals
- Text & topic analysis → Analysis outputs: writes visuals
- Temporal comparison → Analysis outputs: writes findings
- Parquet datasets → Network analysis: relationships
- Network analysis → Gephi exporter: graph data
- Gephi exporter → Graph exports: writes
- Graph exports → Gephi: opens in