# prathameshbende/layer10-enron-memory architecture

> Architecture diagram of the GitHub repository prathameshbende/layer10-enron-memory, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/prathameshbende/layer10-enron-memory
- Repository: https://github.com/prathameshbende/layer10-enron-memory

## Overview

##### Project type & purpose (from README + tree)
This is a **local, script-driven data/NLP pipeline** that:
- Ingests the **Enron email corpus** (external dataset download),
- **Preprocesses/inspects** the data,
- Extracts structured “claims” (or similar semantic units) from emails,
- Builds a **“memory graph”** persisted as `memory_graph.pkl`,
- Supports **querying** that graph,
- Provides **two visualization surfaces**:
- a **Streamlit UI** (`src/visualize_memory.py`)
- a **graph view** generating `memory_graph_visualization.html` (`src/visualize_graph.py`)

##### Data inputs / outputs / artifacts (top-level)
- **External input**: Enron dataset (downloaded from Kaggle; stored locally)
- **Primary persistent artifact**: `memory_graph.pkl` (the built memory graph)
- **Visualization artifact**: `memory_graph_visualization.html`
- **Output samples**: `outputs/context_pack_examples.json`

##### Core pipeline modules (in `src/`)
- `inspect_data.py` — preprocessing/inspection step (turn raw emails into a cleaned/structured form)
- `claim_extractor.py` — semantic extraction component (claims/units to attach to graph)
- `build_graph.py` — graph construction and persistence to `memory_graph.pkl`
- `query_memory.py` — query logic over the graph (loads `memory_graph.pkl`, returns results)
- `visualize_graph.py` — graph visualization generator (likely reads graph, outputs HTML)
- `visualize_memory.py` — Streamlit UI (reads graph, allows interactive exploration/search)
- `build_graph.py` / `query_memory.py` are the “core runtime” modules; the others support ingestion and presentation.

Because there’s no server/client separation here, the most accurate architecture view is **a staged data pipeline + shared persisted graph**:

##### Download corpus (external Kaggle → local raw dataset)
2. `inspect_data.py` reads raw dataset → produces cleaned/inspected representation (even if it’s not explicitly saved, treat this as a pipeline stage)
3. `build_graph.py` consumes cleaned data and uses `claim_extractor.py` → outputs `memory_graph.pkl`
4. `query_memory.py` loads `memory_graph.pkl` → returns query results (likely to stdout/files)
##### Visualization
- `visualize_memory.py` (Streamlit) loads `memory_graph.pkl` → interactive UI
- `visualize_graph.py` loads `memory_graph.pkl` → writes `memory_graph_visualization.html`

##### Key architectural “pattern”
- **Batch ETL + derived artifact + multiple consumers**:
- Producer: `build_graph.py` → `memory_graph.pkl`
- Consumers: `query_memory.py`, `visualize_memory.py`, `visualize_graph.py`

##### Technologies/frameworks to label (without guessing specifics)
- **Python scripts** (CLI-driven pipeline)
- **Streamlit** (explicit in README)
- **Pickle artifact** (`memory_graph.pkl`)
- **HTML output** (`memory_graph_visualization.html`)
- **Kaggle / Enron dataset** (external data source)

##### Components: label clearly
Use concrete labels like:
- “Graph Builder (`src/build_graph.py`)”
- “Claim Extractor module (`src/claim_extractor.py`)”
- “Memory Graph Store (`memory_graph.pkl`)”
- “Streamlit UI (`src/visualize_memory.py`)”

Use different line styles:

##### “As many components as possible” (without overthinking)
For this project, that means:
- Treat `memory_graph.pkl` as a first-class integration node.
- Split “Visualization” into Streamlit vs HTML exporter (they are distinct consumers).

Small annotations near edges/nodes:
- Near `memory_graph.pkl`: “Primary derived artifact; shared contract between builder/query/visualizers”
- Near Streamlit: “Local web UI (http://localhost:8501), reads graph on demand”
- Near `visualize_graph.py`: “Generates static HTML visualization”

##### A small Level 2 runbook flow at bottom mapping to README steps 1–5.

This combination is the most faithful representation of the repo: it’s not a distributed system, so correctness comes from **accurate data artifacts, stage boundaries, and consumer relationships**.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% Level 0 — Context Strip
  %% =========================
  subgraph "Level 0 — Context"
    direction TB

    User["User (Data Scientist/Analyst)"]:::external
    Kaggle["Kaggle / Enron Dataset (External)"]:::external
    Repo["Memory Graph Pipeline + Query + Visualization (This Repo)"]:::boundary
    FS[("Local Filesystem")]:::store
    Readme["README / Runbook"]:::doc
  end

  Kaggle -->|"download"| FS
  User -->|"runs CLI scripts / Streamlit"| Repo
  Repo -->|"reads/writes"| FS
  User -->|"follows"| Readme

  %% ==========================================
  %% Level 1 — Components + Artifacts (Primary)
  %% ==========================================
  subgraph "Level 1 — Components & Artifacts (Core Architecture)"
    direction TB

    subgraph "Source Code (Python)"
      direction TB
      SrcDir["src/ (Core pipeline source)"]:::group

      Inspect["inspect_data.py (Preprocess/Inspect)"]:::script
      Build["build_graph.py (Graph Builder)"]:::script
      Claim["claim_extractor.py (Claim Extraction Module)"]:::module
      Query["query_memory.py (Query Engine)"]:::script
      VizMem["visualize_memory.py (Streamlit UI)"]:::script
      VizGraph["visualize_graph.py (HTML Graph Exporter)"]:::script
    end

    subgraph "Data & Artifacts (Local)"
      direction TB
      Raw[("Raw Enron Corpus (local files)")]:::store
      Clean["Cleaned/Structured Emails (in-memory or intermediate)"]:::data
      GraphPkl[("memory_graph.pkl (Memory Graph Store)")]:::store
      Html["memory_graph_visualization.html (Static Graph View)"]:::doc
      OutDir[("outputs/ (Generated artifacts)")]:::store
      CtxJson["context_pack_examples.json (Example output)"]:::doc
    end

    Browser["Local Browser (Streamlit client)"]:::external
  end

  %% Dataflow (solid)
  FS -->|"contains"| Raw
  Raw -->|"read"| Inspect
  Inspect -->|"produce"| Clean
  Clean -->|"consume"| Build
  Build -->|"write"| GraphPkl

  GraphPkl -->|"read"| Query
  Query -->|"write example output"| CtxJson
  CtxJson -->|"stored in"| OutDir

  GraphPkl -->|"read"| VizGraph
  VizGraph -->|"write"| Html

  GraphPkl -->|"read on demand"| VizMem
  VizMem -->|"serves UI"| Browser

  %% Code dependency (dashed)
  Build -.->|"uses"| Claim

  %% Notes / callouts
  NotePkl["NOTE: Primary derived artifact; shared contract between builder / query / visualizers"]:::note
  NoteStreamlit["NOTE: Local web UI (http://localhost:8501), reads graph on demand"]:::note
  NoteHtml["NOTE: Generates static HTML visualization"]:::note

  GraphPkl --- NotePkl
  VizMem --- NoteStreamlit
  VizGraph --- NoteHtml

  %% ==========================================
  %% Level 2 — Runbook / Happy Path (Sequence)
  %% ==========================================
  subgraph "Level 2 — Operational Happy Path (Runbook Flow)"
    direction TB
    Step0["Step 0: Download Enron corpus (Kaggle → local raw dataset)"]:::step
    Step1["Step 1: Run inspect_data.py (raw → cleaned/structured)"]:::step
    Step2["Step 2: Run build_graph.py (calls claim_extractor.py) → writes memory_graph.pkl"]:::step
    Step3["Step 3a: Run query_memory.py (loads memory_graph.pkl → results/examples)"]:::step
    Step4["Step 3b: Run visualize_memory.py (Streamlit) (loads memory_graph.pkl → interactive UI)"]:::step
    Step5["Step 3c: Run visualize_graph.py (loads memory_graph.pkl → writes HTML)"]:::step
  end

  Step0 --> Step1 --> Step2
  Step2 --> Step3
  Step2 --> Step4
  Step2 --> Step5

  %% =========================
  %% Click Events (Mappings)
  %% =========================
  click Readme "https://github.com/prathameshbende/layer10-enron-memory/blob/main/README.md"
  click GraphPkl "https://github.com/prathameshbende/layer10-enron-memory/blob/main/memory_graph.pkl"
  click Html "https://github.com/prathameshbende/layer10-enron-memory/blob/main/memory_graph_visualization.html"
  click CtxJson "https://github.com/prathameshbende/layer10-enron-memory/blob/main/outputs/context_pack_examples.json"
  click OutDir "https://github.com/prathameshbende/layer10-enron-memory/tree/main/outputs/"
  click Inspect "https://github.com/prathameshbende/layer10-enron-memory/blob/main/src/inspect_data.py"
  click Claim "https://github.com/prathameshbende/layer10-enron-memory/blob/main/src/claim_extractor.py"
  click Build "https://github.com/prathameshbende/layer10-enron-memory/blob/main/src/build_graph.py"
  click Query "https://github.com/prathameshbende/layer10-enron-memory/blob/main/src/query_memory.py"
  click VizGraph "https://github.com/prathameshbende/layer10-enron-memory/blob/main/src/visualize_graph.py"
  click VizMem "https://github.com/prathameshbende/layer10-enron-memory/blob/main/src/visualize_memory.py"
  click SrcDir "https://github.com/prathameshbende/layer10-enron-memory/tree/main/src/"

  %% =========================
  %% Styles
  %% =========================
  classDef external fill:#ffffff,stroke:#374151,stroke-width:1px,color:#111827
  classDef boundary fill:#eef2ff,stroke:#4338ca,stroke-width:2px,color:#111827
  classDef script fill:#dbeafe,stroke:#1d4ed8,stroke-width:2px,color:#0b1220
  classDef module fill:#e5e7eb,stroke:#6b7280,stroke-width:1.5px,color:#111827
  classDef store fill:#dcfce7,stroke:#16a34a,stroke-width:2px,color:#052e16
  classDef doc fill:#ffedd5,stroke:#f97316,stroke-width:2px,color:#431407
  classDef data fill:#ecfeff,stroke:#06b6d4,stroke-width:1.5px,color:#083344
  classDef group fill:#f3f4f6,stroke:#9ca3af,stroke-width:1px,color:#111827
  classDef note fill:#fef9c3,stroke:#ca8a04,stroke-width:1px,color:#422006
  classDef step fill:#fce7f3,stroke:#db2777,stroke-width:1.5px,color:#500724
```
