How prathameshbende/layer10-enron-memory works
Project type & purpose (from README + tree)
This is a local, script-driven data/NLP pipeline that:
• Ingests the Enron email corpus (external dataset download),
• Preprocesses/inspects the data,
• Extracts structured “claims” (or similar semantic units) from emails,
• Builds a “memory graph” persisted as memory_graph.pkl,
• Supports querying that graph,
• Provides two visualization surfaces:
• a Streamlit UI (src/visualize_memory.py)
• a graph view generating memory_graph_visualization.html (src/visualize_graph.py)
Data inputs / outputs / artifacts (top-level)
• External input: Enron dataset (downloaded from Kaggle; stored locally)
• Primary persistent artifact: memory_graph.pkl (the built memory graph)
• Visualization artifact: memory_graph_visualization.html
• Output samples: outputs/context_pack_examples.json
Core pipeline modules (in src/)
• inspect_data.py — preprocessing/inspection step (turn raw emails into a cleaned/structured form)
• claim_extractor.py — semantic extraction component (claims/units to attach to graph)
• build_graph.py — graph construction and persistence to memory_graph.pkl
• query_memory.py — query logic over the graph (loads memory_graph.pkl, returns results)
• visualize_graph.py — graph visualization generator (likely reads graph, outputs HTML)
• visualize_memory.py — Streamlit UI (reads graph, allows interactive exploration/search)
• build_graph.py / query_memory.py are the “core runtime” modules; the others support ingestion and presentation.
Because there’s no server/client separation here, the most accurate architecture view is a staged data pipeline + shared persisted graph:
Download corpus (external Kaggle → local raw dataset)
2. inspect_data.py reads raw dataset → produces cleaned/inspected representation (even if it’s not explicitly saved, treat this as a pipeline stage)
3. build_graph.py consumes cleaned data and uses claim_extractor.py → outputs memory_graph.pkl
4. query_memory.py loads memory_graph.pkl → returns query results (likely to stdout/files)
Visualization
• visualize_memory.py (Streamlit) loads memory_graph.pkl → interactive UI
• visualize_graph.py loads memory_graph.pkl → writes memory_graph_visualization.html
Key architectural “pattern”
• Batch ETL + derived artifact + multiple consumers:
• Producer: build_graph.py → memory_graph.pkl
• Consumers: query_memory.py, visualize_memory.py, visualize_graph.py
Technologies/frameworks to label (without guessing specifics)
• Python scripts (CLI-driven pipeline)
• Streamlit (explicit in README)
• Pickle artifact (memory_graph.pkl)
• HTML output (memory_graph_visualization.html)
• Kaggle / Enron dataset (external data source)
Components: label clearly
Use concrete labels like:
• “Graph Builder (src/build_graph.py)”
• “Claim Extractor module (src/claim_extractor.py)”
• “Memory Graph Store (memory_graph.pkl)”
• “Streamlit UI (src/visualize_memory.py)”
Use different line styles:
“As many components as possible” (without overthinking)
For this project, that means:
• Treat memory_graph.pkl as a first-class integration node.
• Split “Visualization” into Streamlit vs HTML exporter (they are distinct consumers).
Small annotations near edges/nodes:
• Near memory_graph.pkl: “Primary derived artifact; shared contract between builder/query/visualizers”
• Near Streamlit: “Local web UI (http://localhost:8501), reads graph on demand”
• Near visualize_graph.py: “Generates static HTML visualization”
A small Level 2 runbook flow at bottom mapping to README steps 1–5.
This combination is the most faithful representation of the repo: it’s not a distributed system, so correctness comes from accurate data artifacts, stage boundaries, and consumer relationships.