# raam751/prognostype architecture

> Architecture diagram of the GitHub repository raam751/prognostype, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/raam751/prognostype
- Repository: https://github.com/raam751/prognostype

## Overview

1. **What this repo is**
   - A backend-heavy search typeahead system: users type a prefix, the server returns ranked suggestions, and submitted searches update historical and recent popularity.

2. **Top-level runtime shape**
   - The main application runs in **Node.js** with **Express** in `server.js`.
   - Persistence is **SQLite** via `better-sqlite3`.
   - A tiny browser UI lives in `public/index.html`, but it is just a client for the APIs.

3. **Core request paths**
   - `GET /suggest?q=<prefix>` is the read path for autocomplete.
   - `POST /search` is the write path for recording a submitted query.
   - `GET /trending` returns recency-aware popular searches.
   - `GET /cache/debug` and `GET /stats` expose observability for cache and performance.

4. **Autocomplete read flow**
   - `/suggest` first checks the in-memory cache for the prefix.
   - On a hit, suggestions are returned immediately from the owning cache node.
   - On a miss, the server queries SQLite, computes ranking, stores the result back into cache, and returns it.
   - Results are ranked by a recency-aware score: all-time count plus a boost from searches in the last 24 hours.

5. **Write flow for search submissions**
   - `POST /search` does not write every submission directly to SQLite one-by-one.
   - It goes through `batchWriter.js`, which buffers submissions and flushes them in batches.
   - When flushed, it updates the `queries` table and appends to `search_log`.
   - After a search is recorded, all relevant prefix cache entries are invalidated so suggestions can refresh.

6. **Caching boundary**
   - `cache.js` implements a **logical 3-node consistent-hash cache** inside the Node process.
   - Each node has virtual replicas on an MD5 hash ring, so a prefix maps to one node deterministically.
   - The cache is in-memory with TTL-based expiry, so it is fast but ephemeral and lost on restart.
   - This is not a separate cache service; it is embedded application state.

7. **Trending and recency model**
   - The system keeps two persistence views in SQLite:
     - `queries`: all-time popularity per unique query.
     - `search_log`: timestamped events for recency calculations.
   - `trending` and suggestion ranking use the 24-hour window from `search_log` to boost currently active queries over stale but historically large ones.

8. **Persistence model**
   - `loadDB.js` loads the aggregated query counts into SQLite from a generated CSV.
   - The database is the source of truth for counts and search history.
   - `search_log` has indexes on query and timestamp so 24-hour recency lookups remain efficient as the log grows.

9. **Data ingestion pipeline**
   - `ingest.py` is an offline preprocessing step in Python using `pandas`.
   - It joins the raw AOL query log files (`data.csv` and `query.csv`) on `QueryIndex`, aggregates by query text, and emits `query_counts.csv`.
   - That CSV is then loaded into SQLite by `loadDB.js`.

10. **Frontend boundary**
    - `public/index.html` is a minimal client UI that issues debounced suggestion requests as the user types.
    - It is not a separate application tier; it simply exercises the backend endpoints.

11. **Operational behavior**
    - The server tracks cache hit/miss statistics, DB read/write counts, batch flush counts, and `/suggest` latency percentiles.
    - These metrics are surfaced through `/cache/debug` and `/stats`, which are useful for understanding cache effectiveness and batch-write savings.

12. **Key architectural trade-off**
    - The design favors low-latency reads and fewer writes over strict durability.
    - Because the cache is in-memory and writes are batched, recent search activity can be lost on crash and may take up to one cache TTL to fully propagate, but the system stays simple and fast.

Main entry points: [`server.js`](https://github.com/raam751/prognostype/blob/main/server.js) (Express app), [`cache.js`](https://github.com/raam751/prognostype/blob/main/cache.js) (Cache ring), [`ingest.py`](https://github.com/raam751/prognostype/blob/main/ingest.py) (Python ingest), [`batchWriter.js`](https://github.com/raam751/prognostype/blob/main/batchWriter.js) (Batch writer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["UI"]
  node_browser_ui["Browser UI<br/>client<br/>[index.html]"]
end

subgraph group_api["Node runtime"]
  node_express_server["Express app<br/>http server<br/>[server.js]"]
  node_suggest_api["Suggest API<br/>read path<br/>[server.js]"]
  node_search_api["Search API<br/>write path<br/>[server.js]"]
  node_trending_api["Trending API<br/>read path<br/>[server.js]"]
  node_stats_api["Stats API<br/>observability<br/>[server.js]"]
  node_cache_debug_api["Cache debug<br/>observability<br/>[server.js]"]
  node_cache_ring{{"Cache ring<br/>in-memory cache<br/>[cache.js]"}}
  node_cache_node(("Cache node<br/>shard<br/>[cache.js]"))
  node_batch_writer["Batch writer<br/>buffered writer<br/>[batchWriter.js]"]
end

subgraph group_persistence["SQLite"]
  node_sqlite_db[("SQLite DB<br/>database")]
  node_queries_table[("Queries<br/>table")]
  node_search_log_table[("Search log<br/>event log")]
end

subgraph group_ingest["Offline ingestion"]
  node_load_db["DB loader<br/>ingest loader<br/>[loadDB.js]"]
  node_ingest_py["Python ingest<br/>preprocess job<br/>[ingest.py]"]
  node_raw_logs["Raw logs<br/>source data"]
  node_query_csv["Query CSV<br/>derived data"]
end

node_browser_ui -->|"debounced requests"| node_express_server
node_express_server -->|"/suggest"| node_suggest_api
node_express_server -->|"/search"| node_search_api
node_express_server -->|"/trending"| node_trending_api
node_express_server -->|"/stats"| node_stats_api
node_express_server -->|"/cache/debug"| node_cache_debug_api
node_suggest_api -->|"lookup prefix"| node_cache_ring
node_cache_ring -->|"owns key"| node_cache_node
node_suggest_api -->|"cache miss query"| node_sqlite_db
node_suggest_api -->|"store result"| node_cache_ring
node_search_api -->|"buffer submit"| node_batch_writer
node_batch_writer -->|"flush batch"| node_sqlite_db
node_sqlite_db -->|"counts"| node_queries_table
node_sqlite_db -->|"events"| node_search_log_table
node_search_api -->|"invalidate prefixes"| node_cache_ring
node_trending_api -->|"read recency"| node_sqlite_db
node_stats_api -->|"hit/miss stats"| node_cache_ring
node_cache_debug_api -->|"inspect shards"| node_cache_ring
node_load_db -->|"load data"| node_sqlite_db
node_ingest_py -->|"read source"| node_raw_logs
node_ingest_py -->|"emit counts"| node_query_csv
node_query_csv -->|"input"| node_load_db

click node_browser_ui "https://github.com/raam751/prognostype/blob/main/public/index.html"
click node_express_server "https://github.com/raam751/prognostype/blob/main/server.js"
click node_suggest_api "https://github.com/raam751/prognostype/blob/main/server.js"
click node_search_api "https://github.com/raam751/prognostype/blob/main/server.js"
click node_trending_api "https://github.com/raam751/prognostype/blob/main/server.js"
click node_stats_api "https://github.com/raam751/prognostype/blob/main/server.js"
click node_cache_debug_api "https://github.com/raam751/prognostype/blob/main/server.js"
click node_cache_ring "https://github.com/raam751/prognostype/blob/main/cache.js"
click node_cache_node "https://github.com/raam751/prognostype/blob/main/cache.js"
click node_batch_writer "https://github.com/raam751/prognostype/blob/main/batchWriter.js"
click node_load_db "https://github.com/raam751/prognostype/blob/main/loadDB.js"
click node_ingest_py "https://github.com/raam751/prognostype/blob/main/ingest.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser_ui toneBlue
class node_express_server,node_suggest_api,node_search_api,node_trending_api,node_stats_api,node_cache_debug_api,node_cache_ring,node_cache_node,node_batch_writer toneAmber
class node_sqlite_db,node_queries_table,node_search_log_table toneMint
class node_load_db,node_ingest_py,node_raw_logs,node_query_csv toneRose
```

## Components

### UI

Minimal browser client for exercising the backend APIs

- [Browser UI](https://github.com/raam751/prognostype/blob/main/public/index.html): client · Debounced autocomplete client · `public/index.html`

### Node runtime

Express app and request handlers

- [Express app](https://github.com/raam751/prognostype/blob/main/server.js): http server · Main Node.js entrypoint and route wiring · `server.js`
- [Suggest API](https://github.com/raam751/prognostype/blob/main/server.js): read path · Serves prefix autocomplete suggestions · `server.js`
- [Search API](https://github.com/raam751/prognostype/blob/main/server.js): write path · Records submitted searches · `server.js`
- [Trending API](https://github.com/raam751/prognostype/blob/main/server.js): read path · Returns recency-aware popular searches · `server.js`
- [Stats API](https://github.com/raam751/prognostype/blob/main/server.js): observability · Exposes cache and performance metrics · `server.js`
- [Cache debug](https://github.com/raam751/prognostype/blob/main/server.js): observability · Debug view of cache ownership and hits · `server.js`
- [Cache ring](https://github.com/raam751/prognostype/blob/main/cache.js): in-memory cache · Logical 3-node consistent-hash cache with TTL expiry · `cache.js`
- [Cache node](https://github.com/raam751/prognostype/blob/main/cache.js): shard · Owning node for a prefix and its cached suggestions · `cache.js`
- [Batch writer](https://github.com/raam751/prognostype/blob/main/batchWriter.js): buffered writer · Buffers search submissions and flushes them in batches · `batchWriter.js`

### SQLite

Persistent counts, history, and lookup data

- **SQLite DB**: database · Source of truth for counts and search history
- **Queries**: table · All-time popularity per unique query
- **Search log**: event log · Timestamped search events for recency scoring

### Offline ingestion

Preprocessing and database loading pipeline

- [DB loader](https://github.com/raam751/prognostype/blob/main/loadDB.js): ingest loader · Loads aggregated query counts into SQLite · `loadDB.js`
- [Python ingest](https://github.com/raam751/prognostype/blob/main/ingest.py): preprocess job · Joins raw AOL logs and aggregates query counts · `ingest.py`
- **Raw logs**: source data · Input AOL query log files
- **Query CSV**: derived data · Aggregated query counts emitted by preprocessing

## Connections

- Browser UI → Express app: debounced requests · User typing triggers API calls
- Express app → Suggest API: /suggest · Routes autocomplete reads
- Express app → Search API: /search · Routes search submissions
- Express app → Trending API: /trending · Routes trending queries
- Express app → Stats API: /stats · Routes metrics endpoint
- Express app → Cache debug: /cache/debug · Routes cache inspection endpoint
- Suggest API → Cache ring: lookup prefix · Checks in-memory cache before hitting SQLite
- Cache ring → Cache node: owns key · Deterministically maps a prefix to one cache node
- Suggest API → SQLite DB: cache miss query · Reads ranked suggestions from persistence
- Suggest API → Cache ring: store result · Caches returned suggestions for future hits
- Search API → Batch writer: buffer submit · Queues search events instead of writing one-by-one
- Batch writer → SQLite DB: flush batch · Writes batched updates into SQLite
- SQLite DB → Queries: counts · Persists all-time query popularity
- SQLite DB → Search log: events · Persists timestamped search history
- Search API → Cache ring: invalidate prefixes · Clears affected cached suggestions after a write
- Trending API → SQLite DB: read recency · Combines all-time counts with 24-hour activity
- Stats API → Cache ring: hit/miss stats · Reports cache effectiveness
- Cache debug → Cache ring: inspect shards · Shows logical shard ownership and contents
- DB loader → SQLite DB: load data · Initializes the database from generated counts
- Python ingest → Raw logs: read source · Consumes raw AOL log files
- Python ingest → Query CSV: emit counts · Produces aggregated query-count CSV
- Query CSV → DB loader: input · Feeds loader with precomputed counts