raam751/prognostype

1. What this repo is

- A backend-heavy search typeahead system: users type a prefix, the server returns ranked suggestions, and submitted searches update historical and recent popularity.

2. Top-level runtime shape

- The main application runs in Node.js with Express in server.js.

- Persistence is SQLite via better-sqlite3.

- A tiny browser UI lives in public/index.html, but it is just a client for the APIs.

3. Core request paths

- GET /suggest?q=<prefix> is the read path for autocomplete.

- POST /search is the write path for recording a submitted query.

- GET /trending returns recency-aware popular searches.

- GET /cache/debug and GET /stats expose observability for cache and performance.

4. Autocomplete read flow

- /suggest first checks the in-memory cache for the prefix.

- On a hit, suggestions are returned immediately from the owning cache node.

- On a miss, the server queries SQLite, computes ranking, stores the result back into cache, and returns it.

- Results are ranked by a recency-aware score: all-time count plus a boost from searches in the last 24 hours.

5. Write flow for search submissions

- POST /search does not write every submission directly to SQLite one-by-one.

- It goes through batchWriter.js, which buffers submissions and flushes them in batches.

- When flushed, it updates the queries table and appends to search_log.

- After a search is recorded, all relevant prefix cache entries are invalidated so suggestions can refresh.

6. Caching boundary

- cache.js implements a logical 3-node consistent-hash cache inside the Node process.

- Each node has virtual replicas on an MD5 hash ring, so a prefix maps to one node deterministically.

- The cache is in-memory with TTL-based expiry, so it is fast but ephemeral and lost on restart.

- This is not a separate cache service; it is embedded application state.

7. Trending and recency model

- The system keeps two persistence views in SQLite:

- queries: all-time popularity per unique query.

- search_log: timestamped events for recency calculations.

- trending and suggestion ranking use the 24-hour window from search_log to boost currently active queries over stale but historically large ones.

8. Persistence model

- loadDB.js loads the aggregated query counts into SQLite from a generated CSV.

- The database is the source of truth for counts and search history.

- search_log has indexes on query and timestamp so 24-hour recency lookups remain efficient as the log grows.

9. Data ingestion pipeline

- ingest.py is an offline preprocessing step in Python using pandas.

- It joins the raw AOL query log files (data.csv and query.csv) on QueryIndex, aggregates by query text, and emits query_counts.csv.

- That CSV is then loaded into SQLite by loadDB.js.

10. Frontend boundary

- public/index.html is a minimal client UI that issues debounced suggestion requests as the user types.

- It is not a separate application tier; it simply exercises the backend endpoints.

11. Operational behavior

- The server tracks cache hit/miss statistics, DB read/write counts, batch flush counts, and /suggest latency percentiles.

- These metrics are surfaced through /cache/debug and /stats, which are useful for understanding cache effectiveness and batch-write savings.

12. Key architectural trade-off

- The design favors low-latency reads and fewer writes over strict durability.

- Because the cache is in-memory and writes are batched, recent search activity can be lost on crash and may take up to one cache TTL to fully propagate, but the system stays simple and fast.