# afar1/fieldtheory-cli architecture

> Architecture diagram of the GitHub repository afar1/fieldtheory-cli, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/afar1/fieldtheory-cli
- Repository: https://github.com/afar1/fieldtheory-cli

## Overview

1. **Overall shape**
   - This is a **local-first Node.js CLI** for syncing X/Twitter bookmarks, storing them on disk, indexing them in SQLite, and exposing search, stats, classification, and visualization commands.
   - The repo is centered around a single command-line tool (`ft`) rather than a server or web app.

2. **Primary entrypoint and command surface**
   - `src/cli.ts` is the main command dispatcher.
   - `bin/ft.mjs` is the executable wrapper that launches the TypeScript-based CLI.
   - Commands in the README map to subsystems: `sync`, `search`, `classify`, `index`, `viz`, `categories`, `domains`, `stats`, `list`, `show`, `auth`, `fetch-media`, `status`, and `path`.

3. **Local data boundary**
   - All persistent state lives in a user data directory, defaulting to `~/.ft-bookmarks/`, with `FT_DATA_DIR` as an override.
   - The main persisted artifacts are:
     - `bookmarks.jsonl` for raw bookmark records
     - `bookmarks.db` for the SQLite search/index store
     - `bookmarks-meta.json` for sync metadata
     - `oauth-token.json` for optional OAuth credentials
   - This is the core architectural boundary: network access is only for syncing from X; everything else operates on local files.

4. **Configuration and environment**
   - `src/config.ts` and `src/paths.ts` centralize runtime configuration and filesystem locations.
   - `src/fs.ts` likely wraps file operations and path handling so the rest of the app stays focused on domain logic.
   - `src/types.ts` defines the shared record shapes used across sync, storage, indexing, and classification.

5. **Sync pipeline**
   - The sync flow is the most important data path:
     - obtain X auth
     - fetch bookmark data
     - normalize records
     - append/update the local raw cache
     - merge into the search database
   - There are two authentication/sync modes:
     - **Chrome session extraction** on macOS using `src/chrome-cookies.ts` and `src/xauth.ts`
     - **OAuth/API mode** using `src/graphql-bookmarks.ts` plus `src/xauth.ts`
   - The default sync uses X’s internal GraphQL API; the official API path is optional and cross-platform.

6. **X integration layer**
   - `src/graphql-bookmarks.ts` appears to encapsulate the X bookmark retrieval logic against GraphQL endpoints.
   - `src/xauth.ts` and `src/chrome-cookies.ts` are the auth boundary, handling session/token acquisition without persisting browser cookies as a separate credential store.
   - This layer is the only place that should know about X-specific request formats and authentication details.

7. **Bookmark domain and service layer**
   - `src/bookmarks.ts` and `src/bookmarks-service.ts` likely represent the core domain model and orchestration layer for bookmark operations.
   - `bookmarks-service` is the glue between sync input, storage, indexing, classification, and query commands.
   - `src/bookmarks-db.ts` is the persistence/search boundary, probably built on SQLite with FTS5 and BM25 ranking as described in the README.

8. **Search and retrieval**
   - Search is local and database-backed: `ft search` queries the SQLite index rather than calling X.
   - `src/db.ts` likely abstracts SQLite access and schema management.
   - The architecture separates raw data storage from query optimization: raw JSONL is the source-of-truth cache, while SQLite is the derived search/index layer.

9. **Classification subsystem**
   - `src/bookmark-classify.ts` provides rule-based or regex-based classification.
   - `src/bookmark-classify-llm.ts` adds LLM-based classification for richer category/domain labeling.
   - Classification is applied after sync and preserved across incremental reindexing, which implies the stored metadata is designed to survive repeated syncs.

10. **Media handling**
    - `src/bookmark-media.ts` supports downloading static media assets for bookmarked items.
    - This is a separate post-processing path from sync/search, likely operating on already-synced bookmark records.

11. **Visualization and reporting**
    - `src/bookmarks-viz.ts` powers terminal-based dashboard/reporting features like categories, domains, stats, and sparklines.
    - These outputs are read-only consumers of the local store and index, not separate data stores.

12. **Data flow summary**
    - External source: X bookmarks API or browser session-derived auth
    - Ingestion: sync modules fetch and normalize bookmark data
    - Raw cache: append/store into `bookmarks.jsonl`
    - Derived index: merge into `bookmarks.db`
    - Enrichment: optional classification and media fetch
    - Consumption: search, list, show, stats, viz, and agent-driven shell queries

13. **Platform and runtime constraints**
    - Runtime is **Node.js 20+** with TypeScript.
    - Chrome session sync is documented as macOS-only because it depends on Chrome cookie extraction and macOS Keychain behavior.
    - API/OAuth mode removes that OS dependency and is the cross-platform sync path.

14. **Tests and assurance**
    - The `tests/` directory covers major subsystems: classification, bookmarks DB/service, Chrome cookie extraction, config, and GraphQL bookmark syncing.
    - This suggests the architecture is organized around isolated modules with clear seams for auth, sync, persistence, and classification.

15. **Non-core assets**
    - `website/` is a small static marketing/documentation site and is not part of the runtime data pipeline.
    - Operationally, the CLI is the product; the website is ancillary.

Main entry points: [`src/bookmarks-service.ts`](https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks-service.ts) (Service), [`src/cli.ts`](https://github.com/afar1/fieldtheory-cli/blob/main/src/cli.ts) (CLI router), [`src/bookmarks-db.ts`](https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks-db.ts) (SQLite DB), [`src/bookmarks.ts`](https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks.ts) (Bookmarks)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_g1["CLI runtime"]
  node_n1["ft.mjs<br/>launcher<br/>[ft.mjs]"]
  node_n2["CLI router<br/>command dispatcher<br/>[cli.ts]"]
  node_n3["Config<br/>[config.ts]"]
end

subgraph group_g2["Local state"]
  node_n4["Paths<br/>filesystem layout<br/>[paths.ts]"]
  node_n5["FS helpers<br/>file ops<br/>[fs.ts]"]
  node_n9[("Raw store<br/>jsonl cache")]
  node_n10["Meta file<br/>sync metadata"]
  node_n11["OAuth token<br/>credential file"]
  node_n12[("SQLite DB<br/>index store<br/>[bookmarks-db.ts]")]
  node_n13[("DB core<br/>sqlite layer<br/>[db.ts]")]
end

subgraph group_g3["Sync boundary"]
  node_n14["X auth<br/>auth boundary<br/>[xauth.ts]"]
  node_n15["Chrome cookies<br/>macOS extractor<br/>[chrome-cookies.ts]"]
  node_n16["GraphQL sync<br/>x ingest"]
end

subgraph group_g4["Domain services"]
  node_n6["Types<br/>shared models<br/>[types.ts]"]
  node_n7["Bookmarks<br/>domain model<br/>[bookmarks.ts]"]
  node_n8["Service<br/>orchestrator"]
  node_n17["Classifier<br/>rules engine"]
  node_n18["LLM classifier<br/>llm enrichment"]
  node_n19["Media fetcher<br/>asset download<br/>[bookmark-media.ts]"]
end

subgraph group_g5["Read-only views"]
  node_n20["Viz/reporting<br/>terminal views<br/>[bookmarks-viz.ts]"]
end

node_n1 -->|"launches"| node_n2
node_n2 -->|"loads"| node_n3
node_n3 -->|"resolves"| node_n4
node_n4 -->|"uses"| node_n5
node_n2 -->|"routes to"| node_n8
node_n8 -->|"operates on"| node_n7
node_n7 -->|"shaped by"| node_n6
node_n8 -->|"persists"| node_n9
node_n8 -->|"updates"| node_n10
node_n8 -->|"indexes"| node_n12
node_n12 -->|"built on"| node_n13
node_n8 -->|"authenticates via"| node_n14
node_n14 -->|"may use"| node_n15
node_n8 -->|"syncs from"| node_n16
node_n16 -->|"normalizes"| node_n7
node_n8 -->|"classifies"| node_n17
node_n8 -->|"enriches with"| node_n18
node_n8 -->|"fetches media"| node_n19
node_n2 -->|"renders"| node_n20
node_n12 -->|"queries"| node_n20
node_n9 -->|"derives"| node_n12

click node_n1 "https://github.com/afar1/fieldtheory-cli/blob/main/bin/ft.mjs"
click node_n2 "https://github.com/afar1/fieldtheory-cli/blob/main/src/cli.ts"
click node_n3 "https://github.com/afar1/fieldtheory-cli/blob/main/src/config.ts"
click node_n4 "https://github.com/afar1/fieldtheory-cli/blob/main/src/paths.ts"
click node_n5 "https://github.com/afar1/fieldtheory-cli/blob/main/src/fs.ts"
click node_n6 "https://github.com/afar1/fieldtheory-cli/blob/main/src/types.ts"
click node_n7 "https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks.ts"
click node_n8 "https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks-service.ts"
click node_n12 "https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks-db.ts"
click node_n13 "https://github.com/afar1/fieldtheory-cli/blob/main/src/db.ts"
click node_n14 "https://github.com/afar1/fieldtheory-cli/blob/main/src/xauth.ts"
click node_n15 "https://github.com/afar1/fieldtheory-cli/blob/main/src/chrome-cookies.ts"
click node_n16 "https://github.com/afar1/fieldtheory-cli/blob/main/src/graphql-bookmarks.ts"
click node_n17 "https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmark-classify.ts"
click node_n18 "https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmark-classify-llm.ts"
click node_n19 "https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmark-media.ts"
click node_n20 "https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks-viz.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_n1,node_n2,node_n3 toneBlue
class node_n4,node_n5,node_n9,node_n10,node_n11,node_n12,node_n13 toneAmber
class node_n14,node_n15,node_n16 toneMint
class node_n6,node_n7,node_n8,node_n17,node_n18,node_n19 toneRose
class node_n20 toneIndigo
```

## Components

### CLI runtime

Command entrypoint and configuration for the Node.js tool

- [ft.mjs](https://github.com/afar1/fieldtheory-cli/blob/main/bin/ft.mjs): launcher · Executable wrapper that starts the TypeScript CLI · `bin/ft.mjs`
- [CLI router](https://github.com/afar1/fieldtheory-cli/blob/main/src/cli.ts): command dispatcher · Main command parser and subcommand entrypoint · `src/cli.ts`
- [Config](https://github.com/afar1/fieldtheory-cli/blob/main/src/config.ts): runtime config · Environment and option handling · `src/config.ts`

### Local state

Persistent files and SQLite index on disk

- [Paths](https://github.com/afar1/fieldtheory-cli/blob/main/src/paths.ts): filesystem layout · Centralizes data directory and file locations · `src/paths.ts`
- [FS helpers](https://github.com/afar1/fieldtheory-cli/blob/main/src/fs.ts): file ops · Shared filesystem and path utilities · `src/fs.ts`
- **Raw store**: jsonl cache · Persistent bookmark log on disk
- **Meta file**: sync metadata · Tracks last sync state and bookkeeping
- **OAuth token**: credential file · Optional stored OAuth credentials
- [SQLite DB](https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks-db.ts): index store · Derived search and query database · `src/bookmarks-db.ts`
- [DB core](https://github.com/afar1/fieldtheory-cli/blob/main/src/db.ts): sqlite layer · SQLite access and schema management · `src/db.ts`

### Sync boundary

Authentication and X bookmark ingestion

- [X auth](https://github.com/afar1/fieldtheory-cli/blob/main/src/xauth.ts): auth boundary · Obtains session or OAuth access for X · `src/xauth.ts`
- [Chrome cookies](https://github.com/afar1/fieldtheory-cli/blob/main/src/chrome-cookies.ts): macOS extractor · Reads Chrome session cookies for local auth · `src/chrome-cookies.ts`
- [GraphQL sync](https://github.com/afar1/fieldtheory-cli/blob/main/src/graphql-bookmarks.ts): x ingest · Fetches bookmarks from X GraphQL endpoints · `src/graphql-bookmarks.ts`

### Domain services

Core bookmark model, storage, and processing

- [Types](https://github.com/afar1/fieldtheory-cli/blob/main/src/types.ts): shared models · Common record and domain shapes · `src/types.ts`
- [Bookmarks](https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks.ts): domain model · Core bookmark entities and normalization · `src/bookmarks.ts`
- [Service](https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks-service.ts): orchestrator · Coordinates sync, storage, classification, and queries · `src/bookmarks-service.ts`
- [Classifier](https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmark-classify.ts): rules engine · Rule-based bookmark classification · `src/bookmark-classify.ts`
- [LLM classifier](https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmark-classify-llm.ts): llm enrichment · LLM-assisted category and domain labeling · `src/bookmark-classify-llm.ts`
- [Media fetcher](https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmark-media.ts): asset download · Downloads bookmarked media for local use · `src/bookmark-media.ts`

### Read-only views

Search, reporting, and visualization commands

- [Viz/reporting](https://github.com/afar1/fieldtheory-cli/blob/main/src/bookmarks-viz.ts): terminal views · Generates stats, categories, domains, and sparklines · `src/bookmarks-viz.ts`

## Connections

- ft.mjs → CLI router: launches · Executable starts the CLI dispatcher
- CLI router → Config: loads · CLI reads runtime configuration
- Config → Paths: resolves · Configuration determines local paths
- Paths → FS helpers: uses · Path handling flows through filesystem helpers
- CLI router → Service: routes to · Commands delegate into the bookmark service
- Service → Bookmarks: operates on · Service works with bookmark domain objects
- Bookmarks → Types: shaped by · Shared types define bookmark record structure
- Service → Raw store: persists · Service appends and reads raw bookmark cache
- Service → Meta file: updates · Sync metadata is maintained alongside the cache
- Service → SQLite DB: indexes · Service merges data into the SQLite index
- SQLite DB → DB core: built on · Index layer depends on core SQLite access
- Service → X auth: authenticates via · Service obtains X access through auth boundary
- X auth → Chrome cookies: may use · macOS Chrome cookie extraction is one auth path
- Service → GraphQL sync: syncs from · Bookmark ingestion comes from X GraphQL sync
- GraphQL sync → Bookmarks: normalizes · Fetched records are converted into domain bookmarks
- Service → Classifier: classifies · Service applies rule-based labeling
- Service → LLM classifier: enriches with · Optional LLM classification adds richer labels
- Service → Media fetcher: fetches media · Optional media download runs as a post-processing step
- CLI router → Viz/reporting: renders · Reporting commands consume the local store and index
- SQLite DB → Viz/reporting: queries · Visualization and search read from SQLite
- Raw store → SQLite DB: derives · Raw cache is the source for the derived index
