How afar1/fieldtheory-cli works
1. Overall shape
- This is a local-first Node.js CLI for syncing X/Twitter bookmarks, storing them on disk, indexing them in SQLite, and exposing search, stats, classification, and visualization commands.
- The repo is centered around a single command-line tool (ft) rather than a server or web app.
2. Primary entrypoint and command surface
- src/cli.ts is the main command dispatcher.
- bin/ft.mjs is the executable wrapper that launches the TypeScript-based CLI.
- Commands in the README map to subsystems: sync, search, classify, index, viz, categories, domains, stats, list, show, auth, fetch-media, status, and path.
3. Local data boundary
- All persistent state lives in a user data directory, defaulting to ~/.ft-bookmarks/, with FT_DATA_DIR as an override.
- The main persisted artifacts are:
- bookmarks.jsonl for raw bookmark records
- bookmarks.db for the SQLite search/index store
- bookmarks-meta.json for sync metadata
- oauth-token.json for optional OAuth credentials
- This is the core architectural boundary: network access is only for syncing from X; everything else operates on local files.
4. Configuration and environment
- src/config.ts and src/paths.ts centralize runtime configuration and filesystem locations.
- src/fs.ts likely wraps file operations and path handling so the rest of the app stays focused on domain logic.
- src/types.ts defines the shared record shapes used across sync, storage, indexing, and classification.
5. Sync pipeline
- The sync flow is the most important data path:
- obtain X auth
- fetch bookmark data
- normalize records
- append/update the local raw cache
- merge into the search database
- There are two authentication/sync modes:
- Chrome session extraction on macOS using src/chrome-cookies.ts and src/xauth.ts
- OAuth/API mode using src/graphql-bookmarks.ts plus src/xauth.ts
- The default sync uses X’s internal GraphQL API; the official API path is optional and cross-platform.
6. X integration layer
- src/graphql-bookmarks.ts appears to encapsulate the X bookmark retrieval logic against GraphQL endpoints.
- src/xauth.ts and src/chrome-cookies.ts are the auth boundary, handling session/token acquisition without persisting browser cookies as a separate credential store.
- This layer is the only place that should know about X-specific request formats and authentication details.
7. Bookmark domain and service layer
- src/bookmarks.ts and src/bookmarks-service.ts likely represent the core domain model and orchestration layer for bookmark operations.
- bookmarks-service is the glue between sync input, storage, indexing, classification, and query commands.
- src/bookmarks-db.ts is the persistence/search boundary, probably built on SQLite with FTS5 and BM25 ranking as described in the README.
8. Search and retrieval
- Search is local and database-backed: ft search queries the SQLite index rather than calling X.
- src/db.ts likely abstracts SQLite access and schema management.
- The architecture separates raw data storage from query optimization: raw JSONL is the source-of-truth cache, while SQLite is the derived search/index layer.
9. Classification subsystem
- src/bookmark-classify.ts provides rule-based or regex-based classification.
- src/bookmark-classify-llm.ts adds LLM-based classification for richer category/domain labeling.
- Classification is applied after sync and preserved across incremental reindexing, which implies the stored metadata is designed to survive repeated syncs.
10. Media handling
- src/bookmark-media.ts supports downloading static media assets for bookmarked items.
- This is a separate post-processing path from sync/search, likely operating on already-synced bookmark records.
11. Visualization and reporting
- src/bookmarks-viz.ts powers terminal-based dashboard/reporting features like categories, domains, stats, and sparklines.
- These outputs are read-only consumers of the local store and index, not separate data stores.
12. Data flow summary
- External source: X bookmarks API or browser session-derived auth
- Ingestion: sync modules fetch and normalize bookmark data
- Raw cache: append/store into bookmarks.jsonl
- Derived index: merge into bookmarks.db
- Enrichment: optional classification and media fetch
- Consumption: search, list, show, stats, viz, and agent-driven shell queries
13. Platform and runtime constraints
- Runtime is Node.js 20+ with TypeScript.
- Chrome session sync is documented as macOS-only because it depends on Chrome cookie extraction and macOS Keychain behavior.
- API/OAuth mode removes that OS dependency and is the cross-platform sync path.
14. Tests and assurance
- The tests/ directory covers major subsystems: classification, bookmarks DB/service, Chrome cookie extraction, config, and GraphQL bookmark syncing.
- This suggests the architecture is organized around isolated modules with clear seams for auth, sync, persistence, and classification.
15. Non-core assets
- website/ is a small static marketing/documentation site and is not part of the runtime data pipeline.
- Operationally, the CLI is the product; the website is ancillary.
Main entry points: bookmarks-service.ts Service · cli.ts CLI router · bookmarks-db.ts SQLite DB · bookmarks.ts Bookmarks
Components 20
CLI runtime
Command entrypoint and configuration for the Node.js tool
- ft.mjs · Executable wrapper that starts the TypeScript CLI
bin/ft.mjs - CLI router · Main command parser and subcommand entrypoint
src/cli.ts - Config · Environment and option handling
src/config.ts
Local state
Persistent files and SQLite index on disk
- Paths · Centralizes data directory and file locations
src/paths.ts - FS helpers · Shared filesystem and path utilities
src/fs.ts - Raw store · Persistent bookmark log on disk
- Meta file · Tracks last sync state and bookkeeping
- OAuth token · Optional stored OAuth credentials
- SQLite DB · Derived search and query database
src/bookmarks-db.ts - DB core · SQLite access and schema management
src/db.ts
Sync boundary
Authentication and X bookmark ingestion
- X auth · Obtains session or OAuth access for X
src/xauth.ts - Chrome cookies · Reads Chrome session cookies for local auth
src/chrome-cookies.ts - GraphQL sync · Fetches bookmarks from X GraphQL endpoints
src/graphql-bookmarks.ts
Domain services
Core bookmark model, storage, and processing
- Types · Common record and domain shapes
src/types.ts - Bookmarks · Core bookmark entities and normalization
src/bookmarks.ts - Service · Coordinates sync, storage, classification, and queries
src/bookmarks-service.ts - Classifier · Rule-based bookmark classification
src/bookmark-classify.ts - LLM classifier · LLM-assisted category and domain labeling
src/bookmark-classify-llm.ts - Media fetcher · Downloads bookmarked media for local use
src/bookmark-media.ts
Read-only views
Search, reporting, and visualization commands
- Viz/reporting · Generates stats, categories, domains, and sparklines
src/bookmarks-viz.ts
Connections 21
- ft.mjs → CLI router: launches · Executable starts the CLI dispatcher
- CLI router → Config: loads · CLI reads runtime configuration
- Config → Paths: resolves · Configuration determines local paths
- Paths → FS helpers: uses · Path handling flows through filesystem helpers
- CLI router → Service: routes to · Commands delegate into the bookmark service
- Service → Bookmarks: operates on · Service works with bookmark domain objects
- Bookmarks → Types: shaped by · Shared types define bookmark record structure
- Service → Raw store: persists · Service appends and reads raw bookmark cache
- Service → Meta file: updates · Sync metadata is maintained alongside the cache
- Service → SQLite DB: indexes · Service merges data into the SQLite index
- SQLite DB → DB core: built on · Index layer depends on core SQLite access
- Service → X auth: authenticates via · Service obtains X access through auth boundary
- X auth → Chrome cookies: may use · macOS Chrome cookie extraction is one auth path
- Service → GraphQL sync: syncs from · Bookmark ingestion comes from X GraphQL sync
- GraphQL sync → Bookmarks: normalizes · Fetched records are converted into domain bookmarks
- Service → Classifier: classifies · Service applies rule-based labeling
- Service → LLM classifier: enriches with · Optional LLM classification adds richer labels
- Service → Media fetcher: fetches media · Optional media download runs as a post-processing step
- CLI router → Viz/reporting: renders · Reporting commands consume the local store and index
- SQLite DB → Viz/reporting: queries · Visualization and search read from SQLite
- Raw store → SQLite DB: derives · Raw cache is the source for the derived index