How mrbrianevans/companies-house-stream works
Companies Stream visualizes Companies House database changes as live, stream-specific event feeds, with additional history, statistics, samples, schemas, test-session, and MCP capabilities. The principal path is the Companies House stream listener publishing events to Redis Streams; the server consumes those events and publishes WebSocket messages for the browser to render. The graph also shows documented API branches and sampled persistence/export behavior. Several routes and client workers are represented at their exact enclosing paths; their internal wiring is not inferred where source excerpts were unavailable.
Main entry points: server.ts API server · sqlitePersistence.ts SQLite event buffer · mcp.ts MCP tools · listenOnStream.ts Stream listener
Components 27
Stream ingestion
- Stream listener
server/src/streams/listenOnStream.ts - Redis publisher
server/src/chStreamToRedis/streamToRedis.ts - Event types
server/src/types/eventTypes.ts
Server and APIs
- API server
server/src/api/server.ts - Redis stream reader
server/src/api/listenRedisStream.ts - Event stream service
server/src/api/streamEventsFromRedis.ts - Events WebSocket
server/src/api/routes/eventsWebsocket.ts - Event history
server/src/api/routes/eventHistory.ts - Misc endpoints
server/src/api/routes/misc.ts - Health check
server/src/api/routes/healthCheck.ts - Visitor endpoints
server/src/api/routes/visitors.ts - Visitor counter
server/src/api/visitorCounter.ts - MCP tools
server/src/api/routes/mcp.ts - Test sessions
server/src/api/routes/testSessions.ts - Schema maintenance
server/src/api/maintainSchemas.ts
Browser client
- Event manager
client-pure/src/scripts/eventManager.ts - WebSocket worker
client-pure/src/scripts/websocketWorker.ts - Worker wrapper
client-pure/src/scripts/workerWrapper.ts - Samples and schemas
client-pure/src/pages/samples.ts - Statistics page
client-pure/src/pages/stats.ts
Persistence and export
- SQLite event buffer
server/src/persistence/sqlitePersistence.ts - Nightly export
server/src/persistence/runNightlyExport.ts
Other components
- Companies House stream
- Redis
- Browser user
- MCP client
- DuckLake archive
Connections 39
- Redis publisher → Stream listener: opens streams
- Stream listener → Companies House stream: requests events
- Stream listener → Redis: logs responses
- Redis publisher → Redis: writes events and timepoints
- Redis → Redis stream reader: provides stream events
- API server → Redis stream reader: starts reader
- API server → Events WebSocket: registers route
- API server → Event history: registers route
- API server → Health check: registers route
- API server → Misc endpoints: registers route
- API server → Visitor endpoints: registers route
- API server → MCP tools: registers route
- API server → Test sessions: registers route
- Redis stream reader → Redis: reads events
- API server → Redis: updates schemas and company numbers
- API server → Events WebSocket: publishes events
- Events WebSocket → Browser user: sends WebSocket events
- Browser user → Event manager: views live feed
- Event manager → Worker wrapper: starts connection
- Worker wrapper → WebSocket worker: dispatches worker
- WebSocket worker → Event manager: emits events
- Event history → Event stream service: requests history
- Event stream service → Redis: reads stream events
- Test sessions → Event stream service: streams session events
- Test sessions → Redis: stores session state
- Visitor endpoints → Visitor counter: uses counters
- Visitor counter → Redis: reads and updates counts
- MCP client → MCP tools: calls MCP tools
- MCP tools → Redis: reads stream data and schemas
- MCP tools → Visitor counter: queries visitor counts
- Samples and schemas → Browser user: shows samples and schemas
- Samples and schemas → Redis: requests sample data and schemas
- Statistics page → Browser user: shows statistics
- SQLite event buffer → Redis: reads Redis streams
- SQLite event buffer → SQLite event buffer: stores buffered events
- SQLite event buffer → Nightly export: starts nightly export
- Nightly export → SQLite event buffer: reads and removes exported rows
- Nightly export → DuckLake archive: downloads and uploads archive
- Event types → Redis publisher: types events