# mrbrianevans/companies-house-stream architecture

> Architecture diagram of the GitHub repository mrbrianevans/companies-house-stream, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mrbrianevans/companies-house-stream
- Repository: https://github.com/mrbrianevans/companies-house-stream

## Overview

Companies Stream visualizes Companies House database changes as live, stream-specific event feeds, with additional history, statistics, samples, schemas, test-session, and MCP capabilities. The principal path is the Companies House stream listener publishing events to Redis Streams; the server consumes those events and publishes WebSocket messages for the browser to render. The graph also shows documented API branches and sampled persistence/export behavior. Several routes and client workers are represented at their exact enclosing paths; their internal wiring is not inferred where source excerpts were unavailable.

Main entry points: [`server/src/api/server.ts`](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/server.ts) (API server), [`server/src/persistence/sqlitePersistence.ts`](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/persistence/sqlitePersistence.ts) (SQLite event buffer), [`server/src/api/routes/mcp.ts`](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/mcp.ts) (MCP tools), [`server/src/streams/listenOnStream.ts`](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/streams/listenOnStream.ts) (Stream listener)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Stream ingestion"]
  node_stream_listener["Stream listener<br/>[listenOnStream.ts]"]
  node_stream_publisher["Redis publisher<br/>[streamToRedis.ts]"]
  node_stream_types["Event types<br/>[eventTypes.ts]"]
end

subgraph group_service["Server and APIs"]
  node_event_server["API server<br/>[server.ts]"]
  node_redis_listener["Redis stream reader"]
  node_event_streamer["Event stream service"]
  node_websocket_route["Events WebSocket<br/>[eventsWebsocket.ts]"]
  node_history_route["Event history<br/>[eventHistory.ts]"]
  node_misc_route["Misc endpoints<br/>[misc.ts]"]
  node_health_route["Health check<br/>[healthCheck.ts]"]
  node_visitors_route["Visitor endpoints<br/>[visitors.ts]"]
  node_visitor_counter["Visitor counter<br/>[visitorCounter.ts]"]
  node_mcp_route["MCP tools<br/>[mcp.ts]"]
  node_test_sessions["Test sessions<br/>[testSessions.ts]"]
  node_schema_maintenance["Schema maintenance<br/>[maintainSchemas.ts]"]
end

subgraph group_client["Browser client"]
  node_event_manager["Event manager<br/>[eventManager.ts]"]
  node_websocket_worker["WebSocket worker<br/>[websocketWorker.ts]"]
  node_worker_wrapper["Worker wrapper<br/>[workerWrapper.ts]"]
  node_samples_page["Samples and schemas<br/>[samples.ts]"]
  node_stats_page["Statistics page<br/>[stats.ts]"]
end

subgraph group_archive["Persistence and export"]
  node_sqlite_buffer[("SQLite event buffer")]
  node_nightly_export["Nightly export"]
end

node_companies_house(("Companies House stream"))
node_redis[("Redis")]
node_browser(("Browser user"))
node_mcp_consumer(("MCP client"))
node_ducklake[("DuckLake archive")]

node_stream_publisher -->|"opens streams"| node_stream_listener
node_stream_listener -->|"requests events"| node_companies_house
node_stream_listener -->|"logs responses"| node_redis
node_stream_publisher -->|"writes events and timepoints"| node_redis
node_redis -->|"provides stream events"| node_redis_listener
node_event_server -->|"starts reader"| node_redis_listener
node_event_server -->|"registers route"| node_websocket_route
node_event_server -->|"registers route"| node_history_route
node_event_server -->|"registers route"| node_health_route
node_event_server -->|"registers route"| node_misc_route
node_event_server -->|"registers route"| node_visitors_route
node_event_server -->|"registers route"| node_mcp_route
node_event_server -->|"registers route"| node_test_sessions
node_redis_listener -->|"reads events"| node_redis
node_event_server -->|"updates schemas and company numbers"| node_redis
node_event_server -->|"publishes events"| node_websocket_route
node_websocket_route -->|"sends WebSocket events"| node_browser
node_browser -->|"views live feed"| node_event_manager
node_event_manager -->|"starts connection"| node_worker_wrapper
node_worker_wrapper -->|"dispatches worker"| node_websocket_worker
node_websocket_worker -->|"emits events"| node_event_manager
node_history_route -->|"requests history"| node_event_streamer
node_event_streamer -->|"reads stream events"| node_redis
node_test_sessions -->|"streams session events"| node_event_streamer
node_test_sessions -->|"stores session state"| node_redis
node_visitors_route -->|"uses counters"| node_visitor_counter
node_visitor_counter -->|"reads and updates counts"| node_redis
node_mcp_consumer -->|"calls MCP tools"| node_mcp_route
node_mcp_route -->|"reads stream data and schemas"| node_redis
node_mcp_route -->|"queries visitor counts"| node_visitor_counter
node_samples_page -->|"shows samples and schemas"| node_browser
node_samples_page -->|"requests sample data and schemas"| node_redis
node_stats_page -->|"shows statistics"| node_browser
node_sqlite_buffer -->|"reads Redis streams"| node_redis
node_sqlite_buffer -->|"stores buffered events"| node_sqlite_buffer
node_sqlite_buffer -->|"starts nightly export"| node_nightly_export
node_nightly_export -->|"reads and removes exported rows"| node_sqlite_buffer
node_nightly_export -.->|"downloads and uploads archive"| node_ducklake
node_stream_types -->|"types events"| node_stream_publisher

click node_stream_listener "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/streams/listenOnStream.ts"
click node_stream_publisher "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/chStreamToRedis/streamToRedis.ts"
click node_event_server "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/server.ts"
click node_redis_listener "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/listenRedisStream.ts"
click node_event_streamer "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/streamEventsFromRedis.ts"
click node_websocket_route "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/eventsWebsocket.ts"
click node_history_route "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/eventHistory.ts"
click node_misc_route "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/misc.ts"
click node_health_route "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/healthCheck.ts"
click node_visitors_route "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/visitors.ts"
click node_visitor_counter "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/visitorCounter.ts"
click node_mcp_route "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/mcp.ts"
click node_test_sessions "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/testSessions.ts"
click node_schema_maintenance "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/maintainSchemas.ts"
click node_event_manager "https://github.com/mrbrianevans/companies-house-stream/blob/master/client-pure/src/scripts/eventManager.ts"
click node_websocket_worker "https://github.com/mrbrianevans/companies-house-stream/blob/master/client-pure/src/scripts/websocketWorker.ts"
click node_worker_wrapper "https://github.com/mrbrianevans/companies-house-stream/blob/master/client-pure/src/scripts/workerWrapper.ts"
click node_samples_page "https://github.com/mrbrianevans/companies-house-stream/blob/master/client-pure/src/pages/samples.ts"
click node_stats_page "https://github.com/mrbrianevans/companies-house-stream/blob/master/client-pure/src/pages/stats.ts"
click node_sqlite_buffer "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/persistence/sqlitePersistence.ts"
click node_nightly_export "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/persistence/runNightlyExport.ts"
click node_stream_types "https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/types/eventTypes.ts"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_stream_listener,node_stream_publisher,node_stream_types,node_browser,node_mcp_consumer toneBlue
class node_event_server,node_redis_listener,node_event_streamer,node_websocket_route,node_history_route,node_misc_route,node_health_route,node_visitors_route,node_visitor_counter,node_mcp_route,node_test_sessions,node_schema_maintenance,node_redis,node_ducklake toneAmber
class node_event_manager,node_websocket_worker,node_worker_wrapper,node_samples_page,node_stats_page toneMint
class node_sqlite_buffer,node_nightly_export toneRose
class node_companies_house toneIndigo
```

## Components

### Stream ingestion

- [Stream listener](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/streams/listenOnStream.ts): `server/src/streams/listenOnStream.ts`
- [Redis publisher](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/chStreamToRedis/streamToRedis.ts): `server/src/chStreamToRedis/streamToRedis.ts`
- [Event types](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/types/eventTypes.ts): `server/src/types/eventTypes.ts`

### Server and APIs

- [API server](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/server.ts): `server/src/api/server.ts`
- [Redis stream reader](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/listenRedisStream.ts): `server/src/api/listenRedisStream.ts`
- [Event stream service](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/streamEventsFromRedis.ts): `server/src/api/streamEventsFromRedis.ts`
- [Events WebSocket](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/eventsWebsocket.ts): `server/src/api/routes/eventsWebsocket.ts`
- [Event history](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/eventHistory.ts): `server/src/api/routes/eventHistory.ts`
- [Misc endpoints](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/misc.ts): `server/src/api/routes/misc.ts`
- [Health check](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/healthCheck.ts): `server/src/api/routes/healthCheck.ts`
- [Visitor endpoints](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/visitors.ts): `server/src/api/routes/visitors.ts`
- [Visitor counter](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/visitorCounter.ts): `server/src/api/visitorCounter.ts`
- [MCP tools](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/mcp.ts): `server/src/api/routes/mcp.ts`
- [Test sessions](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/routes/testSessions.ts): `server/src/api/routes/testSessions.ts`
- [Schema maintenance](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/api/maintainSchemas.ts): `server/src/api/maintainSchemas.ts`

### Browser client

- [Event manager](https://github.com/mrbrianevans/companies-house-stream/blob/master/client-pure/src/scripts/eventManager.ts): `client-pure/src/scripts/eventManager.ts`
- [WebSocket worker](https://github.com/mrbrianevans/companies-house-stream/blob/master/client-pure/src/scripts/websocketWorker.ts): `client-pure/src/scripts/websocketWorker.ts`
- [Worker wrapper](https://github.com/mrbrianevans/companies-house-stream/blob/master/client-pure/src/scripts/workerWrapper.ts): `client-pure/src/scripts/workerWrapper.ts`
- [Samples and schemas](https://github.com/mrbrianevans/companies-house-stream/blob/master/client-pure/src/pages/samples.ts): `client-pure/src/pages/samples.ts`
- [Statistics page](https://github.com/mrbrianevans/companies-house-stream/blob/master/client-pure/src/pages/stats.ts): `client-pure/src/pages/stats.ts`

### Persistence and export

- [SQLite event buffer](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/persistence/sqlitePersistence.ts): `server/src/persistence/sqlitePersistence.ts`
- [Nightly export](https://github.com/mrbrianevans/companies-house-stream/blob/master/server/src/persistence/runNightlyExport.ts): `server/src/persistence/runNightlyExport.ts`

### Other components

- **Companies House stream**
- **Redis**
- **Browser user**
- **MCP client**
- **DuckLake archive**

## Connections

- Redis publisher → Stream listener: opens streams
- Stream listener → Companies House stream: requests events
- Stream listener → Redis: logs responses
- Redis publisher → Redis: writes events and timepoints
- Redis → Redis stream reader: provides stream events
- API server → Redis stream reader: starts reader
- API server → Events WebSocket: registers route
- API server → Event history: registers route
- API server → Health check: registers route
- API server → Misc endpoints: registers route
- API server → Visitor endpoints: registers route
- API server → MCP tools: registers route
- API server → Test sessions: registers route
- Redis stream reader → Redis: reads events
- API server → Redis: updates schemas and company numbers
- API server → Events WebSocket: publishes events
- Events WebSocket → Browser user: sends WebSocket events
- Browser user → Event manager: views live feed
- Event manager → Worker wrapper: starts connection
- Worker wrapper → WebSocket worker: dispatches worker
- WebSocket worker → Event manager: emits events
- Event history → Event stream service: requests history
- Event stream service → Redis: reads stream events
- Test sessions → Event stream service: streams session events
- Test sessions → Redis: stores session state
- Visitor endpoints → Visitor counter: uses counters
- Visitor counter → Redis: reads and updates counts
- MCP client → MCP tools: calls MCP tools
- MCP tools → Redis: reads stream data and schemas
- MCP tools → Visitor counter: queries visitor counts
- Samples and schemas → Browser user: shows samples and schemas
- Samples and schemas → Redis: requests sample data and schemas
- Statistics page → Browser user: shows statistics
- SQLite event buffer → Redis: reads Redis streams
- SQLite event buffer → SQLite event buffer: stores buffered events
- SQLite event buffer → Nightly export: starts nightly export
- Nightly export → SQLite event buffer: reads and removes exported rows
- Nightly export → DuckLake archive: downloads and uploads archive
- Event types → Redis publisher: types events