# yahya-debug/john_log architecture

> Architecture diagram of the GitHub repository yahya-debug/john_log, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-15). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/yahya-debug/john_log
- Repository: https://github.com/yahya-debug/john_log

## Overview

John Log accepts structured logs over HTTP, validates and buffers batches, then transactionally stores raw logs and minute/hour rollups in Postgres. Operators use the dashboard to ingest, search, aggregate, tail, inspect dead letters, and view service/database statistics; retention manages daily partitions. Edges are limited to relationships evidenced by the README, excerpts, or SOURCE INDEX references; some route internals and dashboard page-to-API calls are unsampled.

Main entry points: [`dashboard/src/App.tsx`](https://github.com/yahya-debug/john_log/blob/main/dashboard/src/App.tsx) (Dashboard shell), [`dashboard/src/api.ts`](https://github.com/yahya-debug/john_log/blob/main/dashboard/src/api.ts) (Dashboard API client), [`src/http/app.ts`](https://github.com/yahya-debug/john_log/blob/main/src/http/app.ts) (HTTP service), [`src/db/logs.ts`](https://github.com/yahya-debug/john_log/blob/main/src/db/logs.ts) (Log storage queries)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Ingestion"]
  node_http["HTTP service<br/>[app.ts]"]
  node_ingest_api["Ingest endpoint<br/>[logs.ts]"]
  node_validation["Batch validation<br/>[validate.ts]"]
  node_buffer["Write buffer<br/>[writeBuffer.ts]"]
end

subgraph group_query["Query and aggregation"]
  node_aggregate_api["Aggregate endpoint<br/>[aggregate.ts]"]
  node_aggregate_runner["Aggregate routing<br/>[aggregateQuery.ts]"]
  node_filters["Query filters<br/>[filters.ts]"]
  node_live["Live aggregates<br/>[liveAggregate.ts]"]
  node_cache["Aggregate cache<br/>[aggregateCache.ts]"]
  node_logs_query["Log search<br/>[logsQuery.ts]"]
end

subgraph group_storage["Postgres and retention"]
  node_logdb["Log storage queries<br/>[logs.ts]"]
  node_postgres[("Postgres")]
  node_retention["Partition retention<br/>[partitions.ts]"]
  node_retention_job["Retention scheduler<br/>[job.ts]"]
end

subgraph group_operations["Operator dashboard"]
  node_admin_api["Admin endpoints<br/>[admin.ts]"]
  node_db_stats["Database statistics<br/>[stats.ts]"]
  node_dashboard["Dashboard shell<br/>[App.tsx]"]
  node_dashboard_api["Dashboard API client<br/>[api.ts]"]
  node_ingest_page["Ingest console<br/>[IngestPage.tsx]"]
  node_logs_page["Log explorer<br/>[LogsPage.tsx]"]
  node_aggregate_page["Aggregate charts<br/>[AggregatePage.tsx]"]
  node_tail_page["Live tail<br/>[TailPage.tsx]"]
  node_dead_page["Dead-letter console<br/>[DeadLetterPage.tsx]"]
  node_stats_page["Stats console<br/>[StatsPage.tsx]"]
end

node_producers(("Log producers"))
node_operators(("Operators"))

node_producers -->|"submit logs"| node_http
node_http -->|"routes requests"| node_ingest_api
node_ingest_api -->|"validates batch"| node_validation
node_ingest_api -->|"queues accepted logs"| node_buffer
node_buffer -->|"writes transaction"| node_logdb
node_logdb -->|"reads and writes"| node_postgres
node_buffer -->|"records committed logs"| node_live
node_aggregate_api -->|"runs query"| node_aggregate_runner
node_aggregate_runner -->|"checks live buckets"| node_live
node_aggregate_runner -->|"loads rollups or logs"| node_logdb
node_aggregate_runner -->|"builds scan conditions"| node_filters
node_aggregate_runner -->|"caches results"| node_cache
node_http -->|"routes requests"| node_aggregate_api
node_http -->|"starts schedule"| node_retention_job
node_retention_job -->|"runs retention"| node_retention
node_retention -->|"maintains partitions"| node_postgres
node_http -->|"routes requests"| node_admin_api
node_admin_api -->|"loads statistics"| node_db_stats
node_db_stats -->|"queries statistics"| node_postgres
node_admin_api -->|"manages dead letters"| node_logdb
node_operators -->|"uses console"| node_dashboard
node_dashboard -->|"selects tab"| node_ingest_page
node_dashboard -->|"selects tab"| node_logs_page
node_dashboard -->|"selects tab"| node_aggregate_page
node_dashboard -->|"selects tab"| node_tail_page
node_dashboard -->|"selects tab"| node_dead_page
node_dashboard -->|"selects tab"| node_stats_page
node_ingest_page -->|"submits logs"| node_dashboard_api
node_logs_page -->|"requests logs"| node_dashboard_api
node_aggregate_page -->|"requests aggregates"| node_dashboard_api
node_tail_page -->|"requests tail"| node_dashboard_api
node_dead_page -->|"manages dead letters"| node_dashboard_api
node_stats_page -->|"requests statistics"| node_dashboard_api
node_dashboard_api -->|"calls service endpoints"| node_http
node_ingest_api -.->|"queries logs"| node_logdb
node_logs_query -->|"queries matching logs"| node_logdb

click node_http "https://github.com/yahya-debug/john_log/blob/main/src/http/app.ts"
click node_ingest_api "https://github.com/yahya-debug/john_log/blob/main/src/http/routes/logs.ts"
click node_validation "https://github.com/yahya-debug/john_log/blob/main/src/ingestion/validate.ts"
click node_buffer "https://github.com/yahya-debug/john_log/blob/main/src/ingestion/writeBuffer.ts"
click node_logdb "https://github.com/yahya-debug/john_log/blob/main/src/db/logs.ts"
click node_aggregate_api "https://github.com/yahya-debug/john_log/blob/main/src/http/routes/aggregate.ts"
click node_aggregate_runner "https://github.com/yahya-debug/john_log/blob/main/src/query/aggregateQuery.ts"
click node_filters "https://github.com/yahya-debug/john_log/blob/main/src/query/filters.ts"
click node_live "https://github.com/yahya-debug/john_log/blob/main/src/query/liveAggregate.ts"
click node_cache "https://github.com/yahya-debug/john_log/blob/main/src/query/aggregateCache.ts"
click node_logs_query "https://github.com/yahya-debug/john_log/blob/main/src/query/logsQuery.ts"
click node_retention "https://github.com/yahya-debug/john_log/blob/main/src/retention/partitions.ts"
click node_retention_job "https://github.com/yahya-debug/john_log/blob/main/src/retention/job.ts"
click node_admin_api "https://github.com/yahya-debug/john_log/blob/main/src/http/routes/admin.ts"
click node_db_stats "https://github.com/yahya-debug/john_log/blob/main/src/db/stats.ts"
click node_dashboard "https://github.com/yahya-debug/john_log/blob/main/dashboard/src/App.tsx"
click node_dashboard_api "https://github.com/yahya-debug/john_log/blob/main/dashboard/src/api.ts"
click node_ingest_page "https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/IngestPage.tsx"
click node_logs_page "https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/LogsPage.tsx"
click node_aggregate_page "https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/AggregatePage.tsx"
click node_tail_page "https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/TailPage.tsx"
click node_dead_page "https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/DeadLetterPage.tsx"
click node_stats_page "https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/StatsPage.tsx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_http,node_ingest_api,node_validation,node_buffer toneBlue
class node_aggregate_api,node_aggregate_runner,node_filters,node_live,node_cache,node_logs_query toneAmber
class node_logdb,node_postgres,node_retention,node_retention_job toneMint
class node_admin_api,node_db_stats,node_dashboard,node_dashboard_api,node_ingest_page,node_logs_page,node_aggregate_page,node_tail_page,node_dead_page,node_stats_page toneRose
class node_producers,node_operators toneIndigo
```

## Components

### Ingestion

- [HTTP service](https://github.com/yahya-debug/john_log/blob/main/src/http/app.ts): `src/http/app.ts`
- [Ingest endpoint](https://github.com/yahya-debug/john_log/blob/main/src/http/routes/logs.ts): `src/http/routes/logs.ts`
- [Batch validation](https://github.com/yahya-debug/john_log/blob/main/src/ingestion/validate.ts): `src/ingestion/validate.ts`
- [Write buffer](https://github.com/yahya-debug/john_log/blob/main/src/ingestion/writeBuffer.ts): `src/ingestion/writeBuffer.ts`

### Query and aggregation

- [Aggregate endpoint](https://github.com/yahya-debug/john_log/blob/main/src/http/routes/aggregate.ts): `src/http/routes/aggregate.ts`
- [Aggregate routing](https://github.com/yahya-debug/john_log/blob/main/src/query/aggregateQuery.ts): `src/query/aggregateQuery.ts`
- [Query filters](https://github.com/yahya-debug/john_log/blob/main/src/query/filters.ts): `src/query/filters.ts`
- [Live aggregates](https://github.com/yahya-debug/john_log/blob/main/src/query/liveAggregate.ts): `src/query/liveAggregate.ts`
- [Aggregate cache](https://github.com/yahya-debug/john_log/blob/main/src/query/aggregateCache.ts): `src/query/aggregateCache.ts`
- [Log search](https://github.com/yahya-debug/john_log/blob/main/src/query/logsQuery.ts): `src/query/logsQuery.ts`

### Postgres and retention

- [Log storage queries](https://github.com/yahya-debug/john_log/blob/main/src/db/logs.ts): `src/db/logs.ts`
- **Postgres**
- [Partition retention](https://github.com/yahya-debug/john_log/blob/main/src/retention/partitions.ts): `src/retention/partitions.ts`
- [Retention scheduler](https://github.com/yahya-debug/john_log/blob/main/src/retention/job.ts): `src/retention/job.ts`

### Operator dashboard

- [Admin endpoints](https://github.com/yahya-debug/john_log/blob/main/src/http/routes/admin.ts): `src/http/routes/admin.ts`
- [Database statistics](https://github.com/yahya-debug/john_log/blob/main/src/db/stats.ts): `src/db/stats.ts`
- [Dashboard shell](https://github.com/yahya-debug/john_log/blob/main/dashboard/src/App.tsx): `dashboard/src/App.tsx`
- [Dashboard API client](https://github.com/yahya-debug/john_log/blob/main/dashboard/src/api.ts): `dashboard/src/api.ts`
- [Ingest console](https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/IngestPage.tsx): `dashboard/src/pages/IngestPage.tsx`
- [Log explorer](https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/LogsPage.tsx): `dashboard/src/pages/LogsPage.tsx`
- [Aggregate charts](https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/AggregatePage.tsx): `dashboard/src/pages/AggregatePage.tsx`
- [Live tail](https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/TailPage.tsx): `dashboard/src/pages/TailPage.tsx`
- [Dead-letter console](https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/DeadLetterPage.tsx): `dashboard/src/pages/DeadLetterPage.tsx`
- [Stats console](https://github.com/yahya-debug/john_log/blob/main/dashboard/src/pages/StatsPage.tsx): `dashboard/src/pages/StatsPage.tsx`

### Other components

- **Log producers**
- **Operators**

## Connections

- Log producers → HTTP service: submit logs (evidence: `README.md`)
- HTTP service → Ingest endpoint: routes requests (evidence: `src/http/app.ts`)
- Ingest endpoint → Batch validation: validates batch (evidence: `src/http/routes/logs.ts`)
- Ingest endpoint → Write buffer: queues accepted logs (evidence: `src/http/routes/logs.ts`)
- Write buffer → Log storage queries: writes transaction (evidence: `src/ingestion/writeBuffer.ts`)
- Log storage queries → Postgres: reads and writes (evidence: `README.md`)
- Write buffer → Live aggregates: records committed logs (evidence: `src/ingestion/writeBuffer.ts`)
- Aggregate endpoint → Aggregate routing: runs query (evidence: `src/http/routes/aggregate.ts`)
- Aggregate routing → Live aggregates: checks live buckets (evidence: `src/query/aggregateQuery.ts`)
- Aggregate routing → Log storage queries: loads rollups or logs (evidence: `src/query/aggregateQuery.ts`)
- Aggregate routing → Query filters: builds scan conditions (evidence: `src/query/aggregateQuery.ts`)
- Aggregate routing → Aggregate cache: caches results (evidence: `src/query/aggregateQuery.ts`)
- HTTP service → Aggregate endpoint: routes requests (evidence: `src/http/app.ts`)
- HTTP service → Retention scheduler: starts schedule (evidence: `src/http/app.ts`)
- Retention scheduler → Partition retention: runs retention (evidence: `src/retention/job.ts`)
- Partition retention → Postgres: maintains partitions (evidence: `src/retention/partitions.ts`)
- HTTP service → Admin endpoints: routes requests (evidence: `src/http/app.ts`)
- Admin endpoints → Database statistics: loads statistics (evidence: `src/http/routes/admin.ts`)
- Database statistics → Postgres: queries statistics (evidence: `src/db/stats.ts`)
- Admin endpoints → Log storage queries: manages dead letters (evidence: `src/http/routes/admin.ts`)
- Operators → Dashboard shell: uses console (evidence: `README.md`)
- Dashboard shell → Ingest console: selects tab (evidence: `dashboard/src/App.tsx`)
- Dashboard shell → Log explorer: selects tab (evidence: `dashboard/src/App.tsx`)
- Dashboard shell → Aggregate charts: selects tab (evidence: `dashboard/src/App.tsx`)
- Dashboard shell → Live tail: selects tab (evidence: `dashboard/src/App.tsx`)
- Dashboard shell → Dead-letter console: selects tab (evidence: `dashboard/src/App.tsx`)
- Dashboard shell → Stats console: selects tab (evidence: `dashboard/src/App.tsx`)
- Ingest console → Dashboard API client: submits logs (evidence: `dashboard/src/pages/IngestPage.tsx`)
- Log explorer → Dashboard API client: requests logs (evidence: `dashboard/src/pages/LogsPage.tsx`)
- Aggregate charts → Dashboard API client: requests aggregates (evidence: `dashboard/src/pages/AggregatePage.tsx`)
- Live tail → Dashboard API client: requests tail (evidence: `dashboard/src/pages/TailPage.tsx`)
- Dead-letter console → Dashboard API client: manages dead letters (evidence: `dashboard/src/pages/DeadLetterPage.tsx`)
- Stats console → Dashboard API client: requests statistics (evidence: `dashboard/src/pages/StatsPage.tsx`)
- Dashboard API client → HTTP service: calls service endpoints (evidence: `dashboard/src/api.ts`)
- Ingest endpoint → Log storage queries: queries logs
- Log search → Log storage queries: queries matching logs (evidence: `src/query/logsQuery.ts`)