# nntmux/newznab-tmux architecture

> Architecture diagram of the GitHub repository nntmux/newznab-tmux, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nntmux/newznab-tmux
- Repository: https://github.com/nntmux/newznab-tmux

## Overview

##### Project type and purpose
- This is a full-stack Laravel-based usenet indexing system (“NNTmux”) that:
- Continuously fetches headers from Usenet NNTP servers via CLI scripts (the “Blacklight” library)
- Processes and collates headers into “releases” and metadata
- Exposes a web UI (Blade/Vue) plus a REST/RSS/API interface
- Provides an admin console, real-time metrics (Laravel Horizon/Tmux), search (Manticore/Elasticsearch)

##### File structure
- Top–level:
- Blacklight/… → stand-alone PHP classes/scripts for NNTP header retrieval, backfilling, multi-threaded processing
- app/… → Laravel MVC (controllers, models, jobs, commands, listeners, middleware)
- resources/views, resources/js, public/assets → frontend templates, assets, Vue/Livewire components, Filament admin
- config/, database/ → configs and migrations/seeders
- docker/, docker-compose.yml, sail → containerized deployment
- storage/ covers & nzb/… → file storage for covers, raw nzb files, caches
- Key patterns:
- MVC + Job Queue (Redis + Horizon)
- CLI Commands invoked by scheduler or tmux to harvest/process
- External service adapters (AniDB, TMDb, Trakt, IRC) in Blacklight/libraries & app/Extensions

##### Extract architecture insights from README
- Multi-threaded processing via Tmux
- MySQL/MariaDB primary store (InnoDB heavy usage)
- Search engine (Manticore/ElasticSearchSiteSearch)
- Requires Redis (queue, cache), optional Memcached
- Docker-based via Laravel Sail

##### Main components & interactions
a. Components:
1. NNTP Harvesters (Blacklight CLI)
2. Post-processing workers (Blacklight & Laravel Jobs)
3. Laravel App
– HTTP/API server (Controllers + Blade/Vue UI)
– CLI Commands (app/Console/Commands) for sync tasks
– Job Queue (Redis + Horizon)
4. Database (MySQL/MariaDB)
5. Search Engine (Manticore or Elasticsearch)
6. Cache/Queue (Redis)
7. File Storage (covers/nzb in storage/)
8. Tmux Manager (monitors workers & DB performance)
9. External services (AniDB, TMDb, Trakt, IRC, NZB indexing endpoints)
10. Docker Host (or on-prem VM) orchestrating containers via docker-compose / Sail

b. Relationships & Data Flow:
- NNTP Harvesters ↔ Usenet NNTP Servers (ingest headers) → Database (staged tables)
- Staged data → Laravel Post-processing jobs → Releases table, metadata tables
- Jobs publish to Redis queue → Worker processes → Database updates & cover download → storage/
- Laravel HTTP/API ← User browser or API client → Controllers → Models → Database / Cache
- Search queries → Search Engine ← synchronized via Laravel Scout / scheduled commands
- Tmux monitors CLI & queue workers, DB stats

c. Architectural patterns & principles:
- MVC for web layer
- Command pattern for CLI jobs
- Producer-consumer via Redis & Horizon
- Adapter pattern for external APIs in Blacklight/libraries & app/Extensions
- Layered separation: harvest → process → store → serve

d. Key technologies/frameworks:
- Laravel (8.x/9.x) with Sail, Horizon, Scout
- Blacklight PHP library for CLI
- MySQL/MariaDB (InnoDB heavy)
- Manticore or Elasticsearch
- Redis (queue, cache)
- Tmux for multi-thread orchestration
- Docker / docker-compose

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Ingestion Layer
  subgraph "Ingestion Layer"
    NNTPS["NNTP Harvesters\n(Blacklight CLI)"]:::ingestion
  end

  %% Processing Layer
  subgraph "Processing Layer"
    PPW["Post-processing Workers"]:::ingestion
    Commands["CLI Commands\n(app/Console/Commands)"]:::ingestion
    TmuxMgr["Tmux Manager"]:::ingestion
  end

  %% Persistence Layer
  subgraph "Persistence Layer"
    DB["MySQL/MariaDB"]:::persistence
    Search["Search Engine\n(Manticore/ElasticSearch)"]:::persistence
    Cache["Redis\n(Queue/Cache)"]:::persistence
    Storage["File Storage\n(covers, nzb)"]:::persistence
  end

  %% Presentation Layer
  subgraph "Presentation Layer"
    Browser["User Browser/API Client"]:::presentation
    HTTP["Laravel HTTP/API Server"]:::presentation
    WebUI["Web UI\n(Blade/Vue)"]:::presentation
  end

  %% External & Infrastructure
  subgraph "External & Infrastructure"
    NNTP["Usenet NNTP Servers"]:::infra
    External["External Services\n(AniDB, TMDb, Trakt, IRC)"]:::infra
    Docker["Docker Host/VM\n(docker-compose)"]:::infra
  end

  %% Data Flows
  NNTP -->|ingest headers| NNTPS
  NNTPS -->|write staged data| DB
  DB -->|trigger jobs| PPW
  Commands -->|dispatch tasks| PPW
  PPW -->|update metadata| DB
  PPW -->|download covers| Storage
  PPW -->|publish messages| Cache
  Cache -->|queue consumers| PPW
  Browser -->|HTTP/API requests| HTTP
  HTTP -->|read/write| DB
  HTTP -->|cache layer| Cache
  HTTP -->|search queries| Search
  HTTP -->|serve assets| Storage
  HTTP -->|render UI| WebUI
  Commands -->|sync search indexes| Search
  TmuxMgr -.->|monitor| PPW
  TmuxMgr -.->|monitor| DB
  Docker -->|hosts containers| NNTPS
  Docker -->|hosts containers| HTTP
  External -->|API calls| NNTPS
  External -->|API calls| HTTP

  %% Click Events
  click NNTPS "https://github.com/nntmux/newznab-tmux/tree/master/Blacklight/"
  click PPW "https://github.com/nntmux/newznab-tmux/tree/master/Blacklight/processing/"
  click Commands "https://github.com/nntmux/newznab-tmux/tree/master/app/Console/Commands/"
  click TmuxMgr "https://github.com/nntmux/newznab-tmux/blob/master/Blacklight/TmuxRun.php"
  click DB "https://github.com/nntmux/newznab-tmux/blob/master/config/database.php"
  click Search "https://github.com/nntmux/newznab-tmux/blob/master/config/elasticsearch.php"
  click Cache "https://github.com/nntmux/newznab-tmux/blob/master/config/cache.php"
  click Storage "https://github.com/nntmux/newznab-tmux/tree/master/storage/"
  click HTTP "https://github.com/nntmux/newznab-tmux/tree/master/app/Http/Controllers/"
  click WebUI "https://github.com/nntmux/newznab-tmux/tree/master/resources/views/"
  click Docker "https://github.com/nntmux/newznab-tmux/blob/master/docker-compose.yml"

  %% Styles
  classDef ingestion fill:#cce5ff,stroke:#3399ff,color:#000;
  classDef persistence fill:#d4edda,stroke:#28a745,color:#000;
  classDef presentation fill:#ffe5b4,stroke:#ff9900,color:#000;
  classDef infra fill:#e2e3e5,stroke:#6c757d,color:#000;
```
