How nntmux/newznab-tmux works
Project type and purpose
• This is a full-stack Laravel-based usenet indexing system (“NNTmux”) that:
• Continuously fetches headers from Usenet NNTP servers via CLI scripts (the “Blacklight” library)
• Processes and collates headers into “releases” and metadata
• Exposes a web UI (Blade/Vue) plus a REST/RSS/API interface
• Provides an admin console, real-time metrics (Laravel Horizon/Tmux), search (Manticore/Elasticsearch)
File structure
• Top–level:
• Blacklight/… → stand-alone PHP classes/scripts for NNTP header retrieval, backfilling, multi-threaded processing
• app/… → Laravel MVC (controllers, models, jobs, commands, listeners, middleware)
• resources/views, resources/js, public/assets → frontend templates, assets, Vue/Livewire components, Filament admin
• config/, database/ → configs and migrations/seeders
• docker/, docker-compose.yml, sail → containerized deployment
• storage/ covers & nzb/… → file storage for covers, raw nzb files, caches
• Key patterns:
• MVC + Job Queue (Redis + Horizon)
• CLI Commands invoked by scheduler or tmux to harvest/process
• External service adapters (AniDB, TMDb, Trakt, IRC) in Blacklight/libraries & app/Extensions
Extract architecture insights from README
• Multi-threaded processing via Tmux
• MySQL/MariaDB primary store (InnoDB heavy usage)
• Search engine (Manticore/ElasticSearchSiteSearch)
• Requires Redis (queue, cache), optional Memcached
• Docker-based via Laravel Sail
Main components & interactions
a. Components:
1. NNTP Harvesters (Blacklight CLI)
2. Post-processing workers (Blacklight & Laravel Jobs)
3. Laravel App
– HTTP/API server (Controllers + Blade/Vue UI)
– CLI Commands (app/Console/Commands) for sync tasks
– Job Queue (Redis + Horizon)
4. Database (MySQL/MariaDB)
5. Search Engine (Manticore or Elasticsearch)
6. Cache/Queue (Redis)
7. File Storage (covers/nzb in storage/)
8. Tmux Manager (monitors workers & DB performance)
9. External services (AniDB, TMDb, Trakt, IRC, NZB indexing endpoints)
10. Docker Host (or on-prem VM) orchestrating containers via docker-compose / Sail
b. Relationships & Data Flow:
• NNTP Harvesters ↔ Usenet NNTP Servers (ingest headers) → Database (staged tables)
• Staged data → Laravel Post-processing jobs → Releases table, metadata tables
• Jobs publish to Redis queue → Worker processes → Database updates & cover download → storage/
• Laravel HTTP/API ← User browser or API client → Controllers → Models → Database / Cache
• Search queries → Search Engine ← synchronized via Laravel Scout / scheduled commands
• Tmux monitors CLI & queue workers, DB stats
c. Architectural patterns & principles:
• MVC for web layer
• Command pattern for CLI jobs
• Producer-consumer via Redis & Horizon
• Adapter pattern for external APIs in Blacklight/libraries & app/Extensions
• Layered separation: harvest → process → store → serve
d. Key technologies/frameworks:
• Laravel (8.x/9.x) with Sail, Horizon, Scout
• Blacklight PHP library for CLI
• MySQL/MariaDB (InnoDB heavy)
• Manticore or Elasticsearch
• Redis (queue, cache)
• Tmux for multi-thread orchestration
• Docker / docker-compose