How iptv-org/epg works
1. This repository is a Node.js/TypeScript toolchain for generating TV EPG data, not a single app. The main runtime job is “grab guide data from many site-specific sources and write it into XMLTV/JSON/GZip outputs.”
2. The central execution path is the epg grab command under scripts/commands/epg/grab.ts. It orchestrates a set of site adapters, reads channel selections from *.channels.xml files, applies CLI options like site list, language filtering, concurrency, timeout, delay, proxy, and output format, then produces guide.xml and optional guide.json / guide.xml.gz.
3. Each source site lives in its own folder under sites/<domain>/ and is effectively an isolated adapter boundary. A site folder usually contains a site config, a site-specific test, one or more *.channels.xml mapping files, fixture data under __data__, and a short readme. Some sites have multiple regions/locales, which is why many folders emit multiple channel XML files.
4. The *.channels.xml files are the main domain input for scraping. They map external source IDs to internal XMLTV channel IDs and language metadata. The channel commands (parse, format, edit, lint, validate) exist to maintain and verify this mapping layer before scraping runs.
5. The shared implementation lives in scripts/core, scripts/models, scripts/constants.ts, and the scripts/api.ts helper. This layer holds the reusable parsing, table/HTML utilities, and typed data models for channels, guides, programs, sites, workers, issues, and API guide records. It is the main boundary between generic orchestration and site-specific scraping logic.
6. Site scraping itself is data-source driven: the repo does not hardcode one transport or one HTML shape. Site configs describe how to load pages/APIs, and the fixtures in sites/*/__data__ capture real responses for tests. Those fixtures show the supported source types: HTML pages, JSON APIs, XML feeds, text payloads, compressed files, and occasional session/auth artifacts.
7. There is a separate maintenance pipeline for repository metadata. scripts/commands/guides/* updates GUIDES.md and exports guide metadata, while scripts/commands/sites/* initializes new site folders and refreshes SITES.md. These commands keep the catalog of available sources and community guides in sync with the codebase.
8. The workers commands and workers.txt appear to manage a worker catalog used by the scraping system. The presence of scripts/models/worker.ts, tests/.../workers_load, and workers.json fixtures suggests a structured mapping from channels/guides to worker definitions rather than ad hoc scraping.
9. Tests are organized as command-level integration tests under tests/commands/* plus per-site tests in each site folder. The tests/__data__ tree contains input and expected output snapshots for parsing, validation, guide grabbing, site scaffolding, guide export/update, and worker loading. This indicates the repo is heavily snapshot/fixture-driven.
10. The repository also has a small build/ops layer: ESLint/Prettier for formatting, Husky pre-commit checks, GitHub Actions for check/format/update/Docker publish, and a Dockerfile plus pm2.config.js for containerized scheduling. In Docker, the tool is run as a scheduled guide loader with a mounted channels.xml and a public output directory.
11. External dependencies matter architecturally in two places: source metadata comes from the iptv-org/database repository, and the produced guide is meant to be consumed as a file or served over HTTP. The README’s examples show local serving via serve, and scheduled execution via cron/chronos, so output delivery is file-first rather than API-first.
Main entry points: grab.test.ts Command tests · grab.ts EPG grab · sites Sites · index.ts Core utils
Components 27
Commands
CLI entrypoints and maintenance jobs
- EPG grab · Main runtime orchestration for scraping and guide generation
scripts/commands/epg/grab.ts - Parse channels · Reads and normalizes channel mapping XML
scripts/commands/channels/parse.ts - Format channels · Rewrites channel mapping XML in canonical form
scripts/commands/channels/format.ts - Edit channels · Assists manual channel mapping updates
scripts/commands/channels/edit.ts - Validate channels · Checks mapping integrity against known data
scripts/commands/channels/validate.ts - Update guides · Refreshes GUIDES.md metadata
scripts/commands/guides/update.ts - Export guides · Exports guide metadata to JSON
scripts/commands/guides/export.ts - Update sites · Refreshes SITES.md and site catalog data
scripts/commands/sites/update.ts - Init site · Creates a new site folder template
scripts/commands/sites/init.ts - Load workers · Loads worker definitions from text and JSON inputs
scripts/commands/workers/load.ts - API load · Loads guide metadata into API-facing records
scripts/commands/api/load.ts
Shared layer
Reusable parsing, models, and API helpers
- Core utils · Reusable parsing, table, and execution utilities
scripts/core/index.ts - HTML table · Table rendering/parsing helper used by commands
scripts/core/htmlTable.ts - Data models · Typed models for channels, guides, programs, sites, workers, and issues
scripts/models/index.ts - API helper · Transforms repository data into API guide records
scripts/api.ts - Default config · Shared runtime defaults for command behavior
scripts/default.config.js
Site adapters
Per-source scraping boundaries and fixtures
- Sites · Per-domain site folders containing isolated scraping logic
sites - Site config · Domain-specific scraping rules and transport setup
sites/abc.net.au/abc.net.au.config.js - Channel map · Region and language-specific XMLTV channel mappings
sites/abc.net.au/abc.net.au_adl.channels.xml - Fixtures · Captured HTML/JSON/XML responses for adapter tests
sites/abc.net.au/__data__/content.json - Site test · Per-site integration test for scraper behavior
sites/abc.net.au/abc.net.au.test.js
Support assets
Snapshots, catalogs, and repository ops
- Command tests · Snapshot-driven tests for CLI commands
tests/commands/epg/grab.test.ts - Snapshots · Expected inputs and outputs for command behavior
tests/__data__/expected/epg_grab/base.guide.xml - Test inputs · Input XML, JSON, and catalog files used by command tests
tests/__data__/input/epg_grab/custom.channels.xml - GUIDES.md · Generated guide catalog document
GUIDES.md - SITES.md · Generated site catalog document
SITES.md - workers.txt · Text catalog consumed by worker loading
workers.txt
Connections 30
- EPG grab → Default config: uses defaults · Applies shared runtime settings and CLI overrides
- EPG grab → Core utils: orchestrates · Uses shared parsing, concurrency, and IO helpers
- EPG grab → Data models: structures · Builds guides, programs, sites, and issue records
- EPG grab → API helper: exports · Can emit API-style guide records in addition to files
- EPG grab → Sites: runs adapters · Loads site-specific scraping implementations
- EPG grab → Channel map: reads mappings · Consumes selected channel XML mappings for scraping
- Parse channels → Core utils: parses with · Uses shared XML and text utilities
- Format channels → Core utils: formats with · Relies on shared normalization and table utilities
- Edit channels → Core utils: edits with · Uses shared helpers for interactive channel maintenance
- Validate channels → Data models: validates against · Checks channel and feed identities in typed data
- Update guides → GUIDES.md: writes · Regenerates the guide catalog document
- Export guides → API helper: exports via · Produces JSON guide metadata from repository data
- Update sites → SITES.md: writes · Regenerates the site catalog document
- Init site → Sites: scaffolds · Creates new site adapter folders from templates
- Load workers → workers.txt: reads · Consumes the worker catalog text file
- Load workers → Data models: builds · Materializes worker records from catalog data
- API load → API helper: hydrates · Loads external guide metadata into API records
- API load → Data models: uses models · Converts loaded data into typed records
- Sites → Site config: contains · Each site folder centers on a domain config
- Sites → Channel map: contains · Channel XML mappings live alongside the adapter
- Sites → Fixtures: contains · Fixtures capture real source responses
- Sites → Site test: contains · Each adapter has a site-specific test
- Site test → Fixtures: verifies · Tests assert behavior against captured responses
- Command tests → Test inputs: consumes · Uses command input snapshots and sample configs
- Command tests → Snapshots: compares · Checks generated outputs against expected snapshots
- Command tests → EPG grab: covers · Validates the main guide-grabbing workflow
- Command tests → Parse channels: covers · Validates channel parsing behavior
- Command tests → Update guides: covers · Validates guide metadata maintenance
- Command tests → Update sites: covers · Validates site catalog maintenance
- Command tests → Load workers: covers · Validates worker catalog loading