# openzim/ted architecture

> Architecture diagram of the GitHub repository openzim/ted, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/openzim/ted
- Repository: https://github.com/openzim/ted

## Overview

##### Project Type and Purpose
- This is a command-line offliner tool written in Python that converts TED Talks into ZIM files. It does so by scraping TED content, downloading videos (with multiple formats available like MP4/webm), and packaging them into offline accessible ZIM files.
- The README emphasizes its usage for both single topics/playlists and bulk operations (ted2zim-multi mode). In addition, there is Docker support which means container deployment is supported too.

##### Analyzing the File Structure
- Top-Level Files and Directories:
- .github folder: contains CI/CD workflows (publishing, tests, QA, warming up stale issues, funding info) which hints at a well-managed open-source project with continuous integration.
- Dockerfile and entrypoint.sh: indicate containerization and shell-level initialization.
- pyproject.toml and pre-commit config: enforce coding conventions and project builds.
- src/ted2zim: the main Python package with core modules such as entrypoint.py, scraper.py, processing.py, constants, and utilities.
- The multi subdirectory inside src/ted2zim shows the existence of an enhanced “multitool” command variant.
- The locale directory (with translations) and templates directory (for HTML assets) show that the project has internationalization components and generates HTML output (likely for ZIM file contents or metadata pages).
- tests folder for unit tests confirming a testable architecture.

##### Main Components and Their Roles
- Command-Line Interface (CLI):
- Entry points: src/ted2zim/entrypoint.py for the main ted2zim functionality and src/ted2zim/multi/entrypoint.py for the multitool advanced mode.
- Tasks handling arguments (topics, links, formats, output options, etc.) based on the examples in the README.
- Scraping and Processing:
- src/ted2zim/scraper.py and src/ted2zim/multi/scraper.py do the core work of scraping TED talk data.
- The processing.py module likely manages data transformation and converting downloaded content into the ZIM format.
- Templating and Assets:
- The templates directory holds HTML files and assets used for generating the final ZIM file content. This is perhaps used to structure or decorate the offline content.
- Localization:
- Files in src/ted2zim/locale indicate that messages and prompts are localized (e.g., for Hindi).
- External Integrations:
- The project requires external command-line tools such as ffmpeg, wget, and curl which must be available on the host.
- Docker support allows containerized operation, and the .github workflows indicate automated testing and deployment (publishing Docker images).
- Multi-mode:
- The “ted2zim-multi” functionality supports the creation of multiple ZIM files in a single run, handling batch operations and metadata JSON configuration.

##### Relationships and Data Flow
- User invokes the CLI (either ted2zim or ted2zim-multi) passing the required arguments.
- The entrypoint collects user input and determines which mode to execute.
- The scraper module fetches TED talk data (by topics, links, or via a JSON metadata file).
- Downloader commands (likely invoking external tools such as wget and curl) retrieve the necessary video content.
- ffmpeg (or equivalent conversion tools) are invoked to ensure correct video formatting based on user preferences.
- Templating modules combine the retrieved content with HTML and assets to generate an offline browsable package.
- The processed data is then packaged into a ZIM file, and the tool outputs the file to the specified directory.
- For multi-mode, the tool repeats the above for each topic or playlist, using the metadata JSON if provided.

##### Architectural Patterns and Design Principles
- Separation of Concerns:
- The design clearly separates input handling (CLI entrypoints), scraping/downloading, processing/formatting, localization, and output packaging.
- Modularity:
- The existence of separate modules (scraper, processing, multi, templates) suggests a modular architecture, making the code easier to maintain.
- Extensibility:
- The “multi” sub-module and the parameterized metadata format allow the system to be easily extended without modifying core functionalities.
- Containerized Deployment:
- Use of Dockerfile and accompanying CI workflows means that deployment and repeatable builds are a priority.

## Diagram (Mermaid)

```mermaid
flowchart TD
    U("User Command Input"):::external

    subgraph "CLI Layer"
        CLI1("CLI (Standard Mode)"):::cli
        CLI2("CLI (Multi-mode)"):::cli
    end

    U --> CLI1
    U --> CLI2

    subgraph "Scraping Modules"
        SCR1("Scraper (Standard Mode)"):::core
        SCR2("Scraper (Multi-mode)"):::core
    end

    CLI1 --> SCR1
    CLI2 --> SCR2

    subgraph "Processing & Packaging"
        PROC("Processing & Conversion"):::core
        TEMP("Templating & Assets"):::core
        LOC("Localization"):::core
        ZIM("ZIM Generator"):::core
    end

    SCR1 --> PROC
    SCR2 --> PROC
    PROC --> TEMP
    PROC --> LOC
    TEMP --> ZIM
    LOC --> ZIM

    EXT("External Tools (ffmpeg, wget, curl)"):::external
    PROC --> EXT

    subgraph "Deployment & CI/CD"
        DOCKER("Dockerfile"):::infra
        ENTP("Entrypoint Script"):::infra
        CI("CI/CD Workflows"):::infra
    end

    ZIM --> DOCKER
    ZIM --> ENTP
    DOCKER --> CI
    ENTP --> CI

    click CLI1 "https://github.com/openzim/ted/blob/main/src/ted2zim/entrypoint.py"
    click CLI2 "https://github.com/openzim/ted/blob/main/src/ted2zim/multi/entrypoint.py"
    click SCR1 "https://github.com/openzim/ted/blob/main/src/ted2zim/scraper.py"
    click SCR2 "https://github.com/openzim/ted/blob/main/src/ted2zim/multi/scraper.py"
    click PROC "https://github.com/openzim/ted/blob/main/src/ted2zim/processing.py"
    click PROC "https://github.com/openzim/ted/blob/main/src/ted2zim/utils.py"
    click TEMP "https://github.com/openzim/ted/tree/main/src/ted2zim/templates"
    click LOC "https://github.com/openzim/ted/tree/main/src/ted2zim/locale"
    click DOCKER "https://github.com/openzim/ted/tree/main/Dockerfile"
    click ENTP "https://github.com/openzim/ted/blob/main/entrypoint.sh"
    click CI "https://github.com/openzim/ted/tree/main/.github/workflows"

    classDef cli fill:#ffebcd,stroke:#333,stroke-width:2px;
    classDef core fill:#add8e6,stroke:#333,stroke-width:2px;
    classDef external fill:#ffe4e1,stroke:#333,stroke-width:2px;
    classDef infra fill:#98fb98,stroke:#333,stroke-width:2px;
```
