# rafikfarhad/articlepub architecture

> Architecture diagram of the GitHub repository rafikfarhad/articlepub, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rafikfarhad/articlepub
- Repository: https://github.com/rafikfarhad/articlepub

## Overview

1. **Project shape**
   - `articlepub` is a Python CLI package that converts an article URL into a Kindle-friendly EPUB, with an optional second path that uploads an existing EPUB to Calibre-Web.
   - The architecture is a small orchestration layer over a content-ingestion and transformation pipeline, plus two external integrations: Anthropic for optional LLM-assisted fetching/cleanup and Calibre-Web for upload.

2. **User entry points**
   - The executable entry point is `src/articlepub/__main__.py`, which routes into `cli.py`.
   - The public commands are `add` for building an EPUB, `doctor` for preflight validation, `upload` for sending an EPUB to Calibre-Web, and `tui` for interactive use.
   - `cli_ui.py` and `tui.py` separate interactive terminal presentation from the core pipeline so the same backend can run in both scripted and human-friendly modes.

3. **Command-layer boundary**
   - `cli.py` is the main boundary between argv/flags and application logic.
   - `cli_support.py` likely centralizes flag parsing, config assembly, validation, and shared command behavior.
   - The CLI adapts output style based on whether stdout/stderr are interactive, which matters for architecture because presentation is intentionally decoupled from processing.

4. **Core conversion pipeline**
   - `pipeline.py` is the orchestrator that ties together fetching, extraction, sanitization, LLM refinement, EPUB generation, and optional metadata storage.
   - The pipeline appears to support multiple fetch modes: local-only, LLM-only, and auto fallback from local to LLM.
   - This is the central data flow: URL in, article representation through several transformation stages, EPUB and diagnostics out.

5. **Content ingestion and extraction**
   - `fetch.py` handles retrieving the source page locally.
   - `extract.py` turns fetched HTML into the main article content.
   - `text.py` likely normalizes or analyzes text content used by extraction/validation steps.
   - This subsystem forms the non-LLM path and is the first choice in `auto` mode.

6. **LLM integration boundary**
   - `src/articlepub/llm/` isolates provider-specific behavior behind a base interface.
   - `anthropic.py` is the only implemented provider in the tree; `base.py` and `types.py` define the abstraction and data contracts.
   - The README indicates this integration uses Anthropic’s Messages API directly and can invoke the `web_fetch` server tool in `llm` mode to fetch a URL without local scraping.

7. **Content cleanup and safety**
   - `sanitize.py` and `redaction.py` suggest a post-processing layer that removes unsafe/undesired markup, normalizes article HTML, and redacts sensitive content from stored artifacts or diagnostics.
   - `prompts.py` holds the LLM prompts used when the system asks the model to clean or fetch content.
   - `strict` mode, mentioned in the README, adds a validation boundary that checks sentence preservation when local source text is available.

8. **Output generation**
   - `epub.py` is the packaging boundary that converts the finalized article model into an EPUB file.
   - `models.py` likely defines the canonical article/book data structures shared across fetch, extract, sanitize, and EPUB generation.
   - The EPUB is the main artifact of the system; the rest of the pipeline exists to produce a clean, Kindle-ready book from arbitrary web content.

9. **External upload path**
   - `upload.py` is a separate integration path for Calibre-Web.
   - It takes an already-built EPUB and pushes it to an external service using URL, credentials, and optionally an API key.
   - Architecturally, this is intentionally split from conversion so upload can be used independently of article extraction.

10. **Diagnostics, artifacts, and observability**
   - `diagnostics.py` and `stats.py` provide runtime reporting: timing, step-level status, and LLM token usage when available.
   - `raw_store.py` manages raw run artifacts under `OUT/.raw`, which the README describes as an optional debug trail that can be preserved or cleaned up.
   - This creates a clear operational boundary between normal production output and debugging evidence.

11. **Interactive experience**
   - `cli_ui.py` and `tui.py` provide colored banners, spinner/status feedback, and “status faces” for interactive terminals.
   - In non-interactive contexts, the same commands degrade to plain output for scripting.
   - This separation keeps the core pipeline deterministic while allowing richer UX when a human is watching.

12. **Validation and preflight**
   - `doctor` is a lightweight validation mode that checks URL, output path, flag compatibility, Calibre settings, and Anthropic configuration before doing the expensive work.
   - This makes the architecture safer for long-running or API-backed jobs by moving failure detection ahead of conversion.

13. **Testing boundary**
   - The `tests/` tree mirrors the main subsystems: CLI, fetch/extract, pipeline, EPUB generation, upload, diagnostics, stats, redaction, and Anthropic behavior.
   - That test layout reflects the repo’s modular architecture: each stage is meant to be independently verifiable, with fixtures for representative article HTML.

14. **High-level data flow**
   - Inputs: article URL, CLI flags, optional Anthropic key/model settings, and optional Calibre-Web connection details.
   - Processing: fetch locally or via LLM, extract and sanitize article content, optionally refine or validate it, then assemble the EPUB.
   - Outputs: EPUB file in the output directory, optional `.raw` debugging artifacts, runtime diagnostics, and optionally an uploaded copy in Calibre-Web.

Main entry points: [`src/articlepub/pipeline.py`](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/pipeline.py) (Pipeline), [`src/articlepub/cli.py`](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/cli.py) (CLI), [`src/articlepub/llm/anthropic.py`](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/llm/anthropic.py) (Anthropic), [`src/articlepub/llm/base.py`](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/llm/base.py) (LLM base)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_cli["CLI & UX"]
  node_node_entry(("__main__<br/>entrypoint<br/>[__main__.py]"))
  node_node_cli["CLI<br/>command router<br/>[cli.py]"]
  node_node_cli_support["CLI support<br/>shared flags<br/>[cli_support.py]"]
  node_node_cli_ui["CLI UI<br/>terminal view<br/>[cli_ui.py]"]
  node_node_tui["TUI<br/>interactive mode<br/>[tui.py]"]
end

subgraph group_group_pipeline["Conversion Pipeline"]
  node_node_pipeline["Pipeline<br/>orchestrator<br/>[pipeline.py]"]
  node_node_fetch["Fetch<br/>[fetch.py]"]
  node_node_extract["Extract<br/>article parser<br/>[extract.py]"]
  node_node_text["Text<br/>[text.py]"]
  node_node_sanitize["Sanitize<br/>content cleanup<br/>[sanitize.py]"]
  node_node_redaction["Redaction<br/>safety filter<br/>[redaction.py]"]
  node_node_models[("Models<br/>data model<br/>[models.py]")]
  node_node_epub["EPUB<br/>package builder<br/>[epub.py]"]
end

subgraph group_group_llm["LLM Integration"]
  node_node_llm_base["LLM base<br/>provider interface<br/>[base.py]"]
  node_node_llm_types["LLM types<br/>llm contracts<br/>[types.py]"]
  node_node_anthropic["Anthropic<br/>llm provider<br/>[anthropic.py]"]
  node_node_prompts["Prompts<br/>prompt library<br/>[prompts.py]"]
end

subgraph group_group_ops["Ops & Output"]
  node_node_diagnostics["Diagnostics<br/>runtime reporting<br/>[diagnostics.py]"]
  node_node_stats["Stats<br/>usage metrics<br/>[stats.py]"]
  node_node_raw_store[("Raw store<br/>artifact store<br/>[raw_store.py]")]
  node_node_upload["Upload<br/>calibre-web client<br/>[upload.py]"]
end

node_node_entry -->|"invokes"| node_node_cli
node_node_cli -->|"uses"| node_node_cli_support
node_node_cli -->|"add"| node_node_pipeline
node_node_cli -->|"upload"| node_node_upload
node_node_cli -->|"tui"| node_node_tui
node_node_cli -->|"reports"| node_node_diagnostics
node_node_tui -->|"renders"| node_node_cli_ui
node_node_pipeline -->|"local fetch"| node_node_fetch
node_node_pipeline -->|"parse"| node_node_extract
node_node_pipeline -->|"clean"| node_node_sanitize
node_node_pipeline -->|"redact"| node_node_redaction
node_node_pipeline -->|"build"| node_node_epub
node_node_extract -->|"analyze"| node_node_text
node_node_sanitize -->|"shape"| node_node_models
node_node_pipeline -->|"passes"| node_node_models
node_node_pipeline -->|"trace"| node_node_diagnostics
node_node_pipeline -->|"measure"| node_node_stats
node_node_pipeline -->|"store"| node_node_raw_store
node_node_pipeline -->|"fallback"| node_node_llm_base
node_node_llm_base -->|"defines"| node_node_llm_types
node_node_anthropic -->|"implements"| node_node_llm_base
node_node_anthropic -->|"uses"| node_node_prompts
node_node_pipeline -.->|"llm mode"| node_node_anthropic
node_node_upload -->|"report"| node_node_diagnostics

click node_node_entry "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/__main__.py"
click node_node_cli "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/cli.py"
click node_node_cli_support "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/cli_support.py"
click node_node_cli_ui "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/cli_ui.py"
click node_node_tui "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/tui.py"
click node_node_pipeline "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/pipeline.py"
click node_node_fetch "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/fetch.py"
click node_node_extract "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/extract.py"
click node_node_text "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/text.py"
click node_node_sanitize "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/sanitize.py"
click node_node_redaction "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/redaction.py"
click node_node_models "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/models.py"
click node_node_epub "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/epub.py"
click node_node_llm_base "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/llm/base.py"
click node_node_llm_types "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/llm/types.py"
click node_node_anthropic "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/llm/anthropic.py"
click node_node_prompts "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/prompts.py"
click node_node_diagnostics "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/diagnostics.py"
click node_node_stats "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/stats.py"
click node_node_raw_store "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/raw_store.py"
click node_node_upload "https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/upload.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_node_entry,node_node_cli,node_node_cli_support,node_node_cli_ui,node_node_tui toneBlue
class node_node_pipeline,node_node_fetch,node_node_extract,node_node_text,node_node_sanitize,node_node_redaction,node_node_models,node_node_epub toneAmber
class node_node_llm_base,node_node_llm_types,node_node_anthropic,node_node_prompts toneMint
class node_node_diagnostics,node_node_stats,node_node_raw_store,node_node_upload toneRose
```

## Components

### CLI & UX

Command parsing and terminal presentation layers.

- [__main__](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/__main__.py): entrypoint · Python module entry that hands control to the CLI. · `src/articlepub/__main__.py`
- [CLI](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/cli.py): command router · Top-level argv boundary for add, doctor, upload, and tui commands. · `src/articlepub/cli.py`
- [CLI support](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/cli_support.py): shared flags · Shared parsing, config assembly, validation, and command helpers. · `src/articlepub/cli_support.py`
- [CLI UI](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/cli_ui.py): terminal view · Interactive terminal presentation, banners, and status output. · `src/articlepub/cli_ui.py`
- [TUI](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/tui.py): interactive mode · Human-friendly terminal workflow built on the same backend. · `src/articlepub/tui.py`

### Conversion Pipeline

Fetch, extract, clean, and package article content.

- [Pipeline](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/pipeline.py): orchestrator · Coordinates fetching, extraction, cleanup, refinement, and EPUB creation. · `src/articlepub/pipeline.py`
- [Fetch](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/fetch.py): local fetch · Retrieves source pages directly from a URL. · `src/articlepub/fetch.py`
- [Extract](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/extract.py): article parser · Turns fetched HTML into the main article content. · `src/articlepub/extract.py`
- [Text](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/text.py): text normalization · Shared text normalization and analysis used by extraction and validation. · `src/articlepub/text.py`
- [Sanitize](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/sanitize.py): content cleanup · Normalizes article HTML and removes undesirable markup. · `src/articlepub/sanitize.py`
- [Redaction](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/redaction.py): safety filter · Redacts sensitive content from stored artifacts or diagnostics. · `src/articlepub/redaction.py`
- [Models](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/models.py): data model · Canonical article and book structures shared across stages. · `src/articlepub/models.py`
- [EPUB](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/epub.py): package builder · Converts the finalized article model into an EPUB file. · `src/articlepub/epub.py`

### LLM Integration

Provider abstraction and Anthropic-backed fetch/cleanup path.

- [LLM base](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/llm/base.py): provider interface · Abstract provider contract and shared LLM data boundaries. · `src/articlepub/llm/base.py`
- [LLM types](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/llm/types.py): llm contracts · Typed request and response structures for provider calls. · `src/articlepub/llm/types.py`
- [Anthropic](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/llm/anthropic.py): llm provider · Anthropic Messages API implementation and web-fetch assisted mode. · `src/articlepub/llm/anthropic.py`
- [Prompts](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/prompts.py): prompt library · Prompt templates for cleanup, fetching, and validation tasks. · `src/articlepub/prompts.py`

### Ops & Output

Diagnostics, raw artifacts, and external publishing.

- [Diagnostics](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/diagnostics.py): runtime reporting · Step status, timings, and run-level reporting. · `src/articlepub/diagnostics.py`
- [Stats](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/stats.py): usage metrics · Tracks timing and token usage when available. · `src/articlepub/stats.py`
- [Raw store](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/raw_store.py): artifact store · Manages optional .raw debug artifacts for a run. · `src/articlepub/raw_store.py`
- [Upload](https://github.com/rafikfarhad/articlepub/blob/master/src/articlepub/upload.py): calibre-web client · Uploads an existing EPUB to Calibre-Web. · `src/articlepub/upload.py`

## Connections

- __main__ → CLI: invokes · Module entry dispatches into the command router.
- CLI → CLI support: uses · CLI shares config and validation helpers.
- CLI → Pipeline: add · Builds an EPUB from an article URL.
- CLI → Upload: upload · Uploads an existing EPUB independently of conversion.
- CLI → TUI: tui · Starts interactive terminal workflow.
- CLI → Diagnostics: reports · Surface command and validation status to the user.
- TUI → CLI UI: renders · Uses interactive terminal presentation.
- Pipeline → Fetch: local fetch · Retrieves source HTML as the first non-LLM path.
- Pipeline → Extract: parse · Extracts article content from fetched markup.
- Pipeline → Sanitize: clean · Normalizes and filters article content.
- Pipeline → Redaction: redact · Removes sensitive content from artifacts or diagnostics.
- Pipeline → EPUB: build · Packages the final article into an EPUB.
- Extract → Text: analyze · Uses shared text processing for extraction and validation.
- Sanitize → Models: shape · Produces canonical article data for downstream stages.
- Pipeline → Models: passes · Shared data model threads through the conversion stages.
- Pipeline → Diagnostics: trace · Records step-level execution and outcome information.
- Pipeline → Stats: measure · Collects timings and token usage when available.
- Pipeline → Raw store: store · Writes optional raw run artifacts for debugging.
- Pipeline → LLM base: fallback · Delegates to LLM-backed fetching or cleanup when needed.
- LLM base → LLM types: defines · Provider interface is backed by typed request/response contracts.
- Anthropic → LLM base: implements · Anthropic is the concrete provider implementation.
- Anthropic → Prompts: uses · Provider assembles model prompts from shared templates.
- Pipeline → Anthropic: llm mode · Optional LLM-assisted fetch/cleanup path.
- Upload → Diagnostics: report · Upload path reuses runtime diagnostics.