How rafikfarhad/articlepub works
1. Project shape
- articlepub is a Python CLI package that converts an article URL into a Kindle-friendly EPUB, with an optional second path that uploads an existing EPUB to Calibre-Web.
- The architecture is a small orchestration layer over a content-ingestion and transformation pipeline, plus two external integrations: Anthropic for optional LLM-assisted fetching/cleanup and Calibre-Web for upload.
2. User entry points
- The executable entry point is src/articlepub/__main__.py, which routes into cli.py.
- The public commands are add for building an EPUB, doctor for preflight validation, upload for sending an EPUB to Calibre-Web, and tui for interactive use.
- cli_ui.py and tui.py separate interactive terminal presentation from the core pipeline so the same backend can run in both scripted and human-friendly modes.
3. Command-layer boundary
- cli.py is the main boundary between argv/flags and application logic.
- cli_support.py likely centralizes flag parsing, config assembly, validation, and shared command behavior.
- The CLI adapts output style based on whether stdout/stderr are interactive, which matters for architecture because presentation is intentionally decoupled from processing.
4. Core conversion pipeline
- pipeline.py is the orchestrator that ties together fetching, extraction, sanitization, LLM refinement, EPUB generation, and optional metadata storage.
- The pipeline appears to support multiple fetch modes: local-only, LLM-only, and auto fallback from local to LLM.
- This is the central data flow: URL in, article representation through several transformation stages, EPUB and diagnostics out.
5. Content ingestion and extraction
- fetch.py handles retrieving the source page locally.
- extract.py turns fetched HTML into the main article content.
- text.py likely normalizes or analyzes text content used by extraction/validation steps.
- This subsystem forms the non-LLM path and is the first choice in auto mode.
6. LLM integration boundary
- src/articlepub/llm/ isolates provider-specific behavior behind a base interface.
- anthropic.py is the only implemented provider in the tree; base.py and types.py define the abstraction and data contracts.
- The README indicates this integration uses Anthropic’s Messages API directly and can invoke the web_fetch server tool in llm mode to fetch a URL without local scraping.
7. Content cleanup and safety
- sanitize.py and redaction.py suggest a post-processing layer that removes unsafe/undesired markup, normalizes article HTML, and redacts sensitive content from stored artifacts or diagnostics.
- prompts.py holds the LLM prompts used when the system asks the model to clean or fetch content.
- strict mode, mentioned in the README, adds a validation boundary that checks sentence preservation when local source text is available.
8. Output generation
- epub.py is the packaging boundary that converts the finalized article model into an EPUB file.
- models.py likely defines the canonical article/book data structures shared across fetch, extract, sanitize, and EPUB generation.
- The EPUB is the main artifact of the system; the rest of the pipeline exists to produce a clean, Kindle-ready book from arbitrary web content.
9. External upload path
- upload.py is a separate integration path for Calibre-Web.
- It takes an already-built EPUB and pushes it to an external service using URL, credentials, and optionally an API key.
- Architecturally, this is intentionally split from conversion so upload can be used independently of article extraction.
10. Diagnostics, artifacts, and observability
- diagnostics.py and stats.py provide runtime reporting: timing, step-level status, and LLM token usage when available.
- raw_store.py manages raw run artifacts under OUT/.raw, which the README describes as an optional debug trail that can be preserved or cleaned up.
- This creates a clear operational boundary between normal production output and debugging evidence.
11. Interactive experience
- cli_ui.py and tui.py provide colored banners, spinner/status feedback, and “status faces” for interactive terminals.
- In non-interactive contexts, the same commands degrade to plain output for scripting.
- This separation keeps the core pipeline deterministic while allowing richer UX when a human is watching.
12. Validation and preflight
- doctor is a lightweight validation mode that checks URL, output path, flag compatibility, Calibre settings, and Anthropic configuration before doing the expensive work.
- This makes the architecture safer for long-running or API-backed jobs by moving failure detection ahead of conversion.
13. Testing boundary
- The tests/ tree mirrors the main subsystems: CLI, fetch/extract, pipeline, EPUB generation, upload, diagnostics, stats, redaction, and Anthropic behavior.
- That test layout reflects the repo’s modular architecture: each stage is meant to be independently verifiable, with fixtures for representative article HTML.
14. High-level data flow
- Inputs: article URL, CLI flags, optional Anthropic key/model settings, and optional Calibre-Web connection details.
- Processing: fetch locally or via LLM, extract and sanitize article content, optionally refine or validate it, then assemble the EPUB.
- Outputs: EPUB file in the output directory, optional .raw debugging artifacts, runtime diagnostics, and optionally an uploaded copy in Calibre-Web.
Main entry points: pipeline.py Pipeline · cli.py CLI · anthropic.py Anthropic · base.py LLM base
Components 21
CLI & UX
Command parsing and terminal presentation layers.
- __main__ · Python module entry that hands control to the CLI.
src/articlepub/__main__.py - CLI · Top-level argv boundary for add, doctor, upload, and tui commands.
src/articlepub/cli.py - CLI support · Shared parsing, config assembly, validation, and command helpers.
src/articlepub/cli_support.py - CLI UI · Interactive terminal presentation, banners, and status output.
src/articlepub/cli_ui.py - TUI · Human-friendly terminal workflow built on the same backend.
src/articlepub/tui.py
Conversion Pipeline
Fetch, extract, clean, and package article content.
- Pipeline · Coordinates fetching, extraction, cleanup, refinement, and EPUB creation.
src/articlepub/pipeline.py - Fetch · Retrieves source pages directly from a URL.
src/articlepub/fetch.py - Extract · Turns fetched HTML into the main article content.
src/articlepub/extract.py - Text · Shared text normalization and analysis used by extraction and validation.
src/articlepub/text.py - Sanitize · Normalizes article HTML and removes undesirable markup.
src/articlepub/sanitize.py - Redaction · Redacts sensitive content from stored artifacts or diagnostics.
src/articlepub/redaction.py - Models · Canonical article and book structures shared across stages.
src/articlepub/models.py - EPUB · Converts the finalized article model into an EPUB file.
src/articlepub/epub.py
LLM Integration
Provider abstraction and Anthropic-backed fetch/cleanup path.
- LLM base · Abstract provider contract and shared LLM data boundaries.
src/articlepub/llm/base.py - LLM types · Typed request and response structures for provider calls.
src/articlepub/llm/types.py - Anthropic · Anthropic Messages API implementation and web-fetch assisted mode.
src/articlepub/llm/anthropic.py - Prompts · Prompt templates for cleanup, fetching, and validation tasks.
src/articlepub/prompts.py
Ops & Output
Diagnostics, raw artifacts, and external publishing.
- Diagnostics · Step status, timings, and run-level reporting.
src/articlepub/diagnostics.py - Stats · Tracks timing and token usage when available.
src/articlepub/stats.py - Raw store · Manages optional .raw debug artifacts for a run.
src/articlepub/raw_store.py - Upload · Uploads an existing EPUB to Calibre-Web.
src/articlepub/upload.py
Connections 24
- __main__ → CLI: invokes · Module entry dispatches into the command router.
- CLI → CLI support: uses · CLI shares config and validation helpers.
- CLI → Pipeline: add · Builds an EPUB from an article URL.
- CLI → Upload: upload · Uploads an existing EPUB independently of conversion.
- CLI → TUI: tui · Starts interactive terminal workflow.
- CLI → Diagnostics: reports · Surface command and validation status to the user.
- TUI → CLI UI: renders · Uses interactive terminal presentation.
- Pipeline → Fetch: local fetch · Retrieves source HTML as the first non-LLM path.
- Pipeline → Extract: parse · Extracts article content from fetched markup.
- Pipeline → Sanitize: clean · Normalizes and filters article content.
- Pipeline → Redaction: redact · Removes sensitive content from artifacts or diagnostics.
- Pipeline → EPUB: build · Packages the final article into an EPUB.
- Extract → Text: analyze · Uses shared text processing for extraction and validation.
- Sanitize → Models: shape · Produces canonical article data for downstream stages.
- Pipeline → Models: passes · Shared data model threads through the conversion stages.
- Pipeline → Diagnostics: trace · Records step-level execution and outcome information.
- Pipeline → Stats: measure · Collects timings and token usage when available.
- Pipeline → Raw store: store · Writes optional raw run artifacts for debugging.
- Pipeline → LLM base: fallback · Delegates to LLM-backed fetching or cleanup when needed.
- LLM base → LLM types: defines · Provider interface is backed by typed request/response contracts.
- Anthropic → LLM base: implements · Anthropic is the concrete provider implementation.
- Anthropic → Prompts: uses · Provider assembles model prompts from shared templates.
- Pipeline → Anthropic: llm mode · Optional LLM-assisted fetch/cleanup path.
- Upload → Diagnostics: report · Upload path reuses runtime diagnostics.