# arvindang/illustrative architecture

> Architecture diagram of the GitHub repository arvindang/illustrative, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arvindang/illustrative
- Repository: https://github.com/arvindang/illustrative

## Overview

1. **What this repo is**
   - A Python 3.12, batch-first pipeline that turns public-domain text into a graphic novel.
   - The main runtime dependency is Google Gemini / Vertex AI for long-context analysis and image generation.
   - The system is designed as a multi-stage content factory, not a generic web app.

2. **Primary entrypoints**
   - `app.py` runs the interactive Streamlit UI for test/ops workflows.
   - `production_run.py` runs the full end-to-end book processing path.
   - The UI and CLI both drive the same underlying pipeline, but with different operational modes.

3. **User-facing surface**
   - `ui/` contains the Streamlit pages: auth, home, dashboard, generate, and novel detail.
   - `ui/session.py` and `ui/db_operations.py` suggest the UI is stateful and backed by persisted job/user records.
   - `auth_utils.py` and `ui/pages/auth.py` form the access boundary before a user can launch or inspect jobs.

4. **Core pipeline orchestration**
   - `ui/pipeline.py` appears to coordinate the main workflow from uploaded source text to finished output.
   - The pipeline is split into three major functional stages: scripting, illustration, and composition/export.
   - Intermediate artifacts are file-based and tracked in the repo’s asset/output directories.

5. **Scripting subsystem**
   - `agents/scripting/` is a multi-pass adaptation engine that ingests an entire novel and produces structured story assets.
   - Passes include beat analysis, adaptation filtering, pacing, character deep dives, dialogue polish, asset manifesting, and final scriptwriting.
   - Output artifacts include JSON files like full script, assets, beats, and character arcs; this stage is the source of narrative structure for later visual generation.

6. **Reference and panel generation**
   - The illustration side is split into reference creation and panel generation.
   - `agents/reference_agent.py` and `agents/panel_agent.py`, plus the provider-specific equivalents under `agents/openai/`, encapsulate model calls for character/object/location references and page/panel images.
   - The `agents/openai/` layer contains API client, batch manager, and file manager code, indicating async/batch-oriented model interaction and artifact transfer.

7. **Layout, composition, and export**
   - `agents/layout_agent.py`, `agents/compositor_agent.py`, and `agents/export_agent.py` handle page assembly after images are generated.
   - The compositor stage places panels on pages, overlays captions and speech bubbles, and prepares final page images.
   - Export produces reader-facing deliverables such as PDF and EPUB.

8. **Quality gates and validation**
   - `validators/` is a distinct boundary around correctness and consistency.
   - Pre-validators check inputs and structure before generation; post-validators check composition and narrative consistency afterward.
   - This is important because the pipeline depends on preserving character identity, visual continuity, and page-level formatting across many generated assets.

9. **Persistence and domain model**
   - `models/` and `migrations/` indicate a relational database is used for users, novels, progress, errors, and cost tracking.
   - `models/novel.py` and `models/user.py` define the core persisted entities.
   - Alembic migrations show the schema evolved to support progress reporting and cost accounting, which are operationally important for long-running generation jobs.

10. **Artifact storage**
   - `assets/input/` is the source drop zone for uploaded text.
   - `assets/output/` is the working and final artifact area for generated characters, pages, and composed outputs.
   - `storage/bucket.py` suggests an abstraction for cloud/object storage, likely used when moving beyond local filesystem-only runs.

11. **Operational and deployment boundaries**
   - The presence of `Procfile`, `railway.toml`, and `nixpacks.toml` shows the app is meant to deploy in hosted PaaS environments.
   - `.github/workflows/pages.yml` and the `site/` directory indicate a separate static documentation/demo site, including sample PDF viewing assets.
   - `production_run.py` is the boundary between demo UI and full batch production use.

12. **Testing and reliability**
   - Tests cover smoke, integration, isolation, scripting-only, references, cost calculation, and full pipeline runs.
   - This mirrors the architecture: each stage can be validated independently, while `tests/test_full_pipeline.py` exercises the integrated workflow.
   - `scripts/` contains environment and syntax checks, reinforcing that the system is operationally sensitive to API credentials, model availability, and pipeline configuration.

13. **Mental model for an architecture diagram**
   - Draw it as a left-to-right content factory: input text and user intent enter on the left, scripted narrative assets are created in the middle, image/reference generation happens next, and page composition/export happens on the right.
   - Put the database and storage beneath the pipeline as shared infrastructure.
   - Put Gemini/Vertex AI outside the system boundary as the external model service the agents depend on.

Main entry points: [`agents/scripting/agent.py`](https://github.com/arvindang/illustrative/blob/main/agents/scripting/agent.py) (Script Agent), [`agents/export_agent.py`](https://github.com/arvindang/illustrative/blob/main/agents/export_agent.py) (Exporter), [`ui/pipeline.py`](https://github.com/arvindang/illustrative/blob/main/ui/pipeline.py) (UI Pipeline), [`agents/panel_agent.py`](https://github.com/arvindang/illustrative/blob/main/agents/panel_agent.py) (Panel Agent)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["UI & Ops"]
  node_app_py["Streamlit App<br/>entrypoint<br/>[app.py]"]
  node_ui_pipeline["UI Pipeline<br/>orchestrator<br/>[pipeline.py]"]
  node_ui_pages_auth["Auth Page<br/>ui page<br/>[auth.py]"]
  node_ui_pages_dashboard["Dashboard<br/>ui page<br/>[dashboard.py]"]
  node_ui_pages_generate["Generate Page<br/>ui page<br/>[generate.py]"]
end

subgraph group_pipeline["Pipeline"]
  node_production_run_py["Prod Run<br/>entrypoint<br/>[production_run.py]"]
end

subgraph group_scripting["Scripting"]
  node_script_agent["Script Agent<br/>scripting orchestrator<br/>[agent.py]"]
  node_beat_analysis["Beat Analysis<br/>scripting pass<br/>[beat_analysis.py]"]
  node_adaptation_filter["Adaptation Filter<br/>scripting pass"]
  node_character_deep_dive["Character Dive<br/>scripting pass"]
  node_scriptwriter["Scriptwriter<br/>scripting pass<br/>[scriptwriter.py]"]
end

subgraph group_visuals["Illustration"]
  node_reference_agent["Reference Agent<br/>image planning<br/>[reference_agent.py]"]
  node_panel_agent["Panel Agent<br/>image generation<br/>[panel_agent.py]"]
end

subgraph group_output["Assembly & Export"]
  node_layout_agent["Layout Agent<br/>[layout_agent.py]"]
  node_compositor_agent["Compositor<br/>composition"]
  node_export_agent["Exporter<br/>[export_agent.py]"]
  node_validators_pre["Pre-Validators<br/>validation<br/>[pre_validators.py]"]
  node_validators_post["Post-Validators<br/>validation<br/>[post_validators.py]"]
end

subgraph group_infra["Infra & External"]
  node_db_models[("Domain Models<br/>persistence model<br/>[novel.py]")]
  node_storage_bucket[("Bucket Storage<br/>artifact storage<br/>[bucket.py]")]
  node_gemini_vertex{{"Gemini / Vertex AI<br/>external model service"}}
end

node_app_py -->|"launches"| node_ui_pipeline
node_production_run_py -->|"starts batch flow"| node_script_agent
node_ui_pages_auth -->|"gates access"| node_ui_pipeline
node_ui_pages_dashboard -->|"reads status"| node_db_models
node_ui_pages_generate -->|"submits job"| node_ui_pipeline
node_ui_pipeline -->|"prechecks"| node_validators_pre
node_validators_pre -->|"allows"| node_script_agent
node_script_agent -->|"runs"| node_beat_analysis
node_script_agent -->|"runs"| node_adaptation_filter
node_script_agent -->|"runs"| node_character_deep_dive
node_script_agent -->|"produces"| node_scriptwriter
node_script_agent -->|"hands off assets"| node_reference_agent
node_reference_agent -->|"model calls"| node_gemini_vertex
node_panel_agent -->|"image calls"| node_gemini_vertex
node_reference_agent -->|"stores refs"| node_storage_bucket
node_panel_agent -->|"stores panels"| node_storage_bucket
node_script_agent -->|"tracks progress"| node_db_models
node_panel_agent -->|"feeds pages"| node_layout_agent
node_layout_agent -->|"defines layout"| node_compositor_agent
node_compositor_agent -->|"checks output"| node_validators_post
node_validators_post -->|"approves export"| node_export_agent
node_compositor_agent -->|"exports pages"| node_export_agent
node_export_agent -->|"publishes artifacts"| node_storage_bucket
node_export_agent -->|"records cost"| node_db_models

click node_app_py "https://github.com/arvindang/illustrative/blob/main/app.py"
click node_production_run_py "https://github.com/arvindang/illustrative/blob/main/production_run.py"
click node_ui_pipeline "https://github.com/arvindang/illustrative/blob/main/ui/pipeline.py"
click node_ui_pages_auth "https://github.com/arvindang/illustrative/blob/main/ui/pages/auth.py"
click node_ui_pages_dashboard "https://github.com/arvindang/illustrative/blob/main/ui/pages/dashboard.py"
click node_ui_pages_generate "https://github.com/arvindang/illustrative/blob/main/ui/pages/generate.py"
click node_script_agent "https://github.com/arvindang/illustrative/blob/main/agents/scripting/agent.py"
click node_beat_analysis "https://github.com/arvindang/illustrative/blob/main/agents/scripting/passes/beat_analysis.py"
click node_adaptation_filter "https://github.com/arvindang/illustrative/blob/main/agents/scripting/passes/adaptation_filter.py"
click node_character_deep_dive "https://github.com/arvindang/illustrative/blob/main/agents/scripting/passes/character_deep_dive.py"
click node_scriptwriter "https://github.com/arvindang/illustrative/blob/main/agents/scripting/passes/scriptwriter.py"
click node_reference_agent "https://github.com/arvindang/illustrative/blob/main/agents/reference_agent.py"
click node_panel_agent "https://github.com/arvindang/illustrative/blob/main/agents/panel_agent.py"
click node_layout_agent "https://github.com/arvindang/illustrative/blob/main/agents/layout_agent.py"
click node_compositor_agent "https://github.com/arvindang/illustrative/blob/main/agents/compositor_agent.py"
click node_export_agent "https://github.com/arvindang/illustrative/blob/main/agents/export_agent.py"
click node_validators_pre "https://github.com/arvindang/illustrative/blob/main/validators/pre_validators.py"
click node_validators_post "https://github.com/arvindang/illustrative/blob/main/validators/post_validators.py"
click node_db_models "https://github.com/arvindang/illustrative/blob/main/models/novel.py"
click node_storage_bucket "https://github.com/arvindang/illustrative/blob/main/storage/bucket.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app_py,node_ui_pipeline,node_ui_pages_auth,node_ui_pages_dashboard,node_ui_pages_generate toneBlue
class node_production_run_py toneAmber
class node_script_agent,node_beat_analysis,node_adaptation_filter,node_character_deep_dive,node_scriptwriter toneMint
class node_reference_agent,node_panel_agent toneRose
class node_layout_agent,node_compositor_agent,node_export_agent,node_validators_pre,node_validators_post toneIndigo
class node_db_models,node_storage_bucket,node_gemini_vertex toneTeal
```

## Components

### UI & Ops

Interactive Streamlit surface and job launching boundaries.

- [Streamlit App](https://github.com/arvindang/illustrative/blob/main/app.py): entrypoint · Interactive UI entrypoint for test and operations workflows. · `app.py`
- [UI Pipeline](https://github.com/arvindang/illustrative/blob/main/ui/pipeline.py): orchestrator · Coordinates the main workflow from uploaded text to generated outputs. · `ui/pipeline.py`
- [Auth Page](https://github.com/arvindang/illustrative/blob/main/ui/pages/auth.py): ui page · Access boundary before users can inspect or launch jobs. · `ui/pages/auth.py`
- [Dashboard](https://github.com/arvindang/illustrative/blob/main/ui/pages/dashboard.py): ui page · Operational overview for job status and progress. · `ui/pages/dashboard.py`
- [Generate Page](https://github.com/arvindang/illustrative/blob/main/ui/pages/generate.py): ui page · User-facing launch point for generation requests. · `ui/pages/generate.py`

### Pipeline

Core end-to-end generation flow from source text to exported book assets.

- [Prod Run](https://github.com/arvindang/illustrative/blob/main/production_run.py): entrypoint · Batch-first end-to-end production processing entrypoint. · `production_run.py`

### Scripting

Novel-to-script adaptation passes that create the narrative backbone.

- [Script Agent](https://github.com/arvindang/illustrative/blob/main/agents/scripting/agent.py): scripting orchestrator · Multi-pass adaptation engine that converts a novel into structured story assets. · `agents/scripting/agent.py`
- [Beat Analysis](https://github.com/arvindang/illustrative/blob/main/agents/scripting/passes/beat_analysis.py): scripting pass · Extracts narrative beats and story structure. · `agents/scripting/passes/beat_analysis.py`
- [Adaptation Filter](https://github.com/arvindang/illustrative/blob/main/agents/scripting/passes/adaptation_filter.py): scripting pass · Filters and reshapes source material for graphic-novel adaptation. · `agents/scripting/passes/adaptation_filter.py`
- [Character Dive](https://github.com/arvindang/illustrative/blob/main/agents/scripting/passes/character_deep_dive.py): scripting pass · Builds character behavior, identity, and continuity notes. · `agents/scripting/passes/character_deep_dive.py`
- [Scriptwriter](https://github.com/arvindang/illustrative/blob/main/agents/scripting/passes/scriptwriter.py): scripting pass · Produces the final structured comic script and scene assets. · `agents/scripting/passes/scriptwriter.py`

### Illustration

Reference and panel image generation stages.

- [Reference Agent](https://github.com/arvindang/illustrative/blob/main/agents/reference_agent.py): image planning · Creates character, object, and location references for visual continuity. · `agents/reference_agent.py`
- [Panel Agent](https://github.com/arvindang/illustrative/blob/main/agents/panel_agent.py): image generation · Generates page and panel imagery from scripted scenes. · `agents/panel_agent.py`

### Assembly & Export

Page composition, validation, persistence, and delivery artifacts.

- [Layout Agent](https://github.com/arvindang/illustrative/blob/main/agents/layout_agent.py): layout · Decides page and panel placement before composition. · `agents/layout_agent.py`
- [Compositor](https://github.com/arvindang/illustrative/blob/main/agents/compositor_agent.py): composition · Assembles pages, captions, and speech bubbles into final page images. · `agents/compositor_agent.py`
- [Exporter](https://github.com/arvindang/illustrative/blob/main/agents/export_agent.py): export · Produces reader-facing deliverables such as PDF and EPUB. · `agents/export_agent.py`
- [Pre-Validators](https://github.com/arvindang/illustrative/blob/main/validators/pre_validators.py): validation · Checks inputs and structure before generation begins. · `validators/pre_validators.py`
- [Post-Validators](https://github.com/arvindang/illustrative/blob/main/validators/post_validators.py): validation · Checks composition and narrative consistency after generation. · `validators/post_validators.py`

### Infra & External

Shared persistence, storage, and external model dependencies.

- [Domain Models](https://github.com/arvindang/illustrative/blob/main/models/novel.py): persistence model · Persisted users, novels, progress, errors, and cost tracking records. · `models/novel.py`
- [Bucket Storage](https://github.com/arvindang/illustrative/blob/main/storage/bucket.py): artifact storage · Abstraction for moving artifacts between local and cloud/object storage. · `storage/bucket.py`
- **Gemini / Vertex AI**: external model service · External long-context and image generation service used by the agents.

## Connections

- Streamlit App → UI Pipeline: launches · The Streamlit app enters the shared pipeline through the UI orchestration layer.
- Prod Run → Script Agent: starts batch flow · The production entrypoint drives the end-to-end batch processing path.
- Auth Page → UI Pipeline: gates access · Authentication must succeed before jobs can be launched or inspected.
- Dashboard → Domain Models: reads status · Dashboard surfaces persisted jobs, progress, and errors.
- Generate Page → UI Pipeline: submits job · Generation requests hand off to the pipeline coordinator.
- UI Pipeline → Pre-Validators: prechecks · Pipeline validates inputs and structure before heavy generation work.
- Pre-Validators → Script Agent: allows · Validated inputs proceed into the scripting stage.
- Script Agent → Beat Analysis: runs · The scripting orchestrator executes the narrative analysis passes.
- Script Agent → Adaptation Filter: runs · Source material is filtered for adaptation suitability.
- Script Agent → Character Dive: runs · Characters are expanded for continuity and visual consistency.
- Script Agent → Scriptwriter: produces · Final structured script and story assets are emitted here.
- Script Agent → Reference Agent: hands off assets · Script-derived structure informs reference generation.
- Reference Agent → Gemini / Vertex AI: model calls · Reference creation depends on external Gemini / Vertex AI services.
- Panel Agent → Gemini / Vertex AI: image calls · Panel generation uses the external model service for visual outputs.
- Reference Agent → Bucket Storage: stores refs · Generated reference artifacts are persisted for reuse.
- Panel Agent → Bucket Storage: stores panels · Page and panel images are written to artifact storage.
- Script Agent → Domain Models: tracks progress · Long-running jobs persist progress and operational state.
- Panel Agent → Layout Agent: feeds pages · Generated panels move into page layout planning.
- Layout Agent → Compositor: defines layout · Layout decisions drive final page assembly.
- Compositor → Post-Validators: checks output · Composed pages are validated for consistency and formatting.
- Post-Validators → Exporter: approves export · Validated output proceeds to deliverable packaging.
- Compositor → Exporter: exports pages · Final page images are packaged for reader-facing formats.
- Exporter → Bucket Storage: publishes artifacts · Exports are stored as final deliverables and working assets.
- Exporter → Domain Models: records cost · Operational metadata such as cost and completion state is persisted.