# levysoft/chatgpt-token-cost-analysis architecture

> Architecture diagram of the GitHub repository levysoft/chatgpt-token-cost-analysis, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/levysoft/chatgpt-token-cost-analysis
- Repository: https://github.com/levysoft/chatgpt-token-cost-analysis

## Overview

##### Project Type & Purpose
- This is a data analysis tool focused on processing ChatGPT conversation exports (in JSON format) to compute token counts and associated costs.
- Its bifurcated design supports both a Python script (for detailed processing, cost grouping, and CSV output) and a web application that provides an offline, local interface for the same analysis.
- The README confirms that the goal is to offer flexible, privacy-enhanced token cost analysis for ChatGPT exports.

##### Analyzing the File Structure
- Top-Level Files: Files like README.md, requirements.txt, LICENSE, and CHANGELOG.md indicate a well-documented open-source project.
- Python Component: The file “chatgpt_token_cost_analysis.py” drives the Python CLI version. It handles JSON file loading, message extraction, token counting with the “tiktoken” library and custom encoding (with help from “cl100k_base.js” for the web version), cost computations based on predefined global variables, and CSV output.
- Web Component: The HTML file (chatgpt_token_cost_analysis.html) combines HTML and JavaScript to provide a GUI for local analysis. It references “cl100k_base.js” as a fallback tokenization engine, ensuring that users can run the analysis offline.
- Assets Folder: Contains images (screenshots) that illustrate how the web tool works. This helps in visually understanding the user interface portion.
- Configuration/Requirements: The “requirements.txt” clearly defines the necessary Python dependencies.

##### Main Components
a. Input:
– The ChatGPT “conversations.json” file that the user supplies.
b. Python Script Block:
– JSON Loader: Reads and validates the JSON export.
– Data Transformer: Functions for extracting messages and converting the data into a structured Pandas DataFrame.
– Token Calculator: Uses “tiktoken” (and related logic for tokenizing with a fallback) to count tokens.
– Cost Calculator: Applies the cost formulas for input and output tokens, grouping by month.
– Output Generator: Saves the processed results into “costs_per_month.csv” for later review.
c. HTML + JavaScript Application Block:
– UI Component: The single-page HTML interface that allows file upload, model selection (drop-down), and result display.
– Tokenizer Loader Component: Attempts first to load the GPT tokenizer from the remote “https://unpkg.com/gpt-tokenizer”, falls back to “cl100k_base.js” locally, and if necessary, resorts to an approximate word-based count.
– Processing Engine: Runs JavaScript functions to replicate (or approximate) the Python processing steps for token counting and cost computation.
d. External Dependencies & Libraries:
– Python Libraries: pandas, json, datetime, tiktoken.
– JavaScript Libraries: gpt-tokenizer (when available) and any local fallback.
– External URLs: Reference to remote sources like “https://unpkg.com/gpt-tokenizer”.

##### Relationships and Data Flow
- Start at the user’s input file (conversations.json).
- For the Python path, show the flow:
Conversations.json  →  JSON Loader → Message Extraction → Token Calculation → Cost Computation → CSV Output.
Uploaded Conversations.json → File Input Component → Tokenizer Loader (remote/local/fallback) → JavaScript Processing Engine → Dynamic Result Display.

##### Architectural Patterns and Design Principles
- Layered/Pipeline Architecture: Both the Python and web paths are essentially pipelines that transform raw JSON data through several computation stages.
- Separation of Concerns: The system is split between a backend script (Python) and a frontend interface (HTML/JavaScript), even though they share common functionality.
- Offline/Privacy-Enhanced Design: Highlight the importance of local processing (especially in the web version) to maintain user privacy.

## Diagram (Mermaid)

```mermaid
flowchart TD
    INPUT["Input: conversations.json"]:::input

    subgraph "Python Analysis Pipeline"
        PCLI["Python CLI Script"]:::python
        PJSON["JSON Loader"]:::python
        PMEX["Message Extraction"]:::python
        PTOK["Token Calculator"]:::python
        PCOST["Cost Calculator"]:::python
        PCSV["CSV Output (costs_per_month.csv)"]:::python
    end

    subgraph "Web Analysis Interface"
        WHMT["HTML Interface"]:::web
        WINP["File Input Component"]:::web
        WTOK["Tokenizer Loader (remote/local fallback)"]:::web
        WPROC["JS Processing Engine"]:::web
        WUI["Dynamic Result Display"]:::web
    end

    subgraph "Project Metadata & Assets"
        CONFIG["Config & Dependencies"]:::config
        DOC1["README.md"]:::docs
        DOC2["CHANGELOG.md"]:::docs
        DOC3["LICENSE"]:::docs
        VAS["Visual Assets (UI Screenshots)"]:::assets
    end

    INPUT --> PCLI
    PCLI --> PJSON
    PJSON --> PMEX
    PMEX --> PTOK
    PTOK --> PCOST
    PCOST --> PCSV

    INPUT --> WHMT
    WHMT --> WINP
    WINP --> WTOK
    WTOK --> WPROC
    WPROC --> WUI

    %% Click Events
    click PCLI "https://github.com/levysoft/chatgpt-token-cost-analysis/blob/main/chatgpt_token_cost_analysis.py"
    click WHMT "https://github.com/levysoft/chatgpt-token-cost-analysis/blob/main/chatgpt_token_cost_analysis.html"
    click WTOK "https://github.com/levysoft/chatgpt-token-cost-analysis/blob/main/cl100k_base.js"
    click CONFIG "https://github.com/levysoft/chatgpt-token-cost-analysis/blob/main/requirements.txt"
    click DOC1 "https://github.com/levysoft/chatgpt-token-cost-analysis/blob/main/README.md"
    click DOC2 "https://github.com/levysoft/chatgpt-token-cost-analysis/blob/main/CHANGELOG.md"
    click DOC3 "https://github.com/levysoft/chatgpt-token-cost-analysis/tree/main/LICENSE"
    click VAS "https://github.com/levysoft/chatgpt-token-cost-analysis/tree/main/assets"

    %% Style Definitions
    classDef python fill:#AED6F1,stroke:#1F618D,stroke-width:2px,color:#1B2631;
    classDef web fill:#F9E79F,stroke:#B9770E,stroke-width:2px,color:#7D6608;
    classDef config fill:#FAD7A0,stroke:#D35400,stroke-width:2px,color:#7D3C98;
    classDef docs fill:#D5F5E3,stroke:#1E8449,stroke-width:2px,color:#145A32;
    classDef assets fill:#E8DAEF,stroke:#8E44AD,stroke-width:2px,color:#4A235A;
    classDef input fill:#FDEBD0,stroke:#D35400,stroke-width:2px,color:#641E16;
```
