How levysoft/chatgpt-token-cost-analysis works
Project Type & Purpose
• This is a data analysis tool focused on processing ChatGPT conversation exports (in JSON format) to compute token counts and associated costs.
• Its bifurcated design supports both a Python script (for detailed processing, cost grouping, and CSV output) and a web application that provides an offline, local interface for the same analysis.
• The README confirms that the goal is to offer flexible, privacy-enhanced token cost analysis for ChatGPT exports.
Analyzing the File Structure
• Top-Level Files: Files like README.md, requirements.txt, LICENSE, and CHANGELOG.md indicate a well-documented open-source project.
• Python Component: The file “chatgpt_token_cost_analysis.py” drives the Python CLI version. It handles JSON file loading, message extraction, token counting with the “tiktoken” library and custom encoding (with help from “cl100k_base.js” for the web version), cost computations based on predefined global variables, and CSV output.
• Web Component: The HTML file (chatgpt_token_cost_analysis.html) combines HTML and JavaScript to provide a GUI for local analysis. It references “cl100k_base.js” as a fallback tokenization engine, ensuring that users can run the analysis offline.
• Assets Folder: Contains images (screenshots) that illustrate how the web tool works. This helps in visually understanding the user interface portion.
• Configuration/Requirements: The “requirements.txt” clearly defines the necessary Python dependencies.
Main Components
a. Input:
– The ChatGPT “conversations.json” file that the user supplies.
b. Python Script Block:
– JSON Loader: Reads and validates the JSON export.
– Data Transformer: Functions for extracting messages and converting the data into a structured Pandas DataFrame.
– Token Calculator: Uses “tiktoken” (and related logic for tokenizing with a fallback) to count tokens.
– Cost Calculator: Applies the cost formulas for input and output tokens, grouping by month.
– Output Generator: Saves the processed results into “costs_per_month.csv” for later review.
c. HTML + JavaScript Application Block:
– UI Component: The single-page HTML interface that allows file upload, model selection (drop-down), and result display.
– Tokenizer Loader Component: Attempts first to load the GPT tokenizer from the remote “https://unpkg.com/gpt-tokenizer”, falls back to “cl100k_base.js” locally, and if necessary, resorts to an approximate word-based count.
– Processing Engine: Runs JavaScript functions to replicate (or approximate) the Python processing steps for token counting and cost computation.
d. External Dependencies & Libraries:
– Python Libraries: pandas, json, datetime, tiktoken.
– JavaScript Libraries: gpt-tokenizer (when available) and any local fallback.
– External URLs: Reference to remote sources like “https://unpkg.com/gpt-tokenizer”.
Relationships and Data Flow
• Start at the user’s input file (conversations.json).
• For the Python path, show the flow:
Conversations.json → JSON Loader → Message Extraction → Token Calculation → Cost Computation → CSV Output.
Uploaded Conversations.json → File Input Component → Tokenizer Loader (remote/local/fallback) → JavaScript Processing Engine → Dynamic Result Display.
Architectural Patterns and Design Principles
• Layered/Pipeline Architecture: Both the Python and web paths are essentially pipelines that transform raw JSON data through several computation stages.
• Separation of Concerns: The system is split between a backend script (Python) and a frontend interface (HTML/JavaScript), even though they share common functionality.
• Offline/Privacy-Enhanced Design: Highlight the importance of local processing (especially in the web version) to maintain user privacy.