# davidlinjiahao/frank_analyzer architecture

> Architecture diagram of the GitHub repository davidlinjiahao/frank_analyzer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/davidlinjiahao/frank_analyzer
- Repository: https://github.com/davidlinjiahao/frank_analyzer

## Overview

##### Project Type and Purpose
- The project is a specialized analytic tool that uses multiple LLMs to process tweets, calculate a consensus prediction, and output structured results in CSV format.
- The README clearly indicates that the tool:
- Processes tweets using various models (e.g., Claude 3, GPT-4, Gemini Pro, etc.)
- Calculates a consensus prediction via majority voting
- Outputs and tracks progress in real time
- Utilizes error handling and asynchronous processing to manage API calls and retries

##### File Structure Analysis
- Top-Level Files and Directories:
- Python scripts (e.g., frank_classifier.py, frank_context.py, frank_evaluator.py, frank_extractor.py) which form the core modules for different processing phases.
- A “fixes” directory that contains Python scripts and text files to handle error corrections or data fixes.
- Several CSV files for storing results (e.g., results.csv, classified_results.csv, evaluated_results.csv) which indicate different stages in processing.
- A test directory with tests for components (test_grok.py, test_perplexity.py, verify_pattern.py) suggesting unit tests for key functionalities.
- Node-related files (package.json, package-lock.json) and agent.js, which might indicate some ancillary automated tasks or integration tests via JavaScript.
- Architectural Patterns:
- The modular division (extractor, classifier, evaluator, context) hints at a pipeline architecture where data flows sequentially through distinct processing stages.
- The “fixes” directory demonstrates an extension point where error correction is handled separately from the core processing.

##### Critical Components
a. Input and Data Acquisition:
- Tweets are the primary input (likely coming from “frank_tweets.json” and possibly integrated with twitter-scraper-finetune).
- Environment setup is crucial (via .env for API keys).

b. Core Processing Pipeline:
- Frank Extractor: Module to ingest tweets.
- Frank Classifier: Module to send tweets to multiple LLMs (Claude 3, GPT-4, etc.), get predictions, and capture reasoning.
- Frank Evaluator: Module for calculating consensus predictions, possibly using majority voting.
- Frank Context: Module to manage tweet contexts, perhaps to retain additional metadata.

c. Error Handling and Fixes:
- “fixes” folder components that apply corrective scripts on processed data (e.g., fix error contexts, reprocess tweets, fix token issues).

d. Data Output:
- CSV files as final output.
- Real-time progress tracking (shown on the terminal perhaps).

e. Supporting Components:
- Asynchronous processing mechanics – ensure to denote if and where the asynchronous operations occur.
- External API integrations for the multiple LLMs:
- OpenAI (GPT-4), Anthropic (Claude 3), Google (for DeepSeek), Perplexity API, Grok API.
- Testing modules ensuring the reliability of individual processing steps.

##### Final Recommendations
- Distinguish between internal modules (Python scripts like frank_classifier.py, frank_context.py, etc.) and external dependencies (LLM APIs using keys in .env).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input Components
    A("Tweet Data Source"):::input
    B("Environment Setup for API Keys"):::input

    %% Core Processing Modules
    C("Tweet Extractor"):::module
    D("Tweet Classifier"):::module

    %% LLM API Integrations Subgraph
    subgraph "LLM APIs"
        E("Claude 3 API"):::external
        F("GPT-4 API"):::external
        G("Gemini Pro API"):::external
        H("DeepSeek API"):::external
        I("Perplexity API"):::external
        J("Grok API"):::external
    end

    K("Consensus Evaluator"):::module
    L("Context Management"):::module

    %% Error Handling Module
    M("Error Handling & Corrections"):::error

    %% Results Output Subgraph (CSV Files)
    subgraph "CSV Outputs"
        N1("results.csv"):::output
        N2("classified_results.csv"):::output
        N3("evaluated_results.csv"):::output
        N4("extracted_results.csv"):::output
    end

    %% Supporting Components
    O("Testing Components"):::external
    P("Automation/Integration Scripts"):::external

    %% Data Flow
    A -->|"provides tweets"| C
    B -->|"provides API keys"| D
    C -->|"extracted tweets"| D
    D -->|"queries"| E
    D -->|"queries"| F
    D -->|"queries"| G
    D -->|"queries"| H
    D -->|"queries"| I
    D -->|"queries"| J
    E -->|"returns result"| D
    F -->|"returns result"| D
    G -->|"returns result"| D
    H -->|"returns result"| D
    I -->|"returns result"| D
    J -->|"returns result"| D
    D -->|"sends output"| K
    D -->|"sends metadata"| L
    K -->|"final predictions"| N1
    K -->|"final predictions"| N2
    L -->|"context data"| N3
    L -->|"context data"| N4
    D -- "error detected" --> M
    M -- "corrected data" --> C

    %% Supporting Nodes (not part of main pipeline)
    O --- C
    O --- D
    P --- D

    %% Click Events
    click A "https://github.com/davidlinjiahao/frank_analyzer/blob/main/frank_tweets.json"
    click B "https://github.com/davidlinjiahao/frank_analyzer/blob/main/.env.template"
    click C "https://github.com/davidlinjiahao/frank_analyzer/blob/main/frank_extractor.py"
    click D "https://github.com/davidlinjiahao/frank_analyzer/blob/main/frank_classifier.py"
    click K "https://github.com/davidlinjiahao/frank_analyzer/blob/main/frank_evaluator.py"
    click L "https://github.com/davidlinjiahao/frank_analyzer/blob/main/frank_context.py"
    click M "https://github.com/davidlinjiahao/frank_analyzer/tree/main/fixes/"
    click N1 "https://github.com/davidlinjiahao/frank_analyzer/blob/main/results.csv"
    click N2 "https://github.com/davidlinjiahao/frank_analyzer/blob/main/classified_results.csv"
    click N3 "https://github.com/davidlinjiahao/frank_analyzer/blob/main/evaluated_results.csv"
    click N4 "https://github.com/davidlinjiahao/frank_analyzer/blob/main/extracted_results.csv"
    click O "https://github.com/davidlinjiahao/frank_analyzer/tree/main/test/"
    click P "https://github.com/davidlinjiahao/frank_analyzer/blob/main/agent.js"

    %% Styles
    classDef input fill:#fceabb,stroke:#e08d3c,stroke-width:2px;
    classDef module fill:#c1e1ec,stroke:#2a9df4,stroke-width:2px;
    classDef external fill:#d0f0c0,stroke:#4caf50,stroke-width:2px;
    classDef error fill:#f8d7da,stroke:#f5c6cb,stroke-width:2px;
    classDef output fill:#d6e9c6,stroke:#3c763d,stroke-width:2px;
```
