# larryliebovitch/ai_ac4a_fall2023 architecture

> Architecture diagram of the GitHub repository larryliebovitch/ai_ac4a_fall2023, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/larryliebovitch/ai_ac4a_fall2023
- Repository: https://github.com/larryliebovitch/ai_ac4a_fall2023

## Overview

##### Project Type and Purpose
- This project is centered around text processing and machine learning, with a focus on generating embeddings for articles and classifying them. It utilizes a ChromaDB vector database and multiple classification approaches (similarity search, logistic regression, and neural networks). The system is essentially a research or prototype AI project that explores different ways of classifying text using embeddings.

##### File Structure and Identify Main Components
- Embeddings Module (Embeddings.py): Reads CSV files containing articles, converts the articles into vector representations, and stores them in a ChromaDB vector database.
- Classification Modules:
– PIRvsNIR Module (PIRvsNIR.py): Compares articles against pre-defined PIR and NIR embeddings to perform classification.
– Queries Module (Queries.py): Retrieves the most similar article from the vector database and uses its metadata for classification.
– Regression Model (RegressionModel.py): Implements a Logistic Regression classifier using the embeddings.
– Neural Networks (Neural_Networks.ipynb): Trains neural networks as an alternative classifier.
- Evaluation/Testing Module (QueryBigData.py): Evaluates the classification accuracy using datasets of different sizes.
- Utility Module (Tokens.py): Provides token counting to analyze the text files.

##### Relationships and Data Flow
- Data Input: Articles are fed from CSV files. Before classification, they are processed by tokens (Tokens.py) and then converted into embeddings by Embeddings.py.
- Embedding Storage: The generated embeddings are stored in a ChromaDB vector database. This is the central repository from which similar items are later queried.
- Classification Paths:
– The PIRvsNIR module takes the embeddings and compares them against two baseline embeddings (PIR vs. NIR).
– The Queries module interacts with the vector database to perform nearest neighbor lookups.
– The RegressionModel and Neural Network classifier modules use the embeddings for training and prediction.
- Evaluation: The QueryBigData.py script tests the performance and accuracy of one of the classification strategies across various sample sizes.

##### Architectural Patterns and Key Technology Involvement
- Pipeline Pattern: Notice the clear pipeline – raw articles are tokenized, converted into embeddings, stored in a vector database, and then fed into various classification algorithms. This layered processing can be separated into distinct components.
- Data Flow and Micro-components: Each Python file/module represents a micro-component in the overall system. Their interaction can be viewed as a series of connected micro-services or discrete steps in a data processing pipeline.
- Third Party Dependency: ChromaDB is central in storing and querying embeddings.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Preprocessing Layer
    subgraph "Preprocessing"
        A["CSV Files"]:::dataSource
        B(("Tokenization Utility")):::processing
        C(("Embedding Generator")):::processing
    end

    %% Storage Layer
    subgraph "Storage"
        D["ChromaDB"]:::storage
    end

    %% Classification Layer
    subgraph "Classification"
        E["PIR vs. NIR Classifier"]:::classification
        F["Nearest Neighbor Query Classifier"]:::classification
        G["Logistic Regression Classifier"]:::classification
        H["Neural Network Classifier"]:::classification
    end

    %% Evaluation Layer
    subgraph "Evaluation"
        I(("Evaluation Module")):::evaluation
    end

    %% Data Flow Connections
    A -->|"RawText"| B
    B -->|"TokenizedText"| C
    C -->|"Embeddings"| D
    D -->|"EmbeddingsQuery"| E
    D -->|"NearestNeighborLookup"| F
    D -->|"EmbeddingsData"| G
    D -->|"EmbeddingsData"| H
    F -->|"ClassificationOutput"| I

    %% Click Events for Component Mapping
    click B "https://github.com/larryliebovitch/ai_ac4a_fall2023/blob/main/Tokens.py"
    click C "https://github.com/larryliebovitch/ai_ac4a_fall2023/blob/main/Embeddings.py"
    click E "https://github.com/larryliebovitch/ai_ac4a_fall2023/blob/main/PIRvsNIR.py"
    click F "https://github.com/larryliebovitch/ai_ac4a_fall2023/blob/main/Queries.py"
    click G "https://github.com/larryliebovitch/ai_ac4a_fall2023/blob/main/RegressionModel.py"
    click H "https://github.com/larryliebovitch/ai_ac4a_fall2023/blob/main/Neural_Networks.ipynb"
    click I "https://github.com/larryliebovitch/ai_ac4a_fall2023/blob/main/QueryBigData.py"

    %% Style Definitions
    classDef dataSource fill:#ffcc00,stroke:#333,stroke-width:2px;
    classDef processing fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef storage fill:#90ee90,stroke:#333,stroke-width:2px;
    classDef classification fill:#ff9999,stroke:#333,stroke-width:2px;
    classDef evaluation fill:#d3d3d3,stroke:#333,stroke-width:2px;
```
