How larryliebovitch/ai_ac4a_fall2023 works
Project Type and Purpose
• This project is centered around text processing and machine learning, with a focus on generating embeddings for articles and classifying them. It utilizes a ChromaDB vector database and multiple classification approaches (similarity search, logistic regression, and neural networks). The system is essentially a research or prototype AI project that explores different ways of classifying text using embeddings.
File Structure and Identify Main Components
• Embeddings Module (Embeddings.py): Reads CSV files containing articles, converts the articles into vector representations, and stores them in a ChromaDB vector database.
• Classification Modules:
– PIRvsNIR Module (PIRvsNIR.py): Compares articles against pre-defined PIR and NIR embeddings to perform classification.
– Queries Module (Queries.py): Retrieves the most similar article from the vector database and uses its metadata for classification.
– Regression Model (RegressionModel.py): Implements a Logistic Regression classifier using the embeddings.
– Neural Networks (Neural_Networks.ipynb): Trains neural networks as an alternative classifier.
• Evaluation/Testing Module (QueryBigData.py): Evaluates the classification accuracy using datasets of different sizes.
• Utility Module (Tokens.py): Provides token counting to analyze the text files.
Relationships and Data Flow
• Data Input: Articles are fed from CSV files. Before classification, they are processed by tokens (Tokens.py) and then converted into embeddings by Embeddings.py.
• Embedding Storage: The generated embeddings are stored in a ChromaDB vector database. This is the central repository from which similar items are later queried.
• Classification Paths:
– The PIRvsNIR module takes the embeddings and compares them against two baseline embeddings (PIR vs. NIR).
– The Queries module interacts with the vector database to perform nearest neighbor lookups.
– The RegressionModel and Neural Network classifier modules use the embeddings for training and prediction.
• Evaluation: The QueryBigData.py script tests the performance and accuracy of one of the classification strategies across various sample sizes.
Architectural Patterns and Key Technology Involvement
• Pipeline Pattern: Notice the clear pipeline – raw articles are tokenized, converted into embeddings, stored in a vector database, and then fed into various classification algorithms. This layered processing can be separated into distinct components.
• Data Flow and Micro-components: Each Python file/module represents a micro-component in the overall system. Their interaction can be viewed as a series of connected micro-services or discrete steps in a data processing pipeline.
• Third Party Dependency: ChromaDB is central in storing and querying embeddings.