# nomomon/text-2-emoji architecture

> Architecture diagram of the GitHub repository nomomon/text-2-emoji, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nomomon/text-2-emoji
- Repository: https://github.com/nomomon/text-2-emoji

## Overview

##### Project Type and Purpose
- The project is an NLP-based machine learning application for emoji prediction on tweets. It uses a machine learning pipeline that involves data cleaning, embedding generation (using word2vec among other methods), and model training (including grid search and using neural network classifiers).
- The README indicates three core processing steps (preprocess, embedding generation, model training) and demonstrates that the final trained model and embeddings are then served via an API (powered likely by FastAPI and uvicorn) with a front-end demo using Streamlit.

##### Analyzing the File Structure
- Top-Level Files: Files like preprocess_dataset.py, create_embedding.py, and create_model.py form the sequential pipeline stages.
- Directories:
- data: Contains raw data (data/raw), intermediate cleaned files (data/bronze), and refined processed data (data/silver). This shows a multi-stage data processing pipeline.
- text2emoji: Acts as the package holding modularized code for data processing, feature extraction (text processing and embedding generation), and model definitions (multiple strategies, grid search, NN classifier, transformer-based approaches). This is an abstraction layer for core functionalities.
- out: Stores the output of different experiments (mobert, unfrozen_bert, word2vec) including best models, grid search results, losses, etc.
- tests: Contains tests for main functionalities in data processing, feature extraction, and model building.
- Notebooks: For exploratory data analysis and preliminary experiments.
- Configuration Files: Files like Pipfile, .flake8, .gitignore, and pre-commit settings indicate adherence to coding quality and dependency management.

##### Architectural Components Identified
- Data Storage & Pre-processing:
- Data Repository: Organized into raw, bronze (initially cleaned), and silver (further processed) stages.
- Preprocessing Module: Handled by preprocess_dataset.py and supplemented by text2emoji/data modules (for text processing and random embeddings).
- Feature Extraction / Embedding Generation:
- create_embedding.py creates sentence embeddings, supported by code in text2emoji/features (e.g., embedding_processing.py).
- Output typically stored as numpy arrays or CSVs.
- Model Training:
- create_model.py trains the emoji classifier. The process uses grid search and leverages code in text2emoji/models (with modules for grid search of transformers, NN classifiers, bootstrap evaluations, etc.).
- Output is stored in the “out” directories.
- API Service:
- api.py provides a RESTful API endpoint to serve predictions (loading the trained model and embeddings into memory).
- Uvicorn is used to launch the API, with built-in Swagger documentation.
- Front-end:
- streamlit_demo.py acts as a front-end demonstration for interacting with the API.
- Utility and Testing:
- Tests (in tests/) ensure that each module behaves as expected.
- Notebooks for data exploration and visualization provide insight into the data and performance trends.

##### Relationships and Data Flow
- Sequential Pipeline: Data flows from raw (data source) → preprocessing (clean and store in bronze CSVs) → embedding generation (create numpy arrays) → model training (build and validate neural network classifier).
- The API then acts as the serving layer, loading pre-trained models and processed embeddings to handle prediction calls.
- The front-end (Streamlit) communicates with the API to obtain results.
- Additional support through exploratory notebooks helps verify and understand data distributions and transformation outcomes.

##### Architectural Patterns and Technologies
- Separation of Concerns: Clearly defined layers for data pre-processing, feature extraction, and model training with distinct scripts and a modular package (text2emoji).
- Pipeline Architecture: A step-wise data processing pipeline (preprocess, embed, model training) culminating in deployment.
- API & Front-End Separation: The back-end (API) serves the prediction model separately from the user interface (Streamlit demo).
- Technologies: Python, Pipenv for dependency management, FastAPI/Uvicorn for the API, and Streamlit for the front-end. Additionally, machine learning frameworks (likely PyTorch/TensorFlow) underly the model training although not explicitly mentioned, alongside utilities (numpy, possibly scikit-learn for grid search).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Storage & Pre-processing
    subgraph "Data Storage & Pre-processing"
        RAW["Raw Data"]:::data
        PREPROC["Preprocessing (preprocess_dataset.py)"]:::process
        TEXTPROC["Text Processing (text2emoji/data/text_processing.py)"]:::process
        BRONZE["Bronze Data"]:::data
        SILVER["Silver Data"]:::data
        RAW -->|"processes"| PREPROC
        PREPROC -->|"outputs"| BRONZE
        BRONZE -->|"refines to"| SILVER
        PREPROC -->|"utilizes"| TEXTPROC
    end

    %% Feature Extraction / Embedding Generation
    subgraph "Feature Extraction / Embedding Generation"
        EMBED["Embedding Generation (create_embedding.py)"]:::feature
        EMBEDPROC["Embedding Processing (text2emoji/features/embedding_processing.py)"]:::feature
        EMBEDGEN["Embedding Generation Module (text2emoji/data/embedding_generation.py)"]:::feature
        SILVER -->|"supplies data to"| EMBED
        EMBED -->|"processes embeddings via"| EMBEDPROC
        EMBEDPROC -->|"generates features using"| EMBEDGEN
    end

    %% Model Training
    subgraph "Model Training"
        MODEL_SCRIPT["Model Training (create_model.py)"]:::model
        MODELS["ML Modules (text2emoji/models)"]:::model
        EMBEDGEN -->|"feeds features to"| MODEL_SCRIPT
        MODEL_SCRIPT -->|"trains models in"| MODELS
    end

    %% API Service
    subgraph "API Service"
        API["API Service (api.py)"]:::api
        MODELS -->|"deploys model to"| API
    end

    %% Front-End & Utilities
    subgraph "Front-End & Utilities"
        STREAMLIT["Front-End Demo (streamlit_demo.py)"]:::frontend
        API -->|"serves predictions to"| STREAMLIT
        NOTEBOOKS["Notebooks"]:::utility
        TESTS["Tests"]:::utility
        NOTEBOOKS ---|"feedback"| MODEL_SCRIPT
        TESTS ---|"validate"| MODEL_SCRIPT
    end

    %% Styles
    classDef data fill:#FFFACD,stroke:#DAA520,stroke-width:2px,color:black;
    classDef process fill:#ADD8E6,stroke:#1E90FF,stroke-width:2px,color:black;
    classDef feature fill:#90EE90,stroke:#32CD32,stroke-width:2px,color:black;
    classDef model fill:#FFB6C1,stroke:#FF69B4,stroke-width:2px,color:black;
    classDef api fill:#DDA0DD,stroke:#BA55D3,stroke-width:2px,color:black;
    classDef frontend fill:#FFD700,stroke:#FFA500,stroke-width:2px,color:black;
    classDef utility fill:#E6E6FA,stroke:#9370DB,stroke-width:2px,color:black;

    %% Click Events
    click RAW "https://github.com/nomomon/text-2-emoji/tree/main/data/raw"
    click BRONZE "https://github.com/nomomon/text-2-emoji/tree/main/data/bronze"
    click SILVER "https://github.com/nomomon/text-2-emoji/tree/main/data/silver"
    click PREPROC "https://github.com/nomomon/text-2-emoji/blob/main/preprocess_dataset.py"
    click TEXTPROC "https://github.com/nomomon/text-2-emoji/blob/main/text2emoji/data/text_processing.py"
    click EMBED "https://github.com/nomomon/text-2-emoji/blob/main/create_embedding.py"
    click EMBEDPROC "https://github.com/nomomon/text-2-emoji/blob/main/text2emoji/features/embedding_processing.py"
    click EMBEDGEN "https://github.com/nomomon/text-2-emoji/blob/main/text2emoji/data/embedding_generation.py"
    click MODEL_SCRIPT "https://github.com/nomomon/text-2-emoji/blob/main/create_model.py"
    click MODELS "https://github.com/nomomon/text-2-emoji/tree/main/text2emoji/models"
    click API "https://github.com/nomomon/text-2-emoji/blob/main/api.py"
    click STREAMLIT "https://github.com/nomomon/text-2-emoji/blob/main/streamlit_demo.py"
    click TESTS "https://github.com/nomomon/text-2-emoji/tree/main/tests"
    click NOTEBOOKS "https://github.com/nomomon/text-2-emoji/tree/main/notebooks"
```
