How nomomon/text-2-emoji works
Project Type and Purpose
• The project is an NLP-based machine learning application for emoji prediction on tweets. It uses a machine learning pipeline that involves data cleaning, embedding generation (using word2vec among other methods), and model training (including grid search and using neural network classifiers).
• The README indicates three core processing steps (preprocess, embedding generation, model training) and demonstrates that the final trained model and embeddings are then served via an API (powered likely by FastAPI and uvicorn) with a front-end demo using Streamlit.
Analyzing the File Structure
• Top-Level Files: Files like preprocess_dataset.py, create_embedding.py, and create_model.py form the sequential pipeline stages.
• Directories:
• data: Contains raw data (data/raw), intermediate cleaned files (data/bronze), and refined processed data (data/silver). This shows a multi-stage data processing pipeline.
• text2emoji: Acts as the package holding modularized code for data processing, feature extraction (text processing and embedding generation), and model definitions (multiple strategies, grid search, NN classifier, transformer-based approaches). This is an abstraction layer for core functionalities.
• out: Stores the output of different experiments (mobert, unfrozen_bert, word2vec) including best models, grid search results, losses, etc.
• tests: Contains tests for main functionalities in data processing, feature extraction, and model building.
• Notebooks: For exploratory data analysis and preliminary experiments.
• Configuration Files: Files like Pipfile, .flake8, .gitignore, and pre-commit settings indicate adherence to coding quality and dependency management.
Architectural Components Identified
• Data Storage & Pre-processing:
• Data Repository: Organized into raw, bronze (initially cleaned), and silver (further processed) stages.
• Preprocessing Module: Handled by preprocess_dataset.py and supplemented by text2emoji/data modules (for text processing and random embeddings).
• Feature Extraction / Embedding Generation:
• create_embedding.py creates sentence embeddings, supported by code in text2emoji/features (e.g., embedding_processing.py).
• Output typically stored as numpy arrays or CSVs.
• Model Training:
• create_model.py trains the emoji classifier. The process uses grid search and leverages code in text2emoji/models (with modules for grid search of transformers, NN classifiers, bootstrap evaluations, etc.).
• Output is stored in the “out” directories.
• API Service:
• api.py provides a RESTful API endpoint to serve predictions (loading the trained model and embeddings into memory).
• Uvicorn is used to launch the API, with built-in Swagger documentation.
• Front-end:
• streamlit_demo.py acts as a front-end demonstration for interacting with the API.
• Utility and Testing:
• Tests (in tests/) ensure that each module behaves as expected.
• Notebooks for data exploration and visualization provide insight into the data and performance trends.
Relationships and Data Flow
• Sequential Pipeline: Data flows from raw (data source) → preprocessing (clean and store in bronze CSVs) → embedding generation (create numpy arrays) → model training (build and validate neural network classifier).
• The API then acts as the serving layer, loading pre-trained models and processed embeddings to handle prediction calls.
• The front-end (Streamlit) communicates with the API to obtain results.
• Additional support through exploratory notebooks helps verify and understand data distributions and transformation outcomes.
Architectural Patterns and Technologies
• Separation of Concerns: Clearly defined layers for data pre-processing, feature extraction, and model training with distinct scripts and a modular package (text2emoji).
• Pipeline Architecture: A step-wise data processing pipeline (preprocess, embed, model training) culminating in deployment.
• API & Front-End Separation: The back-end (API) serves the prediction model separately from the user interface (Streamlit demo).
• Technologies: Python, Pipenv for dependency management, FastAPI/Uvicorn for the API, and Streamlit for the front-end. Additionally, machine learning frameworks (likely PyTorch/TensorFlow) underly the model training although not explicitly mentioned, alongside utilities (numpy, possibly scikit-learn for grid search).