How 1666sapple/bangla-ner-pos works
Project Type and Purpose
• This project is a full-stack NLP application. It integrates machine learning model training (in the model/ module) and inference (in the app/ module) to offer NER and POS tagging via a FastAPI server.
• The README clearly describes that the FastAPI application accepts input Bangla text and returns annotated results. It also includes a web front-end component (templates and static files) for user interaction.
Analyzing the File Structure
• The top-level contains:
• The app/ directory: Contains the API endpoints, HTML templates, static assets, and utility scripts which serve as the front-end and the interface for querying the model.
• The model/ directory: Contains code for model configuration, dataset handling, model architecture, training scripts, and utility functions.
• The data/ directory: Contains the dataset used for training and validation.
• The scripts/ directory: Includes scripts for running the application (run_app.sh) and for training the model (train_model.py).
• The structure implies a separation of concerns: one part (app/) handles API/inference and user interactions, and another (model/) handles training and model-related configurations.
a. Main Components:
• Frontend Web UI: Represented by files in app/templates and app/static (HTML, CSS, JavaScript). It allows users to enter Bangla text.
• FastAPI Server: Driven by files like app/main.py and app/api/routes.py. This component manages API requests, calls prediction functions, and returns results.
• Inference Engine: Within app/models/ner_pos_model.py and app/utils/predict.py. This loads the model weights (from app/models-weight/) and executes inference (NER and POS tagging).
• Machine Learning Module: Comprising model/ (config.py, dataset.py, model.py, train.py, utils.py) which handles training, data processing, and configuration settings.
• Data Component: The data/ directory for training datasets.
• Auxiliary Scripts: Located in scripts/ (train_model.py and run_app.sh) to set up training and to start the FastAPI server.
• Storage and Configuration: Model weights stored in app/models-weight/ and configuration parameters in model/config.py.
b. Relationships and Interactions:
• User Interaction flows from the frontend through a web browser to the FastAPI API endpoints.
• API endpoints (from app/api/routes.py) pass input data to utility functions (in app/utils/predict.py) which call the inference model (in app/models/ner_pos_model.py).
• The inference engine uses pre-trained weights stored in app/models-weight/ to perform predictions.
• The training module (model/) is executed separately (triggered by scripts/train_model.py) and, once trained, updates the weights in app/models-weight/.
• The system configuration and dataset handling (from model/config.py and model/dataset.py) are underlying supports for the entire machine learning pipeline.
c. Architectural Patterns/Design Principles:
• Separation of Concerns: API layer separated from training/model logic.
• MVC-like Pattern: Although not strictly MVC, the separation between presentation (templates/static) and business logic (API, models, utils) is evident.
• Modular design: Clear boundaries between application serving (app/) and heavy computation/model training (model/).
d. Relevant Technologies and Frameworks:
• FastAPI and Uvicorn for the web server and API.
• PyTorch (torch) and transformer frameworks (from Hugging Face) for the underlying model architecture.
• Jinja2 for templating (HTML).
• Additional libraries include pandas, scikit-learn, and numpy for data processing.
• Shell scripts to bootstrap training and running of the application.
Elements
• Clear Labels: Label components such as "Web UI," "FastAPI Server," "API Routes Module," "Prediction Utility & Inference Engine," "Model Training Pipeline," "Model Weights Storage," "Dataset," and "Configuration Module."