How l2-regulasyon/teknofest2022 works
Main Components
• Data & Resources:
• Data Directory: Contains input data (legal texts) and resources (e.g., image for the UI/logo).
• Notebooks: Contains multiple Jupyter notebooks which are used for exploratory data analysis, model pipeline testing, text extraction, and specification documentation.
• Model Artifacts:
• Models Directory: Stores model artifacts (e.g., a binary model file “model.cbm” and a TF-IDF vectorizer “tfidf.pickle”). These are produced by the training process and are consumed during inference.
• Core Application Logic:
• Source Code (src folder): Contains modules for constants, utility functions, machine learning model implementation, and evaluation utilities.
• run.py and run_debug.py: Act as main execution entry points for production runs and debugging/training sessions respectively.
• scorer.py: Enables standalone evaluation of predictions against ground truth.
• Environment / Dependencies:
• environment.yml, req.in, and req.txt: Define and manage the project’s dependencies and environment setup (using conda and pip).
Establish Relationships and Interactions
• Data Flow:
• Input data (from the “data” directory) is utilized by the execution scripts (run.py/run_debug.py) which call modules in the “src” directory.
• The utilities in src (like utils.py and eval_utils.py) perform preprocessing, feature extraction (including TF-IDF handling using the tfidf.pickle), and model inference by leveraging the model artifact (model.cbm).
• Interaction between Components:
• Execution Scripts (run.py, run_debug.py) → Call functions from src/model.py and src/utils.py to process input data and load necessary artifacts.
• Model Pipeline: Within these modules, constants in src/constants.py and utility functions orchestrate the transformation pipeline (e.g., from raw data to standardized text processing) feeding to the model.
• Evaluation Component: scorer.py and functions in src/eval_utils.py compute scores and statistics compared to the ground truth, with experiments and results documented in the provided Notebooks.
• Development vs. Production:
• Notebooks provide an experimental and iterative interface whereas run.py and run_debug.py offer command-line driven pipeline execution.
• The scorer.py is used as a separate evaluation module that can be invoked once predictions have been generated.
Highlighting Architectural Patterns and Design Principles
• Modular Structure:
• Distinct separation between data handling, model logic (prediction/inference/training), and evaluation.
• The use of separate scripts and notebooks supports the iterative development/test cycle.
• Environment Consistency:
• Environment management via environment.yml ensures reproducibility across development and production.
• Pipeline Design:
• The system is designed as a pipeline where each step (data ingestion, preprocessing, model inference, evaluation) is clearly delineated.
• The shared use of utility modules (e.g., constants.py, utils.py) promotes code reusability.
Relevant Technologies, Frameworks, and Libraries
• Python and Jupyter Notebooks for experimentation and interactive development.
• Conda for environment management.
• Possibly machine learning libraries (e.g., scikit-learn for TF-IDF or lightgbm/xgboost if model.cbm is built using these) – could be inferred from the file naming and model pipeline notebooks.
• Pip tools (pip-compile) for dependency management.