How quocnguyenx43/virecax works
• Offline Learning (Baseline) Component:
• Represented by the "baseline" folder.
• Contains scripts for data preprocessing, training (e.g., cnn_train_eval.sh, lstm_train_eval.sh, simple_train_eval.sh), evaluation (run_evaluation_cls_task.py, run_evaluation_generation_task.py), and prediction (run_prediction_full.py).
• Includes key source files like functions.py, models.py, my_datasets.py, my_import.py, and several utility scripts.
• Online Learning (System) Component:
• Represented by the "system" directory.
• Contains online learning strategies where you find components for client handling (under client_handler), online training (in processor such as online_training_1.py, online_training_2.py, online_training_3.py), and streaming experiments.
• Includes configuration files (config.yaml) and scripts to start, manage, and shutdown services (server_start.py, manager.py, shutdown.py, zoo_start.py).
• Additionally, there is evidence of streaming or messaging integrations (e.g., clear_data_kafka.py) to handle real-time data.
• Experimental & Evaluation Component:
• Under system/experiments, you have several subcomponents: scripts and notebooks for evaluating latency, pushing data to MongoDB, generating plots (with mermaid_diagram.html), and result analysis (results folder with multiple strategies and CSV/XLSX files).
• This area demonstrates how various online learning strategies and model performance evaluations flow into test and results generation.
• Additional Components:
• Model configurations: “system/models” which includes JSON files for different strategies (e.g., task_1_cnn_distil.json, task_2_lstm_visobert.json, etc.).
• Extension component: Found in system/extension, which contains files for a browser extension (manifest.json, background.js, popup.html, etc.) indicating potential user interface integrations or data collection front-ends.
• Data and logging artifacts: Various logs and results files spread across both the baseline and system directories.
Relationships and Data Flow
• Offline to Online Interaction:
• The baseline offline component is used for initial model training and experimentation. The output (trained models, evaluation logs, and JSON configuration files) feeds into the online system.
• Online System Pipeline:
• External data sources or user requests enter via client_handler (in the system/client_handler folder) or the browser extension.
• Incoming data is preprocessed (using preprocess.py and utils.py) before being handled by the prediction and online training modules (in the processor folder).
• A configuration file (config.yaml) governs the operational parameters for these services.
• The online training modules continuously update or adjust the deployed models in real time via streaming (as suggested by the presence of Kafka-related scripts) and logging systems.
• Evaluation and Experimentation:
• Experiment results are aggregated in system/experiments, where further analysis is done (with Jupyter notebooks and result CSVs/XLSX files).
• Data flows from online predictions into logging and result generation systems, which then feed back into model improvement cycles.
Architectural Patterns and Technical Stack
• Separation of Concerns:
• Clearly distinct offline training (baseline) and online learning (system) pipelines.
• Streaming & Real-Time Processing:
• Use of client_handler and Kafka-related scripts for handling data streams.
• Modular Design:
• Directories like "processor", "client_handler", "models", and "experiments" indicate a modular approach where each component can be developed and updated independently.
• Experimentation & Model Evaluation:
• An extensive structure in the experiments directory shows emphasis on testing different strategies, tracking latency, and logging performance.