How jacoo-zhao/ai-studio-clearml works
Main Components
• Data Source: Represented by the work_dataset directory, containing the training and testing CSV files. This is where your raw data resides.
• Dataset Artifact Upload: Script (step1_dataset_artifact.py) that uploads the dataset to ClearML as an artifact. This component “pushes” data into the pipeline.
• Data Preprocessing: Script (step2_data_preprocessing.py) that processes the raw data and prepares it for training. The preprocessed data may in turn be used as an artifact or passed directly.
• Model Training: Script (step3_train_model.py) that uses the preprocessed data to train machine learning models. The trained models are saved in model_artifacts.
• Pipeline Orchestration: The main.py and pipeline_from_tasks.py files serve as the entry point to execute the full pipeline by linking the above tasks. Additionally, demo_functions.py and demo_using_artifacts_example.py demonstrate utility functions and artifact loading.
• ClearML Infrastructure: A central element that connects these tasks. This includes the ClearML Server (or hosted service) for tracking experiments, managing task templates, and observing the execution pipeline, as well as the ClearML Agent that runs tasks from a specified queue.
Relationships and Interactions
• The raw data from work_dataset feeds into the dataset upload component (step1_dataset_artifact.py), which registers the data as an artifact on ClearML.
• The data preprocessing component (step2_data_preprocessing.py) then takes over, consuming the uploaded data artifact, processing it, and possibly creating another artifact or intermediate dataset.
• The training component (step3_train_model.py) takes in the preprocessed data and trains a model, saving the final model artifacts (both .pkl and .pth) in the model_artifacts directory.
• The main pipeline orchestrator (main.py) chains these steps together by instantiating a ClearML Pipeline, scheduling tasks with PipelineController.add_step(...), and finally executing the pipeline—often via a queued ClearML Agent.
• If using the local function-based approach (as noted in the README, though still TBD), the design would simply wrap the underlying logic in functions rather than separate tasks. The overall interactions remain similar.
Architectural Patterns and Design Principles
• Modular Pipeline Stages: Each step (dataset upload, preprocessing, training) is separated into its own script for reusability and ease of maintenance.
• Artifact Management: The project emphasizes saving stage outputs (datasets and models) as ClearML artifacts.
• Task-based Orchestration: Utilizing ClearML’s task templates to define and chain tasks represents a microservices-like approach where each stage is independently deployable and monitorable.
• Extensibility: The inclusion of demo scripts and notebook examples (e.g., AI-Studio-Agent.ipynb, AI-Studio-ClearML.ipynb) shows that there are multiple entry points and demonstration paths.