How joachimz/dsba-platform works
Project Type and Purpose
• The DSBA Platform is a toy MLOps platform meant for educational purposes. It supports various operations like data ingestion, preprocessing, model training, evaluation, prediction, and managing a model registry. It further supports multiple interfaces – a CLI, an API (with Docker support), and interactive notebooks for experimentation.
Key Components
a. Core DSBA Library: The main business logic is under the "src/dsba" directory.
• Components include:
• Data Ingestion (modules for files and databases)
• Preprocessing
• Model Training, Evaluation, Prediction
• Model Registry for handling trained models
b. API Component: Under "src/api", including its Dockerfile which indicates containerized deployment for exposing functionality as a service.
c. CLI Component: Located in "src/cli", exposing commands like model listing and predictions.
d. Notebooks: Under "src/notebooks" providing interactive examples (e.g., model_training_example.ipynb) to demonstrate usage.
e. Tests: A dedicated "tests" folder with unit and integration tests to verify functionality.
Relationships and Interactions
• The CLI and API both interface with the DSBA core library. They invoke functionalities (e.g., training, prediction, registry management) offered by the library.
• The CLI can be visualized as a command-line front-end that directly calls DSBA functions.
• The API (possibly containerized via Docker) exposes endpoints which internally call the DSBA library functions.
• Notebooks serve as examples; they import and execute DSBA modules similar to how CLI/API do.
• The tests are layered to validate each module independently (unit tests) and check interactions between modules (integration tests).
Architectural Patterns and Design Principles
• Separation of Concerns: Data ingestion, model functions, and registry management are clearly compartmentalized.
• Modularity: Different interfaces (CLI, API, notebooks) rely on the same core DSBA functionality, promoting code reuse.
• Containerization: The existence of Dockerfiles (e.g., Dockerfile.api) highlights isolation and deployment patterns commonly seen in microservice-style architectures.
• Environment Configuration: Using environment variables (like DSBA_MODELS_ROOT_PATH) to externalize configuration, typical in modern applications.
Technologies and Frameworks
• Python (version 3.12)
• Modern dependency management via pyproject.toml (instead of setup.py)
• Testing framework (pytest) for validations
• Docker for containerizing the API component
• The potential use of machine learning libraries (for instance, xgboost is mentioned in the README, as well as a potential fallback to scikit-learn)