How nidhaloff/igel works
Project Type and Purpose
• This is a CLI-first, configuration-driven machine-learning orchestration tool (igel) written in Python.
• It provides commands for fit/train, evaluate, predict, experiment, export (to ONNX), and serve (via a REST API).
• It also exposes a Python API (Igel class) and organizes examples, tests, docs, and a FastAPI-based server.
High-Level Architecture Overview
• Layers:
1. Presentation Layer:
• CLI entrypoint (igel/__main__.py)
• Python API (Igel class in igel/igel.py)
• Example scripts (examples/*.py)
2. Core Engine Layer:
• Config parser & validator (igel/configs.py, pyproject.toml, CLI parsers)
• Data ingestion (igel/data.py)
• Preprocessing (igel/preprocessing.py)
• Model orchestration (fit/evaluate/predict in igel/igel.py)
• Auto-ML subpackage (igel/auto/*)
• Hyperparameter support (igel/hyperparams.py)
3. Serving Layer:
• FastAPI server adapter (igel/servers/fastapi_server.py)
• Uvicorn runner (invoked under the hood)
4. Persistence Layer & Artifacts:
• File-system–based model_results folder (serialized models, JSON descriptions, CSV predictions)
• Example data under examples/data
5. External Dependencies:
• Pandas, scikit-learn, FastAPI, Uvicorn, ONNX exporter
Main Components and Interactions
• CLI Tool
• Parses commands (fit, evaluate, predict, experiment, export, serve) and flags (--data_path, --yaml_path, --model_results_dir, etc.).
• Instantiates the core Igel engine with a cmd argument.
• Config Loader
• Reads YAML/JSON using igel/configs.py.
• Produces a configuration object driving the pipeline.
• Data Layer
• igel/data.py uses pandas under the hood, configured by dataset.read_data_options.
• Splits and dispatches DataFrames to preprocessing.
• Preprocessing
• Handles missing values, encoding, scaling as directed by config.
• Outputs processed features and targets.
• Model Lifecycle
1. Fit/Train: calls the chosen algorithm (scikit-learn estimator or AutoML)
2. Evaluate: loads model artifact, scores on new data, writes evaluation.json
3. Predict: loads model artifact, writes predictions.csv
4. Export: converts sklearn model to ONNX
• Auto-ML
• Special pipelines under igel/auto/ for Image/Text/Structured tasks.
• Serving
• FastAPI app defines /predict endpoint.
• On start (igel serve), the engine loads model_results and mounts the model.
• Uvicorn runs the ASGI app.
• Tests & Examples
• Unit tests (tests/) validate each layer.
• Example scripts show API usage and serve as template integrations.
Architectural Patterns and Principles
• Layered Architecture: clear separation between CLI/UI, core engine, serving, external libs.
• Pipeline Pattern: data flows through ingestion → preprocessing → modeling → persistence.
• Adapter Pattern: CLI and Python API adapt user requests into core engine calls; FastAPI adapter wraps the engine for REST.
• Configuration-Driven: YAML/JSON drives behavior, minimizing code changes.
• Modular Design: each concern lives in its own module or package (auto, servers, preprocessing, etc.).