# nikolamilojica/illuminate architecture

> Architecture diagram of the GitHub repository nikolamilojica/illuminate, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nikolamilojica/illuminate
- Repository: https://github.com/nikolamilojica/illuminate

## Overview

##### Project Type and Purpose
- Illuminate is a “batteries included” ETL framework that provides rapid ETL and scraping capabilities. It is designed to help with data extraction, transformation, and loading, supported by a modular, pluggable architecture.
- The project is built using Python and leverages frameworks and libraries such as Alembic (for migrations), Click (for CLI interfaces), Tornado, and SQLAlchemy. It is conceptually influenced by platforms like Django and Scrapy.
- In a production or development setting, it is installed from PyPI (package name “illuminated”), supports CLI operations, and can be deployed in containerized environments.

##### Analyzing the File Structure
- The top-level directories include configuration files (Dockerfile, tox.ini, .github, etc.), documentation (docs folder), the primary source code (in the “illuminate” folder), and tests.
- Inside the “illuminate” package, the structure is clearly modular, with subdirectories like “adapter”, “cli”, “common”, “decorators”, “exceptions”, “exporter”, “interface”, “manager”, “meta”, “observation”, and “observer”. This suggests a layered design where different responsibilities (data observation, exporting, management, error handling, etc.) are separated.
- The “tests” directory indicates extensive unit and integration tests, reinforcing the importance of each module.

##### Key Components
a. User Interface / CLI:
- CLI component (illuminate/cli.py and related decorators) as the entry point that handles user commands, project setup, database migrations, and starting the observation process.
b. ETL Core Engine:
- Frame the core logic into modules such as:
- Manager (illuminate/manager): coordinating tasks and overseeing the overall ETL process.
- Adapters (illuminate/adapter): acting as connectors to ingest data from various sources.
- Observers/Observations (illuminate/observer, illuminate/observation): responsible for monitoring data flows and capturing events during the ETL process.
- Exporters (illuminate/exporter): handling the transformation and loading of data to various sinks (with specific exporters for SQL and InfluxDB).
c. Error Handling:
d. Interfaces and Abstractions:
- The “interface” folder indicates a set of standardized interfaces potentially used to allow plug-and-play extensions of functionality (for adapters, exporters, etc.).
e. Metadata and Project Definitions:
- The “meta” and “common” folders provide project definitions, logging facilities, and template management, which are key for configuration and extensibility.
f. External Dependencies & Environment:
- External technologies such as:
- SQLAlchemy and Alembic for database interaction and migrations.
- Click for building the CLI.
- Tornado for asynchronous processing if applicable.
- Docker (Dockerfile and docker-compose usage) for deployment.
- Integration with external systems, such as the target databases (main DB, measurements DB) and ancillary tools (pgAdmin, Grafana) as depicted in the project setup instructions.

## Diagram (Mermaid)

```mermaid
graph TD
    %% User Interface / CLI
    subgraph "User Interface / CLI"
        CLI_Main["CLI Main"]:::cli
        CLI_Dec["CLI Decorators"]:::cli
    end

    %% ETL Core Engine
    subgraph "ETL Core Engine"
        Manager["Manager"]:::core
        Adapters["Adapters"]:::core
        Observers["Observers/Observations"]:::core
        Exporters["Exporters"]:::core
    end

    %% Error Handling and Interfaces
    subgraph "Error Handling & Interfaces"
        Exceptions["Exceptions"]:::error
        Interfaces["Interfaces"]:::interface
    end

    %% Metadata and Common Utilities
    subgraph "Metadata & Utilities"
        Metadata["Metadata"]:::utility
        Common["Common"]:::utility
    end

    %% External Dependencies & Deployment
    subgraph "External Dependencies & Deployment"
        DB["SQLAlchemy/Alembic"]:::external
        CLIExt["Click/Tornado"]:::external
        DockerNode["Docker"]:::external
        Monitoring["pgAdmin/Grafana"]:::external
    end

    %% Testing
    subgraph "Testing"
        Tests["Tests"]:::testing
    end

    %% Connections between layers
    CLI_Main -->|"triggers"| Manager
    CLI_Dec -->|"augments"| Manager

    Manager -->|"coordinates"| Adapters
    Manager -->|"coordinates"| Observers
    Manager -->|"coordinates"| Exporters

    Manager -->|"reports"| Exceptions
    Adapters -->|"reports"| Exceptions
    Observers -->|"reports"| Exceptions
    Exporters -->|"reports"| Exceptions

    Interfaces --- Manager
    Interfaces --- Adapters
    Interfaces --- Observers
    Interfaces --- Exporters

    Metadata -->|"supports"| Manager
    Common -->|"supports"| Manager

    Manager -.->|"uses"| DB
    Exporters -.->|"loads_to"| DB
    CLI_Main -.->|"leverages"| CLIExt

    DockerNode -->|"deploys"| Manager
    Monitoring -->|"monitors"| DockerNode
    Tests -->|"validates"| Manager

    %% Click Events for Component Mapping
    click CLI_Main "https://github.com/nikolamilojica/illuminate/blob/master/illuminate/cli.py"
    click CLI_Dec "https://github.com/nikolamilojica/illuminate/blob/master/illuminate/decorators/cli.py"
    click Manager "https://github.com/nikolamilojica/illuminate/blob/master/illuminate/manager/manager.py"
    click Manager "https://github.com/nikolamilojica/illuminate/tree/master/illuminate/manager"
    click Adapters "https://github.com/nikolamilojica/illuminate/blob/master/illuminate/adapter/adapter.py"
    click Adapters "https://github.com/nikolamilojica/illuminate/tree/master/illuminate/adapter"
    click Observers "https://github.com/nikolamilojica/illuminate/blob/master/illuminate/observer/observer.py"
    click Observers "https://github.com/nikolamilojica/illuminate/blob/master/illuminate/observation/observation.py"
    click Exporters "https://github.com/nikolamilojica/illuminate/blob/master/illuminate/exporter/exporter.py"
    click Exporters "https://github.com/nikolamilojica/illuminate/blob/master/illuminate/exporter/sql.py"
    click Exporters "https://github.com/nikolamilojica/illuminate/blob/master/illuminate/exporter/influxdb.py"
    click Exceptions "https://github.com/nikolamilojica/illuminate/tree/master/illuminate/exceptions"
    click Interfaces "https://github.com/nikolamilojica/illuminate/tree/master/illuminate/interface"
    click Metadata "https://github.com/nikolamilojica/illuminate/tree/master/illuminate/meta"
    click Common "https://github.com/nikolamilojica/illuminate/tree/master/illuminate/common"
    click DockerNode "https://github.com/nikolamilojica/illuminate/tree/master/Dockerfile"

    %% Style Definitions
    classDef cli fill:#f9e79f,stroke:#b7950b,stroke-width:2px;
    classDef core fill:#aed6f1,stroke:#2471a3,stroke-width:2px;
    classDef error fill:#f5b7b1,stroke:#c0392b,stroke-width:2px;
    classDef interface fill:#fad7a0,stroke:#d35400,stroke-width:2px;
    classDef utility fill:#abebc6,stroke:#1e8449,stroke-width:2px;
    classDef external fill:#d2b4de,stroke:#6c3483,stroke-width:2px;
    classDef testing fill:#d6eaf8,stroke:#2e86c1,stroke-width:2px,stroke-dasharray: 5 5;
```
