# bonfire-networks/ecto_sparkles architecture

> Architecture diagram of the GitHub repository bonfire-networks/ecto_sparkles, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bonfire-networks/ecto_sparkles
- Repository: https://github.com/bonfire-networks/ecto_sparkles

## Overview

##### Project type and purpose
- This is an Elixir library (“EctoSparkles”) that extends and wraps Ecto, providing macros and runtime helpers for common patterns: join+preload, reusable joins, schema migrations at runtime, data migrations, N+1 detection, logging, and changeset error formatting.
- It is intended to be pulled in as a dependency by any OTP application that uses Ecto.

##### File structure
- Top‐level:
- mix.exs / mix.lock – defines the OTP library and its dependencies.
- config/ – application configuration (e.g. OTP app name).
- lib/ – the source. Under lib you have:
– query helpers: proload, join_preload, reusable_join, n_plus_1_detector, sanitise_strings
– migration modules: schema_migration (AutoMigrator, Migrator), data_migration (behaviour + runner)
– utilities: erlang_term_binary, json_serde_data
– Log and Changesets.Errors modules
- test/ – unit tests for each feature.
- No frontend or database schema here—it’s a library layering on top of Ecto.

##### README insights
- Features listed and how they integrate:
- Macros expand into Ecto.Query calls.
- Migrator and AutoMigrator wrap Ecto.Migrator at runtime.
- DataMigration is a behaviour plus a runner to backfill data.
- Log uses Telemetry to capture slow or N+1 queries.
- Changesets.Errors transforms Ecto.Changeset errors into human-readable maps.

##### Main components
A. QueryEnhancements
- proload / join_preload macros
- reusable_join macro
- Under the hood: expands into calls to Ecto.Query.join, Ecto.Query.preload
B. SchemaMigration
- EctoSparkles.SchemaMigration.Migrator
- EctoSparkles.SchemaMigration.AutoMigrator (hooks into OTP application start)
- Lines of communication with Ecto.Migrator and the Repo
C. DataMigration
- EctoSparkles.DataMigration behaviour (callback skeleton for data backfills)
- EctoSparkles.DataMigration.Runner (invokes each implementation)
D. Telemetry & Logging
- EctoSparkles.Log (attaches handlers to Telemetry events)
- EctoSparkles.NPlusOneDetector (listens to query events, filters patterns)
E. Utilities
- SanitiseStrings, ErlangTermBinary, JsonSerdeData (helper modules used by other components)
F. Changeset Errors
- EctoSparkles.Changesets.Errors (formats error structs)

##### External dependencies & interactions
- Ecto.Repo / Ecto.Query – Query generation and execution against the database
- Telemetry – instrumentation for logging & N+1 detection
- OTP Application startup – for AutoMigrator hook
- Configuration (config/config.exs) – sets :ecto_sparkles, :otp_app

##### Relationships and data flows
- AutoMigrator invoked at application start → Ecto.Migrator → Repo → DB
- DataMigration.Runner triggered via mix task or runtime call → each module implementing DataMigration → Repo → DB
- Telemetry events emitted by Repo or Ecto → captured by Log and NPlusOneDetector → forwarded to Logger/console or telemetry backends

##### Architectural patterns & design principles
- Macro-based DSL – compile-time code generation for common patterns
- Behaviour + runner pattern – pluggable data migrations
- OTP Application callbacks – AutoMigrator uses application lifecycle
- Telemetry subscription – decouples logging and N+1 detection from application code

##### Detailed checklist
- [ ] Front: Host Application (your OTP app)
- [ ] Library boundary: EctoSparkles
- QueryEnhancements
- SchemaMigration & AutoMigrator
- DataMigration behaviour + Runner
- Log & NPlusOneDetector (Telemetry)
- Utilities & Changeset.Errors
- [ ] External systems: Ecto.Repo, Ecto.Query, Telemetry, Database

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External host application
    HostApp["Host Application"] 

    %% Core Entry and Configuration
    Core["EctoSparkles (lib/ecto_sparkles.ex)"]:::core
    ConfigApp["app config (config/config.exs)"]:::core
    EctoConfig["EctoSparkles config (config/ecto_sparkles.exs)"]:::core
    MixDef["mix.exs"]:::core

    HostApp -->|"depends on"| Core
    HostApp -->|"sets config"| ConfigApp
    HostApp -->|"sets config"| EctoConfig
    HostApp -->|"defines dependency"| MixDef

    %% Library Boundary
    subgraph "EctoSparkles" 
        direction TB

        %% Query Enhancements
        subgraph "QueryEnhancements" 
            direction TB
            QE["Proload/JoinPreload & ReusableJoin macros"]:::query
            SanitiseStrings["SanitiseStrings"]:::query
        end

        %% Schema Migrations
        subgraph "SchemaMigration" 
            direction TB
            AutoMigrator["AutoMigrator"]:::migrations
            Migrator["Migrator"]:::migrations
        end

        %% Data Migrations
        subgraph "DataMigration" 
            direction TB
            DataBehaviour["DataMigration behaviour"]:::dataMig
            Runner["Runner"]:::dataMig
            DMConfig["DataMigration Config"]:::dataMig
        end

        %% Telemetry & Logging
        subgraph "Telemetry & Logging" 
            direction TB
            Log["Log"]:::telemetry
            NPlusOneDetector["NPlusOneDetector"]:::telemetry
        end

        %% Utilities
        subgraph "Utilities" 
            direction TB
            ErlangTermBinary["ErlangTermBinary"]:::utils
            JsonSerdeData["JsonSerdeData"]:::utils
        end

        %% Changeset Errors
        ChangesetErrors["Changesets.Errors"]:::changeset

    end

    %% External Systems
    EctoQuery["Ecto.Query"]
    Repo["Ecto.Repo"]
    EctoMigrator["Ecto.Migrator"]
    TelemetrySys["Telemetry"]
    LoggerSys["Logger/Console"]
    Database["Database"] 

    %% Flows: QueryEnhancements
    Core -->|provides macros| QE
    QE -->|"expands into"| EctoQuery
    SanitiseStrings -->|"utility used by macros"| QE
    EctoQuery -->|"executes query"| Repo
    Repo -->|"DB calls"| Database

    %% Flows: SchemaMigration
    Core -->|start hook| AutoMigrator
    AutoMigrator -->|"triggers"| Migrator
    Migrator -->|"calls Ecto.Migrator"| EctoMigrator
    EctoMigrator -->|"executes migrations"| Repo
    Repo -->|"DB calls"| Database

    %% Flows: DataMigration
    Core -->|provides behaviour| DataBehaviour
    Core -->|provides runner| Runner
    Runner -->|loads config| DMConfig
    Runner -->|"invokes modules"| DataBehaviour
    Runner -->|"runs migrations"| Repo
    Repo -->|"DB calls"| Database

    %% Flows: Telemetry & Logging
    Repo -->|emits events| TelemetrySys
    TelemetrySys -->|listens| Log
    TelemetrySys -->|listens| NPlusOneDetector
    Log -->|"forwards logs"| LoggerSys
    NPlusOneDetector -->|"alerts N+1"| LoggerSys

    %% Utilities usage
    Core -->|includes| ErlangTermBinary
    Core -->|includes| JsonSerdeData

    %% Changeset Errors
    Core -->|formats errors| ChangesetErrors

    %% Click Events
    click SanitiseStrings "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/sanitise_strings.ex"
    click NPlusOneDetector "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/n_plus_1_detector.ex"
    click AutoMigrator "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/schema_migration/auto_migrator.ex"
    click Migrator "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/schema_migration/migrator.ex"
    click DataBehaviour "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/data_migration/data_migration.ex"
    click Runner "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/data_migration/runner.ex"
    click DMConfig "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/data_migration/config.ex"
    click Log "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/log.ex"
    click ErlangTermBinary "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/erlang_term_binary.ex"
    click JsonSerdeData "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/json_serde_data.ex"
    click ChangesetErrors "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/changesets_errors.ex"
    click Core "https://github.com/bonfire-networks/ecto_sparkles/blob/main/lib/ecto_sparkles.ex"
    click ConfigApp "https://github.com/bonfire-networks/ecto_sparkles/blob/main/config/config.exs"
    click EctoConfig "https://github.com/bonfire-networks/ecto_sparkles/blob/main/config/ecto_sparkles.exs"
    click MixDef "https://github.com/bonfire-networks/ecto_sparkles/blob/main/mix.exs"

    %% Styles
    classDef query fill:#D0E6FF,stroke:#000
    classDef migrations fill:#D0FFD6,stroke:#000
    classDef dataMig fill:#F5D0FF,stroke:#000
    classDef telemetry fill:#FFE5B4,stroke:#000
    classDef utils fill:#E8E8E8,stroke:#000
    classDef changeset fill:#FFD1DC,stroke:#000
    classDef core fill:#FFF2CC,stroke:#000
```
