How bonfire-networks/ecto_sparkles works
Project type and purpose
• This is an Elixir library (“EctoSparkles”) that extends and wraps Ecto, providing macros and runtime helpers for common patterns: join+preload, reusable joins, schema migrations at runtime, data migrations, N+1 detection, logging, and changeset error formatting.
• It is intended to be pulled in as a dependency by any OTP application that uses Ecto.
File structure
• Top‐level:
• mix.exs / mix.lock – defines the OTP library and its dependencies.
• config/ – application configuration (e.g. OTP app name).
• lib/ – the source. Under lib you have:
– query helpers: proload, join_preload, reusable_join, n_plus_1_detector, sanitise_strings
– migration modules: schema_migration (AutoMigrator, Migrator), data_migration (behaviour + runner)
– utilities: erlang_term_binary, json_serde_data
– Log and Changesets.Errors modules
• test/ – unit tests for each feature.
• No frontend or database schema here—it’s a library layering on top of Ecto.
README insights
• Features listed and how they integrate:
• Macros expand into Ecto.Query calls.
• Migrator and AutoMigrator wrap Ecto.Migrator at runtime.
• DataMigration is a behaviour plus a runner to backfill data.
• Log uses Telemetry to capture slow or N+1 queries.
• Changesets.Errors transforms Ecto.Changeset errors into human-readable maps.
Main components
A. QueryEnhancements
• proload / join_preload macros
• reusable_join macro
• Under the hood: expands into calls to Ecto.Query.join, Ecto.Query.preload
B. SchemaMigration
• EctoSparkles.SchemaMigration.Migrator
• EctoSparkles.SchemaMigration.AutoMigrator (hooks into OTP application start)
• Lines of communication with Ecto.Migrator and the Repo
C. DataMigration
• EctoSparkles.DataMigration behaviour (callback skeleton for data backfills)
• EctoSparkles.DataMigration.Runner (invokes each implementation)
D. Telemetry & Logging
• EctoSparkles.Log (attaches handlers to Telemetry events)
• EctoSparkles.NPlusOneDetector (listens to query events, filters patterns)
E. Utilities
• SanitiseStrings, ErlangTermBinary, JsonSerdeData (helper modules used by other components)
F. Changeset Errors
• EctoSparkles.Changesets.Errors (formats error structs)
External dependencies & interactions
• Ecto.Repo / Ecto.Query – Query generation and execution against the database
• Telemetry – instrumentation for logging & N+1 detection
• OTP Application startup – for AutoMigrator hook
• Configuration (config/config.exs) – sets :ecto_sparkles, :otp_app
Relationships and data flows
• AutoMigrator invoked at application start → Ecto.Migrator → Repo → DB
• DataMigration.Runner triggered via mix task or runtime call → each module implementing DataMigration → Repo → DB
• Telemetry events emitted by Repo or Ecto → captured by Log and NPlusOneDetector → forwarded to Logger/console or telemetry backends
Architectural patterns & design principles
• Macro-based DSL – compile-time code generation for common patterns
• Behaviour + runner pattern – pluggable data migrations
• OTP Application callbacks – AutoMigrator uses application lifecycle
• Telemetry subscription – decouples logging and N+1 detection from application code
Detailed checklist
• [ ] Front: Host Application (your OTP app)
• [ ] Library boundary: EctoSparkles
• QueryEnhancements
• SchemaMigration & AutoMigrator
• DataMigration behaviour + Runner
• Log & NPlusOneDetector (Telemetry)
• Utilities & Changeset.Errors
• [ ] External systems: Ecto.Repo, Ecto.Query, Telemetry, Database