# nike-inc/brickflow architecture

> Architecture diagram of the GitHub repository nike-inc/brickflow, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nike-inc/brickflow
- Repository: https://github.com/nike-inc/brickflow

## Overview

##### Main Components
- Core Engine & Workflow Management:
- Brickflow’s core logic lives in the “brickflow/engine” directory. It contains modules that manage workflow execution (e.g., task scheduling, project management, dependency resolution, and execution flow).
- The “brickflow/resolver” helps in dependency and configuration resolution.
- Command-Line Interface (CLI):
- Found in “brickflow/cli”, this provides the user-facing commands (like creating projects, adding workflows, deploying to Databricks) and acts as the bridging layer between the user’s inputs and the workflow engine.
- Code Generation & Bundling:
- The “brickflow/codegen” and “brickflow/bundles” directories focus on generating appropriate config files or bundles for different deployment (like polyrepo, monorepo setups) and orchestrating the workflow elements.
- Context Management:
- The “brickflow/context” module encapsulates runtime contextual information such as environment variables, Databricks utilities (dbutils), Spark context, etc.
- Plugin System:
- The “brickflow_plugins” directory shows integration points with external systems. For instance, there are plugins for Airflow (which includes custom operators and hooks) and Databricks-specific functionality (e.g., databricks operators and sensors).
- Hints & Metadata:
- The “brickflow/hints” module (with a “py.typed” file) is used for type hints and likely provides metadata to improve developer experience and ensure type correctness.
- Testing & Documentation:
- A comprehensive “tests” directory (with tests for CLI, engine, codegen, and plugins) alongside “docs” that describe how to use and extend Brickflow. These aren’t part of the runtime architecture but help enforce the quality and usability of the system.
- Examples & Tools:
- The “examples” and “tools” directories illustrate real-world usage scenarios and include utility scripts (like deployment helpers, bundle generators).

##### Relationships and Interactions
- The CLI is the entry point for the user: it triggers project creation, workflow definitions, and deployment commands.
- When a command is issued, the CLI interacts with the core engine (in “brickflow/engine”) to process tasks, manage dependencies, and execute workflows.
- The code generation component (“brickflow/codegen”) prepares deployment bundles (from “brickflow/bundles”) that can be deployed to a Databricks environment.
- Context management ensures that runtime configurations (including environment and Spark contexts) are available to tasks and workflows.
- The plugin system extends core functionality by integrating with external systems (Airflow, Databricks, Secrets management) allowing the main engine to delegate or supplement behavior using these plugins.
- Testing modules validate these interactions while documentation supports developers during integration and maintenance.

##### Architectural Patterns and Principles
- Modularity & Extensibility: The clear separation of submodules (CLI, engine, code generation, plugins) indicates that Brickflow is built using modular principles that allow for easy extension and maintenance.
- Plugin Architecture: By having separate directories for plugins, Brickflow allows external contributors to add extended functionality (e.g., support for new operators or integration tools) without modifying the core.
- Configuration & Convention over Code: The presence of configuration files, YAML project files in sample projects, and bundled schema generation hints at a design that emphasizes explicit configurations and conventions to manage workflow behavior.

##### Instructing the Principal Software Engineer
- That the CLI invokes the Engine, which then uses Context and Codegen to deploy bundles.
- Plugins to the Engine, showing how they can supplement tasks (like Airflow operators or Databricks job controls).
- Incorporate annotations for architectural patterns:
- Areas where the plugin pattern is used.
- Areas where configuration files (from examples, tests, and YAML files) inform behavior.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Components Subgraph
    subgraph "Core Components"
        CLI["CLI – Entry point for project commands"]
        Engine["Engine – Orchestrates workflow tasks"]
        Resolver["Resolver – Manages dependency and configuration resolution"]
        Context["Context Manager – Provides runtime configuration"]
        Hints["Hints & Metadata – Type system and developer support"]
    end

    %% Deployment Pipeline Subgraph
    subgraph "Deployment Pipeline"
        Codegen["Code Generation – Prepares deployment config"]
        Bundler["Deployment Bundling – Creates deployment artifacts"]
    end

    %% Extensions Subgraph
    subgraph "Extensions"
        Plugin["Plugin System – Integrates external systems (Airflow, Databricks)"]
    end

    %% External Systems
    Databricks(("Databricks Workspace"))
    Airflow(("Airflow"))

    %% Connections
    CLI -->|"invokes"| Engine
    Engine -->|"resolves"| Resolver
    Engine -->|"uses"| Context
    Engine -->|"generates"| Codegen
    Codegen -->|"bundles"| Bundler
    Engine -->|"leverages"| Hints
    Engine -->|"augmented_by"| Plugin
    Plugin -->|"integrates_with"| Databricks
    Plugin -->|"integrates_with"| Airflow

    %% Click Events
    click CLI "https://github.com/nike-inc/brickflow/tree/main/brickflow/cli"
    click Engine "https://github.com/nike-inc/brickflow/tree/main/brickflow/engine"
    click Resolver "https://github.com/nike-inc/brickflow/tree/main/brickflow/resolver"
    click Context "https://github.com/nike-inc/brickflow/tree/main/brickflow/context"
    click Hints "https://github.com/nike-inc/brickflow/tree/main/brickflow/hints"
    click Codegen "https://github.com/nike-inc/brickflow/tree/main/brickflow/codegen"
    click Bundler "https://github.com/nike-inc/brickflow/tree/main/brickflow/bundles"
    click Plugin "https://github.com/nike-inc/brickflow/tree/main/brickflow_plugins"

    %% Styles
    classDef core fill:#afd9ee,stroke:#333,stroke-width:2px;
    classDef pipeline fill:#d7e7a9,stroke:#333,stroke-width:2px;
    classDef extension fill:#fdd49e,stroke:#333,stroke-width:2px;
    classDef external fill:#f8cbc1,stroke:#333,stroke-width:2px;

    class CLI,Engine,Resolver,Context,Hints core;
    class Codegen,Bundler pipeline;
    class Plugin extension;
    class Databricks,Airflow external;
```
