# nike-inc/koheesio architecture

> Architecture diagram of the GitHub repository nike-inc/koheesio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nike-inc/koheesio
- Repository: https://github.com/nike-inc/koheesio

## Overview

##### Project Type and Purpose
- Koheesio is a Python framework designed to build efficient data pipelines. Its primary focus is on modular, testable data tasks that serve as building blocks for larger data processing workflows.

##### File Structure
- The top-level directories include standard configuration and docs folders (e.g., .github, docs, tests, etc.). The main implementation lives under the “src/koheesio” directory.
- Core modules like “context.py”, “logger.py”, and “steps” (which contain individual task implementations).
- Specific submodules for “asyncio”, “integrations”, “pandas”, “secrets”, “spark”, and “sso” that indicate integrations with external systems and specialized processing.
- The “spark” directory is detailed with readers, writers, transformations, and utilities, which means heavy emphasis on handling Spark operations.
- The “tests” directory also mirrors many modules, ensuring well-covered test cases and further indicating the importance of modular, testable units.

a. Main Components to Include:
- Core Framework Components:
– Step: Representing the individual task elements.
– Context: Providing the execution and configuration environment.
– Logger: For logging and observability.
- Integration Modules:
– Specific Spark integrations, including modules for readers, writers, transformations, delta support, and utilities.
– Additional specialized modules for async operations (e.g., the “asyncio” directory) as well as “pandas” readers.
- External Dependencies & Extras:
– Mention Pydantic (for configuration and type-safety as highlighted in the README).
– Include any external systems such as external data sources, cloud services, or data warehouses that the integrations target.

b. Relationships and Interactions:
- Directional data flow from the Step components toward integration processes. For example:
– Steps invoke methods on the Context and Logger for configuration and logging.
- How the “asyncio” and “sso” components interact with other modules if asynchronous or security-based tasks are required.

c. Architectural Patterns and Design Principles:
- Modularity: Each component (e.g., steps, integrations, utilities) is self-contained and reusable.
- Strong Typing/Validation: Indicate the use of Pydantic for configurations.
- Integration-Oriented Design: Show how the framework is designed to plug into multiple external systems without being a full-fledged orchestrator.

d. Relevant Technologies:
- Python plays the central role.
- Pydantic is used for settings and validation.
- The testing framework (pytest) and CI/CD workflow tools (GitHub Actions) can be mentioned as peripheral but important deployment/test support elements.

##### Be Detailed
- You capture all primary areas, including the deep modular structure under “src/koheesio” (like the detailed Spark handling capabilities).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Framework Subgraph
    subgraph "Core Framework"
        Step["Step"]:::core
        Context["Context"]:::core
        Logger["Logger"]:::core
    end

    %% Integrations Subgraph
    subgraph "Integrations"
        GenericInt["Generic Integrations"]:::int
        BoxInt["Box Integration"]:::int
        SnowflakeInt["Snowflake Integration"]:::int
        SparkInt["Spark Integration"]:::int
        AsyncioMod["Asyncio Module"]:::int
        PandasInt["Pandas Integration"]:::int
        SSOInt["SSO Integration"]:::int

        %% Nested Spark Components Subgraph (Vertical Layout)
        subgraph "Spark Components"
            SparkRead["Spark Readers"]:::spark
            SparkTrans["Spark Transformations"]:::spark
            SparkWrite["Spark Writers"]:::spark
            SFTPInt["SFTP Integration"]:::spark
        end
    end

    %% External Systems Subgraph
    subgraph "External Dependencies"
        Pydantic["Pydantic (Validation)"]:::external
    end

    %% Relationships
    %% Core interactions
    Step -->|"uses"| Context
    Step -->|"logs to"| Logger

    %% Integration invocations from Steps
    Step -->|"calls"| GenericInt
    Step -->|"calls"| BoxInt
    Step -->|"calls"| SnowflakeInt
    Step -->|"calls"| SparkInt
    Step -->|"calls"| AsyncioMod
    Step -->|"calls"| PandasInt
    Step -->|"calls"| SSOInt

    %% Spark Integration internal flow
    SparkInt -->|"processes via"| SparkRead
    SparkRead -->|"transforms"| SparkTrans
    SparkTrans -->|"writes"| SparkWrite
    SFTPInt -->|"feeds"| SparkWrite

    %% Link External Dependencies (configuration/validation)
    Context -->|"validates_config_with"| Pydantic


    %% Click Events
    click Step "https://github.com/nike-inc/koheesio/tree/main/src/koheesio/steps"
    click Context "https://github.com/nike-inc/koheesio/blob/main/src/koheesio/context.py"
    click Logger "https://github.com/nike-inc/koheesio/blob/main/src/koheesio/logger.py"
    click GenericInt "https://github.com/nike-inc/koheesio/tree/main/src/koheesio/integrations"
    click BoxInt "https://github.com/nike-inc/koheesio/blob/main/src/koheesio/integrations/box.py"
    click SnowflakeInt "https://github.com/nike-inc/koheesio/tree/main/src/koheesio/integrations/snowflake"
    click SparkInt "https://github.com/nike-inc/koheesio/tree/main/src/koheesio/integrations/spark"
    click SparkRead "https://github.com/nike-inc/koheesio/tree/main/src/koheesio/spark/readers"
    click SparkTrans "https://github.com/nike-inc/koheesio/tree/main/src/koheesio/spark/transformations"
    click SparkWrite "https://github.com/nike-inc/koheesio/tree/main/src/koheesio/spark/writers"
    click SFTPInt "https://github.com/nike-inc/koheesio/blob/main/src/koheesio/integrations/spark/sftp.py"
    click AsyncioMod "https://github.com/nike-inc/koheesio/tree/main/src/koheesio/asyncio"
    click PandasInt "https://github.com/nike-inc/koheesio/tree/main/src/koheesio/pandas"
    click SSOInt "https://github.com/nike-inc/koheesio/tree/main/src/koheesio/sso"

    %% Styles
    classDef core fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef int fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef spark fill:#87CEFA,stroke:#333,stroke-width:2px;
    classDef external fill:#FFB6C1,stroke:#333,stroke-width:2px;
```
