# woven-planet/wicker architecture

> Architecture diagram of the GitHub repository woven-planet/wicker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/woven-planet/wicker
- Repository: https://github.com/woven-planet/wicker

## Overview

##### Project Context and Purpose
- From the README, Wicker is intended for managing machine learning datasets (storage and serving) and it encourages extensibility via plugins. The framework is used in ML workflows, which likely involves handling data schemas, persistence mechanisms, and integrations with technologies (e.g., Spark, DynamoDB, WandB, Flyte).
- The file structure reinforces this purpose, with distinct directories for core functionalities (wicker/core), plugin integrations (wicker/plugins), schema handling (wicker/schema), and even dedicated areas for testing.

##### Core Components
a. Core Module (wicker/core):
- Contains the central functionality such as configuration, definitions, persistence, storage, file locking, column file operations, shuffling, and writing.
- This is the backbone of the dataset storage and serving mechanism.
b. Schema Module (wicker/schema):
- Manages data codecs, data parsing and loading, serialization, and validation.
- How schema definitions help ensure data consistency throughout the system.
c. Plugin Architecture (wicker/plugins):
- Enables integration with external systems (e.g., DynamoDB, Spark, WandB, Flyte).
- That these plugins extend Wicker’s functionalities, making external communications and enhancements possible.
d. Testing and Development Support:
- Though not part of the “runtime” architecture, testing code (in both tests/ and wicker/testing) shows a commitment to robust design and might inform how modules interact.
e. Auxiliary and Supporting Elements:
- Project configuration files (setup.py, setup.cfg, pyproject.toml, Makefile) along with CI/CD workflows (.github/workflows) represent the development, deployment, and quality assurance parts of the project.

##### Relationships and Data Flow
- Data flow should be illustrated from dataset ingestion → transformation/validation (via schema) → persistence/storage using core logic.

##### Architectural Patterns and Principles
- The modular structure (separation into core, plugins, schema) implies a design guided by separation of concerns, extensibility, and loose coupling.
- This layered approach helps to isolate dataset management functionality from integration logic.
- Clearly denote dependency inversion where higher-level modules (core) are extended with plugins without modifying them directly.

##### Technical Stack and Development Elements
- Technologies include Python (evident by .py files, setup, and testing tools), with CI/CD managed by GitHub workflows.
- The use of Makefiles and configuration files highlights an emphasis on automation, consistent testing, and type checking (via mypy.ini and Pytest in tests).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Wicker Modules"
        direction TB
        Core["Core Module"]:::module
        Schema["Schema Module"]:::module
        Plugins["Plugin Integrations"]:::plugin
        External["External Systems"]:::external
    end

    subgraph "Support & Tools"
        direction TB
        Testing["Testing Support"]:::testing
        DevTools["CI/CD & Development Tools"]:::dev
    end

    Core -->|"validates"| Schema
    Core -->|"usesPlugins"| Plugins
    Plugins -->|"callsExternal"| External

    Testing -.->|"tests"| Core
    Testing -.->|"tests"| Plugins
    DevTools -.->|"automates"| Testing

    %% Click Events
    click Core "https://github.com/woven-planet/wicker/tree/main/wicker/core"
    click Schema "https://github.com/woven-planet/wicker/tree/main/wicker/schema"
    click Plugins "https://github.com/woven-planet/wicker/tree/main/wicker/plugins"
    click Testing "https://github.com/woven-planet/wicker/tree/main/tests"
    click DevTools "https://github.com/woven-planet/wicker/tree/main/.github/workflows"

    %% Styles
    classDef module fill:#add8e6,stroke:#000,stroke-width:2px;
    classDef plugin fill:#90ee90,stroke:#000,stroke-width:2px;
    classDef testing fill:#ffff99,stroke:#000,stroke-width:2px;
    classDef dev fill:#ffb6c1,stroke:#000,stroke-width:2px;
    classDef external fill:#d3d3d3,stroke:#000,stroke-width:2px;
```
