# uwdata/arquero architecture

> Architecture diagram of the GitHub repository uwdata/arquero, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/uwdata/arquero
- Repository: https://github.com/uwdata/arquero

## Overview

##### Main Components

a) Core Components:
- Table Engine (src/table/)
- Expression Engine (src/expression/)
- Operations Engine (src/op/)
- Verbs Engine (src/verbs/)
- Format Handlers (src/format/)

b) Supporting Components:
- Utility Functions (src/util/)
- Helper Functions (src/helpers/)

##### Component Relationships

**[Input Sources] → [Format Handlers] → [Table Engine] → [Output Formats]**
- ↑↓
- [Expression Engine] ←→ [Operations Engine]
- ↑↓
- [Verbs Engine] ←→ [Helper Functions]

##### Specific Implementation Details

- Rectangles: Core processing components
- Hexagons: Transform operations
- Rounded rectangles: Input/Output handlers

- Blue: Core table operations
- Green: Data input/output
- Orange: Processing/transformation
- Purple: Extension points
- Gray: Utility functions

Top: Input/Output
- ↓         ↓
- [Format Handlers Layer]

Middle: Core Processing
**[Table Engine]←→[Expression Engine]**
- ↕               ↕
**[Operations]←→[Transformation Verbs]**

Bottom: Extensions & Utilities
**[Helper Functions] [Utility Functions]**
- ↑                ↑
**[Extension Points] [Custom Functions]**

##### Additional Notes

- Version information and key dependencies
- Browser vs. Node.js specific components
- Performance-critical paths with bold lines
- Data flow direction indicators

Remember to maintain a balance between detail and clarity, focusing on the architectural relationships while avoiding overwhelming technical implementation details.

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph "Input Layer"
        CSV["CSV"]:::input
        JSON["JSON"]:::input
        Arrow["Arrow"]:::input
        Text["Text"]:::input
        Fixed["Fixed-Width"]:::input
    end

    subgraph "Format Handlers"
        FH_IN["Input Handlers"]:::format
        FH_OUT["Output Handlers"]:::format
        click FH_IN "https://github.com/uwdata/arquero/tree/main/src/format/"
        click FH_OUT "https://github.com/uwdata/arquero/tree/main/src/format/"
    end

    subgraph "Core Processing"
        TE["Table Engine"]:::core
        EE["Expression Engine"]:::core
        OE["Operations Engine"]:::core
        VE["Verbs Engine"]:::core

        click TE "https://github.com/uwdata/arquero/tree/main/src/table/"
        click EE "https://github.com/uwdata/arquero/tree/main/src/expression/"
        click OE "https://github.com/uwdata/arquero/tree/main/src/op/"
        click VE "https://github.com/uwdata/arquero/tree/main/src/verbs/"

        subgraph "Table Components"
            CT["ColumnTable"]:::table
            BT["BitSet"]:::table
            TB["Table Base"]:::table

            click CT "https://github.com/uwdata/arquero/blob/main/src/table/ColumnTable.js"
            click BT "https://github.com/uwdata/arquero/blob/main/src/table/BitSet.js"
            click TB "https://github.com/uwdata/arquero/blob/main/src/table/Table.js"
        end

        subgraph "Expression Components"
            EP["Parser"]:::expr
            EC["Compiler"]:::expr
            EG["Code Generator"]:::expr

            click EP "https://github.com/uwdata/arquero/blob/main/src/expression/parse.js"
            click EC "https://github.com/uwdata/arquero/blob/main/src/expression/compile.js"
            click EG "https://github.com/uwdata/arquero/blob/main/src/expression/codegen.js"
        end

        subgraph "Operation Components"
            AF["Aggregate Functions"]:::op
            WF["Window Functions"]:::op
            CF["Core Functions"]:::op

            click AF "https://github.com/uwdata/arquero/blob/main/src/op/aggregate-functions.js"
            click WF "https://github.com/uwdata/arquero/blob/main/src/op/window-functions.js"
            click CF "https://github.com/uwdata/arquero/tree/main/src/op/functions/"
        end
    end

    subgraph "Support Layer"
        UF["Utility Functions"]:::util
        HF["Helper Functions"]:::util

        click UF "https://github.com/uwdata/arquero/tree/main/src/util/"
        click HF "https://github.com/uwdata/arquero/tree/main/src/helpers/"
    end

    subgraph "Output Layer"
        O_Arrow["Arrow IPC"]:::output
        O_CSV["CSV"]:::output
        O_JSON["JSON"]:::output
        O_HTML["HTML"]:::output
        O_MD["Markdown"]:::output
    end

    %% Input connections
    CSV & JSON & Arrow & Text & Fixed --> FH_IN
    FH_IN --> TE

    %% Core processing connections
    TE <--> EE
    TE <--> OE
    EE <--> OE
    OE <--> VE
    TE <--> VE

    %% Table component connections
    TE --- CT & BT & TB

    %% Expression component connections
    EE --- EP & EC & EG

    %% Operation component connections
    OE --- AF & WF & CF

    %% Support connections
    UF --> TE
    HF --> VE
    UF --> OE
    HF --> OE

    %% Output connections
    TE --> FH_OUT
    FH_OUT --> O_Arrow & O_CSV & O_JSON & O_HTML & O_MD

    %% Styling
    classDef core fill:#3498db,stroke:#2980b9,color:white
    classDef input fill:#2ecc71,stroke:#27ae60,color:white
    classDef output fill:#2ecc71,stroke:#27ae60,color:white
    classDef format fill:#e74c3c,stroke:#c0392b,color:white
    classDef util fill:#95a5a6,stroke:#7f8c8d,color:white
    classDef table fill:#9b59b6,stroke:#8e44ad,color:white
    classDef expr fill:#f1c40f,stroke:#f39c12,color:black
    classDef op fill:#e67e22,stroke:#d35400,color:white
```
