# bitswan-space/bitswan architecture

> Architecture diagram of the GitHub repository bitswan-space/bitswan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/bitswan-space/bitswan
- Repository: https://github.com/bitswan-space/bitswan

## Overview

##### Main Components

a. Core System Components:
- Pipeline Engine
- Jupyter Integration Layer
- Service Layer
- Configuration Management
- Testing Framework

b. Data Flow Components:
- Sources (Various input connectors)
- Processors (Data transformation)
- Sinks (Output destinations)
- Lookup Systems
- Triggers

##### Architecture Patterns to Highlight

- Modular architecture (evident from the directory structure)
- Plugin-based system (multiple connectors and processors)
- Event-driven processing
- Asynchronous operations (based on the async components)

##### Specific Implementation Details

**┌─────────────────────────────────────────────────────────┐**
- Client Interface Layer                  │
- Jupyter Notebooks                                    │
- CLI Interface                                        │
- ┘
- ↕
**┌─────────────────────────────────────────────────────────┐**
- Pipeline Core                         │
- ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
- Sources   │→ │ Processors  │→ │    Sinks    │     │
- ┘  └─────────────┘  └─────────────┘     │
- ↕            ↕                 ↕              │
- ┌─────────────────────────────────────────────┐       │
- Service Layer                   │       │
- ┘       │
- ┘
- ↕
**┌─────────────────────────────────────────────────────────┐**
- Integration Layer                       │
- ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
- AMQP     │  │   Kafka     │  │  Elastic    │     │
- ┘  └─────────────┘  └─────────────┘     │
- ┌─────────────┐  ┌─────────────┐  ┌─────────────┐     │
- MongoDB    │  │  PostgreSQL  │  │    HTTP     │     │
- ┘  └─────────────┘  └─────────────┘     │
- ┘

##### Component Relationships

- Data flow direction between components
- Service dependencies
- Integration points with external systems
- Configuration management flow
- Testing framework interaction points

##### Additional Elements

Supporting components:
- Configuration management system
- Metrics and monitoring
- Testing framework integration
- Security components (authentication/authorization)
- Error handling and recovery paths

##### Notes

Annotations for:
- Asynchronous processing capabilities
- Scaling considerations
- Plugin architecture
- Testing capabilities
- Configuration requirements

- Clear hierarchical representation
- Proper component relationships
- Detailed annotations
- Export to common formats

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client
        JN["Jupyter Notebooks"]:::client
        CLI["CLI Interface"]:::client
    end

    subgraph Core
        PE["Pipeline Engine"]:::core
        PB["Pipeline Builder"]:::core
        SL["Service Layer"]:::core
    end

    subgraph DataFlow
        direction TB
        Sources["Sources"]:::source
        Processors["Processors"]:::processor
        Sinks["Sinks"]:::sink
        Lookups["Lookup Systems"]:::lookup
    end

    subgraph Integration
        direction TB
        AMQP["AMQP"]:::integration
        Kafka["Kafka"]:::integration
        ES["Elasticsearch"]:::integration
        Mongo["MongoDB"]:::integration
        Postgres["PostgreSQL"]:::integration
        HTTP["HTTP"]:::integration
    end

    subgraph Support
        Config["Configuration Management"]:::support
        Testing["Testing Framework"]:::support
        Metrics["Metrics/Monitoring"]:::support
        WebAPI["Web API"]:::support
        WebServer["Web Server"]:::support
        Declarative["Declarative Framework"]:::support
        Expression["Expression Engine"]:::support
    end

    JN --> PE
    CLI --> PE
    PE --> Sources
    Sources --> Processors
    Processors --> Sinks
    Lookups --> Processors
    
    SL --> Integration
    PE --> SL
    PB --> PE

    Sinks --> Integration
    Sources --> Integration

    classDef client fill:#e1f5fe,stroke:#01579b
    classDef core fill:#e8f5e9,stroke:#2e7d32
    classDef source fill:#f1f8e9,stroke:#558b2f
    classDef processor fill:#f3e5f5,stroke:#7b1fa2
    classDef sink fill:#fff3e0,stroke:#ef6c00
    classDef lookup fill:#fff8e1,stroke:#f9a825
    classDef integration fill:#fafafa,stroke:#424242
    classDef support fill:#e8eaf6,stroke:#3f51b5

    click JN "https://github.com/bitswan-space/BitSwan/tree/master/bspump/jupyter/"
    click CLI "https://github.com/bitswan-space/BitSwan/blob/master/bspump/__main__.py"
    click PE "https://github.com/bitswan-space/BitSwan/blob/master/bspump/pipeline.py"
    click PB "https://github.com/bitswan-space/BitSwan/blob/master/bspump/pumpbuilder.py"
    click SL "https://github.com/bitswan-space/BitSwan/blob/master/bspump/service.py"
    click Sources "https://github.com/bitswan-space/BitSwan/blob/master/bspump/abc/source.py"
    click Processors "https://github.com/bitswan-space/BitSwan/blob/master/bspump/abc/processor.py"
    click Sinks "https://github.com/bitswan-space/BitSwan/blob/master/bspump/abc/sink.py"
    click Lookups "https://github.com/bitswan-space/BitSwan/blob/master/bspump/abc/lookup.py"
    click AMQP "https://github.com/bitswan-space/BitSwan/tree/master/bspump/amqp/"
    click Kafka "https://github.com/bitswan-space/BitSwan/tree/master/bspump/kafka/"
    click ES "https://github.com/bitswan-space/BitSwan/tree/master/bspump/elasticsearch/"
    click Mongo "https://github.com/bitswan-space/BitSwan/tree/master/bspump/mongodb/"
    click Postgres "https://github.com/bitswan-space/BitSwan/tree/master/bspump/postgresql/"
    click HTTP "https://github.com/bitswan-space/BitSwan/tree/master/bspump/http/"
    click Config "https://github.com/bitswan-space/BitSwan/blob/master/bspump/asab/config.py"
    click Testing "https://github.com/bitswan-space/BitSwan/tree/master/bspump/unittest/"
    click Metrics "https://github.com/bitswan-space/BitSwan/tree/master/bspump/asab/metrics/"
    click WebAPI "https://github.com/bitswan-space/BitSwan/tree/master/bspump/asab/web/"
    click WebServer "https://github.com/bitswan-space/BitSwan/tree/master/bspump/web/"
    click Declarative "https://github.com/bitswan-space/BitSwan/tree/master/bspump/declarative/"
    click Expression "https://github.com/bitswan-space/BitSwan/tree/master/bspump/declarative/expression/"
```
