# ilya-pimenov/bigquerybot architecture

> Architecture diagram of the GitHub repository ilya-pimenov/bigquerybot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ilya-pimenov/bigquerybot
- Repository: https://github.com/ilya-pimenov/bigquerybot

## Overview

##### Main Components

a) Core Components:
- BigQueryBot Base Class
- ExtendedBigQueryBot Class (extends base functionality)
- Google BigQuery API Integration
- Google Cloud Storage Integration
- External Services Integration (Mandrill, AWS S3)

b) Job Processing Components:
- Query Executor
- Load Job Handler
- Extract Job Handler
- Parallel Execution Controller

**Top Level (Client Layer)**

**Middle Layer (Bot Classes)**

**Service Layer (External Services)**

##### Component Relationships

- BigQueryBot as the core component
- ExtendedBigQueryBot as an extension of the base bot
- Integration points with external services

##### Specific Elements

- Source Management
- Query Execution
- Load Operations
- Extract Operations
- Parallel Processing

- Email Notifications (Mandrill)
- S3 Upload Capability
- URL Signing
- Dependency Graph Generation

c) Integration Points:
- Google BigQuery API
- Google Cloud Storage
- Mandrill API
- AWS S3

##### Data Flow Representation

Following flows:
- Input: Configuration (service account, credentials)
- Processing: Query/Load/Extract operations
- Output: Results to Cloud Storage/S3
- Notifications: Email alerts

##### Implementation Details

Technical aspects:
- Async operation handling
- Event-based architecture ('ready' event)
- Waterfall/Parallel execution patterns
- File handling capabilities

##### Additional Notes

- Async nature of operations
- Clear separation between base and extended functionality
- Error handling flows

Use a combination of:

- Interactive elements (for dependency graph features)
- Export to HTML (as shown in the project's graph feature)
- Zoom capabilities
- Clear text labels

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph Client
        CLI["Client Application"]:::client
        click CLI "https://github.com/ilya-pimenov/BigQueryBot/blob/master/example/main.coffee"
        EP["Entry Point"]:::client
        click EP "https://github.com/ilya-pimenov/BigQueryBot/blob/master/src/index.coffee"
    end

    subgraph Core["Core Components"]
        BQB["BigQueryBot Base Class"]:::core
        click BQB "https://github.com/ilya-pimenov/BigQueryBot/blob/master/src/big_query_bot.coffee"
        EBQB["ExtendedBigQueryBot Class"]:::extended
        click EBQB "https://github.com/ilya-pimenov/BigQueryBot/blob/master/src/extended_big_query_bot.coffee"
        UTIL["Utility Functions"]:::util
        click UTIL "https://github.com/ilya-pimenov/BigQueryBot/blob/master/src/misc.coffee"
    end

    subgraph Processing["Job Processing"]
        QE["Query Executor"]:::process
        LH["Load Job Handler"]:::process
        EH["Extract Job Handler"]:::process
        PC["Parallel Controller"]:::process
        click PC "https://github.com/ilya-pimenov/BigQueryBot/blob/master/graphs/parallel.html"
    end

    subgraph Visualization["Visualization Components"]
        DG["Dependency Graph"]:::viz
        click DG "https://github.com/ilya-pimenov/BigQueryBot/tree/master/src/views/dependency/"
        DGJ["Graph JavaScript"]:::viz
        click DGJ "https://github.com/ilya-pimenov/BigQueryBot/blob/master/src/views/dependency/d3-graph.js"
        DGC["Graph Styling"]:::viz
        click DGC "https://github.com/ilya-pimenov/BigQueryBot/blob/master/src/views/dependency/style.css"
        DGT["Graph Template"]:::viz
        click DGT "https://github.com/ilya-pimenov/BigQueryBot/blob/master/src/views/dependency/template.html"
        REP["Report Generation"]:::viz
        click REP "https://github.com/ilya-pimenov/BigQueryBot/blob/master/src/views/report.jade"
    end

    subgraph External["External Services"]
        BQ["Google BigQuery API"]:::external
        GCS["Google Cloud Storage"]:::external
        MAN["Mandrill API"]:::external
        S3["AWS S3"]:::external
    end

    %% Connections
    CLI --> EP
    EP --> BQB
    BQB --> EBQB
    BQB --> UTIL
    EBQB --> QE
    EBQB --> LH
    EBQB --> EH
    EBQB --> PC
    
    QE --> BQ
    LH --> BQ
    EH --> BQ
    PC --> BQ
    
    EBQB --> DG
    DG --> DGJ
    DG --> DGC
    DG --> DGT
    
    EBQB --> REP
    EBQB --> MAN
    EBQB --> S3
    EBQB --> GCS

    %% Styles
    classDef core fill:#2196F3,stroke:#1565C0,color:white
    classDef extended fill:#FF9800,stroke:#F57C00,color:white
    classDef external fill:#4CAF50,stroke:#388E3C,color:white
    classDef process fill:#9C27B0,stroke:#7B1FA2,color:white
    classDef util fill:#757575,stroke:#616161,color:white
    classDef viz fill:#00BCD4,stroke:#0097A7,color:white
    classDef client fill:#FFC107,stroke:#FFA000,color:white
```
