# turbot/tailpipe architecture

> Architecture diagram of the GitHub repository turbot/tailpipe, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/turbot/tailpipe
- Repository: https://github.com/turbot/tailpipe

## Overview

##### Project type and purpose
- Tailpipe is a Go‑based, command‑line application for collecting, storing, and querying log data locally.
- It uses a plugin system to ingest data from external cloud sources (AWS, Azure, GCP), writes data in Parquet format, and runs interactive SQL queries over DuckDB.

##### File structure
- **cmd/**: Cobra‑based CLI commands (collect, query, plugin, etc.)
- **internal/**: core packages organized by responsibility:
- cmdconfig – command‐specific configuration hooks and runtime mappings
- collector – orchestrates data collection and status reporting
- config – loads HCL config (connections, sources, partitions, formats)
- database – DuckDB wrapper, schema, partitions, Parquet integration
- parquet – conversion, compaction, file management
- file_watcher – monitors files for changes during collection
- interactive – interactive query REPL, autocomplete, query history
- – formats and prints tables, partitions, plugins, sources
- plugin – plugin manager for installing, loading, describing extensions
- metaquery – “help” and autocomplete handlers for custom meta‑commands
- parse – HCL‑based config parser and diagnostics
- logger – structured logging within the application
- ociinstaller – handles OCI‑based plugin downloads
- query – SQL execution over DuckDB
- **scripts/**: installer shell script
- **tests/**: acceptance and unit tests

##### README for insights
- Emphasizes a local, in‑process analytics stack: Parquet + DuckDB.
- Plugins extend connectivity to cloud sources.
- Two primary user flows:
a) `tailpipe collect <table>` – ingest logs into Parquet + DuckDB
b) `tailpipe query` – interactive or one‑off SQL queries

a. Main components
1. CLI Front‑end (Cobra)
2. Configuration Loader (HCL parser)
3. Plugin Manager (OCI installer, local plugin registry)
4. Collector Pipeline
- Source Connectors (AWS/Azure/GCP plugins)
- File Watcher
- Parquet Converter & Compactor
5. Local Storage
- Parquet files (on local disk)
- DuckDB embedded database
6. Query Engine
- SQL parser & executor (DuckDB wrapper)
- Interactive TUI (autocomplete, history)
7. Formatter (tabular output)
8. External Systems
- Cloud APIs (e.g. AWS S3, CloudWatch)
- OCI registries for plugins

b. Relationships & data flow
1. **User** invokes **CLI Front‑end** → hands off to **Command Dispatcher**
2. CLI reads HCL via **Configuration Loader** → determines connection & partition spec
3. For “collect”:
- **Plugin Manager** loads appropriate source plugin → establishes **Source Connector**
- **Collector** pulls log events → writes raw data to temp storage
- **File Watcher** detects new files → triggers **Parquet Converter**
- **Parquet Converter** writes optimized Parquet files → notifies **DuckDB** to attach tables/partitions
4. For “query”:
- **Interactive TUI** captures SQL → hands off to **Query Engine**
- **Query Engine** sends plan to **DuckDB** instance → retrieves resultset
- **Display Formatter** renders the resulttable to the terminal
5. Plugin installation:
- **CLI Front‑end** → **OCI Installer** → downloads & registers plugin binaries

c. Architectural patterns & design principles
- **Modular separation**: each internal package has a single responsibility
- **Plug‑in architecture**: decouples source connectors and CLI extensions via OCI artifacts
- **Pipeline pattern**: collector → file watcher → converter → database
- **Embedded DB**: in‑process DuckDB for zero‑configuration analytics

d. Key technologies & frameworks
- Go + Cobra (CLI)
- HCL (config language)
- Parquet (storage format)
- DuckDB (in‑memory/local analytics)
- OCI registries (plugin distribution)
- Bats (acceptance testing)

##### Detail is king
- Plugin manager separately, linking it to both CLI and collector
- Config loader and error/diagnostic flows (optional) for completeness

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Actors and Systems
    User["User"]:::external
    CloudAPIs["Cloud APIs (AWS, Azure, GCP)"]:::external
    OCIRegistry["OCI Registry"]:::external

    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        CLI["CLI Front-end (Cobra)"]:::internal
        Config["Configuration Loader (HCL)"]:::internal
        PluginMgr["Plugin Manager"]:::internal
        OCIInst["OCI Installer"]:::internal
    end

    %% Ingestion Pipeline
    subgraph "Ingestion Pipeline"
        direction TB
        Collector["Collector Pipeline"]:::internal
        SourceConn["Source Connector"]:::internal
        FileWatcher["File Watcher"]:::internal
        ParquetConv["Parquet Converter & Compactor"]:::internal
    end

    %% Query Pipeline
    subgraph "Query Pipeline"
        direction TB
        TUI["Interactive TUI"]:::internal
        QueryEng["Query Engine (DuckDB)"]:::internal
        DisplayFmt["Display Formatter"]:::internal
    end

    %% Storage Layer
    subgraph "Local Storage"
        direction TB
        ParquetStore["Parquet Files"]:::storage
        DuckDBStore["DuckDB Embedded DB"]:::storage
    end

    %% Relationships
    User -->|"runs collect/query"| CLI
    CLI -->|"loads config"| Config
    CLI -->|"manages plugins"| PluginMgr
    CLI -->|"installs plugins"| OCIInst

    PluginMgr -->|"loads plugin"| SourceConn
    OCIInst -->|"downloads plugin"| OCIRegistry

    CLI -->|"collect command"| Collector
    Collector -->|"uses source"| SourceConn
    SourceConn -->|"fetch logs"| CloudAPIs
    Collector -->|"writes temp"| FileWatcher
    FileWatcher -->|"on new file"| ParquetConv
    ParquetConv -->|"writes parquet"| ParquetStore
    ParquetConv -->|"attach partition"| DuckDBStore

    CLI -->|"query command"| TUI
    TUI -->|"executes SQL"| QueryEng
    QueryEng -->|"reads data"| DuckDBStore
    QueryEng -->|"returns result"| DisplayFmt
    DisplayFmt -->|"shows results"| User

    %% Click Events
    click CLI "https://github.com/turbot/tailpipe/blob/develop/cmd/root.go"
    click Config "https://github.com/turbot/tailpipe/blob/develop/internal/config/load_config.go"
    click Config "https://github.com/turbot/tailpipe/blob/develop/internal/parse/load_config.go"
    click PluginMgr "https://github.com/turbot/tailpipe/blob/develop/internal/plugin/plugin_manager.go"
    click OCIInst "https://github.com/turbot/tailpipe/blob/develop/internal/ociinstaller/mediatypes.go"
    click Collector "https://github.com/turbot/tailpipe/blob/develop/internal/collector/collector.go"
    click SourceConn "https://github.com/turbot/tailpipe/blob/develop/internal/config/source.go"
    click FileWatcher "https://github.com/turbot/tailpipe/blob/develop/internal/file_watcher/connection_watcher.go"
    click ParquetConv "https://github.com/turbot/tailpipe/blob/develop/internal/parquet/convertor.go"
    click ParquetConv "https://github.com/turbot/tailpipe/blob/develop/internal/parquet/compact.go"
    click ParquetStore "https://github.com/turbot/tailpipe/blob/develop/internal/filepaths/parquet.go"
    click DuckDBStore "https://github.com/turbot/tailpipe/blob/develop/internal/database/duck_db.go"
    click QueryEng "https://github.com/turbot/tailpipe/blob/develop/internal/query/execute.go"
    click TUI "https://github.com/turbot/tailpipe/blob/develop/internal/interactive/run.go"
    click DisplayFmt "https://github.com/turbot/tailpipe/blob/develop/internal/display/format.go"

    %% Styling
    classDef internal fill:#cce5ff,stroke:#004085
    classDef external fill:#d4edda,stroke:#155724
    classDef storage fill:#fff3cd,stroke:#856404
```
