How turbot/tailpipe works
Project type and purpose
• Tailpipe is a Go‑based, command‑line application for collecting, storing, and querying log data locally.
• It uses a plugin system to ingest data from external cloud sources (AWS, Azure, GCP), writes data in Parquet format, and runs interactive SQL queries over DuckDB.
File structure
• cmd/: Cobra‑based CLI commands (collect, query, plugin, etc.)
• internal/: core packages organized by responsibility:
• cmdconfig – command‐specific configuration hooks and runtime mappings
• collector – orchestrates data collection and status reporting
• config – loads HCL config (connections, sources, partitions, formats)
• database – DuckDB wrapper, schema, partitions, Parquet integration
• parquet – conversion, compaction, file management
• file_watcher – monitors files for changes during collection
• interactive – interactive query REPL, autocomplete, query history
• – formats and prints tables, partitions, plugins, sources
• plugin – plugin manager for installing, loading, describing extensions
• metaquery – “help” and autocomplete handlers for custom meta‑commands
• parse – HCL‑based config parser and diagnostics
• logger – structured logging within the application
• ociinstaller – handles OCI‑based plugin downloads
• query – SQL execution over DuckDB
• scripts/: installer shell script
• tests/: acceptance and unit tests
README for insights
• Emphasizes a local, in‑process analytics stack: Parquet + DuckDB.
• Plugins extend connectivity to cloud sources.
• Two primary user flows:
a) tailpipe collect <table> – ingest logs into Parquet + DuckDB
b) tailpipe query – interactive or one‑off SQL queries
a. Main components
1. CLI Front‑end (Cobra)
2. Configuration Loader (HCL parser)
3. Plugin Manager (OCI installer, local plugin registry)
4. Collector Pipeline
• Source Connectors (AWS/Azure/GCP plugins)
• File Watcher
• Parquet Converter & Compactor
5. Local Storage
• Parquet files (on local disk)
• DuckDB embedded database
6. Query Engine
• SQL parser & executor (DuckDB wrapper)
• Interactive TUI (autocomplete, history)
7. Formatter (tabular output)
8. External Systems
• Cloud APIs (e.g. AWS S3, CloudWatch)
• OCI registries for plugins
b. Relationships & data flow
1. User invokes CLI Front‑end → hands off to Command Dispatcher
2. CLI reads HCL via Configuration Loader → determines connection & partition spec
3. For “collect”:
• Plugin Manager loads appropriate source plugin → establishes Source Connector
• Collector pulls log events → writes raw data to temp storage
• File Watcher detects new files → triggers Parquet Converter
• Parquet Converter writes optimized Parquet files → notifies DuckDB to attach tables/partitions
4. For “query”:
• Interactive TUI captures SQL → hands off to Query Engine
• Query Engine sends plan to DuckDB instance → retrieves resultset
• Display Formatter renders the resulttable to the terminal
5. Plugin installation:
• CLI Front‑end → OCI Installer → downloads & registers plugin binaries
c. Architectural patterns & design principles
• Modular separation: each internal package has a single responsibility
• Plug‑in architecture: decouples source connectors and CLI extensions via OCI artifacts
• Pipeline pattern: collector → file watcher → converter → database
• Embedded DB: in‑process DuckDB for zero‑configuration analytics
d. Key technologies & frameworks
• Go + Cobra (CLI)
• HCL (config language)
• Parquet (storage format)
• DuckDB (in‑memory/local analytics)
• OCI registries (plugin distribution)
• Bats (acceptance testing)
Detail is king
• Plugin manager separately, linking it to both CLI and collector
• Config loader and error/diagnostic flows (optional) for completeness