How mikekenneth/duckdb_streamlit_arrow_flight works
Main Components
a. Presentation Layer (Client/Web UI)
• Streamlit app (apps/client/web.py) – renders the web interface.
b. API Layer (Flight RPC Server)
• Flight server (apps/server/server.py) – listens on gRPC port, handles incoming Flight RPC calls.
• Query Executor – inside the server module, translates Flight calls into DuckDB SQL execution.
c. Data Layer
• DuckDB file (data/duck.db) – primary relational store.
• (Future) S3/minio endpoint abstraction.
d. Configuration & Orchestration
• .env / example.env – runtime settings (host, port, storage type, file paths).
• docker-compose.yaml – spins up server, client (future), and optional storage services.
Relationships & Data Flow
1. User opens Streamlit UI in browser.
2. UI calls Flight Client library over gRPC to the Flight Server.
3. Flight Server receives RPC, forwards SQL to DuckDB engine.
4. DuckDB executes query against data/duck.db.
6. Server streams flight-ready result back over gRPC.
Architectural Patterns & Technologies
• Data Persistence: Embedded DuckDB file + pluggable object storage.
• UI Framework: Streamlit for rapid prototyping.
• Python modules organized under apps/client and apps/server.
• Presentation (Streamlit UI)
• API/Logic (Flight RPC Client ↔ Flight RPC Server + Query Executor)
• Storage (DuckDB file + Datasets directory / S3)
• “Streamlit Web UI” (port 8501)
• “Flight RPC Client”
• “gRPC Channel”
• “Flight RPC Server”
• “DuckDB Engine”
• “Result Storage (local/S3)”
• UI → Flight Client → gRPC → Flight Server
• Flight Server → DuckDB Engine
• Flight Server ↔ Storage
• Flight Server → gRPC → Flight Client → UI
• Blue for UI components
• Green for API/server components
• Orange for storage components