How picahq/pica works
Main Components
• API Service: This is the primary REST API (found under the “api” directory) which includes domain logic, middleware for authentication/rate limiting, and routing endpoints. It is the entry point for most external and internal requests.
• Archiver: A service (located in “archiver”) dedicated to processing and storing events/logs. It handles different event states (started, completed, failed, etc.), interfacing with storage (such as Google Cloud).
• Database Service: Located in “database”, this component manages the persistence layer (using PostgreSQL and possibly custom initialization logic) and provides a storage abstraction for connections and configurations.
• Cache: The “cache” module implements local and remote caching functions to speed up retrieval of frequently accessed data.
• OAuth Service: Found in the “oauth” directory, this Node/TypeScript service manages authentication and API token refreshing for the 100+ API integrations supported by OneTool.
• Osentities: The “osentities” component is a central module for managing domain entities (e.g., connections, secrets, schemas, configurations). It contains the core logic for data models and their transformations.
• Unified: A service that appears to provide a unified interface or adapter layer (possibly for runtime integrations or cross-module communications).
• CLI: The “cli” tool acts as the command-line interface for developers to interact with the infrastructure (e.g., to initialize configuration, run the server, seed data, etc.).
• Watchdog: A monitoring/health-check service (located in “watchdog”) to supervise system performance and service statuses.
• Resources & Deployment: Docker Compose files, Dockerfiles, and Kubernetes helper scripts indicate that containerization and orchestration (using Docker and possibly k8s) are part of the deployment infrastructure.
Relationships and Interactions
• External Clients (frontends, dashboards, or CLI) communicate with the API.
• The API service directly interacts with the Database service to read/write configurations and persist agent actions.
• The API and OSentities core modules exchange data; OSentities encapsulate the underlying domain models used by other services.
• The OAuth service works either as a standalone microservice or is invoked by the API for authentication flows and token refresh operations.
• The Archiver collects events and logs sent by the API and potentially other services.
• The Cache module is utilized by several services (API, OSentities) to optimize response times.
• The CLI tool serves as a developer interface to trigger commands that interact with these core services.
• Watchdog monitors system activities, ensuring system components (API, Database, OAuth, etc.) are healthy, and can trigger alerts if issues are detected.
• Containerization (Dockerfiles, Docker Compose) ties everything together in a production/development environment, showing how isolated services communicate on an orchestrated network layer.
Architectural Patterns and Principles
• Microservices Architecture: Separate services (API, archiver, database, oauth, etc.) each serve a specific domain role.
• Clear Separation of Concerns: Business logic, data persistence, authentication, caching, and monitoring are decoupled.
• Containerization and Orchestration: Deployment is managed via Docker and Docker Compose, with possible orchestration hints from Kubernetes helper files.
• Polyglot Persistence & Development: While most services are implemented in Rust, the OAuth service leverages Node/TypeScript, showing intentional technology choices for specific workloads.
Technologies, Frameworks, and Tools
• Languages: Rust (for most services), TypeScript/Node.js (for OAuth service).
• Containerization: Docker, Docker Compose (multiple Dockerfiles across services).
• CI/CD: GitHub workflows under the “.github/workflows” directory.
• Deployment: Kubernetes (as suggested by k8s driver in API) and Docker orchestration.
• External Dependencies: Integration with third-party APIs via OneTool, which connects to 100+ APIs and tools.
• CI Tools: Linting, testing, merge, and seeding scripts – these indicate emphasis on continuous integration.
Specific Elements
• Use clear labels for each service component (e.g., “API Service”, “Archiver”, “Database Service”, “OAuth Service”, “Cache”, “Osentities”, “Unified”, “CLI”, “Watchdog”).
• Don’t forget to illustrate CI/CD pipelines as part of development and deployment (represented using the GitHub workflows).