How javfg/pis works
Main Components
a) Core Pipeline Components:
• Configuration Management System
• Task Registry
• Step Runner
• Storage System
• Manifest Manager
• Validation System
b) Task Execution Flow:
• Pre-tasks Phase
• Main Tasks Phase
• Validation Phase
• Upload Phase
┌─ Input Layer ─────┐ ┌─ Processing Layer ──┐ ┌─ Storage Layer ──┐
• CLI │ │ - Step Runner │ │ - Google Storage │
• Config Parser │ │ - Task Registry │ │ - Local Storage │
• ENV Variables │ │ - Task Executor │ └─────────────────┘
• ┘ │ - Validators │
• ┘
Component Details
a) Core Components (Rectangles):
• Configuration System (src/pis/config/)
• Task Registry (src/pis/task/task_registry.py)
• Step Runner (src/pis/step/)
• Storage Handlers (src/pis/storage/)
b) Task Types (Rounded Rectangles):
• Pre-tasks
• Main Tasks
• Validators
• Configuration Flow
• Task Execution Flow
• Data Movement
• Validation Results
Specific Implementation Details
Key architectural elements:
┌─ Task System ──────────────┐
• ┌─ Base Task ───────────┐ │
• run() │ │
• validate() │ │
• upload() │ │
• ┘ │
• ┌─ Concrete Tasks ──────┐ │
• Download │ │
• ElasticSearch │ │
• Explode │ │
• ┘ │
• ┘
Important Relationships
• Configuration flow from CLI/ENV/YAML to Task Registry
• Task Registry to Step Runner relationship
• Step Runner to Storage System interaction
• Manifest generation and management flow
• Validation system integration points
Additional Elements
Supporting components:
• Logging System (src/pis/util/logger.py)
• Error Handling (src/pis/util/errors.py)
• File System Utilities (src/pis/util/fs.py)
Execution Flow Representation
Four-phase execution process:
[Init] → [Staging] → [Validation] → [Upload]
• ↓ ↓ ↓ ↓
[Pretasks] [Tasks] [Validators] [Storage]
Notes for Implementation
• Keep the parallel execution nature visible
• Manifest file generation and management
• Integration points with external services
• Error handling and validation flows
• Configuration inheritance and override patterns
Remember to maintain a balance between detail and clarity, focusing on the pipeline nature of the system while showing the parallel execution capabilities and the four-phase approach to task execution.