How nike-inc/koheesio works
Project Type and Purpose
• Koheesio is a Python framework designed to build efficient data pipelines. Its primary focus is on modular, testable data tasks that serve as building blocks for larger data processing workflows.
File Structure
• The top-level directories include standard configuration and docs folders (e.g., .github, docs, tests, etc.). The main implementation lives under the “src/koheesio” directory.
• Core modules like “context.py”, “logger.py”, and “steps” (which contain individual task implementations).
• Specific submodules for “asyncio”, “integrations”, “pandas”, “secrets”, “spark”, and “sso” that indicate integrations with external systems and specialized processing.
• The “spark” directory is detailed with readers, writers, transformations, and utilities, which means heavy emphasis on handling Spark operations.
• The “tests” directory also mirrors many modules, ensuring well-covered test cases and further indicating the importance of modular, testable units.
a. Main Components to Include:
• Core Framework Components:
– Step: Representing the individual task elements.
– Context: Providing the execution and configuration environment.
– Logger: For logging and observability.
• Integration Modules:
– Specific Spark integrations, including modules for readers, writers, transformations, delta support, and utilities.
– Additional specialized modules for async operations (e.g., the “asyncio” directory) as well as “pandas” readers.
• External Dependencies & Extras:
– Mention Pydantic (for configuration and type-safety as highlighted in the README).
– Include any external systems such as external data sources, cloud services, or data warehouses that the integrations target.
b. Relationships and Interactions:
• Directional data flow from the Step components toward integration processes. For example:
– Steps invoke methods on the Context and Logger for configuration and logging.
• How the “asyncio” and “sso” components interact with other modules if asynchronous or security-based tasks are required.
c. Architectural Patterns and Design Principles:
• Modularity: Each component (e.g., steps, integrations, utilities) is self-contained and reusable.
• Strong Typing/Validation: Indicate the use of Pydantic for configurations.
• Integration-Oriented Design: Show how the framework is designed to plug into multiple external systems without being a full-fledged orchestrator.
d. Relevant Technologies:
• Python plays the central role.
• Pydantic is used for settings and validation.
• The testing framework (pytest) and CI/CD workflow tools (GitHub Actions) can be mentioned as peripheral but important deployment/test support elements.
Be Detailed
• You capture all primary areas, including the deep modular structure under “src/koheesio” (like the detailed Spark handling capabilities).