How intertwin-eu/dcnios works
Project type and purpose
• DCNiOS is an open-source, Python-based command-line tool (CLI) whose purpose is to automate the creation and deployment of event-driven data processing flows in Apache NiFi. It connects storage systems (dCache, S3, Kafka, etc.) to an OSCAR cluster, handling sources, transformations (“alterations”), and destinations.
File structure
• Top level:
• cli/ – the core Python CLI code
– dcnios-cli.py (entry point)
– apis/ (HTTP wrappers: NiFi, AWS, auxiliary)
– sources/, alterations/, destinations/ (module definitions + templates)
– template/ (JSON skeletons for new flows)
– description.yaml, env.py
• docker/ – Dockerfile, helper scripts, simple-client demo
• docpage/ – Docusaurus-based documentation site
• .github/ – CI workflows (deploy documentation)
• requirements.txt, CODE_OF_CONDUCT, CONTRIBUTING, LICENSE
• This indicates a single-repo CLI tool, with modular subpackages for each flow component, plus docs and containerization support.
README for additional insights
• Confirms the workflow:
a. User writes a YAML spec (using CLI)
b. CLI issues HTTP calls to a NiFi cluster to assemble processors and connections
c. NiFi captures events from sources, applies alterations, enqueues them, then dispatches to OSCAR
d. OSCAR processes data via containerized jobs
• Technology stack: Python, Apache NiFi, HTTP REST APIs, OSCAR, AWS SDKs.
a. Main components
• DCNiOS CLI (local machine or CI)
• Apache NiFi cluster
• External data sources (dCache, S3, Kafka, SQS)
• NiFi dataflow (Sources → Alterations → Internal Queue → Destinations)
• OSCAR cluster (serverless computing)
• Optional: Infrastructure Manager (IM) / Kubernetes for NiFi deployment
• Documentation site (docpage) and Docker support (auxiliary)
b. Relationships & interactions
1. User invokes DCNiOS CLI with a YAML description
2. CLI → NiFi REST API: creates Process Groups, templates, controller services
3. NiFi → Source connectors: polls dCache/Kafka/S3/SQS
4. Within NiFi: data passes through Alteration processors (encode, decode, merge)
5. NiFi enqueues events, then NiFi → OSCAR REST API: submits processing jobs
6. OSCAR cluster schedules and runs containerized tasks
7. CLI can also orchestrate AWS resources or IM for Kubernetes if required
8. Docker folder illustrates how to build an alternative NiFi client image
c. Architectural patterns & principles
• Event-driven, pipeline pattern
• Modular plugin architecture: sources/alterations/destinations
• Infrastructure as code: YAML-driven flow creation
• Separation of concerns: CLI vs. runtime (NiFi) vs. compute (OSCAR)
d. Key technologies/frameworks
• Python 3.7+ (cli code)
• Apache NiFi REST API
• AWS SDK (boto3 via aws.py)
• OSCAR Python client
• Docker for packaging
• Docusaurus for documentation
Detail without overthinking
• Break NiFi cluster into its logical subcomponents (sources, alter, queue, dest)
• Templates and YAML spec used by CLI as an input artifact