# intertwin-eu/dcnios architecture

> Architecture diagram of the GitHub repository intertwin-eu/dcnios, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/intertwin-eu/dcnios
- Repository: https://github.com/intertwin-eu/dcnios

## Overview

##### Project type and purpose
- DCNiOS is an open-source, Python-based command-line tool (CLI) whose purpose is to automate the creation and deployment of event-driven data processing flows in Apache NiFi. It connects storage systems (dCache, S3, Kafka, etc.) to an OSCAR cluster, handling sources, transformations (“alterations”), and destinations.

##### File structure
- Top level:
- cli/ – the core Python CLI code
– dcnios-cli.py (entry point)
– apis/ (HTTP wrappers: NiFi, AWS, auxiliary)
– sources/, alterations/, destinations/ (module definitions + templates)
– template/ (JSON skeletons for new flows)
– description.yaml, env.py
- docker/ – Dockerfile, helper scripts, simple-client demo
- docpage/ – Docusaurus-based documentation site
- .github/ – CI workflows (deploy documentation)
- requirements.txt, CODE_OF_CONDUCT, CONTRIBUTING, LICENSE
- This indicates a single-repo CLI tool, with modular subpackages for each flow component, plus docs and containerization support.

##### README for additional insights
- Confirms the workflow:
a. User writes a YAML spec (using CLI)
b. CLI issues HTTP calls to a NiFi cluster to assemble processors and connections
c. NiFi captures events from sources, applies alterations, enqueues them, then dispatches to OSCAR
d. OSCAR processes data via containerized jobs
- Technology stack: Python, Apache NiFi, HTTP REST APIs, OSCAR, AWS SDKs.

a. Main components
- DCNiOS CLI (local machine or CI)
- Apache NiFi cluster
- External data sources (dCache, S3, Kafka, SQS)
- NiFi dataflow (Sources → Alterations → Internal Queue → Destinations)
- OSCAR cluster (serverless computing)
- Optional: Infrastructure Manager (IM) / Kubernetes for NiFi deployment
- Documentation site (docpage) and Docker support (auxiliary)
b. Relationships & interactions
1. User invokes DCNiOS CLI with a YAML description
2. CLI → NiFi REST API: creates Process Groups, templates, controller services
3. NiFi → Source connectors: polls dCache/Kafka/S3/SQS
4. Within NiFi: data passes through Alteration processors (encode, decode, merge)
5. NiFi enqueues events, then NiFi → OSCAR REST API: submits processing jobs
6. OSCAR cluster schedules and runs containerized tasks
7. CLI can also orchestrate AWS resources or IM for Kubernetes if required
8. Docker folder illustrates how to build an alternative NiFi client image
c. Architectural patterns & principles
- Event-driven, pipeline pattern
- Modular plugin architecture: sources/alterations/destinations
- Infrastructure as code: YAML-driven flow creation
- Separation of concerns: CLI vs. runtime (NiFi) vs. compute (OSCAR)
d. Key technologies/frameworks
- Python 3.7+ (cli code)
- Apache NiFi REST API
- AWS SDK (boto3 via aws.py)
- OSCAR Python client
- Docker for packaging
- Docusaurus for documentation

##### Detail without overthinking
- Break NiFi cluster into its logical subcomponents (sources, alter, queue, dest)
- Templates and YAML spec used by CLI as an input artifact

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Systems
    subgraph "External Systems"
        direction TB
        dCacheExt["dCache"]:::external
        S3Ext["S3"]:::external
        KafkaExt["Kafka"]:::external
        SQSExt["SQS"]:::external
    end

    %% DCNiOS CLI Layer
    subgraph "DCNiOS CLI" 
        direction TB
        cliEntr["dcnios-cli.py"]:::control
        cliEnv["env.py"]:::control
        cliDesc["description.yaml"]:::control

        subgraph "APIs"
            direction TB
            wrapNifi["nifiManagment.py"]:::control
            wrapAWS["aws.py"]:::control
            wrapAux["auxiliaryFunctions.py"]:::control
        end

        subgraph "Sources Module"
            direction TB
            dcache["dcache.py"]:::control
            kafka["kafka.py"]:::control
            s3sqs["s3sqs.py"]:::control
            genericSrc["generic.py"]:::control
        end

        subgraph "Alterations Module"
            direction TB
            alterProc["alteration.py"]:::control
        end

        subgraph "Destinations Module"
            direction TB
            oscarDest["oscar.py"]:::control
            s3Dest["s3aws.py"]:::control
        end

        subgraph "Template Definitions"
            direction TB
            srcT["sources/"]:::control
            altT["alterations/"]:::control
            destT["destinations/"]:::control
        end
    end

    %% NiFi Cluster
    subgraph "NiFi Cluster" 
        direction TB
        nifiSources["Sources"]:::data
        nifiAlter["Alterations"]:::data
        nifiQueue["Internal Queue"]:::data
        nifiDest["Destinations"]:::data
    end

    %% OSCAR Cluster
    OSCAR["OSCAR Cluster"]:::control

    %% Connections
    cliEntr -->|"reads YAML spec"| cliDesc
    cliEntr -->|"loads env config"| cliEnv
    cliEntr -->|"calls NiFi REST API"| wrapNifi
    wrapNifi -->|"HTTP REST"| nifiSources

    dCacheExt -->|"polling"| nifiSources
    KafkaExt -->|"polling"| nifiSources
    S3Ext -->|"polling"| nifiSources
    SQSExt -->|"polling"| nifiSources

    nifiSources -->|"data stream"| nifiAlter
    nifiAlter -->|"data stream"| nifiQueue
    nifiQueue -->|"data stream"| nifiDest
    nifiDest -->|"submit jobs via REST"| OSCAR

    wrapAWS -.->|"orchestrate infra"| OSCAR

    %% Click Events
    click cliEntr "https://github.com/intertwin-eu/dcnios/blob/main/cli/dcnios-cli.py"
    click cliEnv "https://github.com/intertwin-eu/dcnios/blob/main/cli/env.py"
    click cliDesc "https://github.com/intertwin-eu/dcnios/blob/main/cli/description.yaml"
    click wrapNifi "https://github.com/intertwin-eu/dcnios/blob/main/cli/apis/nifiManagment.py"
    click wrapAWS "https://github.com/intertwin-eu/dcnios/blob/main/cli/apis/aws.py"
    click wrapAux "https://github.com/intertwin-eu/dcnios/blob/main/cli/apis/auxiliaryFunctions.py"
    click dcache "https://github.com/intertwin-eu/dcnios/blob/main/cli/sources/dcache.py"
    click kafka "https://github.com/intertwin-eu/dcnios/blob/main/cli/sources/kafka.py"
    click s3sqs "https://github.com/intertwin-eu/dcnios/blob/main/cli/sources/s3sqs.py"
    click genericSrc "https://github.com/intertwin-eu/dcnios/blob/main/cli/sources/generic.py"
    click alterProc "https://github.com/intertwin-eu/dcnios/blob/main/cli/alterations/alteration.py"
    click oscarDest "https://github.com/intertwin-eu/dcnios/blob/main/cli/destinations/oscar.py"
    click s3Dest "https://github.com/intertwin-eu/dcnios/blob/main/cli/destinations/s3aws.py"
    click srcT "https://github.com/intertwin-eu/dcnios/tree/main/cli/template/sources/"
    click altT "https://github.com/intertwin-eu/dcnios/tree/main/cli/template/alterations/"
    click destT "https://github.com/intertwin-eu/dcnios/tree/main/cli/template/destinations/"

    %% Styles
    classDef control fill:#a3d5ff,stroke:#000,stroke-width:1px
    classDef data fill:#a3ffae,stroke:#000,stroke-width:1px
    classDef external fill:#d3d3d3,stroke:#000,stroke-width:1px
```
