# diracgrid/dirac architecture

> Architecture diagram of the GitHub repository diracgrid/dirac, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/diracgrid/dirac
- Repository: https://github.com/diracgrid/dirac

## Overview

##### Main Components
- Client/Portal Layer – Representing the user interfaces (e.g., command‐line tools, web portal, REST APIs) that submit jobs and manage workflows.
- Core Services – The heart of DIRAC, which includes:
– The DIRAC Core components (found as “src/DIRAC/Core”) handling common functionality such as networking, job submission, configuration, utility functions, and the Agent/Client frameworks.
– The FrameworkSystem where many of the administrative and management tasks are implemented.
- AccountingSystem – Responsible for tracking resource usage, managing reports, and handling fee/account data.
- ConfigurationSystem – Managing site and resource configuration; it includes agents for updating configurations and synchronizing external data.
- DataManagementSystem (DMS) – Handling file catalogues, data replication, and transfer operations.
- MonitoringSystem – Responsible for sending and receiving monitoring data for job and resource status; includes modules for data operation monitoring.
- ProductionSystem – Governing production workflows, including job steps and validation.
- RequestManagementSystem – Handling job requests, scheduling, and operations on job states.
- ResourceStatusSystem – Monitoring and managing site and resource statuses through various policies (e.g., downtime, free disk space).
- TransformationSystem – Supporting transformation workflows, including file recovery, task management, and output validation.
- WorkloadManagementSystem – Managing pilot jobs, job agents, scheduling and execution, integration with pilot wrappers and remote execution.
- Agents, Clients, and Scripts – Many directories are dedicated to agent implementations (e.g., Monitoring agents, Pilot agents, etc.) and a wide range of client scripts that are used for operational commands. Represent these as secondary components that interact both with their parent subsystem services and the core messaging/infrastructure.

- How the Clients interface with the Core and subsequently access the various subsystems and services over defined protocols (often over REST or via messaging systems such as MQ components).
– From user/job submission clients to the WorkloadManagementSystem.
– Between the various Agents (e.g., Pilot agents, RequestExecuting agents, Configuration agents) and the Core management services.
– From the subsystems down to their underlying Database layers (for Accounting, File Catalogs, Monitoring DBs, etc.) and messaging queues.
- Connections to external systems and resource providers, showing that DIRAC creates a layer of abstraction over disparate computing elements, storage systems, and authentication/authorization services.

##### Architectural Patterns and Design Principles
- Modular and Layered Design – Emphasize that each subsystem is encapsulated as a relatively independent module under the broader DIRAC umbrella.
- Service-Oriented Architecture (SOA) – Many components (agents, clients, and services) interact asynchronously and are loosely coupled.
- Distributed Communication – Illustrate usage of messaging systems (MQ, Tornado-based REST servers) that facilitate distributed and remote execution.
- Database and Configuration Abstraction – Show that each subsystem contains its own DB components (noted by “DB” directories with SQL files) to store state information and operational metrics.

##### Relevant Technologies & Frameworks
- Python 3.9 – Make a note that the entire code base is written in Python using its ecosystem.
- Tornado – As seen in the “src/DIRAC/Core/Tornado” directories, used for asynchronous servers and potentially for RESTful interfaces.
- Messaging and Logging Systems – Mention use of message queues (MQ) and logging frameworks (standardLogging, etc.) for communication and monitoring.
- Databases – Various SQL-based DBs within subsystems (AccountingDB, FileCatalogDB, MonitoringDB, etc.) are integral to the system.
- Pre-commit and Testing Tools – Although not part of the runtime architecture, these tools reflect the design quality and integration testing integration.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client/Portal Layer
    subgraph "Client/Portal Layer"
        C1("Command-Line Tools & Web Portal"):::client
        C2("API Interfaces"):::client
    end

    %% Core Services Layer
    subgraph "Core Services"
        CR("DIRAC Core Components"):::core
        FS("Framework System"):::core
    end

    %% Subsystems and Services
    subgraph "Subsystems and Services"
        A("AccountingSystem"):::subsystem
        CONF("ConfigurationSystem"):::subsystem
        DMS("DataManagementSystem"):::subsystem
        MON("MonitoringSystem"):::subsystem
        PROD("ProductionSystem"):::subsystem
        REQ("RequestManagementSystem"):::subsystem
        RS("ResourceStatusSystem"):::subsystem
        SMS("StorageManagementSystem"):::subsystem
        TRANS("TransformationSystem"):::subsystem
        WLM("WorkloadManagementSystem"):::subsystem
    end

    %% Agents, Clients, and Scripts
    subgraph "Agents/Clients & Scripts"
        OPS("Operational Agents & Scripts"):::agent
    end

    %% External Resources and Databases
    subgraph "External Resources"
        EXDB("Databases & External Systems"):::external
    end

    %% Interactions
    C1 -->|"submitJob"| CR
    C2 -->|"apiCall"| CR
    CR -->|"controlFlow"| FS
    FS -->|"delegateTask"| A
    FS -->|"delegateTask"| CONF
    FS -->|"delegateTask"| DMS
    FS -->|"delegateTask"| MON
    FS -->|"delegateTask"| PROD
    FS -->|"delegateTask"| REQ
    FS -->|"delegateTask"| RS
    FS -->|"delegateTask"| SMS
    FS -->|"delegateTask"| TRANS
    FS -->|"delegateTask"| WLM

    %% Operational interactions  
    OPS -->|"notify"| FS
    OPS -->|"notify"| A
    OPS -->|"notify"| CONF

    %% Subsystems interacting with external DBs
    A -->|"dbSync"| EXDB
    CONF -->|"dbSync"| EXDB
    DMS -->|"dbSync"| EXDB
    MON -->|"dbSync"| EXDB
    PROD -->|"dbSync"| EXDB
    REQ -->|"dbSync"| EXDB
    RS -->|"dbSync"| EXDB
    SMS -->|"dbSync"| EXDB
    TRANS -->|"dbSync"| EXDB
    WLM -->|"dbSync"| EXDB

    %% Click Events for Client/Portal Layer
    click C1 "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/Interfaces"
    click C2 "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/Interfaces/API"

    %% Click Events for Core Services
    click CR "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/Core"
    click FS "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/FrameworkSystem"

    %% Click Events for Subsystems
    click A "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/AccountingSystem"
    click CONF "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/ConfigurationSystem"
    click DMS "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/DataManagementSystem"
    click MON "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/MonitoringSystem"
    click PROD "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/ProductionSystem"
    click REQ "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/RequestManagementSystem"
    click RS "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/ResourceStatusSystem"
    click SMS "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/StorageManagementSystem"
    click TRANS "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/TransformationSystem"
    click WLM "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/WorkloadManagementSystem"

    %% Click Event for Agents/Clients & Scripts
    click OPS "https://github.com/diracgrid/dirac/tree/integration/src/DIRAC/Core/scripts"

    %% Styling Classes
    classDef client fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef core fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef subsystem fill:#FFA500,stroke:#000,stroke-width:1px;
    classDef agent fill:#9370DB,stroke:#000,stroke-width:1px;
    classDef external fill:#D3D3D3,stroke:#000,stroke-width:1px;
```
