# arfin-parween/the_night_owl architecture

> Architecture diagram of the GitHub repository arfin-parween/the_night_owl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/arfin-parween/the_night_owl
- Repository: https://github.com/arfin-parween/the_night_owl

## Overview

NightOwl is an AI on-call engineer for a Spark ETL pipeline. The workflow runs or simulates a failure, records run and service evidence in OpenSearch, detects failed runs and wakes OpenClaw, then investigates through read-only CLI tools. The agent proposes an incident; a human approval is required before guarded config remediation, rerun, MTTR measurement and postmortem storage. README documents schema diff, change correlation, upstream logs, hybrid incident search and lineage impact. Some unsampled setup and tool internals are represented at their exact directory paths.

Main entry points: [`nightowl/cli.py`](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/cli.py) (NightOwl CLI), [`nightowl/os_client.py`](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/os_client.py) (OpenSearch Client), [`nightowl/watcher.py`](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/watcher.py) (Failure Watcher), [`nightowl/fixes.py`](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/fixes.py) (Guarded Fixes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_operations["Operations"]
  node_cli["NightOwl CLI<br/>[cli.py]"]
  node_settings["Runtime Settings<br/>[settings.py]"]
  node_setup["OpenSearch Setup<br/>[setup_os.py]"]
  node_watcher["Failure Watcher<br/>[watcher.py]"]
end

subgraph group_etl["ETL and Evidence"]
  node_scenarios["Failure Scenarios<br/>[scenarios.py]"]
  node_datagen["Synthetic Data<br/>[datagen.py]"]
  node_pipeline["Spark ETL<br/>[pipeline.py]"]
  node_eventlog["Structured Events<br/>[eventlog.py]"]
end

subgraph group_investigation["Investigation"]
  node_seed["Seeded Memory<br/>[seed_data.py]"]
  node_osclient["OpenSearch Client<br/>[os_client.py]"]
  node_tools["Investigation Tools<br/>[tools.py]"]
end

subgraph group_remediation["Remediation"]
  node_fixes["Guarded Fixes<br/>[fixes.py]"]
  node_pipelineconfig["Pipeline Config"]
end

subgraph group_integrations["External Integrations"]
  node_opensearch[("OpenSearch")]
  node_openclaw["OpenClaw Agent"]
  node_telegram["Telegram"]
end

node_operator(("Operator"))
node_human(("Approver"))

node_operator -->|"invokes"| node_cli
node_cli -->|"runs"| node_pipeline
node_cli -->|"arms"| node_scenarios
node_scenarios -->|"writes defects"| node_datagen
node_pipeline -->|"emits events"| node_eventlog
node_pipeline -->|"records runs"| node_opensearch
node_eventlog -->|"indexes events"| node_osclient
node_osclient -->|"queries and writes"| node_opensearch
node_cli -->|"sets up"| node_setup
node_setup -->|"configures indices"| node_opensearch
node_setup -->|"loads history"| node_seed
node_cli -->|"starts watching"| node_watcher
node_watcher -->|"polls failures"| node_osclient
node_watcher -->|"wakes agent"| node_openclaw
node_watcher -.->|"pings"| node_telegram
node_openclaw -->|"executes tools"| node_cli
node_cli -->|"dispatches queries"| node_tools
node_tools -->|"queries evidence"| node_osclient
node_tools -->|"investigates memory and lineage"| node_opensearch
node_openclaw -->|"reports RCA"| node_telegram
node_human -->|"approves incident"| node_openclaw
node_openclaw -->|"requests fix"| node_cli
node_cli -->|"dispatches apply"| node_fixes
node_fixes -->|"changes config"| node_pipelineconfig
node_fixes -->|"reruns"| node_pipeline
node_fixes -->|"stores postmortem"| node_osclient

click node_cli "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/cli.py"
click node_scenarios "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/scenarios.py"
click node_datagen "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/datagen.py"
click node_pipeline "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/pipeline.py"
click node_eventlog "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/eventlog.py"
click node_settings "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/settings.py"
click node_setup "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/setup_os.py"
click node_seed "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/seed_data.py"
click node_osclient "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/os_client.py"
click node_watcher "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/watcher.py"
click node_tools "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/tools.py"
click node_fixes "https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/fixes.py"
click node_pipelineconfig "https://github.com/arfin-parween/the_night_owl/blob/main/config/pipeline.default.yaml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_cli,node_settings,node_setup,node_watcher toneBlue
class node_scenarios,node_datagen,node_pipeline,node_eventlog toneAmber
class node_seed,node_osclient,node_tools toneMint
class node_fixes,node_pipelineconfig toneRose
class node_opensearch,node_openclaw,node_telegram,node_operator,node_human toneIndigo
```

## Components

### Operations

- [NightOwl CLI](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/cli.py): `nightowl/cli.py`
- [Runtime Settings](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/settings.py): `nightowl/settings.py`
- [OpenSearch Setup](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/setup_os.py): `nightowl/setup_os.py`
- [Failure Watcher](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/watcher.py): `nightowl/watcher.py`

### ETL and Evidence

- [Failure Scenarios](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/scenarios.py): `nightowl/scenarios.py`
- [Synthetic Data](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/datagen.py): `nightowl/datagen.py`
- [Spark ETL](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/pipeline.py): `nightowl/pipeline.py`
- [Structured Events](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/eventlog.py): `nightowl/eventlog.py`

### Investigation

- [Seeded Memory](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/seed_data.py): `nightowl/seed_data.py`
- [OpenSearch Client](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/os_client.py): `nightowl/os_client.py`
- [Investigation Tools](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/tools.py): `nightowl/tools.py`

### Remediation

- [Guarded Fixes](https://github.com/arfin-parween/the_night_owl/blob/main/nightowl/fixes.py): `nightowl/fixes.py`
- [Pipeline Config](https://github.com/arfin-parween/the_night_owl/blob/main/config/pipeline.default.yaml): `config/pipeline.default.yaml`

### External Integrations

- **OpenSearch**
- **OpenClaw Agent**
- **Telegram**

### Other components

- **Operator**
- **Approver**

## Connections

- Operator → NightOwl CLI: invokes (evidence: `README.md`)
- NightOwl CLI → Spark ETL: runs (evidence: `nightowl/cli.py`)
- NightOwl CLI → Failure Scenarios: arms (evidence: `nightowl/cli.py`)
- Failure Scenarios → Synthetic Data: writes defects (evidence: `nightowl/scenarios.py`)
- Spark ETL → Structured Events: emits events (evidence: `nightowl/pipeline.py`)
- Spark ETL → OpenSearch: records runs (evidence: `README.md`)
- Structured Events → OpenSearch Client: indexes events (evidence: `nightowl/eventlog.py`)
- OpenSearch Client → OpenSearch: queries and writes (evidence: `nightowl/os_client.py`)
- NightOwl CLI → OpenSearch Setup: sets up (evidence: `nightowl/cli.py`)
- OpenSearch Setup → OpenSearch: configures indices (evidence: `nightowl/setup_os.py`)
- OpenSearch Setup → Seeded Memory: loads history (evidence: `nightowl/setup_os.py`)
- NightOwl CLI → Failure Watcher: starts watching (evidence: `nightowl/cli.py`)
- Failure Watcher → OpenSearch Client: polls failures (evidence: `nightowl/watcher.py`)
- Failure Watcher → OpenClaw Agent: wakes agent (evidence: `nightowl/watcher.py`)
- Failure Watcher → Telegram: pings (evidence: `nightowl/watcher.py`)
- OpenClaw Agent → NightOwl CLI: executes tools (evidence: `README.md`)
- NightOwl CLI → Investigation Tools: dispatches queries (evidence: `nightowl/cli.py`)
- Investigation Tools → OpenSearch Client: queries evidence (evidence: `nightowl/tools.py`)
- Investigation Tools → OpenSearch: investigates memory and lineage (evidence: `README.md`)
- OpenClaw Agent → Telegram: reports RCA (evidence: `README.md`)
- Approver → OpenClaw Agent: approves incident (evidence: `README.md`)
- OpenClaw Agent → NightOwl CLI: requests fix (evidence: `README.md`)
- NightOwl CLI → Guarded Fixes: dispatches apply (evidence: `nightowl/cli.py`)
- Guarded Fixes → Pipeline Config: changes config (evidence: `nightowl/fixes.py`)
- Guarded Fixes → Spark ETL: reruns (evidence: `nightowl/fixes.py`)
- Guarded Fixes → OpenSearch Client: stores postmortem (evidence: `nightowl/fixes.py`)
