# maninekkalapudi/decohortproject architecture

> Architecture diagram of the GitHub repository maninekkalapudi/decohortproject, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/maninekkalapudi/decohortproject
- Repository: https://github.com/maninekkalapudi/decohortproject

## Overview

This is a configuration-driven ETL framework: a job definition selects a reader, optional processor, and writer; the orchestrator runs those stages and loads the result into DuckDB. The main workflow and the documented CSV, JSON, text, and REST API inputs are shown, along with the implemented cleaner and DuckDB destination. Reader implementations beyond the API reader were not sampled, so their detailed behavior and external source wiring are not inferred. The graph reflects the available orchestrator, factory, and writer code.

Main entry points: [`main.py`](https://github.com/maninekkalapudi/decohortproject/blob/main/main.py) (Pipeline orchestrator), [`src/readers/factory.py`](https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/factory.py) (ReaderFactory), [`src/readers/csv_reader.py`](https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/csv_reader.py) (CSVReader), [`src/readers/json_reader.py`](https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/json_reader.py) (JSONReader)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Job control"]
  node_jobfile["Job definitions"]
  node_config["JobConfig<br/>[config.py]"]
  node_main["Pipeline orchestrator<br/>[main.py]"]
end

subgraph group_extract["Data extraction"]
  node_readerfactory["ReaderFactory<br/>[factory.py]"]
  node_csvreader["CSVReader<br/>[csv_reader.py]"]
  node_jsonreader["JSONReader<br/>[json_reader.py]"]
  node_txtreader["TXTReader<br/>[text_reader.py]"]
  node_apireader["APIReader<br/>[api_reader.py]"]
end

subgraph group_transform["Data transformation"]
  node_processorfactory["ProcessorFactory<br/>[factory.py]"]
  node_cleaner["BasicCleaner<br/>[basic_cleaner.py]"]
end

subgraph group_load["Data loading"]
  node_writerfactory["WriterFactory<br/>[factory.py]"]
  node_duckwriter["DuckDBWriter<br/>[duckdb_writer.py]"]
end

node_user(("Pipeline operator"))
node_files["Local data files"]
node_apiservice{{"REST API"}}
node_warehouse[("DuckDB warehouse")]
node_analyst(("Warehouse consumer"))

node_user -->|"runs"| node_main
node_main -->|"reads jobs"| node_jobfile
node_main -->|"builds config"| node_config
node_main -->|"selects reader"| node_readerfactory
node_readerfactory -->|"dispatches"| node_csvreader
node_readerfactory -->|"dispatches"| node_jsonreader
node_readerfactory -->|"dispatches"| node_txtreader
node_readerfactory -->|"dispatches"| node_apireader
node_main -->|"extracts"| node_csvreader
node_main -->|"extracts"| node_jsonreader
node_main -->|"extracts"| node_txtreader
node_main -->|"extracts"| node_apireader
node_csvreader -->|"reads"| node_files
node_jsonreader -->|"reads"| node_files
node_txtreader -->|"reads"| node_files
node_apireader -->|"requests data"| node_apiservice
node_main -->|"selects processor"| node_processorfactory
node_processorfactory -->|"dispatches"| node_cleaner
node_main -->|"transforms data"| node_cleaner
node_main -->|"selects writer"| node_writerfactory
node_writerfactory -->|"dispatches"| node_duckwriter
node_main -->|"loads data"| node_duckwriter
node_duckwriter -->|"writes table"| node_warehouse
node_analyst -->|"queries"| node_warehouse

click node_jobfile "https://github.com/maninekkalapudi/decohortproject/blob/main/elt_job_config.json"
click node_config "https://github.com/maninekkalapudi/decohortproject/blob/main/src/utils/config.py"
click node_main "https://github.com/maninekkalapudi/decohortproject/blob/main/main.py"
click node_readerfactory "https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/factory.py"
click node_csvreader "https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/csv_reader.py"
click node_jsonreader "https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/json_reader.py"
click node_txtreader "https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/text_reader.py"
click node_apireader "https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/api_reader.py"
click node_processorfactory "https://github.com/maninekkalapudi/decohortproject/blob/main/src/processors/factory.py"
click node_cleaner "https://github.com/maninekkalapudi/decohortproject/blob/main/src/processors/basic_cleaner.py"
click node_writerfactory "https://github.com/maninekkalapudi/decohortproject/blob/main/src/writers/factory.py"
click node_duckwriter "https://github.com/maninekkalapudi/decohortproject/blob/main/src/writers/duckdb_writer.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_jobfile,node_config,node_main toneBlue
class node_readerfactory,node_csvreader,node_jsonreader,node_txtreader,node_apireader,node_warehouse toneAmber
class node_processorfactory,node_cleaner,node_apiservice toneMint
class node_writerfactory,node_duckwriter toneRose
class node_user,node_files,node_analyst toneIndigo
```

## Components

### Job control

- [Job definitions](https://github.com/maninekkalapudi/decohortproject/blob/main/elt_job_config.json): `elt_job_config.json`
- [JobConfig](https://github.com/maninekkalapudi/decohortproject/blob/main/src/utils/config.py): `src/utils/config.py`
- [Pipeline orchestrator](https://github.com/maninekkalapudi/decohortproject/blob/main/main.py): `main.py`

### Data extraction

- [ReaderFactory](https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/factory.py): `src/readers/factory.py`
- [CSVReader](https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/csv_reader.py): `src/readers/csv_reader.py`
- [JSONReader](https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/json_reader.py): `src/readers/json_reader.py`
- [TXTReader](https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/text_reader.py): `src/readers/text_reader.py`
- [APIReader](https://github.com/maninekkalapudi/decohortproject/blob/main/src/readers/api_reader.py): `src/readers/api_reader.py`

### Data transformation

- [ProcessorFactory](https://github.com/maninekkalapudi/decohortproject/blob/main/src/processors/factory.py): `src/processors/factory.py`
- [BasicCleaner](https://github.com/maninekkalapudi/decohortproject/blob/main/src/processors/basic_cleaner.py): `src/processors/basic_cleaner.py`

### Data loading

- [WriterFactory](https://github.com/maninekkalapudi/decohortproject/blob/main/src/writers/factory.py): `src/writers/factory.py`
- [DuckDBWriter](https://github.com/maninekkalapudi/decohortproject/blob/main/src/writers/duckdb_writer.py): `src/writers/duckdb_writer.py`

### Other components

- **Pipeline operator**
- **Local data files**
- **REST API**
- **DuckDB warehouse**
- **Warehouse consumer**

## Connections

- Pipeline operator → Pipeline orchestrator: runs
- Pipeline orchestrator → Job definitions: reads jobs
- Pipeline orchestrator → JobConfig: builds config
- Pipeline orchestrator → ReaderFactory: selects reader
- ReaderFactory → CSVReader: dispatches
- ReaderFactory → JSONReader: dispatches
- ReaderFactory → TXTReader: dispatches
- ReaderFactory → APIReader: dispatches
- Pipeline orchestrator → CSVReader: extracts
- Pipeline orchestrator → JSONReader: extracts
- Pipeline orchestrator → TXTReader: extracts
- Pipeline orchestrator → APIReader: extracts
- CSVReader → Local data files: reads
- JSONReader → Local data files: reads
- TXTReader → Local data files: reads
- APIReader → REST API: requests data
- Pipeline orchestrator → ProcessorFactory: selects processor
- ProcessorFactory → BasicCleaner: dispatches
- Pipeline orchestrator → BasicCleaner: transforms data
- Pipeline orchestrator → WriterFactory: selects writer
- WriterFactory → DuckDBWriter: dispatches
- Pipeline orchestrator → DuckDBWriter: loads data
- DuckDBWriter → DuckDB warehouse: writes table
- Warehouse consumer → DuckDB warehouse: queries