How maninekkalapudi/decohortproject works
This is a configuration-driven ETL framework: a job definition selects a reader, optional processor, and writer; the orchestrator runs those stages and loads the result into DuckDB. The main workflow and the documented CSV, JSON, text, and REST API inputs are shown, along with the implemented cleaner and DuckDB destination. Reader implementations beyond the API reader were not sampled, so their detailed behavior and external source wiring are not inferred. The graph reflects the available orchestrator, factory, and writer code.
Main entry points: main.py Pipeline orchestrator · factory.py ReaderFactory · csv_reader.py CSVReader · json_reader.py JSONReader
Components 17
Job control
- Job definitions
elt_job_config.json - JobConfig
src/utils/config.py - Pipeline orchestrator
main.py
Data extraction
- ReaderFactory
src/readers/factory.py - CSVReader
src/readers/csv_reader.py - JSONReader
src/readers/json_reader.py - TXTReader
src/readers/text_reader.py - APIReader
src/readers/api_reader.py
Data transformation
- ProcessorFactory
src/processors/factory.py - BasicCleaner
src/processors/basic_cleaner.py
Data loading
- WriterFactory
src/writers/factory.py - DuckDBWriter
src/writers/duckdb_writer.py
Other components
- Pipeline operator
- Local data files
- REST API
- DuckDB warehouse
- Warehouse consumer
Connections 24
- Pipeline operator → Pipeline orchestrator: runs
- Pipeline orchestrator → Job definitions: reads jobs
- Pipeline orchestrator → JobConfig: builds config
- Pipeline orchestrator → ReaderFactory: selects reader
- ReaderFactory → CSVReader: dispatches
- ReaderFactory → JSONReader: dispatches
- ReaderFactory → TXTReader: dispatches
- ReaderFactory → APIReader: dispatches
- Pipeline orchestrator → CSVReader: extracts
- Pipeline orchestrator → JSONReader: extracts
- Pipeline orchestrator → TXTReader: extracts
- Pipeline orchestrator → APIReader: extracts
- CSVReader → Local data files: reads
- JSONReader → Local data files: reads
- TXTReader → Local data files: reads
- APIReader → REST API: requests data
- Pipeline orchestrator → ProcessorFactory: selects processor
- ProcessorFactory → BasicCleaner: dispatches
- Pipeline orchestrator → BasicCleaner: transforms data
- Pipeline orchestrator → WriterFactory: selects writer
- WriterFactory → DuckDBWriter: dispatches
- Pipeline orchestrator → DuckDBWriter: loads data
- DuckDBWriter → DuckDB warehouse: writes table
- Warehouse consumer → DuckDB warehouse: queries