# ericmiguel/datahood architecture

> Architecture diagram of the GitHub repository ericmiguel/datahood, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ericmiguel/datahood
- Repository: https://github.com/ericmiguel/datahood

## Overview

##### Project type and purpose
- This is a CLI toolkit (Python package) called **Datahood**. Its two primary functions are:
a. **Data Transfer** between MongoDB and BSON files via async streaming pipelines.
b. **Schema Discovery** that inspects existing documents and generates Python types (TypedDict or Pydantic).

##### Main components of the system
a. **CLI Layer**
- Entry point (`src/datahood/__main__.py`)
- Command modules under `src/datahood/cli/` (export.py, schema.py, help_text.py)
b. **Core Library**
- **Pipeline** (`src/datahood/pipeline.py`): orchestrates data flow.
- **Connectors** (`src/datahood/connectors/`): adapters for reading data sources.
- Base interface (`base.py`)
- BSON connector (`bson.py`)
- MongoDB connector (`mongodb.py`)
- **Exporters** (`src/datahood/exporters/`): adapters for writing data targets.
- Base interface (`base.py`)
- BSON exporter (`bson.py`)
- MongoDB exporter (`mongodb.py`)
- **Schema Inference** (`src/datahood/schema/infer/`): inspects sample documents to build an abstract schema.
- Base, factory, and backend-specific infer modules (bson.py, mongodb.py)
- **Schema Generation** (`src/datahood/schema/gen/`): converts the abstract schema into Python code.
- Base, TypedDict, Pydantic generators
c. **External Services & Artifacts**
- MongoDB instances (source and/or destination)
- BSON files on disk
- Generated Python model files

##### Relationships and data flow
- User runs a `dh` command → **CLI Layer** parses flags and dispatches to an operation (transfer or schema).
- **Transfer** path:
CLI → Pipeline → Source Connector → Async stream of documents → Pipeline → Destination Exporter
- Source Connector can be MongoDB or BSON
- Destination Exporter can be BSON or MongoDB
- **Schema** path:
CLI → Schema Infer Factory → Abstract Schema → Schema Generator → Writes Python code file
- Inference uses sampling from MongoDB or BSON
- Generation outputs TypedDict or Pydantic models

##### Architectural patterns and design principles
- **Adapter/Plugin Pattern**: Connectors and Exporters implement base interfaces, allowing future sources/targets.
- **Factory Pattern**: Schema inference selects the correct infer implementation at runtime.
- **Separation of Concerns**: CLI, orchestration (pipeline), I/O adapters (connectors/exporters), inference, and code generation are cleanly modularized.
- **Async Streaming**: Enables back-pressure and high-throughput transfers without loading entire collections in memory.

##### Important notes
- Every major folder under `src/datahood` as a separate component.
- Do not overcrowd: group related modules inside a single container where appropriate, but call out the key subcomponents.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% CLI Layer
    subgraph "CLI Layer"
        direction TB
        CLI["dh CLI"]:::cli
        ExportCmd["export command"]:::cli
        SchemaCmd["schema command"]:::cli
        HelpTxt["help_text"]:::cli
    end

    %% Core Library
    subgraph "Core Library"
        direction TB
        Pipeline["datahood.pipeline"]:::core

        subgraph "Connectors (Adapter Pattern)"
            direction TB
            BaseConn["base.py"]:::core
            BSONConn["bson.py"]:::core
            MongoConn["mongodb.py"]:::core
        end

        subgraph "Exporters (Adapter Pattern)"
            direction TB
            BaseExp["base.py"]:::core
            BSONExp["bson.py"]:::core
            MongoExp["mongodb.py"]:::core
        end

        subgraph "Schema Inference (Factory Pattern)"
            direction TB
            InfBase["base.py"]:::core
            BSONInf["bson.py"]:::core
            MongoInf["mongodb.py"]:::core
            InfFact["factory.py"]:::core
        end

        subgraph "Schema Generation"
            direction TB
            GenBase["base.py"]:::core
            TDGen["typeddict.py"]:::core
            PydGen["pydantic.py"]:::core
        end
    end

    %% External & Artifacts
    MongoDBSrc[(MongoDB Source)]:::external
    BSONFileIn["BSON File"]:::artifact
    MongoDBDest[(MongoDB Destination)]:::external
    BSONFileOut["BSON File"]:::artifact
    AbstractSchema["Abstract Schema"]:::artifact
    GenCode["Generated Python File"]:::artifact

    %% Transfer Flow
    CLI -->|"transfer"| Pipeline
    Pipeline --> BaseConn
    BaseConn --> BSONConn
    BaseConn --> MongoConn
    BSONConn -->|reads async stream| BSONFileIn
    MongoConn -->|reads async stream| MongoDBSrc
    Pipeline --> BaseExp
    BaseExp --> BSONExp
    BaseExp --> MongoExp
    BSONExp -->|writes async stream| BSONFileOut
    MongoExp -->|writes async stream| MongoDBDest

    %% Schema Flow
    CLI -->|"schema"| InfFact
    InfFact --> InfBase
    InfFact --> BSONInf
    InfFact --> MongoInf
    InfFact --> AbstractSchema
    AbstractSchema --> GenBase
    GenBase --> TDGen
    GenBase --> PydGen
    TDGen --> GenCode
    PydGen --> GenCode

    %% Click Events
    click CLI "https://github.com/ericmiguel/datahood/blob/main/src/datahood/__main__.py"
    click ExportCmd "https://github.com/ericmiguel/datahood/blob/main/src/datahood/cli/export.py"
    click SchemaCmd "https://github.com/ericmiguel/datahood/blob/main/src/datahood/cli/schema.py"
    click HelpTxt "https://github.com/ericmiguel/datahood/blob/main/src/datahood/cli/help_text.py"
    click Pipeline "https://github.com/ericmiguel/datahood/blob/main/src/datahood/pipeline.py"
    click BaseConn "https://github.com/ericmiguel/datahood/blob/main/src/datahood/connectors/base.py"
    click BSONConn "https://github.com/ericmiguel/datahood/blob/main/src/datahood/connectors/bson.py"
    click MongoConn "https://github.com/ericmiguel/datahood/blob/main/src/datahood/connectors/mongodb.py"
    click BaseExp "https://github.com/ericmiguel/datahood/blob/main/src/datahood/exporters/base.py"
    click BSONExp "https://github.com/ericmiguel/datahood/blob/main/src/datahood/exporters/bson.py"
    click MongoExp "https://github.com/ericmiguel/datahood/blob/main/src/datahood/exporters/mongodb.py"
    click InfBase "https://github.com/ericmiguel/datahood/blob/main/src/datahood/schema/infer/base.py"
    click BSONInf "https://github.com/ericmiguel/datahood/blob/main/src/datahood/schema/infer/bson.py"
    click MongoInf "https://github.com/ericmiguel/datahood/blob/main/src/datahood/schema/infer/mongodb.py"
    click InfFact "https://github.com/ericmiguel/datahood/blob/main/src/datahood/schema/infer/factory.py"
    click GenBase "https://github.com/ericmiguel/datahood/blob/main/src/datahood/schema/gen/base.py"
    click TDGen "https://github.com/ericmiguel/datahood/blob/main/src/datahood/schema/gen/typeddict.py"
    click PydGen "https://github.com/ericmiguel/datahood/blob/main/src/datahood/schema/gen/pydantic.py"

    %% Styles
    classDef cli fill:#D0E8FF,stroke:#137CBD,color:#000;
    classDef core fill:#E0F7E0,stroke:#28A745,color:#000;
    classDef external fill:#F0F0F0,stroke:#888,color:#000;
    classDef artifact fill:#FFE4B5,stroke:#FF8C00,color:#000;
```
