How ericmiguel/datahood works
Project type and purpose
• This is a CLI toolkit (Python package) called Datahood. Its two primary functions are:
a. Data Transfer between MongoDB and BSON files via async streaming pipelines.
b. Schema Discovery that inspects existing documents and generates Python types (TypedDict or Pydantic).
Main components of the system
a. CLI Layer
• Entry point (src/datahood/__main__.py)
• Command modules under src/datahood/cli/ (export.py, schema.py, help_text.py)
b. Core Library
• Pipeline (src/datahood/pipeline.py): orchestrates data flow.
• Connectors (src/datahood/connectors/): adapters for reading data sources.
• Base interface (base.py)
• BSON connector (bson.py)
• MongoDB connector (mongodb.py)
• Exporters (src/datahood/exporters/): adapters for writing data targets.
• Base interface (base.py)
• BSON exporter (bson.py)
• MongoDB exporter (mongodb.py)
• Schema Inference (src/datahood/schema/infer/): inspects sample documents to build an abstract schema.
• Base, factory, and backend-specific infer modules (bson.py, mongodb.py)
• Schema Generation (src/datahood/schema/gen/): converts the abstract schema into Python code.
• Base, TypedDict, Pydantic generators
c. External Services & Artifacts
• MongoDB instances (source and/or destination)
• BSON files on disk
• Generated Python model files
Relationships and data flow
• User runs a dh command → CLI Layer parses flags and dispatches to an operation (transfer or schema).
• Transfer path:
CLI → Pipeline → Source Connector → Async stream of documents → Pipeline → Destination Exporter
• Source Connector can be MongoDB or BSON
• Destination Exporter can be BSON or MongoDB
• Schema path:
CLI → Schema Infer Factory → Abstract Schema → Schema Generator → Writes Python code file
• Inference uses sampling from MongoDB or BSON
• Generation outputs TypedDict or Pydantic models
Architectural patterns and design principles
• Adapter/Plugin Pattern: Connectors and Exporters implement base interfaces, allowing future sources/targets.
• Factory Pattern: Schema inference selects the correct infer implementation at runtime.
• Separation of Concerns: CLI, orchestration (pipeline), I/O adapters (connectors/exporters), inference, and code generation are cleanly modularized.
• Async Streaming: Enables back-pressure and high-throughput transfers without loading entire collections in memory.
Important notes
• Every major folder under src/datahood as a separate component.
• Do not overcrowd: group related modules inside a single container where appropriate, but call out the key subcomponents.