How pyladiesams/data-ingestion-modern-stack-apr2025 works
Project type and purpose
• This is not a typical full-stack web app but a hands-on data ingestion workshop repository.
• Its goal is to teach how to ingest data from a REST API into a DuckDB database using the open-source Python library dlt, all orchestrated through Jupyter/Colab notebooks.
File structure
• Top-level items:
• .gitignore, LICENSE, pyproject.toml, .pre-commit-config.yaml – standard repo/config files.
• README.md – describes workshop, dependencies, usage.
• workshop/ – contains the “blank” starter notebook.
• solutions/ – contains the completed solution notebook.
• There is no separate “src” code directory: the notebook cells *are* the pipeline code.
• Dependency management is handled by uv (Astral’s UV tool) as specified in the README and pyproject.toml.
Key insights from the README
• The workshop uses:
• dlt (Data Load Tool) to materialize an ingestion pipeline.
• A REST API as the *source*.
• DuckDB as the *destination* database.
• Jupyter Notebook (or Google Colab) as the interactive execution environment.
• Pre-commit hooks ensure code formatting and linting before commits.
Main components
a. Data Source: REST API endpoint(s).
b. Ingestion Engine:
• Python environment managed by uv.
• Jupyter Notebook or Colab session containing code cells.
• The dlt library, configured in code cells.
c. Storage: DuckDB database file, local or in-memory.
d. Developer Environment:
• uv (dependency management)
• pre-commit for hooks
• Git/GitHub for version control
e. Optional Execution Host:
• Local machine (Jupyter Notebook)
• Remote (Google Colab)
Relationships and data flow
1. Notebook (Jupyter/Colab) → runs Python code using dlt.
2. dlt → issues HTTP requests to the REST API.
3. REST API → returns JSON or other structured data.
4. dlt → transforms/loads rows into DuckDB.
5. DuckDB → persists the ingested data for downstream analysis.
6. Developer can query DuckDB back in the same notebook.
Architectural patterns/design principles
• Modular: the pipeline logic is encapsulated in notebook cells; switching to a script or separate module would be trivial.
• Dependency Isolation: using virtual environments via uv.
• Separation of Concerns:
• Notebook = orchestration & presentation
• dlt = ingestion logic & connectors
• DuckDB = storage engine
Putting it all together
• uv sync & pre-commit install (setup phase)
• Execution of dlt code.
• Off to one side, show the “REST API” (external service).
• At the bottom, show DuckDB (local file) receiving data.