# amazon-science/synthesizrr architecture

> Architecture diagram of the GitHub repository amazon-science/synthesizrr, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amazon-science/synthesizrr
- Repository: https://github.com/amazon-science/synthesizrr

## Overview

##### Project type and purpose
- From the file tree (a Python package under `src/synthesizrr/`) and the README, this is a research‐oriented data‐processing pipeline implementing “SynthesizRR: Generating Diverse Datasets with Retrieval Augmentation.”
- It is neither a classic client-server web app nor a simple library; it’s a full multi‐stage data‐generation workflow that orchestrates data ingestion, retrieval, generation, and evaluation at scale using Ray/Dask.

##### File structure
- Top‐level files and folders:
- `.github/workflows/` – CI: linting, tests, release
- `pyproject.toml`, `requirements.txt` – build and dependencies
- `src/synthesizrr/` – main Python package
- Inside `src/synthesizrr/`:
- `data.py` – dataset‐creation stage
- `corpus.py` – retrieval/corpus‐building stage
- `generation.py` – retrieval‐augmented generation logic
- `metrics.py` – evaluation stage
- `driver.py` and `main.py` – entrypoints/orchestration
- `common.py` – shared constants (DATA_DIR, CORPUS_DIR, RESULTS_DIR)
- No MVC or microservices; however, it uses a modular pipeline pattern and delegates parallelism to Ray + Dask.

##### README for additional insights
- Workflow: set up three directories (DATA_DIR, CORPUS_DIR, RESULTS_DIR), start a Ray cluster, and sequentially run the three scripts in `expts/` (mirrored in `src/synthesizrr/` here).
- External dependencies: Ray cluster (head/workers), Dask-on-Ray, SpaCy, NLTK, and optional S3 storage for large corpora/results.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% Users and Orchestration
  User["User CLI"]:::user
  subgraph "Orchestration Layer"
    main["main.py"]:::orchestration
    driver["driver.py"]:::orchestration
    common["common.py"]:::config
  end

  %% Compute Layer
  subgraph "Compute Layer"
    Head["Ray Head Node"]:::compute
    Workers["Ray Worker Nodes"]:::compute
  end

  %% Pipeline Modules
  subgraph "Pipeline Modules"
    data["Data Ingestion\ndata.py"]:::module
    corpus["Corpus Construction\ncorpus.py"]:::module
    gen["Retrieval-Augmented Generation\ngeneration.py"]:::module
    metrics["Metrics & Evaluation\nmetrics.py"]:::module
  end

  %% Storage Layer
  subgraph "Storage Layer"
    raw["DATA_DIR"]:::storage
    corpDir["CORPUS_DIR"]:::storage
    results["RESULTS_DIR"]:::storage
  end

  %% External Services
  subgraph "External Services"
    Dask["Dask-on-Ray"]:::external
    NLP["SpaCy, NLTK"]:::external
    S3["Optional S3 Storage"]:::external
  end

  %% CI/CD
  subgraph "CI/CD"
    lint["linting.yml"]:::cicd
    tests["tests.yml"]:::cicd
    release["release.yml"]:::cicd
    img["img/"]:::cicd
  end

  %% Connections
  User --> main
  main --> driver
  main --> common
  driver --> Head
  driver --> Workers
  Head --- Workers
  Head --> Dask
  Workers --> Dask

  raw --> data --> results
  raw --> corpus --> results
  corpDir --> corpus
  results --> gen
  raw --> gen
  gen --> results
  results --> metrics --> results

  S3 --> raw
  S3 --> corpDir
  S3 --> results

  NLP --> corpus
  NLP --> gen
  common --> data
  common --> corpus
  common --> gen
  common --> metrics

  lint --> main
  tests --> main
  release --> main
  img --> main

  %% Click Events
  click main "https://github.com/amazon-science/synthesizrr/blob/main/src/synthesizrr/main.py"
  click driver "https://github.com/amazon-science/synthesizrr/blob/main/src/synthesizrr/driver.py"
  click common "https://github.com/amazon-science/synthesizrr/blob/main/src/synthesizrr/common.py"
  click data "https://github.com/amazon-science/synthesizrr/blob/main/src/synthesizrr/data.py"
  click corpus "https://github.com/amazon-science/synthesizrr/blob/main/src/synthesizrr/corpus.py"
  click gen "https://github.com/amazon-science/synthesizrr/blob/main/src/synthesizrr/generation.py"
  click metrics "https://github.com/amazon-science/synthesizrr/blob/main/src/synthesizrr/metrics.py"
  click lint "https://github.com/amazon-science/synthesizrr/blob/main/.github/workflows/linting.yml"
  click tests "https://github.com/amazon-science/synthesizrr/blob/main/.github/workflows/tests.yml"
  click release "https://github.com/amazon-science/synthesizrr/blob/main/.github/workflows/release.yml"
  click img "https://github.com/amazon-science/synthesizrr/tree/main/img/"

  %% Styles
  classDef module fill:#e0ffe0,stroke:#0a0,stroke-width:1px
  classDef storage fill:#add8e6,stroke:#006,stroke-width:1px
  classDef compute fill:#ffa500,stroke:#c60,stroke-width:1px
  classDef external fill:#ffff99,stroke:#aa0,stroke-width:1px
  classDef orchestration fill:#ffe4b5,stroke:#b55,stroke-width:1px
  classDef cicd fill:#f0f0f0,stroke:#888,stroke-width:1px
  classDef user fill:#ffffff,stroke:#000,stroke-width:1px
  classDef config fill:#fff0f5,stroke:#c0a,stroke-width:1px
```
