# angusketo/da_dataextraction architecture

> Architecture diagram of the GitHub repository angusketo/da_dataextraction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/angusketo/da_dataextraction
- Repository: https://github.com/angusketo/da_dataextraction

## Overview

##### Project Type and Purpose
- This is a data‐acquisition and processing pipeline implemented as a series of Jupyter notebooks.
- Goal: extract a Diels–Alder reaction dataset from Reaxys, augment it with donated IDs, and verify reaction correctness.

##### File Structure Analysis
- Four top‐level files—all Jupyter notebooks except README.md:
- Reaxys_Data_Extraction.ipynb
- DonatedIDs_Data_Acquisition.ipynb
- Identify_RC_DA.ipynb
- README.md
- No “src/” or “lib/” folders—a linear notebook-based workflow.

##### README Insights
- Describes sequence:
1. Extract reactions via Reaxys_Data_Extraction.ipynb
2. Acquire donated IDs via DonatedIDs_Data_Acquisition.ipynb
3. Verify with Identify_RC_DA.ipynb
- Dependencies: RDKit 2021.03.2 and optional RXNMapper for atom mapping.

a. Main Components
- External Data Source: Reaxys API / database
- Notebook #1: Reaxys_Data_Extraction
- Notebook #2: DonatedIDs_Data_Acquisition
- Notebook #3: Identify_RC_DA
- Supporting Libraries/Services: RDKit, RXNMapper (external GitHub tool)
- File System/Data Store: intermediate CSV/JSON files (e.g., raw_reactions.csv, donatedIDs.csv, verified_reactions.csv)

b. Relationships & Data Flows
1. Reaxys (external) → Notebook #1 → raw reaction dump
2. donatedIDs.csv (local input) → Notebook #2 (uses raw data as needed) → merged dataset
3. Merged dataset → Notebook #3 (invokes RDKit ± RXNMapper) → final verified dataset
4. All notebooks read/write from a project folder on local disk.

c. Architectural Patterns
- Linear ETL pipeline: Extract → Transform/Augment → Validate
- Loose coupling: each notebook is a self-contained stage
- External library integration points clearly separate from core data flow

d. Key Technologies
- Jupyter notebooks for orchestration and code
- RDKit for cheminformatics processing
- RXNMapper for optional atom‐mapping
- Reaxys API or file exports as data source

##### Essential Elements
- Node labels: notebook names, data filenames, service/library names

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Service
    Reaxys["Reaxys API"]:::external

    %% Pipeline Notebooks
    Extraction["Reaxys_Data_Extraction.ipynb"]:::notebook
    DonatedIDs["DonatedIDs_Data_Acquisition.ipynb"]:::notebook
    Identify["Identify_RC_DA.ipynb"]:::notebook

    %% Data Artifacts
    RawData["raw_reactions.csv"]:::data
    DonatedFile["donatedIDs.csv"]:::data
    MergedData["merged_dataset.csv"]:::data
    FinalData["verified_reactions.csv"]:::data

    %% Documentation
    README["README.md"]:::doc

    %% External Dependencies
    subgraph "External Dependencies"
        direction TB
        RDKit["RDKit"]:::lib
        RXNMapper["RXNMapper"]:::lib
    end

    %% Data Flow
    Reaxys -->|"extract reactions"| Extraction
    Extraction -->|"raw_reactions.csv"| RawData
    RawData --> DonatedIDs
    DonatedFile --> DonatedIDs
    DonatedIDs -->|"merged_dataset.csv"| MergedData
    MergedData --> Identify
    Identify -->|"verified_reactions.csv"| FinalData

    %% Library Calls
    Identify -.->|"use RDKit Mol objects"| RDKit
    Identify -.->|"optional atom mapping"| RXNMapper

    %% Documentation Flow
    README -.-> Extraction
    README -.-> DonatedIDs
    README -.-> Identify

    %% Click Events
    click Extraction "https://github.com/angusketo/da_dataextraction/blob/main/Reaxys_Data_Extraction.ipynb"
    click DonatedIDs "https://github.com/angusketo/da_dataextraction/blob/main/DonatedIDs_Data_Acquisition.ipynb"
    click Identify "https://github.com/angusketo/da_dataextraction/blob/main/Identify_RC_DA.ipynb"
    click README "https://github.com/angusketo/da_dataextraction/blob/main/README.md"

    %% Styles
    classDef notebook fill:#cfe2f3,stroke:#084b8a,color:#084b8a,stroke-width:1px;
    classDef external fill:#d9ead3,stroke:#38761d,color:#38761d,stroke-width:1px;
    classDef data fill:#fce5cd,stroke:#e69138,color:#e69138,stroke-width:1px,stroke-dasharray: 5 2;
    classDef lib fill:#eeeeee,stroke:#a0a0a0,color:#666666,stroke-width:1px,stroke-dasharray: 2 2;
    classDef doc fill:#f4cccc,stroke:#c00000,color:#c00000,stroke-width:1px;
```
