How angusketo/da_dataextraction works
Project Type and Purpose
• This is a data‐acquisition and processing pipeline implemented as a series of Jupyter notebooks.
• Goal: extract a Diels–Alder reaction dataset from Reaxys, augment it with donated IDs, and verify reaction correctness.
File Structure Analysis
• Four top‐level files—all Jupyter notebooks except README.md:
• Reaxys_Data_Extraction.ipynb
• DonatedIDs_Data_Acquisition.ipynb
• Identify_RC_DA.ipynb
• README.md
• No “src/” or “lib/” folders—a linear notebook-based workflow.
README Insights
• Describes sequence:
1. Extract reactions via Reaxys_Data_Extraction.ipynb
2. Acquire donated IDs via DonatedIDs_Data_Acquisition.ipynb
3. Verify with Identify_RC_DA.ipynb
• Dependencies: RDKit 2021.03.2 and optional RXNMapper for atom mapping.
a. Main Components
• External Data Source: Reaxys API / database
• Notebook #1: Reaxys_Data_Extraction
• Notebook #2: DonatedIDs_Data_Acquisition
• Notebook #3: Identify_RC_DA
• Supporting Libraries/Services: RDKit, RXNMapper (external GitHub tool)
• File System/Data Store: intermediate CSV/JSON files (e.g., raw_reactions.csv, donatedIDs.csv, verified_reactions.csv)
b. Relationships & Data Flows
1. Reaxys (external) → Notebook #1 → raw reaction dump
2. donatedIDs.csv (local input) → Notebook #2 (uses raw data as needed) → merged dataset
3. Merged dataset → Notebook #3 (invokes RDKit ± RXNMapper) → final verified dataset
4. All notebooks read/write from a project folder on local disk.
c. Architectural Patterns
• Linear ETL pipeline: Extract → Transform/Augment → Validate
• Loose coupling: each notebook is a self-contained stage
• External library integration points clearly separate from core data flow
d. Key Technologies
• Jupyter notebooks for orchestration and code
• RDKit for cheminformatics processing
• RXNMapper for optional atom‐mapping
• Reaxys API or file exports as data source
Essential Elements
• Node labels: notebook names, data filenames, service/library names