How ipis-mjkim/caueeg-dataset works
Project Type & Purpose
• This is a research dataset repository (“CAUEEG EEG dataset”)
• It provides:
– Raw EEG signals in EDF format
– Event descriptions (.json)
– Annotation files (.json, .xlsx)
• It is consumed by a separate Python/PyTorch library (caueeg-ceednet) for loading, benchmarking, and model training.
File Structure
Top-level items:
• annotation.json, dementia.json, abnormal.json, etc.
• /event/*.json (one file per recording)
• /signal/edf/*.edf (EEG signal files)
There are no code modules here—this is strictly the data layer.
Key Insights from README
• Data access requires an external approval form → download link & password
• Loading functions:
– load_caueeg_full_dataset(...) → returns config_data + full_eeg_dataset
– load_caueeg_task_datasets(...) → returns train/val/test splits for tasks (“dementia”, “abnormal”, “no-overlap”)
• Python/PyTorch is the target environment for downstream ML pipelines and benchmarks.
Main Components
a. External Service (“Data Request & Ethics Approval”)
– Form (Google Forms) → CAU ethics committee → download link & password
b. Data Storage (on disk)
– Annotation files
– Event folder
– Signal folder (EDF files)
c. Data Access Layer
– caueeg-ceednet Python module (DataLoader)
– Key entry points: load_caueeg_full_dataset, load_caueeg_task_datasets, load_caueeg_task_split
d. Benchmark/Task Modules
– CAUEEG-Dementia benchmark
– CAUEEG-Abnormal benchmark
– “No-overlap” variants
e. ML Pipeline
– PyTorch Dataset & DataLoader
– Model training & evaluation stages
– Metric computation, logging, visualization
f. Client / Researcher
– Jupyter notebooks (e.g., 01_DataSet_DataLoader.ipynb)
– Custom scripts that invoke the loader and training code
Relationships & Data Flows
1. Researcher submits form → receives download link → downloads dataset to local path
2. Python DataLoader (caueeg-ceednet) reads files from disk:
• Reads annotation JSON/XLSX → maps serial IDs to labels
• Reads event JSON → optional event streams
• Reads EDF files → loads raw signal arrays
3. DataLoader outputs in-memory Python dicts / PyTorch Datasets
4. Benchmark modules split data into train/val/test based on task parameters
5. ML Pipeline consumes splits → trains models → outputs metrics & plots
Architectural Patterns & Design Principles
• Separation of Concerns:
– Data storage vs. data loading vs. training/evaluation
• Modular Benchmarks: one loader function, multiple task-specific wrappers
• Extensibility: load_event flag toggles inclusion of event data
• Standard Formats: EDF, JSON, Excel ensure compatibility
• EDF (European Data Format) → raw EEG signals
• JSON / Excel → annotations & events
• Python / PyTorch → DataLoader & ML code
• Jupyter Notebook → exploratory analysis
• GitHub → version control & code hosting
Final Checklist
[ ] All major components drawn and labeled
[ ] Key technologies (, EDF, JSON, PyTorch) annotated
[ ] Benchmarks and flags (load_event, no-overlap) represented