How iamgroot42/mimir works
Main Components
• Core Package (Library): The heart of the project is the “mimir” directory. It contains:
• Attack implementations (in mimir/attacks/): Each attack (loss, reference-based, zlib, neighborhood, etc.) is implemented as its own module.
• Utility modules (mimir/utils.py, mimir/data_utils.py, mimir/config.py, etc.): These handle configuration, data loading, model handling, and plotting.
• Model definition and custom dataset support (mimir/models.py and mimir/custom_datasets.py).
• Experiment Runner:
• The “run.py” entry point which takes a configuration file (e.g., configs/mi.json) to launch experiments.
• Various bash scripts under “scripts/” also execute experiments and orchestrate different runs.
• Data Handling:
• “data/” directory includes scripts related to dataset creation and caching.
• “configs/” directory holds JSON configuration files that define experimental settings (paths, flags, etc.).
• Analysis and Visualization:
• “analysis/” includes Python and shell scripts for post‑experiment analysis, plotting results, and visualizing metrics.
• “notebooks/” provide Jupyter notebooks (e.g., figures_for_paper.ipynb) to further explore the results.
• Continuous Integration/Testing:
• The “.github” directory holds GitHub workflows for testing and documentation, ensuring code quality.
• Local/Experimental Results and Datasets:
• The “local/” directory contains experimental outputs, deduplication results, and aggregated measurements.
Relationships and Interactions
• The CLI/Entry-Point: When an experiment is run via run.py or a bash script, it loads a configuration (from the “configs” directory) and passes it to the mimir core.
• Data Flow:
• The configuration defines how and where data is loaded (data_utils.py) – data may come from local directories or from Hugging Face Datasets.
• The core package loads the specified dataset, then runs the chosen attack(s) implementations to compute memorization scores.
• Each attack module in mimir/attacks interacts with the rest of the components (e.g., using model prediction modules, logging, and result aggregation).
• Post-Processing:
• Experiment results are saved temporarily (e.g., in tmp_results/ directories) and later moved to results/ directories.
• Analysis scripts read these results to produce figures and tables, as seen in the “analysis/” and “notebooks/” directories.
• Integration Points: The system interfaces with external libraries (like Hugging Face and compression libraries for zlib) and supports extensibility for additional attacks or datasets.
Architectural Patterns and Design Principles
• Modularity: Clear separation of functionality in distinct modules – attacks, configuration, data handling, and analysis are decoupled.
• Config-Driven Execution: The experiments are parameterized through JSON config files, allowing flexibility for data sources and experimental setup.
• Extensibility: Instructions in the README show that new attacks and datasets can be plugged into the system with minimal changes.
• Reusability & Testing: There is a dedicated “tests/” directory to ensure that the various components work as intended, and continuous integration is set up via GitHub workflows.
Detailed Elements
• “CLI/Entry Point (run.py & scripts/)”
• “Configuration (configs/...)”
• “Core Library (mimir/)”
– “Attack Modules (mimir/attacks/)”
– “Utility / Data Utils (mimir/data_utils.py, mimir/config.py)”
• “Data Preprocessing (data/ and custom dataset support in mimir/custom_datasets.py)”
• “Analysis & Visualization (analysis/, notebooks/)”
• “Results Management (local/, results/, tmp_results/)”
• How configuration data is fed into the experiment runner.
• How the core library processes data and returns results.
• The movement of results from temporary storage to final analysis.
• External integration points:
• External libraries for data (Hugging Face Datasets), compression (zlib), and standard Python packages.
• Where CI/CD fits in (e.g., GitHub workflows and tests directories).