How smartlord7/qidlearninglib works
Project Type and Purpose
• QIDLearningLib is an open-source Python library (not a web or full-stack app) whose purpose is to automate the detection, evaluation, and optimization of quasi-identifiers (QIDs) in tabular datasets. It provides a suite of privacy, utility, causality, and performance metrics, redundancy analysis, multiple optimization strategies, CSV export, and visualization utilities.
File-Structure Analysis
• Top-level:
• src/QIDLearningLib/…: the main package
• src/QIDLearningLib/datasets: built-in sample datasets
• src/QIDLearningLib/doc: generated HTML docs
• src/QIDLearningLib/main.py: CLI/entry point
• .github/workflows: CI (unit tests, lint)
• Core modules under src/QIDLearningLib/QIDLearningLib:
• metrics/ (causality, correlation, data_privacy, data_utility, performance, qid_specific)
• optimizer/ (evolutionary_algorithm, greedy_search, simulated_annealing, tabu_search, util, graph, metric)
• structure/ (defines data structures, grouped_metric)
• util/ (data I/O, stats, time)
• test/ (unit tests)
• Patterns: layered modules (data I/O → metrics → redundancy analysis → optimization → export/visualization), strategy pattern for pluggable optimizers and metrics.
README Insights
• Describes features: metric suite, redundancy analysis, flexible optimization, extensibility, CSV export, visualization
• Highlights interoperability with ARX and Amnesia via CSV output
• No UI—interaction via CLI or Python API
1. CLI / Python API (main.py)
2. Data Loader (util/data.py)
3. Metrics Engine (metrics/*)
4. Redundancy Analyzer (metrics/correlation + metric-redundancy utilities)
5. Optimization Framework (optimizer/*)
– Evolutionary Algorithm
– Simulated Annealing
– Greedy Search
– Tabu Search
6. Result Exporter (CSV writer)
7. Visualization Tools (plotting utilities in util and optional graph module)
8. Built-in Datasets (datasets/*)
9. External Tools (ARX, Amnesia)
10. Test Harness (test/*)
11. Documentation Generator (doc/)
• CLI/API → Data Loader → Metrics Engine → Redundancy Analyzer → Optimization Framework → Result Exporter → Visualization → CLI/API
• Datasets feed into Data Loader
• Optimization Framework calls back into Metrics Engine for evaluating candidate QID sets
• Test Harness exercises Metrics Engine and Optimization Framework in isolation
• Documentation Generator reads docstrings and outputs HTML under doc/
c. Architectural Patterns & Principles:
• Layered architecture: separation between data I/O, computation, optimization, and presentation
• Strategy pattern: interchangeable optimizer modules implementing a common interface
• Modular metric plug-ins: each metric lives in its own file/folder for extensibility
• Dependency inversion: high‐level workflow code depends on abstract metric and optimizer interfaces, not concrete implementations
d. Key Technologies & Frameworks:
• Python 3.x
• pandas, numpy, scipy for data handling and computations
• matplotlib or seaborn (implied) for plotting
• pytest or unittest for test/
• setuptools / pip for packaging
• CI via GitHub Actions (.github/workflows/workflow.yml)
Details to Capture
• The feedback loop where the optimizer invokes metrics to score candidate QID sets
• Test harness and documentation generator as side components that verify and describe the system