How shoumiksaha/ai-polished-text works
Project type and purpose
• APT-Eval is an open-source research tool / CLI pipeline for evaluating AI “polishers” and detectors on a human-written text (HWT) dataset.
• It ingests raw and AI-polished text samples, runs a suite of detection models/metrics, computes evaluation metrics, and outputs structured results for analysis.
File structure
Top-level directories & files:
• data/ – raw and polished CSV/JSON datasets and preprocessing scripts (cal_distance.py, json_to_csv.py, polish scripts).
• detectors/ – detector “plugins” subdivided by model name (radar, detectgpt, gltr, binoculars, commercial wrappers like ZeroGPT).
• raid/ – core helper code inherited from RAID (detect.py, evaluate.py, utils).
• results/ – stored predictions, accuracy/FPR results, domain- and polish-type breakdowns, and plotting scripts.
• CLI scripts – detect_cli.py, evaluate_cli.py, evaluate_for_hybrid.py.
• run_*.sh & set_api_keys.sh – orchestration and environment setup.
• environment.yml – Conda environment specification.
README
• Describes dataset (15K samples), directory layouts (data/polished, data/polished_json).
• Usage: CLI workflows (detect_cli.py, evaluate_cli.py, evaluate_for_hybrid.py, plus helper shell scripts).
• Lists included detectors and how to add new ones.
• Points to RAID library integration.
a. Main components:
1. Data Ingestion & Preprocessing
• Raw HWT CSV (data/HWT_original_data.csv)
• Polished data CSV/JSON (data/polished & data/polished_json)
• Utilities: data/json_to_csv.py, data/cal_distance.py
2. Detector Engine
• CLI front-end: detect_cli.py
• Plugin loader: detectors/detector.py
• Detector modules: detectors/models/{binoculars, detectgpt, fast_detectgpt, gltr, llmdet, radar, gpt2_detector, commercial wrappers}
• External APIs: ZeroGPT, GPTZero, huggingface datasets
3. Evaluation Engine
• CLI front-end: evaluate_cli.py & evaluate_for_hybrid.py
• RAID integration: raid/detect.py, raid/evaluate.py, raid/utils.py
• Threshold calculation and hybrid evaluation
4. Results & Analysis
• JSON result files (predictions.json, results_acc.json, results_fpr.json under results/<model>/)
• Domain- and polish-type breakdowns (results/domain_results, results/domain_results_ratio)
• Plotting scripts: results/plot_domain_results.py, results/plots/, results/domain_results/…
5. Orchestration & Environment
• Shell scripts (run_detector_basic.sh, run_detector_hybrid_polish_prctg.sh, run_detector_hybrid_polish_type.sh)
• Environment setup: environment.yml, set_api_keys.sh
1. Data Ingestion → Detector Engine
• CSV/JSON → detect_cli.py → detectors/detector.py loads specific model → model returns prediction scores
• Predictions written to JSON under results/<model>/
2. Detector Engine → Evaluation Engine
• predictions.json + original CSV → evaluate_cli.py / evaluate_for_hybrid.py → evaluation metrics JSON
3. Evaluation Engine → Results & Analysis
• results_acc.json, results_fpr.json → further breakdown and plot generation scripts → charts & domain/polish tables
4. Orchestration
• Shell scripts sequence detection + evaluation for various polish-types and models
5. Environment
• environment.yml & set_api_keys.sh support reproducibility and external API credential injection
c. Architectural patterns & design principles
• Plugin architecture for detectors: detectors/models folder structure allows easy extension of new detectors.
• Clear separation of concerns: Data layer, Detection layer, Evaluation layer, and Analysis layer.
• Reuse of external library (RAID) for core detection/evaluation logic.
• CLI-driven workflows enabling batch automation via shell scripts.
d. Key technologies/frameworks
• Python 3.x
• Conda environment (environment.yml)
• Huggingface datasets (optional data loading)
• PyTorch / TensorFlow (underlying detector models)
• External APIs: OpenAI (ChatGPT-based detectors), ZeroGPT, GPTZero
• Data formats: CSV, JSON
• Plotting: Matplotlib/Seaborn PDFs
Detail level
• Shell scripts triggering end-to-end pipelines
• Environment setup step feeding API keys into detector modules
• Two evaluation paths: basic (evaluate_cli.py) vs. hybrid (evaluate_for_hybrid.py)