How farhanrhine/auto-feature-selector-tool works
Project type and purpose
• This is a lightweight, Python-based data-preprocessing CLI tool.
• Its sole responsibility is to ingest a dataset, perform cleaning and feature-engineering steps, and output a clean dataset.
File structure
• Top-level files only:
• app.py – the main script (entry point)
• helper.py – reusable functions for each cleaning/transformation step
• requirements.txt – external Python libraries (e.g., pandas, numpy)
• readme.md – workflow description
• No frontend or database; this is a simple command-line pipeline.
README for additional insights
• The workflow enumerates eight sequential stages:
1. Data ingestion
2. Unnecessary-column removal
3. Missing/duplicate data detection & handling
4. Missing-value imputation techniques
5. Data summary/statistics
6. Categorical/object feature transformation
7. Train/test split
8. Feature selection
• End goal: produce or download a cleaned dataset.
a. Main components (modules & artifacts)
• CLI entrypoint (app.py)
• Helper module (helper.py)
• Individual processing stages as sub-modules or functions:
– Data Reader
– Column Pruner
– MissingValueHandler
– ImputationEngine
– DataAnalyzer
– Transformer
– Splitter
– FeatureSelector
• Data stores (files on disk): Raw Dataset, Cleaned Dataset
• External dependencies (pandas, numpy, etc.)
b. Relationships & interactions
– app.py orchestrates the pipeline: imports helper.py, invokes each stage in sequence
– Each stage receives a DataFrame, returns a DataFrame
– Final stage writes output to disk or returns a download URL
c. Architectural patterns & design principles
– Pipeline (chain of responsibility): each step is isolated, testable, and composable
– Modularization: helper.py groups reusable functions
– Single Responsibility: each function/stage does one job
d. Key technologies & libraries
– Python 3.x
– pandas for data manipulation
– numpy for numeric operations
Detailed checklist for the architect
• If a stage splits into two (e.g., branching for different imputation strategies), show a fork and later join.
• Optionally, annotate each module with its primary Python functions (e.g., MissingValueHandler.handle_nulls()).