How surajjayakumar2004/author_identificationn works
Project type and purpose
• From the file tree and minimal README, this is an Author Identification machine-learning pipeline.
• It ingests raw text (data/…), preprocesses it, extracts features, trains multiple models (stylometric, BM25, transformer, BERT, POS-tagger), and then builds an ensemble to predict an author for a new text.
File structure
• Top-level scripts:
• main.py – likely orchestrates full ETL + training pipeline
• predict_author.py – CLI for loading models and running inference on new text
• test_article.txt – sample input
• data/ – holds raw CSV/text, processed JSON, feature matrices and saved vectorizer/model artifacts
• data/processed – cleaned JSON per author + combined CSV/JSON
• data/features – numeric feature files (.npy, .npz, .csv) and pickled models used during feature extraction
• data/model_jobLib – joblib/pickle files for vectorizers, label_encoder, ensemble
• preprocessing/ – data_preprocessing.py, feature_extraction.py, utils.py
• models/ – Python modules implementing each submodel (bert_model.py, stylometric_model.py, bm25_model.py, transformer_model.py, ensemble_model.py)
• pos_tagger_model/ – code and saved TensorBoard logs for fine-tuning a POS-tagger
• reports/ – evaluation.py for metrics and reporting
• requirements.txt – dependencies