# surajjayakumar2004/author_identificationn architecture

> Architecture diagram of the GitHub repository surajjayakumar2004/author_identificationn, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/surajjayakumar2004/author_identificationn
- Repository: https://github.com/surajjayakumar2004/author_identificationn

## Overview

##### Project type and purpose
- From the file tree and minimal README, this is an Author Identification machine-learning pipeline.
- It ingests raw text (data/…), preprocesses it, extracts features, trains multiple models (stylometric, BM25, transformer, BERT, POS-tagger), and then builds an ensemble to predict an author for a new text.

##### File structure
- Top-level scripts:
- main.py – likely orchestrates full ETL + training pipeline
- predict_author.py – CLI for loading models and running inference on new text
- test_article.txt – sample input
- data/ – holds raw CSV/text, processed JSON, feature matrices and saved vectorizer/model artifacts
- data/processed – cleaned JSON per author + combined CSV/JSON
- data/features – numeric feature files (.npy, .npz, .csv) and pickled models used during feature extraction
- data/model_jobLib – joblib/pickle files for vectorizers, label_encoder, ensemble
- preprocessing/ – data_preprocessing.py, feature_extraction.py, utils.py
- models/ – Python modules implementing each submodel (bert_model.py, stylometric_model.py, bm25_model.py, transformer_model.py, ensemble_model.py)
- pos_tagger_model/ – code and saved TensorBoard logs for fine-tuning a POS-tagger
- reports/ – evaluation.py for metrics and reporting
- requirements.txt – dependencies

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        RawData["Raw Data Store\n(authors.csv)"]:::data
        PreData["Preprocessed Data Store\n(data/processed/)"]:::data
    end

    %% Preprocessing Layer
    subgraph "Preprocessing Layer"
        direction TB
        DataPreproc["Data Preprocessing\n(data_preprocessing.py)"]:::processing
        Utils["Utilities\n(utils.py)"]:::processing
    end

    %% Feature Extraction Layer
    subgraph "Feature Extraction Layer"
        direction TB
        FE["Feature Extraction Module\n(feature_extraction.py + data/features/)"]:::processing
    end

    %% POS-Tagger Training Subsystem
    subgraph "POS-Tagger Training"
        direction TB
        POS["POS-Tagger Training Subsystem\n(fine_tune_pos_tagger.py)"]:::model
    end

    %% Model Training Subsystem
    subgraph "Model Training"
        direction TB
        Styl["Stylometric Model Training\n(stylometric_model.py)"]:::model
        BM25["BM25 Model Training\n(bm25_model.py)"]:::model
        Transf["Transformer Model Training\n(transformer_model.py)"]:::model
        BERT["BERT Model Training\n(bert_model.py)"]:::model
    end

    %% Ensemble and Artifacts
    Ensemble["Ensemble Model Builder\n(ensemble_model.py)"]:::model
    Artifacts["Model Artifacts Store\n(data/model_jobLib/)"]:::data

    %% Orchestration and Services
    Main["Pipeline Orchestration Script\n(main.py)"]:::script
    Predict["Prediction Service\n(predict_author.py)"]:::script
    Sample["Sample Input Article\n(test_article.txt)"]:::script
    Req["Project Dependencies\n(requirements.txt)"]:::script
    Report["Reporting/Evaluation Module\n(evaluation.py)"]:::script

    %% Data Flow
    RawData -->|"raw CSV"| DataPreproc
    DataPreproc -->|"clean JSON/CSV"| PreData
    PreData -->|"processed data"| FE
    PreData -->|"processed data"| POS

    FE -->|"feature matrices"| Styl
    FE -->|"feature matrices"| BM25
    FE -->|"feature matrices"| Transf
    FE -->|"feature matrices"| BERT

    Styl -->|"model.pkl"| Artifacts
    BM25 -->|"model.pkl"| Artifacts
    Transf -->|"model.pkl"| Artifacts
    BERT -->|"model.pkl"| Artifacts
    POS -->|"POS model"| Artifacts

    Artifacts -->|"submodel artifacts"| Ensemble
    Ensemble -->|"ensemble_model.pkl"| Artifacts

    Main -->|orchestrates| DataPreproc
    Main -->|orchestrates| FE
    Main -->|orchestrates| Styl
    Main -->|orchestrates| BM25
    Main -->|orchestrates| Transf
    Main -->|orchestrates| BERT
    Main -->|orchestrates| POS
    Main -->|orchestrates| Ensemble
    Main -->|writes metrics| Report

    Predict -->|"loads artifacts"| Artifacts
    Predict -->|"requirements"| Req
    Predict -->|"input"| Sample
    Predict -->|"output author"| Report

    Report -->|"reads metrics"| Artifacts

    %% Click Events
    click RawData "https://github.com/surajjayakumar2004/author_identificationn/blob/main/data/authors.csv"
    click PreData "https://github.com/surajjayakumar2004/author_identificationn/tree/main/data/processed/"
    click DataPreproc "https://github.com/surajjayakumar2004/author_identificationn/blob/main/preprocessing/data_preprocessing.py"
    click Utils "https://github.com/surajjayakumar2004/author_identificationn/blob/main/preprocessing/utils.py"
    click FE "https://github.com/surajjayakumar2004/author_identificationn/blob/main/preprocessing/feature_extraction.py"
    click POS "https://github.com/surajjayakumar2004/author_identificationn/blob/main/pos_tagger_model/fine_tune_pos_tagger.py"
    click Styl "https://github.com/surajjayakumar2004/author_identificationn/blob/main/models/stylometric_model.py"
    click BM25 "https://github.com/surajjayakumar2004/author_identificationn/blob/main/models/bm25_model.py"
    click Transf "https://github.com/surajjayakumar2004/author_identificationn/blob/main/models/transformer_model.py"
    click BERT "https://github.com/surajjayakumar2004/author_identificationn/blob/main/models/bert_model.py"
    click Ensemble "https://github.com/surajjayakumar2004/author_identificationn/blob/main/models/ensemble_model.py"
    click Artifacts "https://github.com/surajjayakumar2004/author_identificationn/tree/main/data/model_jobLib/"
    click Main "https://github.com/surajjayakumar2004/author_identificationn/blob/main/main.py"
    click Predict "https://github.com/surajjayakumar2004/author_identificationn/blob/main/predict_author.py"
    click Sample "https://github.com/surajjayakumar2004/author_identificationn/blob/main/test_article.txt"
    click Req "https://github.com/surajjayakumar2004/author_identificationn/blob/main/requirements.txt"
    click Report "https://github.com/surajjayakumar2004/author_identificationn/blob/main/reports/evaluation.py"

    %% Styles
    classDef data fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef processing fill:#ffe5b4,stroke:#333,stroke-width:1px
    classDef model fill:#c2f0c2,stroke:#333,stroke-width:1px
    classDef script fill:#e0e0e0,stroke:#333,stroke-width:1px
```
