How satish24011993/research_paper_classifier works
Project type and purpose
• This is a full-stack machine-learning pipeline for research-paper abstract classification and disease–entity extraction.
• It includes data ingestion/preprocessing, model fine-tuning (classification), external LLM integration (disease extraction), evaluation, a REST API (FastAPI), and containerized/cloud deployment.
File structure
Top-level files:
• app.py – FastAPI service
• main_pipeline.py – data preprocessing + training + save model
• inference_and_evaluation.py – load model + evaluate + local inference
• deploy.sh & Dockerfile – containerization and deployment scripts
• requirements.txt – dependencies
• results/ – model artifacts, checkpoints, performance reports
• Dataset_1_/ – expected input data structure (Cancer/Non-Cancer .txt files)
Patterns and components:
• “main_pipeline.py” defines ETL → model training → artifact saving.
• “inference_and_evaluation.py” reuses saved artifacts to measure performance or run ad-hoc inference.
• “app.py” wraps inference and disease-extraction logic behind REST endpoints.
• Dockerfile + deploy.sh orchestrate container builds and pushes to GCP Cloud Run or Hugging Face Spaces.
README for additional insights
• Tech stack: Python 3.9+, Transformers (DistilBERT + LoRA), LangChain–Google Gemini for NER, FastAPI, Uvicorn, Docker, GCP SDK, Hugging Face CLI.
• Workflow described end-to-end: from raw abstracts → classification → entity extraction → performance metrics → REST API → Docker → cloud deployment.
• Notes on scalability (batch endpoints, Kafka/Redis streams) and agentic orchestration (optional bonus).
Essential detail checklist
• End-to-end pipeline stages and artifacts
• Separation between training and serving
• Storage location of trained models and label mappings
• External API integration point (Gemini)
• Containerization and cloud deployment flows
• Optional scalability extensions