How 0011ashwin/amity-hack works
Project Overview and Purpose
• The project is a web application – likely built on a Flask (or similar) framework – providing an interface for processing and analyzing documents.
• It includes capabilities such as extracting text from PDF files, classifying documents based on content (with trained machine learning models), detecting plagiarism, and handling query operations on documents.
Analyzing the File Structure
• Top-level Files:
• README.md (“Document Intelligence System”) that briefly defines the project's intent.
• Several Python scripts (run.py, test.py, tool.py) likely used for running or testing various functionalities.
• requirements.txt which will list dependencies and tech stack libraries.
• The app Directory:
• Contains main components of the web application (app.py, __init__.py).
• Includes modules like document_classifier.py, pdf_extractor.py, plagiarism_detector.py, and query_engine.py – each corresponding to a core functionality.
• Has a static directory (for style.css) and a templates directory (with HTML files like index.html, document_classification.html, plagiarism.html, etc.) to serve as the user interface.
• The document_classifier Directory:
• Contains supplementary scripts (e.g., hackathon.py) and training data organized into categories (financial, healthcare, legal).
• Stores critical model files (text_classifier.pkl and vectorizer.pkl) that are used in the classification process.
• Data Directory:
• Hosts sample PDF files used either for demonstration, testing, or as input documents for the system.
Main Components Identified
a. User Interface / Web Layer:
• Frontend templates (HTML files) and static assets (CSS) used to provide the UI.
• Located in the app/templates and app/static directories.
b. Application/Controller Layer:
• The Flask app (app/app.py) that wires up the user interface with the backend functionalities.
• Responsible for routing HTTP requests to appropriate module handlers.
c. Document Processing Modules:
• PDF Extraction Module (app/pdf_extractor.py): Extracts text from PDF documents.
• Document Classifier (app/document_classifier.py): Utilizes ML models stored in document_classifier/text_classifier.pkl and document_classifier/vectorizer.pkl to classify documents.
• Plagiarism Detector (app/plagiarism_detector.py): Checks documents for duplicated or plagiarized content.
• Query Engine (app/query_engine.py): Processes user search or query operations on documents.
d. Training Data and Model Files:
• Document classifier training data available in document_classifier/training_data (organized by domain such as financial, healthcare, legal).
• Persisted model objects (pickle files) that are loaded during runtime for classification tasks.
e. Data Storage:
• Input data (e.g., sample PDFs) located within the data directory.
f. Supporting Scripts and Tools:
• Additional scripts like run.py, test.py, and tool.py for running the server, performing tests, or executing utilities.
Relationships and Interactions
• Web clients interact with the UI layer (HTML templates and CSS), which is backed by the Flask application.
• The application layer (app/app.py) routes incoming requests to specialized modules (PDF extraction, classification, plagiarism detection, query engine).
• Each module interacts with its respective models (e.g., classification module loads models and vectorizers) and input documents (from the data directory or user uploads).
• Processed outputs (extracted text, classification labels, plagiarism reports) then are rendered back to the user via HTML templates.
Architectural Patterns and Technologies
• MVC-Inspired Structure: Separation between the view (HTML/CSS templates), controller (Flask routes in app/app.py), and the model (machine learning models and processing functions).
• Modular Design: Each functionality (extraction, classification, plagiarism detection) is implemented as a separate module, facilitating maintainability and possible future extension.
• Use of Pickle for model persistence: Ensures that the document classification component can quickly load trained models.