How digvijay123321/cosinesim_checker works
Main Components
• Web Interface & User Interaction:
• Represented by app.py and the templates folder (index.html and result.html).
• This layer acts as the entry point of the application where users submit documents or trigger evaluations.
• Document Management:
• The project houses two document directories: Doc1 and Doc2.
• These directories contain different versions of documents (text files, PDFs, and JSON files for metadata) that will be processed and compared.
• Core Processing & Evaluation Engine:
• Multiple Python scripts (eval.py, evaluate_documents.py, evalution.py, and old_eval.py) form the heart of the evaluation process.
• They likely coordinate the comparison tasks, orchestrating how document content is processed and similarity calculated.
• Embedding Generation:
• embedding_generator.py is a dedicated module to generate embeddings.
• It uses five different models (OpenAI, T5-small, BERT, RoBERTa, and Modern BERT) to create vector representations of documents for cosine similarity analysis.
• Data Storage & Metrics:
• The Database directory holds JSON files (company_data.json and comparisons.json) that store company data and evaluation metrics.
• This component is critical for persisting input data, computed embeddings, and similarity scores.
• External and Utility Components:
• Dockerfile indicates containerization for deployment, ensuring consistency across environments.
• saving_to_gcp.py suggests integration with Google Cloud Platform for storing or processing data in the cloud.
• clear_temp.py and test.py, along with additional scripts (t1.py, t2.py, t3.py), likely provide auxiliary functions such as cleanup, testing, or experimental runs.
Relationships & Interactions
• User requests come into the Web Interface (app.py), which routes them to the core processing layers.
• The processing engine retrieves the necessary documents from the Doc1 and Doc2 directories.
• Embedding Generation is triggered to compute document embeddings via embedding_generator.py.
• The evaluation modules then compare the embeddings (using cosine similarity) and produce related metrics.
• Results and comparisons are stored in the Database JSON files.
• Additionally, utility scripts may handle tasks such as clearing temporary files or pushing results to external cloud storage (GCP).
• Deployment concerns (like the Docker container) should be represented as an encapsulating layer to show how the entire system is packaged and run.
Architectural Patterns & Technologies
• Separation of Concerns: Clear demarcation between UI (web templates, app.py), business logic (evaluation and embedding generation), and storage (JSON in Database) reflects a modular architecture.
• Service Integration: Integration with external models, cloud services (GCP), and containerization (Docker) are important details.