How desmondchoy/mtech-policy-recsys works
Main Components
• Data Inputs:
• Policy Documents: “data/policies/raw/” (PDFs) and their processed JSONs in “data/policies/processed/”.
• Transcripts: Raw transcripts in “data/transcripts/raw/” (synthetic and real) and processed transcripts in “data/transcripts/processed/”.
• Coverage Requirements: Defined in “data/coverage_requirements/coverage_requirements.py”.
• Data Generation & Preparation:
• Personalities and Transcript Generation Scripts: “scripts/data_generation/generate_personalities.py” and “scripts/data_generation/generate_transcripts.py” along with corresponding notebooks.
• Policy Extraction: “scripts/extract_policy_tier.py” which uses Google Gemini API.
• Transcript Evaluation and Processing:
• Evaluation: “scripts/evaluation/eval_transcript_main.py” along with evaluators, processors, and the related utilities.
• Transcript Processing: “src/utils/transcript_processing.py”, which regenerates structured (parsed) JSON transcripts.
• Multi-Agent Orchestration (Core Agents):
• Agents in “src/agents/”: These include the analyzer, extractor, voting agent, and cs_agent. The extractor agent (especially in “src/agents/extractor.py”) extracts requirements from transcripts.
• Prompts and LLM Service: “src/prompts/” and “src/models/llm_service.py” indicating the connection to external LLMs (Google Gemini and OpenAI via CrewAI).
• Report/Comparison Generation:
• “scripts/generate_policy_comparison.py” uses extracted requirements and processed policy JSONs to produce Markdown reports which reside in “results/”.
• Web Interface (for user interaction):
• Located in “src/web/app.py”.
Relationships and Interactions
• Raw Policy PDFs flow into the Policy Extraction component (via extract_policy_tier.py) to generate Structured Policy JSON.
• Raw Transcripts and coverage requirements, plus generated personalities, feed into the Transcript Generation script to produce raw transcript files.
• Raw transcripts flow into an Evaluation component (eval_transcript_main.py); evaluation results determine pass/fail decisions.
• Transcripts that pass evaluation are further processed/parsed via the transcript_processing.py script to create standardized JSON transcripts.
• Requirement Extraction:
• Parsed transcripts are then fed to the Extractor Agent (src/agents/extractor.py) that leverages the LLM service (with associated prompts) to extract customer requirements in a structured JSON format saved to “data/extracted_customer_requirements/”.
• Policy Comparison and Reporting:
• The output from the requirement extraction along with the processed policy data is provided as input to the Policy Comparison Report Generation Script. This step utilizes the LLM (via LLM service) to generate comparison reports in Markdown.
• (Future) Analyzer Agent:
• Once all inputs are available (policy, requirements, comparison reports), a designated analyzer agent is expected to generate personalized policy recommendations.
• Finally, the Web UI:
• This component (src/web/app.py) would offer an interactive interface for customers to query or review recommendations and reports.
Architectural Patterns and Design Principles
• Modularization & Separation of Concerns: Notice clear separation between data ingestion/generation, processing, multi-agent orchestration, and reporting.
• Use of External APIs: Emphasize the integration with external LLM APIs (Google Gemini and OpenAI) which power key decision processes in the agents.