How 2003harsh/skillhorizon works
Project Type and Purpose
• From the file tree and README, this is a Python-based backend pipeline (CLI/service) for an AI-powered course recommendation system.
• It orchestrates:
• Job listing scraping (Playwright, Scrapy)
• Skill-gap analysis
• Course data scraping
• Vector embedding & semantic search (FAISS + LangChain)
• LLM reasoning for personalized recommendations
• Data storage in MongoDB
File Structure
• Top-level:
• main.py – entry point, coordinates the pipeline
• requirements.txt – dependencies (LangChain, FAISS, PyMongo, Scrapy, Playwright, etc.)
• utils/ – modular scripts for each pipeline stage:
– get_jobs_json.py: scrape & extract required skills from jobs
– formatted_json_generator.py: normalize/sanitize scraped data
– convertor.py: convert JSON to embeddings input
– create_index.py: build/manage FAISS vector index
– genai.py: orchestrate LLM calls for ranking & explanation
– reterive_output.py: aggregate results and format output
README for Additional Insights
• Workflow stages clearly defined: Job & skill-gap analysis → Course scraping & ranking → Semantic search + LLM personalization.
• External systems: Coursera, Udemy, edX (scraping targets); LLM provider; MongoDB.
a. Main Components
1. User Interface (CLI or API endpoint in main.py)
2. Job Scraper Service (Playwright & Scrapy)
3. Skill-Gap Analyzer (compares scraped skills vs. user skills)
4. Course Scraper Service (Scrapy for course & review data)
5. Data Transformation Layer (formatted_json_generator + convertor)
6. Vector Store (FAISS index managed by create_index)
7. Retrieval & LLM Pipeline (LangChain + genai)
8. Database (MongoDB for raw & processed data)
9. External Services:
• Job boards (LinkedIn, Indeed, etc.)
• Course platforms (Coursera, Udemy, edX)
• LLM API (OpenAI, Cohere, etc.)
b. Relationships & Interactions
• User → main.py: submits profile & preferences
• main.py → Job Scraper: fetch listings → MongoDB
• main.py → Skill-Gap Analyzer: reads from MongoDB → outputs missing skills
• main.py → Course Scraper: fetch course data & reviews → MongoDB
• Data Transformation Layer: reads raw data → outputs normalized JSON
• Vector Store Indexing: ingest normalized JSON → builds/updates FAISS index
• Retrieval Pipeline:
• FAISS search for relevant courses
• genai calls LLM to refine & rank results
• main.py → retrieve_output: collects final recommendations → user
c. Architectural Patterns & Design Principles
• Pipeline / staged ETL pattern
• Modular single-responsibility scripts (utils/)
• Separation of concerns: scraping vs. transformation vs. retrieval
• Vector-based RAG architecture for semantic search + LLM
d. Key Technologies & Frameworks
• Python modules in utils/
• Scrapy & Playwright for ingestion
• MongoDB for persistence
• FAISS + LangChain for vector search & RAG
• LLM provider for natural language reasoning
Details to Capture
• Entry point (main.py) orchestrates everything.
• Each utils script’s responsibility.
• Data persisted in MongoDB vs. indexed in FAISS vs. runtime LLM calls.
• External web scraping targets and LLM endpoint.