How loid122/interactive_courses_scraper works
Project type and purpose
• This is a monolithic full-stack web application (“Interactive_courses_scraper”) built in Flask.
• Its purpose is to fetch and process data from multiple IITM websites, then present interactive visualizations in the browser.
File structure
• Top-level artifacts:
– flaskapp.py (the Flask entry point)
– Requirements.txt (dependencies)
– static/ (JavaScript, CSS, images)
– templates/ (Jinja2 HTML views)
• No database directory—sessions are encrypted and stored client-side.
• No microservice folders—everything lives in one codebase.
Extract key architectural details from the README
• Backend: Flask for HTTP routing and session management.
• Scraping sub-modules: Requests, Selenium, BeautifulSoup, regex.
• Security: AES-GCM (via Fernet) for encrypting session payload in cookies.
• Frontend: Cytoscape.js for graph/visualization, served as static assets.
Main components
A. Client Browser (UI layer)
– Hosts HTML templates and runs Cytoscape.js for interactive graphs
B. Flask Application
– HTTP Server / Router (flaskapp.py)
– Session Manager (Fernet crypto)
– Scraper Orchestrator (coordinates Requests + Selenium + BS4)
C. External IITM Websites
– Target endpoints for data extraction
D. Static Asset Server
– Serves JS/CSS/images from static/
E. (Optional) Logging or Error-handling sub-component
Relationships & interactions
1. Browser → Flask Server (HTTP GET/POST)
2. Flask → Session Manager
• Decrypt/validate session cookie, or create a new one
3. Flask → Scraper Orchestrator
• Dispatch HTTP requests (requests module)
• Spin up headless browser tasks (Selenium)
• Parse HTML (BeautifulSoup, regex)
4. Scraper Orchestrator → External IITM Websites (data fetch)
5. Scraper Orchestrator → Flask
• Returns structured JSON or Python objects
6. Flask → Browser
• Renders Jinja2 templates or returns JSON endpoints
• Embeds Cytoscape.js graphs for interactivity
7. Browser ↔ Cytoscape.js
• Renders and updates graph based on provided data
Architectural patterns or design principles
• Monolithic layered architecture (Presentation → Application → Data-fetching)
• “Serverless” data store for sessions (encrypted cookies rather than DB)
• Separation of concerns: Scraper logic separated from routing logic
• Use of external services (IITM websites) as data providers
Technologies & frameworks to annotate
• Flask (Python web framework)
• Requests, Selenium, BeautifulSoup (web scraping)
• cryptography.Fernet (AES-GCM) for sessions
• Cytoscape.js (client-side graph visualization)
• Jinja2 (templating engine)
Diagramming recommendations
• Use rectangles for components (Flask, Scraper, Session Manager, Browser).
– Blue for backend services
– Green for client/browser
– Orange for external websites
– Gray for static assets
Essential elements to capture
– Notation showing cryptography step in session handling
– Visualization library (Cytoscape.js) as a client-side subcomponent
– External IITM websites as separate blocks