# loid122/interactive_courses_scraper architecture

> Architecture diagram of the GitHub repository loid122/interactive_courses_scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/loid122/interactive_courses_scraper
- Repository: https://github.com/loid122/interactive_courses_scraper

## Overview

##### Project type and purpose
- This is a monolithic full-stack web application (“Interactive_courses_scraper”) built in Flask.
- Its purpose is to fetch and process data from multiple IITM websites, then present interactive visualizations in the browser.

##### File structure
- Top-level artifacts:
– flaskapp.py (the Flask entry point)
– Requirements.txt (dependencies)
– static/ (JavaScript, CSS, images)
– templates/ (Jinja2 HTML views)
- No database directory—sessions are encrypted and stored client-side.
- No microservice folders—everything lives in one codebase.

##### Extract key architectural details from the README
- Backend: Flask for HTTP routing and session management.
- Scraping sub-modules: Requests, Selenium, BeautifulSoup, regex.
- Security: AES-GCM (via Fernet) for encrypting session payload in cookies.
- Frontend: Cytoscape.js for graph/visualization, served as static assets.

##### Main components
A. Client Browser (UI layer)
– Hosts HTML templates and runs Cytoscape.js for interactive graphs
B. Flask Application
– HTTP Server / Router (flaskapp.py)
– Session Manager (Fernet crypto)
– Scraper Orchestrator (coordinates Requests + Selenium + BS4)
C. External IITM Websites
– Target endpoints for data extraction
D. Static Asset Server
– Serves JS/CSS/images from static/
E. (Optional) Logging or Error-handling sub-component

##### Relationships & interactions
1. Browser → Flask Server (HTTP GET/POST)
2. Flask → Session Manager
- Decrypt/validate session cookie, or create a new one
3. Flask → Scraper Orchestrator
- Dispatch HTTP requests (requests module)
- Spin up headless browser tasks (Selenium)
- Parse HTML (BeautifulSoup, regex)
4. Scraper Orchestrator → External IITM Websites (data fetch)
5. Scraper Orchestrator → Flask
- Returns structured JSON or Python objects
6. Flask → Browser
- Renders Jinja2 templates or returns JSON endpoints
- Embeds Cytoscape.js graphs for interactivity
7. Browser ↔ Cytoscape.js
- Renders and updates graph based on provided data

##### Architectural patterns or design principles
- Monolithic layered architecture (Presentation → Application → Data-fetching)
- “Serverless” data store for sessions (encrypted cookies rather than DB)
- Separation of concerns: Scraper logic separated from routing logic
- Use of external services (IITM websites) as data providers

##### Technologies & frameworks to annotate
- Flask (Python web framework)
- Requests, Selenium, BeautifulSoup (web scraping)
- cryptography.Fernet (AES-GCM) for sessions
- Cytoscape.js (client-side graph visualization)
- Jinja2 (templating engine)

##### Diagramming recommendations
- Use rectangles for components (Flask, Scraper, Session Manager, Browser).
– Blue for backend services
– Green for client/browser
– Orange for external websites
– Gray for static assets

##### Essential elements to capture
– Notation showing cryptography step in session handling
– Visualization library (Cytoscape.js) as a client-side subcomponent
– External IITM websites as separate blocks

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Client Layer
    subgraph "Client Browser"
        direction TB
        Browser["Web Browser"]:::client
        subgraph "UI Layer"
            direction TB
            Templates["Jinja2 Templates"]:::client
            Cyt["Cytoscape.js"]:::client
        end
        Browser -->|"renders templates"| Templates
        Browser -->|"runs visualizations"| Cyt
    end

    Browser -->|"HTTP GET/POST"| FlaskApp

    %% Server Layer
    subgraph "Flask Application"
        direction TB
        FlaskApp["Flask App\n(flaskapp.py)"]:::backend
        Session["Session Manager\nAES-GCM (Fernet)"]:::backend
        Scraper["Scraper Orchestrator\n(Requests, Selenium, BS4, regex)"]:::backend
        subgraph "Static Asset Server"
            direction TB
            JS["1.js"]:::static
            CSS["style.css"]:::static
            Images["images/"]:::static
        end
        Cookie["Encrypted Session Cookie"]:::storage

        FlaskApp -->|"decrypt/validate session"| Session
        Session -->|"reads/writes cookie"| Cookie
        FlaskApp -->|"dispatch scraping"| Scraper
        FlaskApp -->|"serves assets"| JS
        FlaskApp -->|"serves assets"| CSS
        FlaskApp -->|"serves assets"| Images
    end

    Scraper -->|"Fetch Data"| External
    External["IITM Websites"]:::external
    Scraper -->|"return data"| FlaskApp
    FlaskApp -->|"Render HTML/JSON"| Browser

    %% Documentation and Dependencies
    subgraph "Project Artifacts"
        direction TB
        Req["Requirements.txt"]:::docs
        Readme["README.md"]:::docs
    end

    %% Click Events
    click FlaskApp "https://github.com/loid122/interactive_courses_scraper/blob/main/flaskapp.py"
    click Session "https://github.com/loid122/interactive_courses_scraper/blob/main/flaskapp.py"
    click Scraper "https://github.com/loid122/interactive_courses_scraper/blob/main/flaskapp.py"
    click JS "https://github.com/loid122/interactive_courses_scraper/blob/main/static/1.js"
    click CSS "https://github.com/loid122/interactive_courses_scraper/blob/main/static/css/style.css"
    click Images "https://github.com/loid122/interactive_courses_scraper/tree/main/static/images/"
    click Templates "https://github.com/loid122/interactive_courses_scraper/tree/main/templates/"
    click Req "https://github.com/loid122/interactive_courses_scraper/blob/main/Requirements.txt"
    click Readme "https://github.com/loid122/interactive_courses_scraper/blob/main/README.md"
    
    %% Template Clicks
    click home "https://github.com/loid122/interactive_courses_scraper/blob/main/templates/home.html"
    click index "https://github.com/loid122/interactive_courses_scraper/blob/main/templates/index.html"
    click login "https://github.com/loid122/interactive_courses_scraper/blob/main/templates/login.html"
    click dashboard "https://github.com/loid122/interactive_courses_scraper/blob/main/templates/dashboard.html"
    click attendance "https://github.com/loid122/interactive_courses_scraper/blob/main/templates/attendance.html"
    click viewgrades "https://github.com/loid122/interactive_courses_scraper/blob/main/templates/viewgrades.html"
    click nfc "https://github.com/loid122/interactive_courses_scraper/blob/main/templates/nfc_expenditure.html"
    click indi "https://github.com/loid122/interactive_courses_scraper/blob/main/templates/indi.html"
    click indo "https://github.com/loid122/interactive_courses_scraper/blob/main/templates/indo.html"

    %% Styles
    classDef client fill:#e0f7e9,stroke:#2d8f3f,color:#000
    classDef backend fill:#d0e3ff,stroke:#1f5bbf,color:#000
    classDef static fill:#f2f2f2,stroke:#999,color:#000
    classDef storage fill:#f0f0f0,stroke:#666,color:#000,stroke-dasharray: 5 5
    classDef external fill:#ffe0b2,stroke:#ff8f00,color:#000
    classDef docs fill:#f9f0ff,stroke:#9c27b0,color:#000
```
