# pawan459/pdf_crawler architecture

> Architecture diagram of the GitHub repository pawan459/pdf_crawler, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pawan459/pdf_crawler
- Repository: https://github.com/pawan459/pdf_crawler

## Overview

##### Main Components

A. Frontend Layer (sih-front-new)
- React Application
- Key Components:
- Authentication (Login/Signup)
- Dashboard
- URL Management
- Report Generation
- Search Interface

B. Backend Layer (sih-back-new)
- Node.js Server
- API Endpoints
- Python Integration
- Authentication System
- PDF Crawling Engine

C. Database Layer
- PostgreSQL Database
- URL Storage
- User Management
- PDF Metadata Storage

D. External Services
- PDF Crawling Service
- Notification System

- [Client Tier] → [Application Tier] → [Data Tier]

##### Specific Elements

Frontend Section:
- Protected routes flow
- User interface components (Login, Dashboard, URL Management)

Backend Section:
- API endpoints
- Authentication middleware
- Python integration for crawling
- URL processing flow

Database Section:
- Main database tables
- Relationships between entities
- Data flow connections

##### Connection Types

- Double lines: Data streaming

##### Key Workflows to Highlight

a) PDF Crawling Flow:
- URL Input → Crawler Service → PDF Detection → Storage → Notification

b) Authentication Flow:
- Login/Signup → JWT Authentication → Protected Routes

c) URL Management Flow:
- URL Addition → Validation → Storage → Monitoring

##### Additional Elements

- A notification system component
- PDF processing pipeline
- Monitoring and scheduling services
- Error handling paths

##### Notes for Implementation

- Maintain proper spacing between components
- Version information
- Brief descriptions for complex flows
- Scaling points
- Security boundaries

##### Technical Details

- Node.js backend framework
- React frontend framework
- PostgreSQL database
- Python crawler integration
- Authentication mechanisms
- API endpoint structure
- File storage system

Remember to:
- Use appropriate scaling
- Critical paths
- Error handling flows
- Monitoring points
- Security boundaries

This system design should effectively communicate the architecture while maintaining clarity and providing sufficient detail for understanding the complete system flow.

## Diagram (Mermaid)

```mermaid
%%{init: {'theme': 'base', 'themeVariables': { 'primaryColor': '#4C78A8', 'secondaryColor': '#82B366', 'tertiaryColor': '#FFB366', 'quaternaryColor': '#FF6666', 'quinaryColor': '#9673A6'}}}%%

flowchart TB
    subgraph Client["Client Layer"]
        direction TB
        FE["React Application"]
        AC["Authentication Components"]:::frontend
        DC["Dashboard Component"]:::frontend
        UC["URL Management Component"]:::frontend
        RC["Report Generation Component"]:::frontend
        SC["Search Interface Component"]:::frontend
        EH["Event Handling System"]:::frontend
        FC["Frontend Config"]:::frontend
    end

    subgraph Server["Server Layer"]
        direction TB
        NJS["Node.js Server"]
        
        subgraph API["API Layer"]
            direction TB
            AE["Authentication Endpoints"]:::api
            UE["URL Management Endpoints"]:::api
            SE["Search Endpoints"]:::api
            RE["Report Generation Endpoints"]:::api
        end
        
        AUTH["Authorization System"]:::security
        BC["Backend Config"]:::config
    end

    subgraph Service["Service Layer"]
        direction TB
        PC["Python Crawler"]:::crawler
        NS["Notification Service"]:::service
    end

    subgraph Data["Data Layer"]
        direction TB
        DB[(PostgreSQL Database)]:::database
    end

    %% Connections
    FE --> AC & DC & UC & RC & SC
    AC --> EH
    DC --> EH
    UC --> EH
    RC --> EH
    SC --> EH
    EH --> AE & UE & SE & RE
    
    AE & UE & SE & RE --> AUTH
    AUTH --> DB
    
    UE --> PC
    PC --> NS
    PC --> DB
    
    %% Click Events
    click AC "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-front-new/src/Components/Login.js"
    click DC "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-front-new/src/Components/dashboard.js"
    click UC "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-front-new/src/Components/manageUrl.js"
    click RC "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-front-new/src/Components/generateReport.js"
    click SC "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-front-new/src/Components/search.js"
    click EH "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-front-new/src/utils/event.js"
    click FC "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-front-new/src/config.json"
    click API "https://github.com/Pawan459/PDF_Crawler/tree/master//SIH/sih-back-new/api/"
    click AE "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-back-new/api/login.js"
    click UE "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-back-new/api/addUrl.js"
    click SE "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-back-new/api/search.js"
    click RE "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-back-new/api/generate.js"
    click AUTH "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-back-new/util/authorize.js"
    click BC "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-back-new/config.json"
    click DB "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-back-new/db.js"
    click PC "https://github.com/Pawan459/PDF_Crawler/blob/master//SIH/sih-back-new/api/python/test.py"

    %% Styling
    classDef frontend fill:#4C78A8,color:white
    classDef api fill:#82B366,color:white
    classDef security fill:#FFB366,color:black
    classDef service fill:#FF6666,color:white
    classDef database fill:#9673A6,color:white
    classDef config fill:#CCCCCC,color:black
    classDef crawler fill:#FF6666,color:white

    %% Legend
    subgraph Legend
        direction TB
        F["Frontend Component"]:::frontend
        A["API Component"]:::api
        S["Security Component"]:::security
        SE["Service Component"]:::service
        D["Database"]:::database
        C["Configuration"]:::config
        CR["Crawler"]:::crawler
    end
```
