How pawan459/pdf_crawler works
Main Components
A. Frontend Layer (sih-front-new)
• React Application
• Key Components:
• Authentication (Login/Signup)
• Dashboard
• URL Management
• Report Generation
• Search Interface
B. Backend Layer (sih-back-new)
• Node.js Server
• API Endpoints
• Python Integration
• Authentication System
• PDF Crawling Engine
C. Database Layer
• PostgreSQL Database
• URL Storage
• User Management
• PDF Metadata Storage
D. External Services
• PDF Crawling Service
• Notification System
• [Client Tier] → [Application Tier] → [Data Tier]
Specific Elements
Frontend Section:
• Protected routes flow
• User interface components (Login, Dashboard, URL Management)
Backend Section:
• API endpoints
• Authentication middleware
• Python integration for crawling
• URL processing flow
Database Section:
• Main database tables
• Relationships between entities
• Data flow connections
Connection Types
• Double lines: Data streaming
Key Workflows to Highlight
a) PDF Crawling Flow:
• URL Input → Crawler Service → PDF Detection → Storage → Notification
b) Authentication Flow:
• Login/Signup → JWT Authentication → Protected Routes
c) URL Management Flow:
• URL Addition → Validation → Storage → Monitoring
Additional Elements
• A notification system component
• PDF processing pipeline
• Monitoring and scheduling services
• Error handling paths
Notes for Implementation
• Maintain proper spacing between components
• Version information
• Brief descriptions for complex flows
• Scaling points
• Security boundaries
Technical Details
• Node.js backend framework
• React frontend framework
• PostgreSQL database
• Python crawler integration
• Authentication mechanisms
• API endpoint structure
• File storage system
Remember to:
• Use appropriate scaling
• Critical paths
• Error handling flows
• Monitoring points
• Security boundaries
This system design should effectively communicate the architecture while maintaining clarity and providing sufficient detail for understanding the complete system flow.