How sudhanshu1402/technical_internship_program works
Project type and purpose
• This is a monorepo of Python “web‑scraping” (crawler) projects built with Scrapy.
• Each top‑level folder (EXTRA_WEBSITES, RETAIL_WEBSITES, TYRES) groups one or more Scrapy spiders that extract structured data from specific target sites and persist it in CSV, Excel, SQL or MongoDB.
File‐structure analysis
• Top‑level: three domain folders + a global README.
• Under each domain folder: one subfolder per target website.
• Each subfolder contains:
– items.py (data schema)
– spider module(s) (*.py)
– pipelines.py (post‑processing & storage)
– middlewares.py (request/response hooks)
– settings.py (crawler configuration)
– sometimes CSV or DOCX storage templates
• No centralized scheduler: each spider group is self‑contained.
README insights
• Uses Python with Scrapy framework plus optional BeautifulSoup, Selenium.
• Storage via CSV, Excel, MongoDB, SQL.
• Main scraping methods: HTML/DOM/XPath parsing.
a. Main components
1. Scheduler/CLI (developer or cron job triggers)
2. Scrapy Engine
• Downloader Middleware
• Scheduler (internal queue)
3. Spider Modules (one per site)
4. Item Pipeline
5. Storage Adapters:
– CSV/Excel writer
– MongoDB client
– SQL client
6. External Websites (targets)
b. Relationships & data flow
1. Scheduler → Scrapy Engine (start requests)
2. Engine → Downloader Middleware → HTTP(s) requests → External Website
3. External Website → HTTP response → Downloader Middleware → Spider parse()
4. Spider parse() → yields Items → Engine → Item Pipeline
5. Pipeline processes items → hands off to Storage Adapter(s)
6. Storage Adapter writes to CSV/Excel/DB
c. Architectural patterns/design principles
• Scrapy’s event‑driven crawler architecture (engine + spider callbacks + pipeline)
• Modular spider design: each site lives in its own submodule for isolation
• Middleware pipeline pattern for request/response customization
• Item pipeline pattern for sequential data transformation and output
d. Key technologies/frameworks
• Python 3.x, Scrapy framework
• Optional: BeautifulSoup, Selenium for dynamic pages
• Storage: CSV modules, Pandas/Excel writer, PyMongo (MongoDB), SQLAlchemy/pyodbc