How sangharshbyss/ecourtsdata works
This repo is a Python automation / data-extraction pipeline (not a server app). Its purpose is to:
1. Query sessions court case records by “Act” (notably SC/ST Act 1989) on an eCourts-like website.
2. Download each case’s HTML/PDF/attachment(s) into a local folder structure (likely per district/court).
Aggregate CSVs into one CSV per district court.
• an external dependency: eCourts website
• local persistence: folders + CSV artifacts
• Python modules that implement: automation → parsing → normalization → aggregation
Data acquisition / web automation (Downloader)
Likely files:
• download.py — entry point for downloading case files/pages
• fill_the_form.py — automation to fill search forms (Act, district, etc.)
• Page Object.py — page-object abstraction for the site’s pages (Selenium-style)
• district.py — district/court enumeration, routing, or iteration logic
• startfresh2.py — orchestration/reset/retry workflow; possibly “run from scratch”
External system: target “sessions court / eCourts portal” website
Key outputs: downloaded files into a local folder per district/court/case
Transformation / parsing (HTML → CSV)
Likely files:
• htmltoCsv_2.py — parser/converter that reads downloaded HTML (or saved pages) and emits CSV rows
• csvModule.py — shared CSV writing utilities / schema helpers
• samplePage.html — a fixture used to test parsing assumptions
Input: downloaded HTML/pages
Output: intermediate CSV files (often one per case or per download batch)
Aggregation (CSV merge per district)
Likely files:
• allCsv.py — combines many CSVs into a single district-level CSV
Input: many per-case/per-batch CSVs
Output: one consolidated CSV per district court
Cross-cutting concerns (should still appear
• Filesystem storage layout (directories, naming conventions)
• Config / parameters (act name, district list, output paths)
• Logging / error handling / retries / throttling (critical for scraping pipelines)
• Manual operator (someone triggers scripts and monitors results)
What relationships/interactions to show (dataflow + control flow)
• Control flow (which script calls which module / stage order)
• Data flow (what artifact is produced/consumed)
Operator (User)
• runs startfresh2.py or download.py (whichever is the real entry point)
Downloader stage
• sends HTTP/browser automation to eCourts website
• receives HTML/pages/docs
• writes files to Local Download Folder
Converter stage (htmltoCsv_2.py)
• reads from Local Download Folder
• uses csvModule.py
• writes Intermediate CSV Folder
Aggregator (allCsv.py)
• reads all intermediate CSVs for a district
• outputs District-level CSV
If there is district iteration:
• A loop/iterator component (likely district.py) feeding district/court identifiers into both download and merge steps.
Architectural pattern(s) to explicitly label
This repo is best described as:
• Batch ETL / Pipeline architecture
• Extract: web download
• Transform: HTML parsing/normalization
• Load: CSV artifacts (local “data lake”)
• Page Object pattern (strongly suggested by Page Object.py)
• Site pages as objects; isolate selectors and navigation logic
• File-based staging (filesystem as the integration boundary between steps)
• stage boundaries: “downloaded pages” and “intermediate CSVs”
CSV Aggregation
Explicit storage nodes between them:
• “Downloaded Case Files Folder”
• “Intermediate CSV Folder”
• “Final District CSV Output Folder”
Inside each stage, show the modules:
• Download/Automation:
• startfresh2.py (Orchestrator)
• download.py (Downloader)
• fill_the_form.py (Form automation)
• Page Object.py (Page objects / selectors)
• district.py (District iteration/metadata)
• Transform:
• htmltoCsv_2.py (Parser/Extractor/Normalizer)
• csvModule.py (CSV schema + writer helpers)
• samplePage.html (Parsing fixture / test input) — show as “Sample/Fixture” not production
• Aggregate:
• allCsv.py (Merger/Reducer)
Title: EcourtsData – Batch Extraction & CSV Aggregation Architecture
Left to right flow:
Operator
→ Orchestrator (startfresh2.py)
→ Download Stage:
• District Iterator (district.py)
• Page Objects (Page Object.py)
• Form Automation (fill_the_form.py)
• Downloader (download.py)
↔ External: eCourts Portal
→ Data Store: Downloaded Case Files Folder
→ Transform Stage:
• HTML→CSV (htmltoCsv_2.py)
• CSV Utilities (csvModule.py)
→ Data Store: Intermediate CSV Folder
→ Aggregate Stage:
• District Merger (allCsv.py)
→ Data Store: Final District CSVs
A note near samplePage.html: “Parser fixture / example input”.
That’s “detailed and separated into many components” while still matching the repo’s real structure and purpose.