# sangharshbyss/ecourtsdata architecture

> Architecture diagram of the GitHub repository sangharshbyss/ecourtsdata, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sangharshbyss/ecourtsdata
- Repository: https://github.com/sangharshbyss/ecourtsdata

## Overview

This repo is a **Python automation / data-extraction pipeline** (not a server app). Its purpose is to:

1. **Query sessions court case records by “Act”** (notably SC/ST Act 1989) on an eCourts-like website.
2. **Download each case’s HTML/PDF/attachment(s)** into a local folder structure (likely per district/court).
##### Aggregate CSVs into one CSV per district court.

- an external dependency: **eCourts website**
- local persistence: **folders + CSV artifacts**
- Python modules that implement: **automation → parsing → normalization → aggregation**

##### Data acquisition / web automation (Downloader)
Likely files:
- `download.py` — entry point for downloading case files/pages
- `fill_the_form.py` — automation to fill search forms (Act, district, etc.)
- `Page Object.py` — page-object abstraction for the site’s pages (Selenium-style)
- `district.py` — district/court enumeration, routing, or iteration logic
- `startfresh2.py` — orchestration/reset/retry workflow; possibly “run from scratch”

**External system:** target “sessions court / eCourts portal” website
**Key outputs:** downloaded files into a local folder per district/court/case

##### Transformation / parsing (HTML → CSV)
Likely files:
- `htmltoCsv_2.py` — parser/converter that reads downloaded HTML (or saved pages) and emits CSV rows
- `csvModule.py` — shared CSV writing utilities / schema helpers
- `samplePage.html` — a fixture used to test parsing assumptions

**Input:** downloaded HTML/pages
**Output:** intermediate CSV files (often one per case or per download batch)

##### Aggregation (CSV merge per district)
Likely files:
- `allCsv.py` — combines many CSVs into a single district-level CSV

**Input:** many per-case/per-batch CSVs
**Output:** one consolidated CSV per district court

##### Cross-cutting concerns (should still appear
- **Filesystem storage layout** (directories, naming conventions)
- **Config / parameters** (act name, district list, output paths)
- **Logging / error handling / retries / throttling** (critical for scraping pipelines)
- **Manual operator** (someone triggers scripts and monitors results)

##### What relationships/interactions to show (dataflow + control flow)
- **Control flow** (which script calls which module / stage order)
- **Data flow** (what artifact is produced/consumed)

##### Operator (User)
- runs `startfresh2.py` or `download.py` (whichever is the real entry point)
##### Downloader stage
- sends HTTP/browser automation to **eCourts website**
- receives HTML/pages/docs
- writes files to **Local Download Folder**
##### Converter stage (`htmltoCsv_2.py`)
- reads from **Local Download Folder**
- uses `csvModule.py`
- writes **Intermediate CSV Folder**
##### Aggregator (`allCsv.py`)
- reads all intermediate CSVs for a district
- outputs **District-level CSV**

If there is district iteration:
- A **loop/iterator** component (likely `district.py`) feeding district/court identifiers into both download and merge steps.

##### Architectural pattern(s) to explicitly label
This repo is best described as:
- **Batch ETL / Pipeline architecture**
- Extract: web download
- Transform: HTML parsing/normalization
- Load: CSV artifacts (local “data lake”)
- **Page Object pattern** (strongly suggested by `Page Object.py`)
- Site pages as objects; isolate selectors and navigation logic
- **File-based staging** (filesystem as the integration boundary between steps)
- stage boundaries: “downloaded pages” and “intermediate CSVs”

##### CSV Aggregation

Explicit storage nodes between them:
- “Downloaded Case Files Folder”
- “Intermediate CSV Folder”
- “Final District CSV Output Folder”

Inside each stage, show the modules:
- Download/Automation:
- `startfresh2.py` (Orchestrator)
- `download.py` (Downloader)
- `fill_the_form.py` (Form automation)
- `Page Object.py` (Page objects / selectors)
- `district.py` (District iteration/metadata)
- Transform:
- `htmltoCsv_2.py` (Parser/Extractor/Normalizer)
- `csvModule.py` (CSV schema + writer helpers)
- `samplePage.html` (Parsing fixture / test input) — show as “Sample/Fixture” not production
- Aggregate:
- `allCsv.py` (Merger/Reducer)

Title: **EcourtsData – Batch Extraction & CSV Aggregation Architecture**

Left to right flow:

Operator
→ Orchestrator (`startfresh2.py`)
→ Download Stage:
- District Iterator (`district.py`)
- Page Objects (`Page Object.py`)
- Form Automation (`fill_the_form.py`)
- Downloader (`download.py`)
↔ External: eCourts Portal
→ Data Store: Downloaded Case Files Folder
→ Transform Stage:
- HTML→CSV (`htmltoCsv_2.py`)
- CSV Utilities (`csvModule.py`)
→ Data Store: Intermediate CSV Folder
→ Aggregate Stage:
- District Merger (`allCsv.py`)
→ Data Store: Final District CSVs

A note near `samplePage.html`: “Parser fixture / example input”.

That’s “detailed and separated into many components” while still matching the repo’s real structure and purpose.

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% Level 1: System Context
  %% =========================
  subgraph "System Context (Batch ETL)"
    OP["Operator\n(Manual run/monitor)"]:::operator
    EXT(("eCourts / Sessions Court Portal\n(External Website)")):::external
    FS[("Local Filesystem\n(Folder-based staging)")]:::store
  end

  OP -->|"runs_scripts"| ORCH
  DL -->|"browser_automation_http"| EXT
  EXT -->|"case_pages_html_pdf_attachments"| DL
  DL -->|"writes_downloads"| DLSTORE
  DLSTORE -->|"reads_html_pages"| XFORM
  XFORM -->|"writes_intermediate_csvs"| CSVSTORE
  CSVSTORE -->|"reads_per_case_or_batch_csvs"| AGG
  AGG -->|"writes_district_level_csv"| OUTSTORE

  %% =========================
  %% Level 2/3: Pipeline Stages (Components)
  %% =========================
  subgraph "EcourtsData Pipeline (This Repo) — Batch ETL / Pipeline"
    direction TB

    subgraph "Stage 0: Run Control"
      direction TB
      ORCH["Run Orchestrator\n(startfresh2.py)\n- cleanup/reset\n- retries/resume"]:::exec
      CFG["Config / Parameters\n(act_name, district_list,\noutput_paths)"]:::config
      LOG["Logging / Error Handling\n(retry/backoff/throttling)"]:::ops

      ORCH -->|"loads_params"| CFG
      ORCH -->|"emits_logs"| LOG
    end

    subgraph "Stage 1: Extract — Download / Automation"
      direction TB
      DIST["District Iterator / Metadata\n(district.py)\n(loop: per_district/court)"]:::exec
      PO["Page Objects\n(Page Object.py)\n(Page Object pattern)"]:::exec
      FORM["Form Automation\n(fill_the_form.py)\n(Act + district search)"]:::exec
      DL["Downloader\n(download.py)\n(fetch_case_pages/files)"]:::exec

      ORCH -->|"orchestrates"| DIST
      ORCH -->|"orchestrates"| DL
      DIST -->|"provides_district_params"| DL
      DL -->|"uses_selectors_navigation"| PO
      DL -->|"fills_search_forms"| FORM
      LOG -.->|"handles_failures"| DL
    end

    DLSTORE[("Downloaded Case Files Folder\n(HTML/PDF/attachments)\n(per_district/court/case)")]:::store

    subgraph "Stage 2: Transform — HTML → CSV"
      direction TB
      XFORM["HTML→CSV Converter\n(htmltoCsv_2.py)\n(parse/extract/normalize)"]:::exec
      CSVU["CSV Utils\n(csvModule.py)\n(schema + writers)"]:::exec
      FIX["Parser Fixture\n(samplePage.html)\n(example input)"]:::fixture

      ORCH -->|"orchestrates"| XFORM
      XFORM -->|"uses"| CSVU
      FIX -.->|"validates_parsing_assumptions"| XFORM
      LOG -.->|"logs_parse_errors"| XFORM
    end

    CSVSTORE[("Intermediate CSV Folder\n(per_case_or_batch_csv)")]:::store

    subgraph "Stage 3: Load/Reduce — Aggregation"
      direction TB
      AGG["District CSV Merger\n(allCsv.py)\n(reduce: per_district)"]:::exec
      ORCH -->|"orchestrates"| AGG
      DIST -->|"district_scope"| AGG
      LOG -.->|"logs_merge_issues"| AGG
    end

    OUTSTORE[("Final District CSV Output Folder\n(one_csv_per_district)")]:::store
  end

  %% Tie filesystem node to stores (contextual)
  DLSTORE --- FS
  CSVSTORE --- FS
  OUTSTORE --- FS

  %% Annotations
  NOTE1["Pattern: File-based staging\n(downloads → intermediate_csv → final_csv)"]:::note
  NOTE2["Batch/Loop: per_district → per_case\n(idempotency via startfresh)"]:::note
  NOTE1 -.-> DLSTORE
  NOTE1 -.-> CSVSTORE
  NOTE1 -.-> OUTSTORE
  NOTE2 -.-> DIST

  %% Documentation
  DOC["Pipeline Documentation\n(README.md)"]:::doc
  OP -.->|"references"| DOC

  %% Click Events (from component mapping)
  click ORCH "https://github.com/sangharshbyss/ecourtsdata/blob/master/startfresh2.py"
  click DL "https://github.com/sangharshbyss/ecourtsdata/blob/master/download.py"
  click FORM "https://github.com/sangharshbyss/ecourtsdata/blob/master/fill_the_form.py"
  click PO "https://github.com/sangharshbyss/ecourtsdata/blob/master/Page Object.py"
  click DIST "https://github.com/sangharshbyss/ecourtsdata/blob/master/district.py"
  click XFORM "https://github.com/sangharshbyss/ecourtsdata/blob/master/htmltoCsv_2.py"
  click CSVU "https://github.com/sangharshbyss/ecourtsdata/blob/master/csvModule.py"
  click AGG "https://github.com/sangharshbyss/ecourtsdata/blob/master/allCsv.py"
  click FIX "https://github.com/sangharshbyss/ecourtsdata/blob/master/samplePage.html"
  click DOC "https://github.com/sangharshbyss/ecourtsdata/blob/master/README.md"

  %% Styles
  classDef operator fill:#111827,stroke:#0b1220,color:#ffffff,stroke-width:1.5px;
  classDef external fill:#e5e7eb,stroke:#6b7280,color:#111827,stroke-width:1.5px;
  classDef exec fill:#dbeafe,stroke:#2563eb,color:#0b1220,stroke-width:1.5px;
  classDef store fill:#dcfce7,stroke:#16a34a,color:#052e16,stroke-width:1.5px;
  classDef ops fill:#ffedd5,stroke:#f97316,color:#7c2d12,stroke-width:1.5px;
  classDef config fill:#ede9fe,stroke:#7c3aed,color:#2e1065,stroke-width:1.5px;
  classDef note fill:#fef9c3,stroke:#ca8a04,color:#713f12,stroke-width:1px,stroke-dasharray: 4 3;
  classDef fixture fill:#ffe4e6,stroke:#e11d48,color:#881337,stroke-width:1.5px;
  classDef doc fill:#cffafe,stroke:#0891b2,color:#083344,stroke-width:1.5px;
```
