# badarulnisa/reverse-engine architecture

> Architecture diagram of the GitHub repository badarulnisa/reverse-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/badarulnisa/reverse-engine
- Repository: https://github.com/badarulnisa/reverse-engine

## Overview

This repository is a collection of registry-specific extraction pipelines, with a secondary HAR/API-analysis and generic collection toolkit. The main user workflow is to choose a registry, enumerate or paginate its records using that registry’s access strategy, fetch details where available, preserve progress, and export a workbook. The README documents seven registry targets and their differing techniques. The sampled source confirms selected request, parsing, token, and traversal behavior; unsampled registry internals are represented only where the tree or README supports them.

Main entry points: [`adgm_scraper/run_adgm.py`](https://github.com/badarulnisa/reverse-engine/blob/main/adgm_scraper/run_adgm.py) (ADGM Crawl), [`jafza_scrapper/run_enrich.py`](https://github.com/badarulnisa/reverse-engine/blob/main/jafza_scrapper/run_enrich.py) (JAFZA Enrichment), [`generated_collectors/api_client.py`](https://github.com/badarulnisa/reverse-engine/blob/main/generated_collectors/api_client.py) (RPC Replay Client), [`jafza_scrapper/places_client.py`](https://github.com/badarulnisa/reverse-engine/blob/main/jafza_scrapper/places_client.py) (Places Client)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_registry_crawls["Registry Crawls"]
  node_dmcc["DMCC Scraper<br/>[main.py]"]
  node_difc["DIFC Scraper<br/>[difc_scraper.py]"]
  node_adgm["ADGM Crawl<br/>[run_adgm.py]"]
  node_dfsa["DFSA Registers<br/>[dfsa_common.py]"]
  node_jafza["JAFZA Enrichment<br/>[run_enrich.py]"]
  node_investdubai["Invest Dubai Crawl"]
  node_qatar["Qatar Chamber"]
  node_sharjah["Sharjah Crawl<br/>[sharjah_scrape.py]"]
end

subgraph group_shared_collection["Collection Support"]
  node_rpc_client["RPC Replay Client<br/>[api_client.py]"]
  node_token_manager["Token Extraction<br/>[token_manager.py]"]
  node_fanout["Prefix Fanout<br/>[fanout_engine.py]"]
  node_record_parser["Record Parsing<br/>[parser.py]"]
  node_checkpoint_export["Checkpoint and Export<br/>[run_adgm.py]"]
end

subgraph group_analysis_toolkit["HAR Analysis"]
  node_har_input["HAR Capture"]
  node_har_parser["HAR Parser<br/>[har_parser.py]"]
  node_audit["Audit and Findings<br/>[main.py]"]
end

subgraph group_enrichment["Data Enrichment"]
  node_places_client["Places Client<br/>[places_client.py]"]
end

node_operator(("Operator"))
node_registry_sites{{"Public Registries"}}
node_places_api{{"Google Places"}}
node_workbook["Excel Workbook"]

node_operator -->|"runs"| node_dmcc
node_operator -->|"runs"| node_difc
node_operator -->|"runs"| node_adgm
node_operator -->|"runs"| node_dfsa
node_operator -->|"runs"| node_jafza
node_operator -->|"runs"| node_investdubai
node_operator -->|"runs"| node_qatar
node_operator -->|"runs"| node_sharjah
node_dmcc -->|"replays calls"| node_rpc_client
node_token_manager -->|"provides tokens"| node_rpc_client
node_rpc_client -->|"requests"| node_registry_sites
node_fanout -->|"queries prefixes"| node_rpc_client
node_difc -->|"sweeps queries"| node_registry_sites
node_adgm -->|"partitions listings"| node_registry_sites
node_adgm -->|"saves progress"| node_checkpoint_export
node_dfsa -->|"fetches listings"| node_registry_sites
node_jafza -->|"collects records"| node_registry_sites
node_investdubai -->|"enumerates records"| node_registry_sites
node_qatar -->|"fetches pages"| node_registry_sites
node_sharjah -->|"paginates results"| node_registry_sites
node_jafza -->|"enriches matches"| node_places_client
node_places_client -.->|"searches places"| node_places_api
node_places_client -->|"returns candidates"| node_jafza
node_har_input -->|"provides capture"| node_har_parser
node_har_parser -->|"normalizes traffic"| node_audit
node_audit -->|"writes report"| node_workbook
node_adgm -->|"exports records"| node_workbook

click node_dmcc "https://github.com/badarulnisa/reverse-engine/blob/main/main.py"
click node_difc "https://github.com/badarulnisa/reverse-engine/blob/main/difc_scraper.py"
click node_adgm "https://github.com/badarulnisa/reverse-engine/blob/main/adgm_scraper/run_adgm.py"
click node_dfsa "https://github.com/badarulnisa/reverse-engine/blob/main/dfsa_scraper/dfsa_common.py"
click node_jafza "https://github.com/badarulnisa/reverse-engine/blob/main/jafza_scrapper/run_enrich.py"
click node_investdubai "https://github.com/badarulnisa/reverse-engine/blob/main/investindubai/visitdubai_bruteforce.py"
click node_qatar "https://github.com/badarulnisa/reverse-engine/blob/main/qatrcid/qatarcid_scraper.py"
click node_sharjah "https://github.com/badarulnisa/reverse-engine/blob/main/sharjah-scrapper/sharjah_scrape.py"
click node_rpc_client "https://github.com/badarulnisa/reverse-engine/blob/main/generated_collectors/api_client.py"
click node_token_manager "https://github.com/badarulnisa/reverse-engine/blob/main/generated_collectors/token_manager.py"
click node_fanout "https://github.com/badarulnisa/reverse-engine/blob/main/generated_collectors/fanout_engine.py"
click node_record_parser "https://github.com/badarulnisa/reverse-engine/blob/main/app/collectors/parser.py"
click node_checkpoint_export "https://github.com/badarulnisa/reverse-engine/blob/main/adgm_scraper/run_adgm.py"
click node_har_parser "https://github.com/badarulnisa/reverse-engine/blob/main/app/core/har_parser.py"
click node_audit "https://github.com/badarulnisa/reverse-engine/blob/main/main.py"
click node_places_client "https://github.com/badarulnisa/reverse-engine/blob/main/jafza_scrapper/places_client.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dmcc,node_difc,node_adgm,node_dfsa,node_jafza,node_investdubai,node_qatar,node_sharjah toneBlue
class node_rpc_client,node_token_manager,node_fanout,node_record_parser,node_checkpoint_export toneAmber
class node_har_input,node_har_parser,node_audit toneMint
class node_places_client toneRose
class node_operator,node_registry_sites,node_places_api,node_workbook toneIndigo
```

## Components

### Registry Crawls

- [DMCC Scraper](https://github.com/badarulnisa/reverse-engine/blob/main/main.py): `main.py`
- [DIFC Scraper](https://github.com/badarulnisa/reverse-engine/blob/main/difc_scraper.py): `difc_scraper.py`
- [ADGM Crawl](https://github.com/badarulnisa/reverse-engine/blob/main/adgm_scraper/run_adgm.py): `adgm_scraper/run_adgm.py`
- [DFSA Registers](https://github.com/badarulnisa/reverse-engine/blob/main/dfsa_scraper/dfsa_common.py): `dfsa_scraper/dfsa_common.py`
- [JAFZA Enrichment](https://github.com/badarulnisa/reverse-engine/blob/main/jafza_scrapper/run_enrich.py): `jafza_scrapper/run_enrich.py`
- [Invest Dubai Crawl](https://github.com/badarulnisa/reverse-engine/blob/main/investindubai/visitdubai_bruteforce.py): `investindubai/visitdubai_bruteforce.py`
- [Qatar Chamber](https://github.com/badarulnisa/reverse-engine/blob/main/qatrcid/qatarcid_scraper.py): `qatrcid/qatarcid_scraper.py`
- [Sharjah Crawl](https://github.com/badarulnisa/reverse-engine/blob/main/sharjah-scrapper/sharjah_scrape.py): `sharjah-scrapper/sharjah_scrape.py`

### Collection Support

- [RPC Replay Client](https://github.com/badarulnisa/reverse-engine/blob/main/generated_collectors/api_client.py): `generated_collectors/api_client.py`
- [Token Extraction](https://github.com/badarulnisa/reverse-engine/blob/main/generated_collectors/token_manager.py): `generated_collectors/token_manager.py`
- [Prefix Fanout](https://github.com/badarulnisa/reverse-engine/blob/main/generated_collectors/fanout_engine.py): `generated_collectors/fanout_engine.py`
- [Record Parsing](https://github.com/badarulnisa/reverse-engine/blob/main/app/collectors/parser.py): `app/collectors/parser.py`
- [Checkpoint and Export](https://github.com/badarulnisa/reverse-engine/blob/main/adgm_scraper/run_adgm.py): `adgm_scraper/run_adgm.py`

### HAR Analysis

- **HAR Capture**
- [HAR Parser](https://github.com/badarulnisa/reverse-engine/blob/main/app/core/har_parser.py): `app/core/har_parser.py`
- [Audit and Findings](https://github.com/badarulnisa/reverse-engine/blob/main/main.py): `main.py`

### Data Enrichment

- [Places Client](https://github.com/badarulnisa/reverse-engine/blob/main/jafza_scrapper/places_client.py): `jafza_scrapper/places_client.py`

### Other components

- **Operator**
- **Public Registries**
- **Google Places**
- **Excel Workbook**

## Connections

- Operator → DMCC Scraper: runs
- Operator → DIFC Scraper: runs
- Operator → ADGM Crawl: runs
- Operator → DFSA Registers: runs
- Operator → JAFZA Enrichment: runs
- Operator → Invest Dubai Crawl: runs
- Operator → Qatar Chamber: runs
- Operator → Sharjah Crawl: runs
- DMCC Scraper → RPC Replay Client: replays calls
- Token Extraction → RPC Replay Client: provides tokens
- RPC Replay Client → Public Registries: requests
- Prefix Fanout → RPC Replay Client: queries prefixes
- DIFC Scraper → Public Registries: sweeps queries
- ADGM Crawl → Public Registries: partitions listings
- ADGM Crawl → Checkpoint and Export: saves progress
- DFSA Registers → Public Registries: fetches listings
- JAFZA Enrichment → Public Registries: collects records
- Invest Dubai Crawl → Public Registries: enumerates records
- Qatar Chamber → Public Registries: fetches pages
- Sharjah Crawl → Public Registries: paginates results
- JAFZA Enrichment → Places Client: enriches matches
- Places Client → Google Places: searches places
- Places Client → JAFZA Enrichment: returns candidates
- HAR Capture → HAR Parser: provides capture
- HAR Parser → Audit and Findings: normalizes traffic
- Audit and Findings → Excel Workbook: writes report
- ADGM Crawl → Excel Workbook: exports records