# aymaan-mulla/gdpr-compliant-multiplatform-scraper architecture

> Architecture diagram of the GitHub repository aymaan-mulla/gdpr-compliant-multiplatform-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper
- Repository: https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper

## Overview

This repository is a collection of local scripts for collecting social-media URLs, obtaining post and reel data through Bright Data, organizing results by collection, downloading linked media, and generating Word reports. The principal flow is URL categorization, Bright Data snapshot retrieval, collection organization, then media enrichment; report generation is a separate downstream utility. The README names a broader stack, but the sampled package and source show no Express, GraphQL, Neo4j, Prisma, or Redis runtime. Some listed scripts and data directories were not sampled, so their links are omitted.

Main entry points: [`organize.js`](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/organize.js) (Collection organizer), [`main.js`](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/main.js) (Post snapshot client), [`main2.js`](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/main2.js) (Reel snapshot client), [`list4.js`](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/list4.js) (URL categorizer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_url_prep["URL preparation"]
  node_saved_collection_scraper["Collection importer<br/>[scraper.js]"]
  node_categorized_urls["URL categorizer<br/>[list4.js]"]
  node_collection_inputs["Collection URL files"]
  node_post_urls["Post URL files"]
  node_reel_urls["Reel URL files"]
end

subgraph group_collection["Snapshot collection"]
  node_brightdata{{"Bright Data"}}
  node_post_snapshots["Post snapshot client<br/>[main.js]"]
  node_reel_snapshots["Reel snapshot client<br/>[main2.js]"]
  node_collection_snapshots["Collection snapshot client<br/>[main.js]"]
  node_post_data["Post scrape files"]
  node_reel_data["Reel scrape files"]
  node_csv_exports["CSV exports"]
end

subgraph group_media["Media organization"]
  node_organizer["Collection organizer<br/>[organize.js]"]
  node_organized_posts["Organized scrape data"]
  node_media_downloader["Media downloader<br/>[organ2.js]"]
  node_local_media["Downloaded media"]
  node_collection_linker["Python collection linker<br/>[doc2.py]"]
end

subgraph group_reporting["Report generation"]
  node_frame_extractor["Video frame extraction<br/>[test2.py]"]
  node_frame_copy["Frame transfer<br/>[test3.py]"]
  node_word_report["Word report generator<br/>[te.py]"]
  node_report_files["Word reports<br/>[te.py]"]
end

node_saved_collection_scraper -->|"writes JSON"| node_collection_inputs
node_collection_inputs -->|"reads URLs"| node_categorized_urls
node_categorized_urls -->|"writes posts"| node_post_urls
node_categorized_urls -->|"writes reels"| node_reel_urls
node_post_snapshots -->|"triggers snapshots"| node_brightdata
node_post_snapshots -->|"polls and downloads"| node_brightdata
node_reel_snapshots -->|"triggers snapshots"| node_brightdata
node_reel_snapshots -->|"polls and downloads"| node_brightdata
node_post_snapshots -->|"writes JSON"| node_post_data
node_reel_snapshots -->|"writes JSON"| node_reel_data
node_collection_snapshots -->|"triggers dataset"| node_brightdata
node_collection_snapshots -->|"writes CSV"| node_csv_exports
node_post_urls -->|"reads collection URLs"| node_organizer
node_reel_urls -->|"reads collection URLs"| node_organizer
node_post_data -->|"loads post data"| node_organizer
node_reel_data -->|"loads reel data"| node_organizer
node_organizer -->|"writes grouped data"| node_organized_posts
node_organized_posts -->|"reads media links"| node_media_downloader
node_media_downloader -->|"fetches media"| node_brightdata
node_media_downloader -->|"writes files"| node_local_media
node_collection_linker -->|"writes collection copies"| node_organized_posts
node_frame_extractor -.->|"produces frames"| node_frame_copy
node_frame_copy -.->|"provides frames"| node_word_report
node_word_report -->|"writes DOCX"| node_report_files

click node_saved_collection_scraper "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/scraper.js"
click node_categorized_urls "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/list4.js"
click node_collection_inputs "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/collections"
click node_post_urls "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/posts"
click node_reel_urls "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/reels"
click node_post_snapshots "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/main.js"
click node_reel_snapshots "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/main2.js"
click node_collection_snapshots "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/collections/main.js"
click node_post_data "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/fol2"
click node_reel_data "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/fol3"
click node_csv_exports "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/collections"
click node_organizer "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/organize.js"
click node_organized_posts "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/scraped"
click node_media_downloader "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/organ2.js"
click node_local_media "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/scraped"
click node_collection_linker "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/doc2.py"
click node_frame_extractor "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/test2.py"
click node_frame_copy "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/test3.py"
click node_word_report "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/te.py"
click node_report_files "https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/te.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_saved_collection_scraper,node_categorized_urls,node_collection_inputs,node_post_urls,node_reel_urls toneBlue
class node_brightdata,node_post_snapshots,node_reel_snapshots,node_collection_snapshots,node_post_data,node_reel_data,node_csv_exports toneAmber
class node_organizer,node_organized_posts,node_media_downloader,node_local_media,node_collection_linker toneMint
class node_frame_extractor,node_frame_copy,node_word_report,node_report_files toneRose
```

## Components

### URL preparation

- [Collection importer](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/scraper.js): `scraper.js`
- [URL categorizer](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/list4.js): `list4.js`
- [Collection URL files](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/collections): `collections`
- [Post URL files](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/posts): `posts`
- [Reel URL files](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/reels): `reels`

### Snapshot collection

- **Bright Data**
- [Post snapshot client](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/main.js): `main.js`
- [Reel snapshot client](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/main2.js): `main2.js`
- [Collection snapshot client](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/collections/main.js): `collections/main.js`
- [Post scrape files](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/fol2): `fol2`
- [Reel scrape files](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/fol3): `fol3`
- [CSV exports](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/collections): `collections`

### Media organization

- [Collection organizer](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/organize.js): `organize.js`
- [Organized scrape data](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/scraped): `scraped`
- [Media downloader](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/organ2.js): `organ2.js`
- [Downloaded media](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/tree/main/scraped): `scraped`
- [Python collection linker](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/doc2.py): `doc2.py`

### Report generation

- [Video frame extraction](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/test2.py): `test2.py`
- [Frame transfer](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/test3.py): `test3.py`
- [Word report generator](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/te.py): `te.py`
- [Word reports](https://github.com/aymaan-mulla/gdpr-compliant-multiplatform-scraper/blob/main/te.py): `te.py`

## Connections

- Collection importer → Collection URL files: writes JSON (evidence: `scraper.js`)
- Collection URL files → URL categorizer: reads URLs (evidence: `list4.js`)
- URL categorizer → Post URL files: writes posts (evidence: `list4.js`)
- URL categorizer → Reel URL files: writes reels (evidence: `list4.js`)
- Post snapshot client → Bright Data: triggers snapshots (evidence: `main.js`)
- Post snapshot client → Bright Data: polls and downloads (evidence: `main.js`)
- Reel snapshot client → Bright Data: triggers snapshots (evidence: `main2.js`)
- Reel snapshot client → Bright Data: polls and downloads (evidence: `main2.js`)
- Post snapshot client → Post scrape files: writes JSON (evidence: `main.js`)
- Reel snapshot client → Reel scrape files: writes JSON (evidence: `main2.js`)
- Collection snapshot client → Bright Data: triggers dataset (evidence: `collections/main.js`)
- Collection snapshot client → CSV exports: writes CSV (evidence: `collections/main.js`)
- Post URL files → Collection organizer: reads collection URLs (evidence: `organize.js`)
- Reel URL files → Collection organizer: reads collection URLs (evidence: `organize.js`)
- Post scrape files → Collection organizer: loads post data (evidence: `organize.js`)
- Reel scrape files → Collection organizer: loads reel data (evidence: `organize.js`)
- Collection organizer → Organized scrape data: writes grouped data (evidence: `organize.js`)
- Organized scrape data → Media downloader: reads media links (evidence: `organ2.js`)
- Media downloader → Bright Data: fetches media (evidence: `organ2.js`)
- Media downloader → Downloaded media: writes files (evidence: `organ2.js`)
- Python collection linker → Organized scrape data: writes collection copies (evidence: `doc2.py`)
- Video frame extraction → Frame transfer: produces frames
- Frame transfer → Word report generator: provides frames
- Word report generator → Word reports: writes DOCX (evidence: `te.py`)