# pratham-asthana/data_sync architecture

> Architecture diagram of the GitHub repository pratham-asthana/data_sync, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pratham-asthana/data_sync
- Repository: https://github.com/pratham-asthana/data_sync

## Overview

This repository syncs SharePoint document metadata into a Neo4j knowledge graph. The main workflow discovers newly added sites, traverses their document libraries, classifies matching files, exports JSON records, and has the CLI orchestrator prepare ingestion for Neo4j. Separate tools list/export arbitrary SharePoint contents, compare file snapshots, and inspect graph schema. Edges reflect README statements and shown calls; some execution and database details are described only at a high level because the relevant source is partial or unsampled.

Main entry points: [`sharepoint_new_site_pipeline.py`](https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_new_site_pipeline.py) (New Site Pipeline), [`main.py`](https://github.com/pratham-asthana/data_sync/blob/main/main.py) (Sync Orchestrator), [`comparison.py`](https://github.com/pratham-asthana/data_sync/blob/main/comparison.py) (Site Comparison), [`list_sharepoint_sites.py`](https://github.com/pratham-asthana/data_sync/blob/main/list_sharepoint_sites.py) (Site Listing)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_discovery["Site Discovery"]
  node_site_listing["Site Listing"]
  node_site_diff["Site Comparison<br/>[comparison.py]"]
  node_site_snapshot[("Site Registry")]
end

subgraph group_extraction["SharePoint Extraction"]
  node_site_pipeline["New Site Pipeline"]
  node_graph_api["Graph Client"]
  node_classification["Document Classification"]
  node_site_exports["Site JSON Exports"]
end

subgraph group_sync["Sync and Ingestion"]
  node_orchestrator["Sync Orchestrator<br/>[main.py]"]
  node_ingester["Cypher Ingestion<br/>[ingest_to_neo4j.py]"]
  node_cypher_output["Generated Cypher"]
end

subgraph group_utilities["Inspection Tools"]
  node_item_export["Nested JSON Export"]
  node_item_listing["Item Listing"]
  node_item_diff["Item Snapshot Comparison"]
  node_schema_extractor["Schema Extractor"]
  node_schema_output["Schema JSON<br/>[neo4j_schema.json]"]
end

node_operator(("Operator"))
node_sharepoint["SharePoint"]
node_neo4j[("Neo4j Graph")]

node_operator -->|"runs sync"| node_orchestrator
node_site_diff -->|"runs listing"| node_site_listing
node_site_listing -->|"lists sites"| node_sharepoint
node_site_listing -->|"writes sites"| node_site_snapshot
node_site_snapshot -->|"loads baseline"| node_site_diff
node_site_pipeline -->|"runs comparison"| node_site_diff
node_site_diff -->|"returns added sites"| node_site_pipeline
node_site_pipeline -->|"traverses drives"| node_graph_api
node_graph_api -->|"reads items"| node_sharepoint
node_site_pipeline -->|"classifies files"| node_classification
node_classification -->|"produces records"| node_site_exports
node_site_pipeline -->|"writes exports"| node_site_exports
node_operator -->|"runs listing"| node_item_listing
node_item_listing -->|"queries Graph"| node_sharepoint
node_operator -->|"runs export"| node_item_export
node_item_export -->|"queries Graph"| node_sharepoint
node_operator -->|"compares snapshots"| node_item_diff
node_item_diff -->|"uses current snapshot"| node_item_export
node_orchestrator -->|"runs pipeline"| node_site_pipeline
node_orchestrator -->|"loads JSON"| node_site_exports
node_orchestrator -->|"generates Cypher"| node_ingester
node_ingester -->|"writes script"| node_cypher_output
node_ingester -.->|"executes Cypher"| node_neo4j
node_operator -->|"runs extraction"| node_schema_extractor
node_schema_extractor -->|"reads schema"| node_neo4j
node_schema_extractor -->|"writes schema"| node_schema_output

click node_site_listing "https://github.com/pratham-asthana/data_sync/blob/main/list_sharepoint_sites.py"
click node_site_diff "https://github.com/pratham-asthana/data_sync/blob/main/comparison.py"
click node_site_snapshot "https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_sites.json"
click node_site_pipeline "https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_new_site_pipeline.py"
click node_graph_api "https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_items_json_with_date.py"
click node_classification "https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_new_site_pipeline.py"
click node_site_exports "https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_new_site_pipeline.py"
click node_orchestrator "https://github.com/pratham-asthana/data_sync/blob/main/main.py"
click node_ingester "https://github.com/pratham-asthana/data_sync/blob/main/ingest_to_neo4j.py"
click node_cypher_output "https://github.com/pratham-asthana/data_sync/blob/main/ingest_generated.cypher"
click node_item_export "https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_items_json_with_date.py"
click node_item_listing "https://github.com/pratham-asthana/data_sync/blob/main/list_sharepoint_items.py"
click node_item_diff "https://github.com/pratham-asthana/data_sync/blob/main/compare_sharepoint_items.py"
click node_schema_extractor "https://github.com/pratham-asthana/data_sync/blob/main/neo4j_schema_extractor.py"
click node_schema_output "https://github.com/pratham-asthana/data_sync/blob/main/neo4j_schema.json"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_site_listing,node_site_diff,node_site_snapshot toneBlue
class node_site_pipeline,node_graph_api,node_classification,node_site_exports,node_neo4j toneAmber
class node_orchestrator,node_ingester,node_cypher_output toneMint
class node_item_export,node_item_listing,node_item_diff,node_schema_extractor,node_schema_output toneRose
class node_operator,node_sharepoint toneIndigo
```

## Components

### Site Discovery

- [Site Listing](https://github.com/pratham-asthana/data_sync/blob/main/list_sharepoint_sites.py): `list_sharepoint_sites.py`
- [Site Comparison](https://github.com/pratham-asthana/data_sync/blob/main/comparison.py): `comparison.py`
- [Site Registry](https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_sites.json): `sharepoint_sites.json`

### SharePoint Extraction

- [New Site Pipeline](https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_new_site_pipeline.py): `sharepoint_new_site_pipeline.py`
- [Graph Client](https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_items_json_with_date.py): `sharepoint_items_json_with_date.py`
- [Document Classification](https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_new_site_pipeline.py): `sharepoint_new_site_pipeline.py`
- [Site JSON Exports](https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_new_site_pipeline.py): `sharepoint_new_site_pipeline.py`

### Sync and Ingestion

- [Sync Orchestrator](https://github.com/pratham-asthana/data_sync/blob/main/main.py): `main.py`
- [Cypher Ingestion](https://github.com/pratham-asthana/data_sync/blob/main/ingest_to_neo4j.py): `ingest_to_neo4j.py`
- [Generated Cypher](https://github.com/pratham-asthana/data_sync/blob/main/ingest_generated.cypher): `ingest_generated.cypher`

### Inspection Tools

- [Nested JSON Export](https://github.com/pratham-asthana/data_sync/blob/main/sharepoint_items_json_with_date.py): `sharepoint_items_json_with_date.py`
- [Item Listing](https://github.com/pratham-asthana/data_sync/blob/main/list_sharepoint_items.py): `list_sharepoint_items.py`
- [Item Snapshot Comparison](https://github.com/pratham-asthana/data_sync/blob/main/compare_sharepoint_items.py): `compare_sharepoint_items.py`
- [Schema Extractor](https://github.com/pratham-asthana/data_sync/blob/main/neo4j_schema_extractor.py): `neo4j_schema_extractor.py`
- [Schema JSON](https://github.com/pratham-asthana/data_sync/blob/main/neo4j_schema.json): `neo4j_schema.json`

### Other components

- **Operator**
- **SharePoint**
- **Neo4j Graph**

## Connections

- Operator → Sync Orchestrator: runs sync (evidence: `README.md`)
- Site Comparison → Site Listing: runs listing (evidence: `comparison.py`)
- Site Listing → SharePoint: lists sites (evidence: `README.md`)
- Site Listing → Site Registry: writes sites (evidence: `list_sharepoint_sites.py`)
- Site Registry → Site Comparison: loads baseline (evidence: `comparison.py`)
- New Site Pipeline → Site Comparison: runs comparison (evidence: `sharepoint_new_site_pipeline.py`)
- Site Comparison → New Site Pipeline: returns added sites (evidence: `sharepoint_new_site_pipeline.py`)
- New Site Pipeline → Graph Client: traverses drives (evidence: `README.md`)
- Graph Client → SharePoint: reads items (evidence: `README.md`)
- New Site Pipeline → Document Classification: classifies files (evidence: `README.md`)
- Document Classification → Site JSON Exports: produces records (evidence: `README.md`)
- New Site Pipeline → Site JSON Exports: writes exports (evidence: `README.md`)
- Operator → Item Listing: runs listing (evidence: `list_sharepoint_items.py`)
- Item Listing → SharePoint: queries Graph (evidence: `list_sharepoint_items.py`)
- Operator → Nested JSON Export: runs export (evidence: `sharepoint_items_json_with_date.py`)
- Nested JSON Export → SharePoint: queries Graph (evidence: `sharepoint_items_json_with_date.py`)
- Operator → Item Snapshot Comparison: compares snapshots (evidence: `compare_sharepoint_items.py`)
- Item Snapshot Comparison → Nested JSON Export: uses current snapshot (evidence: `compare_sharepoint_items.py`)
- Sync Orchestrator → New Site Pipeline: runs pipeline (evidence: `README.md`)
- Sync Orchestrator → Site JSON Exports: loads JSON (evidence: `main.py`)
- Sync Orchestrator → Cypher Ingestion: generates Cypher (evidence: `main.py`)
- Cypher Ingestion → Generated Cypher: writes script (evidence: `README.md`)
- Cypher Ingestion → Neo4j Graph: executes Cypher (evidence: `README.md`)
- Operator → Schema Extractor: runs extraction (evidence: `neo4j_schema_extractor.py`)
- Schema Extractor → Neo4j Graph: reads schema (evidence: `neo4j_schema_extractor.py`)
- Schema Extractor → Schema JSON: writes schema (evidence: `neo4j_schema_extractor.py`)
