How pratham-asthana/data_sync works
This repository syncs SharePoint document metadata into a Neo4j knowledge graph. The main workflow discovers newly added sites, traverses their document libraries, classifies matching files, exports JSON records, and has the CLI orchestrator prepare ingestion for Neo4j. Separate tools list/export arbitrary SharePoint contents, compare file snapshots, and inspect graph schema. Edges reflect README statements and shown calls; some execution and database details are described only at a high level because the relevant source is partial or unsampled.
Main entry points: sharepoint_new_site_pipeline.py New Site Pipeline · main.py Sync Orchestrator · comparison.py Site Comparison · list_sharepoint_sites.py Site Listing
Components 18
Site Discovery
- Site Listing
list_sharepoint_sites.py - Site Comparison
comparison.py - Site Registry
sharepoint_sites.json
SharePoint Extraction
- New Site Pipeline
sharepoint_new_site_pipeline.py - Graph Client
sharepoint_items_json_with_date.py - Document Classification
sharepoint_new_site_pipeline.py - Site JSON Exports
sharepoint_new_site_pipeline.py
Sync and Ingestion
- Sync Orchestrator
main.py - Cypher Ingestion
ingest_to_neo4j.py - Generated Cypher
ingest_generated.cypher
Inspection Tools
- Nested JSON Export
sharepoint_items_json_with_date.py - Item Listing
list_sharepoint_items.py - Item Snapshot Comparison
compare_sharepoint_items.py - Schema Extractor
neo4j_schema_extractor.py - Schema JSON
neo4j_schema.json
Other components
- Operator
- SharePoint
- Neo4j Graph
Connections 26
- Operator → Sync Orchestrator: runs sync
README.md - Site Comparison → Site Listing: runs listing
comparison.py - Site Listing → SharePoint: lists sites
README.md - Site Listing → Site Registry: writes sites
list_sharepoint_sites.py - Site Registry → Site Comparison: loads baseline
comparison.py - New Site Pipeline → Site Comparison: runs comparison
sharepoint_new_site_pipeline.py - Site Comparison → New Site Pipeline: returns added sites
sharepoint_new_site_pipeline.py - New Site Pipeline → Graph Client: traverses drives
README.md - Graph Client → SharePoint: reads items
README.md - New Site Pipeline → Document Classification: classifies files
README.md - Document Classification → Site JSON Exports: produces records
README.md - New Site Pipeline → Site JSON Exports: writes exports
README.md - Operator → Item Listing: runs listing
list_sharepoint_items.py - Item Listing → SharePoint: queries Graph
list_sharepoint_items.py - Operator → Nested JSON Export: runs export
sharepoint_items_json_with_date.py - Nested JSON Export → SharePoint: queries Graph
sharepoint_items_json_with_date.py - Operator → Item Snapshot Comparison: compares snapshots
compare_sharepoint_items.py - Item Snapshot Comparison → Nested JSON Export: uses current snapshot
compare_sharepoint_items.py - Sync Orchestrator → New Site Pipeline: runs pipeline
README.md - Sync Orchestrator → Site JSON Exports: loads JSON
main.py - Sync Orchestrator → Cypher Ingestion: generates Cypher
main.py - Cypher Ingestion → Generated Cypher: writes script
README.md - Cypher Ingestion → Neo4j Graph: executes Cypher
README.md - Operator → Schema Extractor: runs extraction
neo4j_schema_extractor.py - Schema Extractor → Neo4j Graph: reads schema
neo4j_schema_extractor.py - Schema Extractor → Schema JSON: writes schema
neo4j_schema_extractor.py