How radumarias/syncoxiders works
Project Type and Purpose
• The project is a cloud-based synchronization and sharing solution focused on syncing files (and emails) between various providers (Google Drive, Dropbox, S3, SFTP, etc.) and local storage, with encryption provided via rencfs.
• The README also explains that the solution is currently in a proof-of-concept phase (with CLI support) and envisions an MVP with additional features such as a browser UI, enhanced sync functionality, and future direct API integrations (with Kafka & Flink).
File Structure
• Multiple Rust crates indicate a modular design:
– The root includes top-level configuration files (Cargo.toml, Cargo.lock) plus GitHub workflows and configuration folders.
– Key modules include:
◦ file-change-consumer: Likely responsible for ingesting changes.
◦ file-change-router: Possibly handling the routing or decision logic of change events.
◦ file-tree-merge: Contains logic for merging changes from different sources, conflict resolution, and managing directory trees.
◦ file-watcher: Likely monitors file system events and triggers syncs.
• The presence of tests (including shell scripts in file-tree-merge/tests) shows that there is a focus on reliable behavior via integration and script-based system tests.
Main Components
• Frontend & User Interaction:
• CLI application (as per the README “syncoxiders” binary command) and a planned browser UI.
• Input/Change Detection:
• File Watcher: Monitors local filesystem changes.
• File Change Consumer: Consumes detected events for further processing.
• Coordination and Routing:
• File Change Router: Determines how and where to route changes (e.g., local folder to cloud, inter-cloud routing).
• Core Processing Engine:
• File Tree Merge Module: Contains the algorithms to merge file trees, resolve conflicts, and apply necessary changes.
• External Integrations (Adapters):
• Cloud Providers: Currently relying on third-party tools/libraries such as rclone, gdrive-rs, and using rencfs for encryption.
• Future plans include developing native clients for these providers.
• Messaging & Future Data Pipeline:
• The envisioned incorporation of Kafka (for change streaming) and Flink (for windowing/grouping) to feed merged changes into the system.
Relationships and Interactions
• The user initiates an action (via CLI or later via the browser app), which triggers the file watcher and/or file change consumer.
• The file change consumer routes events to the file change router that then categorizes the event type (create, modify, delete, move) for further processing.
• The file tree merge module then gets involved to compare the current file state with previous snapshots, resolve conflicts, and compute a unified changes tree.
• Once the merge is complete, the system pushes the changes to the respective targets (local files and/or remote cloud storages).
• External services (rclone, rencfs, gdrive-rs) are used to interact with the storage providers, with encryption happening as data flows between cloud and local systems.
• Optionally, future components will integrate real-time messaging (Kafka) to manage change propagation across distributed systems.
Specific Architectural Patterns and Key Points
• Modularity: The use of separate Rust crates implies a microservice-like or modular monolith design. Each major function (watching, routing, merging) is a separate module that can be developed and tested independently.
• Data Flow and Event-Driven Design: The file-change consumer and router suggest an event-driven design.
• Extensibility: The architecture leaves room for plugging in new external providers and for evolving the way changes are processed (moving from rclone-run solutions to custom API clients).
• Future Scaling: The inclusion of Kafka and Flink in the planned MVP indicates a design that is prepared to scale out using robust messaging and stream processing frameworks.
Final Recommendations for the Principal Software Engineer
• Parts where future improvements (like Kafka/Flink integrations) are planned, so the design remains flexible.