How gildas-le-drogoff/convert-to-parquet-rust works
convert-to-parquet is a Rust data-conversion application with both CLI and GUI entry points. Its main workflow reads delimited, spreadsheet, JSON, or Parquet data; infers or applies a schema; converts and writes Parquet; and reports validation results. It also supports Parquet inspection and reverse CSV/JSONL export, plus GUI-based analysis, preview, and conversion. The README documents major formats and pipeline behavior; unsampled modules are included where the README or source index establishes their role, with uncertain links omitted.
Main entry points: mod.rs Conversion pipeline · main.rs CLI · app.rs GUI application · worker.rs Background jobs
Components 25
User Interfaces
- CLI
src/main.rs - GUI launcher
src/bin/gui_convert.rs
Input Handling
- Delimited input
src/conversion/csv_blocks.rs - Spreadsheet reader
src/xlsx.rs - JSON handling
src/json.rs - JSON Arrow pipeline
src/json_arrow.rs
Schema and Conversion
- Schema inference
src/schema.rs - Column analysis
src/analysis.rs - Conversion pipeline
src/conversion/mod.rs - JSON conversion
src/conversion/json_pipeline.rs - Parallel blocks
src/conversion/pipeline.rs - Ordered Parquet writer
src/conversion/parquet_writer.rs - Memory limits
src/memory.rs
Outputs and Inspection
- Validation report
src/diag.rs - Parquet inspection
src/inspect.rs - Data preview
src/preview.rs - CSV and JSONL export
src/export.rs - Parquet rewrite
src/parquet_rewrite.rs
GUI Workflow
- GUI application
src/gui/app.rs - File and edit state
src/gui/model.rs - GUI analysis
src/gui/analysis.rs - Background jobs
src/gui/worker.rs - GUI views
src/gui/view/mod.rs - Output planning
src/gui/paths.rs
Other components
- User
Connections 31
- User → CLI: runs conversion
README.md - User → GUI launcher: opens GUI
src/bin/gui_convert.rs - CLI → Conversion pipeline: converts files
README.md - Conversion pipeline → Schema inference: infers schema
src/conversion/mod.rs - Conversion pipeline → Delimited input: streams blocks
src/conversion/mod.rs - Conversion pipeline → Column analysis: tracks metrics
src/conversion/mod.rs - Conversion pipeline → Parallel blocks: runs pipeline
src/conversion/mod.rs - Parallel blocks → Ordered Parquet writer: writes batches
src/conversion/pipeline.rs - Conversion pipeline → Ordered Parquet writer: starts writer
src/conversion/mod.rs - Conversion pipeline → Memory limits: sizes blocks
src/conversion/mod.rs - CLI → JSON conversion: converts JSON
README.md - JSON conversion → JSON handling: reads JSON
src/conversion/json_pipeline.rs - JSON conversion → JSON Arrow pipeline: decodes batches
src/conversion/json_pipeline.rs - JSON conversion → Ordered Parquet writer: sends batches
src/conversion/json_pipeline.rs - Conversion pipeline → Validation report: reports results
src/conversion/mod.rs - CLI → Parquet inspection: views schema
README.md - CLI → CSV and JSONL export: exports formats
README.md - CLI → Parquet rewrite: rewrites Parquet
- CLI → Spreadsheet reader: reads spreadsheets
README.md - GUI launcher → GUI application: starts app
- GUI application → File and edit state: updates state
src/gui/app.rs - GUI application → GUI views: renders views
src/gui/app.rs - GUI application → Background jobs: dispatches jobs
src/gui/app.rs - GUI application → Output planning: plans outputs
src/gui/app.rs - Background jobs → GUI analysis: analyzes files
src/gui/worker.rs - Background jobs → Conversion pipeline: converts sources
src/gui/worker.rs - Background jobs → Spreadsheet reader: reads sheets
src/gui/worker.rs - Background jobs → JSON handling: handles JSON
src/gui/worker.rs - GUI analysis → Data preview: previews data
src/gui/analysis.rs - GUI analysis → Parquet inspection: inspects Parquet
src/gui/analysis.rs - GUI views → Data preview: shows preview