How dataunitylab/jsonoid-discovery works
Main Components
a) Core Processing Components:
• Schema Discovery Engine
• Monoid Processing System
• Transformer Pipeline
• Equivalence Relations Handler
• Spark Integration Layer
b) Input/Output Components:
• JSON Input Handler (NDJSON format)
• Schema Output Generator (JSON Schema format)
Detailed Component Breakdown
a) Schema Discovery Engine:
• Connections to different schema type processors:
• Array Schema Processor
• Number Schema Processor
• Object Schema Processor
• String Schema Processor
b) Monoid Processing System:
• BloomFilter Monoid
• Examples Monoid
• HyperLogLog Monoid
• Type-specific Monoids (Arrays, Numbers, Objects, Strings)
• To the Schema Discovery Engine
c) Transformer Pipeline:
• As a sequential flow with these transformers:
• DefinitionTransformer
• DisjointObjectTransformer
• DynamicObjectTransformer
• EnumTransformer
• MergeAllOfTransformer
d) Equivalence Relations Handler:
• Four equivalence relations:
• Kind
• IntersectingLabel
• TypeMatch
e) Spark Integration Layer:
• Parallel processing capabilities
• JsonoidRDD and JsonoidSpark components
Additional Elements
• Component abbreviations
b) Annotations for:
• Configuration options
• Performance considerations
• Scale-out points
Technical Details to Highlight
• NDJSON input format handling
• JSON Schema draft 2020-12 output
• Spark distributed processing capabilities
• Monoid-based information collection
• Schema transformation pipeline
Integration Points
• External system connections (Apache Spark)
• Validation integration points
• Extension/plugin possibilities