# dataunitylab/jsonoid-discovery architecture

> Architecture diagram of the GitHub repository dataunitylab/jsonoid-discovery, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2024-12-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dataunitylab/jsonoid-discovery
- Repository: https://github.com/dataunitylab/jsonoid-discovery

## Overview

##### Main Components

a) Core Processing Components:
- Schema Discovery Engine
- Monoid Processing System
- Transformer Pipeline
- Equivalence Relations Handler
- Spark Integration Layer

b) Input/Output Components:
- JSON Input Handler (NDJSON format)
- Schema Output Generator (JSON Schema format)

##### Detailed Component Breakdown

a) Schema Discovery Engine:
- Connections to different schema type processors:
- Array Schema Processor
- Number Schema Processor
- Object Schema Processor
- String Schema Processor

b) Monoid Processing System:
- BloomFilter Monoid
- Examples Monoid
- HyperLogLog Monoid
- Type-specific Monoids (Arrays, Numbers, Objects, Strings)
- To the Schema Discovery Engine

c) Transformer Pipeline:
- As a sequential flow with these transformers:
- DefinitionTransformer
- DisjointObjectTransformer
- DynamicObjectTransformer
- EnumTransformer
- MergeAllOfTransformer

d) Equivalence Relations Handler:
- Four equivalence relations:
- Kind
- IntersectingLabel
- TypeMatch

e) Spark Integration Layer:
- Parallel processing capabilities
- JsonoidRDD and JsonoidSpark components

##### Additional Elements

- Component abbreviations

b) Annotations for:
- Configuration options
- Performance considerations
- Scale-out points

##### Technical Details to Highlight

- NDJSON input format handling
- JSON Schema draft 2020-12 output
- Spark distributed processing capabilities
- Monoid-based information collection
- Schema transformation pipeline

##### Integration Points

- External system connections (Apache Spark)
- Validation integration points
- Extension/plugin possibilities

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph Input
        JSON[JSON Input Handler]
    end

    subgraph "Core Processing"
        SDE[Schema Discovery Engine]:::core
        click SDE "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/DiscoverSchema.scala"

        subgraph "Schema Type Processors"
            ASP[Array Schema Processor]:::processor
            NSP[Number Schema Processor]:::processor
            OSP[Object Schema Processor]:::processor
            SSP[String Schema Processor]:::processor
            click ASP "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/schemas/ArraySchema.scala"
            click NSP "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/schemas/NumberSchema.scala"
            click OSP "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/schemas/ObjectSchema.scala"
            click SSP "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/schemas/StringSchema.scala"
        end

        subgraph "Monoid Processing"
            BFM[BloomFilter Monoid]:::monoid
            HLM[HyperLogLog Monoid]:::monoid
            EXM[Examples Monoid]:::monoid
            click BFM "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/utils/BloomFilter.scala"
            click HLM "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/utils/HyperLogLog.scala"
            click EXM "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/schemas/ExamplesProperty.scala"
        end

        subgraph "Transformer Pipeline"
            DFT[Definition Transformer]:::transform
            DOT[Disjoint Object Transformer]:::transform
            DYT[Dynamic Object Transformer]:::transform
            ENT[Enum Transformer]:::transform
            MAT[Merge AllOf Transformer]:::transform
            click DFT "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/transformers/DefinitionTransformer.scala"
            click DOT "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/transformers/DisjointObjectTransformer.scala"
            click DYT "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/transformers/DynamicObjectTransformer.scala"
            click ENT "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/transformers/EnumTransformer.scala"
            click MAT "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/transformers/MergeAllOfTransformer.scala"
        end

        subgraph "Equivalence Relations"
            ERH[Equivalence Relations Handler]:::equiv
            click ERH "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/EquivalenceRelation.scala"
            KND[Kind]:::equiv
            LBL[Label]:::equiv
            ILB[IntersectingLabel]:::equiv
            TMT[TypeMatch]:::equiv
        end
    end

    subgraph "Spark Integration"
        JRDD[JsonoidRDD]:::spark
        JSP[JsonoidSpark]:::spark
        click JRDD "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/spark/JsonoidRDD.scala"
        click JSP "https://github.com/dataunitylab/jsonoid-discovery/blob/main/src/main/scala/io/github/dataunitylab/jsonoid/discovery/spark/JsonoidSpark.scala"
    end

    subgraph Output
        SCH[Schema Output Generator]
    end

    %% Connections
    JSON --> SDE
    SDE --> ASP & NSP & OSP & SSP
    BFM & HLM & EXM --> SDE
    ASP & NSP & OSP & SSP --> DFT
    DFT --> DOT --> DYT --> ENT --> MAT
    ERH --> KND & LBL & ILB & TMT
    ERH --> SDE
    MAT --> SCH
    JRDD --> JSP
    JSP --> SDE

    %% Styles
    classDef core fill:#2374ab
    classDef processor fill:#48a9a6
    classDef monoid fill:#4b4e6d
    classDef transform fill:#d4837f
    classDef equiv fill:#a37774
    classDef spark fill:#845ec2

    %% Legend
    subgraph Legend
        L1[Core Component]:::core
        L2[Schema Processor]:::processor
        L3[Monoid Component]:::monoid
        L4[Transformer]:::transform
        L5[Equivalence Component]:::equiv
        L6[Spark Component]:::spark
    end
```
