How kensho-technologies/graphql-compiler works
Project type and purpose
• This is not a full-stack app; it’s an open-source compiler library (“graphql-compiler”) whose sole purpose is to translate GraphQL queries into optimized database queries (SQL, Cypher/MATCH, Gremlin).
High-level architecture
A. Client / API Layer
• Entry points:
– graphql_to_sql(...)
– graphql_to_match(...)
– graphql_to_gremlin(...)
• Helpers to build schema info:
– SQL: get_sqlalchemy_schema_info(...)
– OrientDB: get_graphql_schema_from_orientdb_schema_data(...)
B. Compiler Frontend
• Parses GraphQL query (using graphql lib).
• Performs initial AST manipulation & validation (compiler_frontend.py, validation.py).
• Outputs an intermediate representation (IR).
C. Macro Expansion & Transformations
• Expand user-defined macros (macros/)
• Schema transformations (rename, split, merge) (schema_transformation/).
D. IR Optimization & Planning
• Apply consistency checks, common IR optimizations (ir_lowering_common/).
• Cost Estimation modules (cost_estimation/) compute selectivities, cardinalities.
• Pagination planning (query_pagination/).
E. IR Lowering per Backend
For each target language there is:
• A lowering pipeline (ir_lowering_sql/, ir_lowering_cypher/, ir_lowering_gremlin/, ir_lowering_match/).
• Emitters (emit_sql.py, emit_cypher.py, emit_gremlin.py, emit_match.py) that walk lowered IR and produce query strings.
F. Post-processing
• Target-specific tweaks (e.g. SQL post_processing).
G. Execution (optional)
• SQLAlchemy engine (create_engine).
• OrientDB / RedisGraph / Neo4j drivers via graphql_compiler.api.*.
H. Testing & Tooling
• End-to-end & unit tests (tests/).
• CLI entry point: graphql_compiler/tool.py.
• Docker Compose to spin up test databases.
Relationships and data flow
Client/API → Frontend → Macro Expansion → IR Optimizer/Planner → IR Lowering → Emit → Post-processing → Generated Query → Database Execution
Architectural patterns and principles
• Compiler pipeline / visitor pattern for AST and IR.
• Pluggable back-end modules.
• Single responsibility: each Python subpackage (e.g., query_formatting, schema_generation) has a focused concern.
• Separation of concerns: front-end vs. lowering vs. emitter vs. post-processing.
Final checklist
– All major compiler subpackages (compiler, macros, schema_generation, query_planning, ir_lowering_*, emit_*, post_processing).
– API layer (graphql_to_* functions, get_*_schema_info).
– External DB back-ends.
– Clear labels and callouts for key modules (e.g., “Cost Estimation: determines filter selectivities”).