How shopify/liquid works
Project type and purpose
• This is an open-source Ruby library implementing the Liquid template engine (liquid.gemspec, lib/liquid).
• Its purpose is to parse, compile, and render user-supplied templates in a safe, stateless, extensible way.
File structure
• Top-level:
• Configuration & CI (.github/, .rubocop*, Gemfile, Rakefile, LICENSE, README.md)
• Source code (lib/liquid/)
• Example server (example/)
• Performance harness (performance/)
• Automated tests (test/)
• Key architectural clues:
• lib/liquid/lexer.rb, tokenizer.rb → lexical analysis
• parser.rb, parser_switching.rb → syntax analysis
• document.rb, parse_tree_visitor.rb → AST representation & traversal
• context.rb, environment.rb, registers.rb → runtime execution context & configuration
• tags/, standardfilters.rb → extension points for tags & filters
• partial_cache.rb, profiler/ → caching & performance hooks
• template_factory.rb & template.rb → user API for parse/compile/render separation
Insights from README
• Emphasizes two-step model: parse/compile vs. render
• Introduces “Environment” as a scoped registry of tags/filters
• Mentions error modes, strict rendering, usage tracking
• Encourages modularity & safe execution of untrusted templates
a. Main components:
1. Client API (Liquid::Template)
2. Lexer/Tokenizer
3. Parser & Parser Switching
4. AST (Document, BlockBody, Tag, Variable, Expression nodes)
5. Template Factory & Compiled Template
6. Environment (registry of tags, filters, file system resolver, error mode)
7. Render Context (variables hash, registers, resource limits)
8. Filter & Tag implementations (standardfilters.rb, lib/liquid/tags/)
9. Partial Cache & Profiler Hooks
10. Output Stream
b. Relationships and interactions:
• The client calls Template.parse(source, environment)
→ Source fed into Lexer → Tokens fed into Parser
→ Parser produces AST (Document) → Compiled Template object stored/cached
• The client calls Template.render(vars, options)
→ Render Context built from Environment + vars + options
→ AST traversed by parse_tree_visitor, evaluating nodes in Context
→ Tags and Variables dispatched to registered implementations
→ Partial_cache used for includes; profiler hooks track execution
→ Output concatenated into final string
c. Architectural patterns & principles:
• Pipeline: lex → parse → compile → render
• Stateless compilation: parse once, render many
• Dependency injection: Environment encapsulates tags/filters
• Visitor pattern: parse_tree_visitor walks AST
• Strategy/Registry: filters and tags registered per environment
• Caching: partial_cache for fragments, TemplateFactory caching
d. Key technologies/frameworks:
• Ruby (MRI, .ruby-version)
• Rake for tasks (Rakefile)
• RuboCop for linting
• GitHub Actions for CI (.github/workflows)
Be detailed and exhaustive
• Major lib/liquid subdirectories and show how they plug into the core pipeline.