# o19s/solr-tmdb architecture

> Architecture diagram of the GitHub repository o19s/solr-tmdb, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/o19s/solr-tmdb
- Repository: https://github.com/o19s/solr-tmdb

## Overview

##### Project type and purpose
- This is a specialized search indexing project, not a typical full-stack app. Its sole purpose is to ingest The Movie Database (TMDB) JSON data, configure an Apache Solr core, and expose a searchable index. It’s used in training as part of the “Think Like a Relevancy Engineer” curriculum.

##### File structure
- Top-level scripts:
- docker.sh / docker.ps1 & docker/ folder → Dockerized Solr setup
- index.sh / index.ps1 → Shell/PowerShell wrappers that POST the TMDB JSON into Solr
- solr_home/ → All Solr configuration for the “tmdb” core (schema.xml, solrconfig.xml, stopwords, synonyms, etc.)
- docker/lib/ → External JARs (Querqy filters) mounted into Solr via configuration
- housekeeping/ → Python scripts (createSolrTmdbDataset.py, tmdbMovies.py) to fetch or transform data and testing scripts
- solr-postman-collection.json → Postman collection for interactive API calls

##### README for additional insights
- The README confirms Docker is the recommended deployment, but manual Solr install can also be used.
- It lays out the workflows: download, start Solr, index data, confirm via queries, optionally use Postman.

##### Components
a. Data Sources
- TMDB JSON file (external data file)
b. Data-Preparation Layer
- Python scripts in housekeeping/create_dataset (fetch/transform → intermediate JSON)
c. Indexing Layer
- index.sh / index.ps1 → invokes HTTP POST to Solr’s /tmdb/update endpoint
d. Search Engine
- Solr server running inside Docker (or locally if not containerized)
- Docker container with mounted volumes:
– solr_home/tmdb/conf (core configuration)
– docker/lib JARs (Querqy extensions)
e. Client & Testing
- Postman (solr-postman-collection.json) or direct CURL/HTTP clients
- housekeeping/testing scripts for validation

##### Relationships & interactions
- Solr container shows two mounted volumes: configuration and extension libraries
- Clients (Postman, test scripts) query Solr’s HTTP GET/POST endpoints and receive JSON responses

##### Architectural patterns and principles
- Separation of concerns: data ingestion, indexing, search
- Containerization: isolated Solr environment for reproducibility
- Config-as-Code: all schema and config files versioned in solr_home
- Extensibility: custom filters (Querqy) loaded from docker/lib

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Source
    JSON["TMDB JSON File"]:::data
    click JSON "https://github.com/o19s/solr-tmdb/blob/master/tmdb_solr.json.zip"

    %% Data-Preparation Layer
    subgraph "Data-Preparation Layer"
        PS1["createSolrTmdbDataset.py"]:::proc
        PS2["tmdbMovies.py"]:::proc
    end
    click PS1 "https://github.com/o19s/solr-tmdb/blob/master/housekeeping/create_dataset/createSolrTmdbDataset.py"
    click PS2 "https://github.com/o19s/solr-tmdb/blob/master/housekeeping/create_dataset/tmdbMovies.py"

    %% Indexing Layer
    subgraph "Indexing Layer"
        IDX1["index.sh"]:::proc
        IDX2["index.ps1"]:::proc
    end
    click IDX1 "https://github.com/o19s/solr-tmdb/blob/master/index.sh"
    click IDX2 "https://github.com/o19s/solr-tmdb/blob/master/index.ps1"

    %% Solr Container
    subgraph "Solr Docker Container" 
        direction TB
        SOLR["Solr Server"]:::container
        subgraph "Mounted Volumes"
            subgraph "Configuration" 
                SOLR_XML["solr.xml"]:::config
                SCHEMA["schema.xml"]:::config
                SOLRCFG["solrconfig.xml"]:::config
                STOPWORDS_JSON["_schema_analysis_stopwords_english.json"]:::config
                STOPWORDS_TXT["stopwords.txt"]:::config
                SYNONYMS["synonyms*.txt"]:::config
                LANG_DIR["lang/"]:::config
                CORE_PROP["core.properties"]:::config
            end
            subgraph "Extensions (Querqy JARs)"
                JAR1["querqy-regex-filter-1.1.0-SNAPSHOT.jar"]:::config
                JAR2["querqy-solr-5.1.lucene810.0-jar-with-dependencies.jar"]:::config
            end
        end
    end
    click SOLR_XML "https://github.com/o19s/solr-tmdb/blob/master/solr_home/solr.xml"
    click SCHEMA "https://github.com/o19s/solr-tmdb/blob/master/solr_home/tmdb/conf/schema.xml"
    click SOLRCFG "https://github.com/o19s/solr-tmdb/blob/master/solr_home/tmdb/conf/solrconfig.xml"
    click STOPWORDS_JSON "https://github.com/o19s/solr-tmdb/blob/master/solr_home/tmdb/conf/_schema_analysis_stopwords_english.json"
    click STOPWORDS_TXT "https://github.com/o19s/solr-tmdb/blob/master/solr_home/tmdb/conf/stopwords.txt"
    click SYNONYMS "https://github.com/o19s/solr-tmdb/blob/master/solr_home/tmdb/conf/synonyms*.txt"
    click LANG_DIR "https://github.com/o19s/solr-tmdb/tree/master/solr_home/tmdb/conf/lang/"
    click CORE_PROP "https://github.com/o19s/solr-tmdb/blob/master/solr_home/tmdb/core.properties"
    click JAR1 "https://github.com/o19s/solr-tmdb/blob/master/docker/lib/querqy-regex-filter-1.1.0-SNAPSHOT.jar"
    click JAR2 "https://github.com/o19s/solr-tmdb/blob/master/docker/lib/querqy-solr-5.1.lucene810.0-jar-with-dependencies.jar"
    click SOLR "https://github.com/o19s/solr-tmdb/tree/master/docker/Dockerfile"

    %% Client & Testing
    subgraph "Client & Testing"
        POST["Postman Collection"]:::client
        TEST1["splainer_puppet_solr.py"]:::proc
        TEST2["splainer_links_solr.csv"]:::proc
        REQ["requirements.txt"]:::proc
    end
    click POST "https://github.com/o19s/solr-tmdb/blob/master/solr-postman-collection.json"
    click TEST1 "https://github.com/o19s/solr-tmdb/blob/master/housekeeping/testing/splainer_puppet_solr.py"
    click TEST2 "https://github.com/o19s/solr-tmdb/blob/master/housekeeping/testing/splainer_links_solr.csv"
    click REQ "https://github.com/o19s/solr-tmdb/blob/master/housekeeping/testing/requirements.txt"

    %% Data Flows
    JSON -->|feeds| PS1
    PS1 --> PS2
    PS2 -->|produces JSON| IDX1
    PS2 -->|produces JSON| IDX2
    IDX1 -->|"HTTP POST to /tmdb/update"| SOLR
    IDX2 -->|"HTTP POST to /tmdb/update"| SOLR
    POST -->|"queries HTTP API"| SOLR
    TEST1 -->|"automated tests"| SOLR
    TEST2 -->|"test data"| TEST1

    classDef data fill:#e0f7fa,stroke:#0288d1,color:#01579b
    classDef proc fill:#e8f5e9,stroke:#2e7d32,color:#1b5e20
    classDef container fill:#ffe0b2,stroke:#ef6c00,color:#e65100
    classDef config fill:#eceff1,stroke:#90a4ae,color:#37474f
    classDef client fill:#f3e5f5,stroke:#8e24aa,color:#4a148c
```
