How zsolt-varju-serco/open-semantic-search works
Main Components
• Search Server/Engine – The core of Open Semantic Search. This includes the underlying search engine (primarily built around Apache Solr, as evidenced by the deb package in src/solr.deb and related Docker configurations).
• ETL Framework – The document processing pipeline which handles tasks like crawling, text extraction (using Tika), OCR, named entity recognition (using spaCy and other tools), and even integration of external services. Look at the src/open-semantic-etl module and its associated requirements.
• User Interfaces – This includes the web-based search UI (accessible via docker-compose and the web interface on port 8080) and any additional search apps or desktop search interfaces (the src/open-semantic-desktop-search module and its Vagrantfile and build scripts point to a desktop appliance).
• Connectors – The system is modular and extensible, with connectors for files, databases, RSS, web scraping, etc., as indicated in the extensive docs/connector directory.
• Documentation & Administration – Modules and documentation for admin interfaces, configurations (read docs/doc/admin/*), and configuration of components.
• Integration & Test Environment – Docker configurations for running integration tests (docker-compose.etl.test.yml, docker-compose.test.yml, along with test scripts and Dockerfile-test). Also, end-to-end tests using Playwright and Jest indicate the presence of an automated testing layer.
• External Dependencies – Submodules and external services such as Apache Tika, Apache Solr, and third-party Python libraries (installed via the requirements.txt in the ETL module).
Relationships and Interactions
• The ETL framework fetches and processes documents, then pushes enriched data to the search engine index.
• The search engine (Solr) serves queries from the front-end UIs.
• There is clear separation between build/packaging (deb packages, Docker images) and runtime services.
• Integration tests and CI/CD pipelines (through Docker and scripts) indicate that components are deployed together in a containerized setup.
Architectural Patterns & Technologies
• Modularity and microservices – Each major function (ETL, search, UI) is encapsulated in its own directory and is built as a separate package/service.
• Containerization – With multiple Dockerfiles (Dockerfile, Dockerfile-cron, Dockerfile-test) and docker-compose files, the architecture is oriented toward containerized deployment.
• Use of Git submodules – Components (like Open Semantic ETL) are managed in separate repositories but integrated via submodules, reinforcing modularity.
• Integration and end-to-end testing – The presence of separate testing configurations shows a commitment to robust integration (via docker-compose.etl.test.yml) and browser automation for UI validation.
Additional Recommendations
• Mention testing components explicitly, showing how integration tests and E2E tests are orchestrated in a containerized environment.