How sarperavci/ctf-writeups-search works
Project type and purpose
• This is a full-stack search application for CTF writeups.
• Frontend: a static, single-page UI built with InstantSearch.js, Bootstrap, and Parcel.
• Backend: a Typesense search engine (self-hosted via Docker or cloud) plus Python indexing scripts to ingest data.
File structure
• Top level
– configuration files (.env.development, .eslintrc.json, .rubocop.yml, parcelrc, docker-compose.yml)
– index.html & manifest.webmanifest (static entry point for the UI)
– package.json & requirements.txt (JS and Python dependencies)
– create_read_only_key.py & check_collection.py (utility scripts to manage Typesense keys/collections)
• scripts/indexer/
– index.py: reads raw writeups JSON and pushes documents to Typesense via its API
– transform_writeups.py: converts raw writeup data to the required JSON schema
• src/
– app.js: initializes InstantSearch.js with the Typesense InstantSearch Adapter
– SCSS files and stop_words.json: UI styling and search stop-word list
README for additional insights
• Architecture is lightly described:
– Frontend served from S3 + CloudFront (static assets)
– Typesense server (Docker-compose in dev or Typesense Cloud in production)
– Python indexing pipeline that reads JSON, transforms, and pushes to Typesense
• Key tech: Typesense, InstantSearch.js, Python, Docker
Main components and interactions
a. User Browser (SPA UI)
– Issues search queries via InstantSearch.js
– Renders results and filters
b. Static Asset Host
– S3 + CloudFront CDN (production) or Parcel dev server (localhost:3000)
c. Search Backend
– Typesense server (self-hosted Docker container or cloud instance)
– Exposes a RESTful search API
d. Indexing Pipeline
– Python scripts (transform_writeups.py → index.py)
– Reads data from scripts/data/writeups.json
– Authenticates to Typesense using admin API key
– Creates/updates collections and pushes documents
e. Utility Scripts
– create_read_only_key.py: generates a search-only API key for client use
– check_collection.py: verifies Typesense collections exist
Architectural patterns and design principles
• Clear separation of concerns:
– UI layer (static assets)
– Search engine layer (Typesense)
– Data ingestion layer (Python scripts)
• Micro-service-like boundaries: each layer can be scaled or hosted independently
• Environment isolation: Docker Compose for local Typesense; environment variables for keys/hosts
Essential details to capture
• Typesense as the core search engine (self-hosted or cloud)
• InstantSearch.js + Typesense InstantSearch Adapter bridging UI to search API
• Python indexing scripts transforming raw data into the Typesense collection
• Deployment targets: Docker Compose locally, S3/CloudFront in prod
• Utility scripts for API key management