# sarperavci/ctf-writeups-search architecture

> Architecture diagram of the GitHub repository sarperavci/ctf-writeups-search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sarperavci/ctf-writeups-search
- Repository: https://github.com/sarperavci/ctf-writeups-search

## Overview

##### Project type and purpose
- This is a full-stack search application for CTF writeups.
- Frontend: a static, single-page UI built with InstantSearch.js, Bootstrap, and Parcel.
- Backend: a Typesense search engine (self-hosted via Docker or cloud) plus Python indexing scripts to ingest data.

##### File structure
- Top level
– configuration files (.env.development, .eslintrc.json, .rubocop.yml, parcelrc, docker-compose.yml)
– index.html & manifest.webmanifest (static entry point for the UI)
– package.json & requirements.txt (JS and Python dependencies)
– create_read_only_key.py & check_collection.py (utility scripts to manage Typesense keys/collections)
- scripts/indexer/
– index.py: reads raw writeups JSON and pushes documents to Typesense via its API
– transform_writeups.py: converts raw writeup data to the required JSON schema
- src/
– app.js: initializes InstantSearch.js with the Typesense InstantSearch Adapter
– SCSS files and stop_words.json: UI styling and search stop-word list

##### README for additional insights
- Architecture is lightly described:
– Frontend served from S3 + CloudFront (static assets)
– Typesense server (Docker-compose in dev or Typesense Cloud in production)
– Python indexing pipeline that reads JSON, transforms, and pushes to Typesense
- Key tech: Typesense, InstantSearch.js, Python, Docker

##### Main components and interactions
a. User Browser (SPA UI)
– Issues search queries via InstantSearch.js
– Renders results and filters
b. Static Asset Host
– S3 + CloudFront CDN (production) or Parcel dev server (localhost:3000)
c. Search Backend
– Typesense server (self-hosted Docker container or cloud instance)
– Exposes a RESTful search API
d. Indexing Pipeline
– Python scripts (transform_writeups.py → index.py)
– Reads data from scripts/data/writeups.json
– Authenticates to Typesense using admin API key
– Creates/updates collections and pushes documents
e. Utility Scripts
– create_read_only_key.py: generates a search-only API key for client use
– check_collection.py: verifies Typesense collections exist

##### Architectural patterns and design principles
- Clear separation of concerns:
– UI layer (static assets)
– Search engine layer (Typesense)
– Data ingestion layer (Python scripts)
- Micro-service-like boundaries: each layer can be scaled or hosted independently
- Environment isolation: Docker Compose for local Typesense; environment variables for keys/hosts

##### Essential details to capture
- Typesense as the core search engine (self-hosted or cloud)
- InstantSearch.js + Typesense InstantSearch Adapter bridging UI to search API
- Python indexing scripts transforming raw data into the Typesense collection
- Deployment targets: Docker Compose locally, S3/CloudFront in prod
- Utility scripts for API key management

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Subgraphs
    subgraph "Frontend (SPA UI)" 
        direction TB
        Browser["Browser UI"]:::frontend
        IndexHTML["index.html"]:::frontend
        Manifest["manifest.webmanifest"]:::frontend
        AppJS["src/app.js"]:::frontend
        AppSCSS["src/app.scss"]:::frontend
        BootstrapSCSS["src/bootstrap.scss"]:::frontend
        IndexSCSS["src/index.scss"]:::frontend
        StopWords["src/utils/stop_words.json"]:::frontend
        PackageJSON_FE["package.json"]:::frontend
        ParcelRC_FE[".parcelrc"]:::frontend
    end

    subgraph "Static Asset Hosting" 
        direction TB
        ParcelDev["Parcel Dev Server"]:::infra
        CDN["S3 + CloudFront CDN"]:::infra
        ParcelRC_HOST[".parcelrc"]:::infra
        PackageJSON_HOST["package.json"]:::infra
    end

    subgraph "Search Backend (Typesense)" 
        direction TB
        Typesense["Typesense Server"]:::search
        DockerCompose["docker-compose.yml"]:::search
        EnvDev[".env.development"]:::search
    end

    subgraph "Indexing Pipeline (Python)" 
        direction TB
        Transform["scripts/indexer/transform_writeups.py"]:::pipeline
        Indexer["scripts/indexer/index.py"]:::pipeline
        Requirements["requirements.txt"]:::pipeline
        DataSource["scripts/data/writeups.json"]:::pipeline
    end

    subgraph "Utility Scripts" 
        direction TB
        ReadOnlyKey["create_read_only_key.py"]:::pipeline
        CheckCollection["check_collection.py"]:::pipeline
    end

    subgraph "Linting & Configuration" 
        direction TB
        ESLintRC[".eslintrc.json"]:::config
        ESLintIgnore[".eslintignore"]:::config
        RuboConfig[".rubocop.yml"]:::config
        EditorConfig[".editorconfig"]:::config
        Prettier[".prettierrc"]:::config
        NVMRC[".nvmrc"]:::config
        GitIgnore[".gitignore"]:::config
    end

    subgraph "Developer / Local Machine" 
        DevMachine["Developer Machine"]:::infra
    end

    %% Connections
    Browser -->|"HTTP Search Query"| Typesense
    Browser -->|"Loads Assets"| ParcelDev
    ParcelDev -->|"Serves SPA"| Browser
    CDN -->|"Serves SPA"| Browser

    DevMachine -->|"docker-compose up"| DockerCompose
    DockerCompose -->|"Starts"| Typesense

    Transform -->|"reads JSON"| DataSource
    Transform -->|"outputs schema JSON"| Indexer
    Indexer -->|"HTTPS Indexing Call"| Typesense

    ReadOnlyKey -->|"manages API Keys"| Typesense
    CheckCollection -->|"verifies Collection"| Typesense

    %% Click Events - Frontend
    click IndexHTML "https://github.com/sarperavci/ctf-writeups-search/blob/main/index.html"
    click Manifest "https://github.com/sarperavci/ctf-writeups-search/blob/main/manifest.webmanifest"
    click AppJS "https://github.com/sarperavci/ctf-writeups-search/blob/main/src/app.js"
    click AppSCSS "https://github.com/sarperavci/ctf-writeups-search/blob/main/src/app.scss"
    click BootstrapSCSS "https://github.com/sarperavci/ctf-writeups-search/blob/main/src/bootstrap.scss"
    click IndexSCSS "https://github.com/sarperavci/ctf-writeups-search/blob/main/src/index.scss"
    click StopWords "https://github.com/sarperavci/ctf-writeups-search/blob/main/src/utils/stop_words.json"
    click PackageJSON_FE "https://github.com/sarperavci/ctf-writeups-search/blob/main/package.json"
    click ParcelRC_FE "https://github.com/sarperavci/ctf-writeups-search/blob/main/.parcelrc"

    %% Click Events - Static Hosting
    click ParcelRC_HOST "https://github.com/sarperavci/ctf-writeups-search/blob/main/.parcelrc"
    click PackageJSON_HOST "https://github.com/sarperavci/ctf-writeups-search/blob/main/package.json"

    %% Click Events - Search Backend
    click DockerCompose "https://github.com/sarperavci/ctf-writeups-search/blob/main/docker-compose.yml"
    click EnvDev "https://github.com/sarperavci/ctf-writeups-search/blob/main/.env.development"

    %% Click Events - Indexing Pipeline
    click Transform "https://github.com/sarperavci/ctf-writeups-search/blob/main/scripts/indexer/transform_writeups.py"
    click Indexer "https://github.com/sarperavci/ctf-writeups-search/blob/main/scripts/indexer/index.py"
    click Requirements "https://github.com/sarperavci/ctf-writeups-search/blob/main/requirements.txt"

    %% Click Events - Utility Scripts
    click ReadOnlyKey "https://github.com/sarperavci/ctf-writeups-search/blob/main/create_read_only_key.py"
    click CheckCollection "https://github.com/sarperavci/ctf-writeups-search/blob/main/check_collection.py"

    %% Click Events - Linting & Configuration
    click ESLintRC "https://github.com/sarperavci/ctf-writeups-search/blob/main/.eslintrc.json"
    click ESLintIgnore "https://github.com/sarperavci/ctf-writeups-search/blob/main/.eslintignore"
    click RuboConfig "https://github.com/sarperavci/ctf-writeups-search/blob/main/.rubocop.yml"
    click EditorConfig "https://github.com/sarperavci/ctf-writeups-search/blob/main/.editorconfig"
    click Prettier "https://github.com/sarperavci/ctf-writeups-search/blob/main/.prettierrc"
    click NVMRC "https://github.com/sarperavci/ctf-writeups-search/blob/main/.nvmrc"
    click GitIgnore "https://github.com/sarperavci/ctf-writeups-search/blob/main/.gitignore"

    %% Styles
    classDef frontend fill:#cce5ff,stroke:#2b6cb0,color:#1a202c
    classDef infra fill:#e2e3e5,stroke:#4a5568,color:#1a202c
    classDef search fill:#ffe5b4,stroke:#dd6b20,color:#1a202c
    classDef pipeline fill:#d4edda,stroke:#2f855a,color:#1a202c
    classDef config fill:#f0f4f8,stroke:#718096,color:#1a202c
```
