# nyu-llm-ctf/llm_ctf_dataset_dev architecture

> Architecture diagram of the GitHub repository nyu-llm-ctf/llm_ctf_dataset_dev, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nyu-llm-ctf/llm_ctf_dataset_dev
- Repository: https://github.com/nyu-llm-ctf/llm_ctf_dataset_dev

## Overview

##### Project type and purpose
- This is a data‐centric repository (“LLM_CTF_Dataset_Dev”) containing raw Capture-The-Flag (CSAW CTF) challenge materials organized by year and category (crypto, web, pwn, rev, misc). There is no conventional “application” code; the repository’s purpose is to serve as a dataset for ML/CTF research or tooling.

##### File structure
- Top‐level directories:
- 2013, 2014, 2015, 2016 – each containing subfolders by CTF phase (CSAW-Finals, CSAW-Quals) and challenge categories (crypto, web, pwn, rev, misc).
- README.md at the root (identifying the repo).
- Each challenge folder typically includes: Dockerfile/docker-compose.yml (deployment), challenge.json (metadata), source binaries/scripts, input/output assets, solution code, and sometimes flag files.
- There are no src/, lib/, or tests folders as in an application; instead the file tree itself is the raw dataset.

##### Diagramming strategy
a. Main components
- Raw File Storage – the on‐disk hierarchy: years → phases → categories → individual challenge directories.
- Metadata Indexer – a component that reads challenge.json files and builds a normalized catalog.
- Processing Pipeline – scripts or services that extract features (e.g., text extraction, binary parsing, container configuration) from raw challenge contents.
- Database – a persistent store (SQL or NoSQL) for processed metadata and extracted features.
- API Layer – RESTful service exposing dataset queries (by year, category, keywords).
- Frontend/UI (optional) – web dashboard to browse/search the dataset.
- Consumer/ML Pipeline – external ML training systems or CTF tooling that consume the API or database.

b. Relationships and data flows
1. Raw File Storage feeds into the Metadata Indexer and Processing Pipeline.
2. Both Indexer and Pipeline write structured records to the Database.
3. API Layer reads from Database to serve requests.
4. Frontend/UI calls API Layer.
5. Consumer/ML Pipeline calls API Layer or reads directly from Database.

c. Architectural patterns/design principles
- Microservices for Indexer, Processing Pipeline, and API Layer to allow independent scaling.
- Event-driven ingestion: when new challenge directory is added, an event triggers indexing and processing.
- Separation of concerns: raw storage vs. processing vs. serving.

d. Key technologies/frameworks (examples)
- File storage: local disk or object store (S3).
- Message bus for events: Kafka or RabbitMQ.
- Processing: Python or Node.js scripts in Docker containers.
- Database: PostgreSQL or MongoDB.
- API: Flask, Express.js, or FastAPI.
- Frontend: React or simple server-rendered pages.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Raw File Storage"
        direction TB
        A2013["2013/"]:::storage
        A2014["2014/"]:::storage
        A2015["2015/"]:::storage
        A2016["2016/"]:::storage
        README["README.md"]:::storage
    end

    subgraph "Ingestion & Processing"
        direction TB
        EB["Event Bus"]:::processing
        MI["Metadata Indexer"]:::processing
        PP["Processing Pipeline"]:::processing
    end

    subgraph "Database Layer"
        direction TB
        DB[(Database)]:::persistence
    end

    subgraph "API & UI"
        direction TB
        API["API Service"]:::interface
        UI["Frontend/UI"]:::interface
    end

    Consumer["ML Consumer"]:::external

    %% Connections
    A2013 -->|triggers| EB
    A2014 -->|triggers| EB
    A2015 -->|triggers| EB
    A2016 -->|triggers| EB
    EB -->|indexes| MI
    EB -->|processes| PP
    MI -->|writes| DB
    PP -->|writes| DB
    DB -->|serves data| API
    UI -->|calls| API
    Consumer -->|calls| API
    Consumer -->|reads| DB

    %% Click Events
    click A2013 "https://github.com/nyu-llm-ctf/llm_ctf_dataset_dev/tree/main/2013/"
    click A2014 "https://github.com/nyu-llm-ctf/llm_ctf_dataset_dev/tree/main/2014/"
    click A2015 "https://github.com/nyu-llm-ctf/llm_ctf_dataset_dev/tree/main/2015/"
    click A2016 "https://github.com/nyu-llm-ctf/llm_ctf_dataset_dev/tree/main/2016/"
    click README "https://github.com/nyu-llm-ctf/llm_ctf_dataset_dev/blob/main//README.md"

    %% Styles
    classDef storage fill:#ADD8E6,stroke:#000;
    classDef processing fill:#FFA500,stroke:#000;
    classDef persistence fill:#90EE90,stroke:#000;
    classDef interface fill:#D8BFD8,stroke:#000;
    classDef external fill:#D3D3D3,stroke:#000;
```
