# fredfung007/cityflow-nl architecture

> Architecture diagram of the GitHub repository fredfung007/cityflow-nl, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fredfung007/cityflow-nl
- Repository: https://github.com/fredfung007/cityflow-nl

## Overview

##### Project Type and Purpose
- This is an open-source baseline toolkit for a research challenge (AI City NLP vehicle retrieval).
- It provides data utilities, a configurable pipeline, a Siamese-style retrieval model, evaluation scripts, and example outputs.

##### File Structure
Top-level directories:
- data/
- JSON data files (train-tracks, test-tracks, test-queries)
- Script to extract frames (extract_vdo_frms.py)
- baseline/
- Configuration (config.py, default.yaml)
- Dataset definition (vehicle_retrieval_dataset.py)
- Model code (siamese_baseline_model.py)
- Workflow orchestration (workflow.py)
- Evaluation (evaluate.py)
- Submission prep (prepare_output_json.py)
- Utilities (utils.py)
- Requirements (requirements.txt)
- Sample results (baseline-results.json)

##### README for Insights
- The core pipeline: frame extraction → data loading → model inference → ranking → evaluation → submission JSON.
- Emphasis on MRR and Recall metrics.
- Configurable via default.yaml.

##### Main Components and Interactions
a. Data Storage & Input
- Source JSONs: train-tracks.json, test-tracks.json, test-queries.json
- Frame extraction script reads video sources (CityFlow benchmark) and writes image frames
b. Configuration Service
- default.yaml (hyperparameters, file paths)
- config.py loads and exposes settings
c. Data Preparation Module
- vehicle_retrieval_dataset.py: wraps JSON + frame files into dataset objects
- utils.py: helper functions (e.g., text processing, image loading)
d. Workflow Orchestrator
- workflow.py: CLI entry point that ties together dataset, model, evaluation, and output generation
e. Model Module
- siamese_baseline_model.py: implements the Siamese-style embedding model for text + image similarity
f. Inference & Ranking
- Within workflow.py, runs model over all query embeddings vs. track embeddings to rank candidates
g. Evaluation Module
- evaluate.py: computes MRR, Recall@5, Recall@10 against ground truth
h. Submission Preparation
- prepare_output_json.py: converts ranked lists into the required submission JSON format
i. Outputs
- baseline-results.json: sample submission / model outputs

##### Architectural Patterns & Design Principles
- Modular pipeline: each stage (data prep, model, evaluation) is a separate module
- Configuration-driven: default.yaml drives file paths, model parameters
- Single-process CLI orchestration (no microservices)
- Clear separation of concerns: data, model, evaluation, I/O

##### Detailed Component Breakdown
- Raw JSON files (train-tracks.json, test-tracks.json, test-queries.json)
- Extracted Frames (output of extract_vdo_frms.py)
[Configuration – Green Rectangle]
- default.yaml → config.py
[Data Prep – Green Oval]
- vehicle_retrieval_dataset.py
- utils.py
[Model – Green Rectangle]
- siamese_baseline_model.py
[Workflow Orchestrator – Green Diamond]
- workflow.py (CLI entry)
[Evaluation – Green Oval]
- evaluate.py
[Submission Prep – Green Oval]
- prepare_output_json.py
- baseline-results.json (submission)

##### Summary of Steps for the Engineer
1. Sketch the Data Layer on the left, Configuration above or next to it.
3. Dataset loader to the Model component.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        TrainJSON[(train-tracks.json)]:::blue
        TestTracksJSON[(test-tracks.json)]:::blue
        TestQueriesJSON[(test-queries.json)]:::blue
        ExtractScript((Frame Extraction Script)):::green
        Frames[(Extracted Frames)]:::blue
    end

    %% Configuration
    subgraph "Configuration"
        direction TB
        DefaultYAML["default.yaml"]:::green
        ConfigPy["config.py"]:::green
    end

    %% Baseline Pipeline
    subgraph "Baseline Pipeline"
        direction TB
        DatasetOval((vehicle_retrieval_dataset.py)):::green
        UtilsOval((utils.py)):::green
        ModelBox["siamese_baseline_model.py"]:::green
        WorkflowDiamond{"workflow.py"}:::green
        EvalOval((evaluate.py)):::green
        SubmitOval((prepare_output_json.py)):::green
        RequirementsTxt["requirements.txt"]:::green
    end

    %% Output Layer
    Output[(baseline-results.json)]:::orange

    %% Connections
    TrainJSON --> ExtractScript
    TestTracksJSON --> ExtractScript
    TestQueriesJSON --> ExtractScript
    ExtractScript --> Frames

    Frames --> DatasetOval
    TrainJSON --> DatasetOval
    TestTracksJSON --> DatasetOval
    TestQueriesJSON --> DatasetOval
    UtilsOval --> DatasetOval

    DefaultYAML --> ConfigPy
    ConfigPy --> DatasetOval
    ConfigPy --> ModelBox
    ConfigPy --> WorkflowDiamond

    WorkflowDiamond --> DatasetOval
    WorkflowDiamond --> ModelBox
    WorkflowDiamond --> EvalOval
    WorkflowDiamond --> SubmitOval

    DatasetOval -->|embeddings| ModelBox
    ModelBox -->|rankedList| EvalOval
    ModelBox -->|rankedList| SubmitOval
    EvalOval --> SubmitOval
    SubmitOval --> Output

    RequirementsTxt --> WorkflowDiamond

    %% Click Events
    click TrainJSON "https://github.com/fredfung007/cityflow-nl/blob/main/data/train-tracks.json"
    click TestTracksJSON "https://github.com/fredfung007/cityflow-nl/blob/main/data/test-tracks.json"
    click TestQueriesJSON "https://github.com/fredfung007/cityflow-nl/blob/main/data/test-queries.json"
    click DefaultYAML "https://github.com/fredfung007/cityflow-nl/blob/main/baseline/default.yaml"
    click ConfigPy "https://github.com/fredfung007/cityflow-nl/blob/main/baseline/config.py"
    click DatasetOval "https://github.com/fredfung007/cityflow-nl/blob/main/baseline/vehicle_retrieval_dataset.py"
    click UtilsOval "https://github.com/fredfung007/cityflow-nl/blob/main/baseline/utils.py"
    click WorkflowDiamond "https://github.com/fredfung007/cityflow-nl/blob/main/baseline/workflow.py"
    click ModelBox "https://github.com/fredfung007/cityflow-nl/blob/main/baseline/siamese_baseline_model.py"
    click EvalOval "https://github.com/fredfung007/cityflow-nl/blob/main/baseline/evaluate.py"
    click SubmitOval "https://github.com/fredfung007/cityflow-nl/blob/main/baseline/prepare_output_json.py"
    click Output "https://github.com/fredfung007/cityflow-nl/blob/main/baseline/baseline-results.json"
    click RequirementsTxt "https://github.com/fredfung007/cityflow-nl/blob/main/baseline/requirements.txt"

    %% Styles
    classDef blue fill:#blue,stroke:#000,color:#fff;
    classDef green fill:#2ecc71,stroke:#000,color:#000;
    classDef orange fill:#f39c12,stroke:#000,color:#000;
```
