# youcef3939/godspeed architecture

> Architecture diagram of the GitHub repository youcef3939/godspeed, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/youcef3939/godspeed
- Repository: https://github.com/youcef3939/godspeed

## Overview

1. **Overall shape**
   - Godspeed is a layered retrieval pipeline: local text files are ingested, normalized and chunked, embedded into vectors, stored in a FAISS index, then queried through a retriever and exposed via a FastAPI service.
   - The architecture is intentionally split so the same core pipeline can later swap in remote services or distributed infrastructure.

2. **Primary runtime boundary**
   - The current runtime is a Python application.
   - The API entrypoint is `godspeed/api/main.py`, which serves the query surface.
   - Core search logic is not embedded in the API; it sits behind a `SearchPipeline` orchestration layer.

3. **Input / ingestion subsystem**
   - `godspeed/ingestion/text_ingestor.py` reads a local corpus of `.txt` files from a directory.
   - This is the repository’s current source boundary: raw text files come in, and document metadata such as source path is preserved for citation later.

4. **Processing subsystem**
   - `godspeed/processing/chunker.py` is responsible for transforming ingested text into smaller semantic or fixed-size chunks.
   - This stage is where text becomes indexable units, and chunk identity becomes important for retrieval and citations.

5. **Embedding subsystem**
   - `godspeed/embedding/sentence_transformer.py` wraps a Sentence Transformers model.
   - The README indicates the model is `all-MiniLM-L6-v2`, so the current implementation depends on local embedding generation rather than an external embedding API.

6. **Storage / indexing subsystem**
   - `godspeed/storage/faiss_store.py` provides the vector index layer using FAISS.
   - This is a local vector store, meaning retrieval is backed by in-process or local filesystem–style indexing rather than a managed database.
   - The store is the key persistence/search boundary between offline indexing and online querying.

7. **Retrieval subsystem**
   - `godspeed/retrieval/retriever.py` performs the query-time similarity search and ranking.
   - It likely consumes the query embedding, searches FAISS, and returns scored chunk hits with associated metadata and citations.
   - This is the main “answer selection” boundary in the system.

8. **Orchestration layer**
   - `godspeed/pipeline.py` ties ingestion, chunking, embedding, storage, and retrieval into one end-to-end flow.
   - This is the central coordination point and the best place to draw the system’s main data path: corpus → chunks → embeddings/index → query → ranked hits.

9. **API layer**
   - `godspeed/api/main.py` exposes the pipeline over HTTP with at least `GET /health` and `POST /query`.
   - The API returns structured, machine-readable responses with ranks, scores, chunk IDs, document IDs, and source citations.
   - The API is a thin boundary: it should not own indexing logic, only request/response handling.

10. **Domain and contracts**
   - `godspeed/domain/models.py` defines the core entities used across layers, such as documents, chunks, hits, and query/result shapes.
   - `godspeed/interfaces/contracts.py` defines abstract contracts so ingestion, embedding, storage, and retrieval can be swapped without rewriting the pipeline.
   - These two modules are the main abstraction boundary between “what the system means” and “how the current implementation works.”

11. **Core utilities**
   - `godspeed/core/logging.py` centralizes logging behavior.
   - This is infrastructure support rather than business logic, but it matters because the pipeline has multiple stages and benefits from consistent traceability.

12. **Testing and validation**
   - `tests/test_pipeline.py` suggests the main verification focus is the integrated pipeline rather than isolated components only.
   - That fits the architecture: the important system behavior is end-to-end retrieval quality and correct wiring between ingestion, indexing, and response formatting.

13. **Key architectural constraint**
   - The repo currently favors a **local-first** implementation: local corpus, local embedding model, local FAISS index.
   - The interfaces package shows the design already anticipates replacing those pieces with distributed or managed equivalents later, without changing the API or orchestration contract.

14. **Diagram-friendly data flow**
   - A good architecture diagram would show one left-to-right path: local text corpus enters ingestion, moves through chunking and embedding, is indexed in FAISS, then query traffic enters the API, passes through the pipeline and retriever, and exits as ranked citation-backed results.

Main entry points: [`godspeed/pipeline.py`](https://github.com/youcef3939/godspeed/blob/main/godspeed/pipeline.py) (Pipeline), [`godspeed/retrieval/retriever.py`](https://github.com/youcef3939/godspeed/blob/main/godspeed/retrieval/retriever.py) (Retriever), [`godspeed/api/main.py`](https://github.com/youcef3939/godspeed/blob/main/godspeed/api/main.py) (API), [`godspeed/embedding/sentence_transformer.py`](https://github.com/youcef3939/godspeed/blob/main/godspeed/embedding/sentence_transformer.py) (Embedder)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_api_main["API<br/>[main.py]"]
  node_query(("Query<br/>Request"))
  node_ranked_results["Results<br/>Response payload"]
end

subgraph group_pipeline["Pipeline"]
  node_pipeline_main["Pipeline<br/>Orchestrator<br/>[pipeline.py]"]
  node_text_ingestor["Ingestor<br/>Text loader<br/>[text_ingestor.py]"]
  node_chunker["Chunker<br/>Text processor<br/>[chunker.py]"]
  node_embedder["Embedder<br/>SentenceTransformer"]
  node_faiss_store[("Vector store<br/>FAISS index<br/>[faiss_store.py]")]
  node_retriever["Retriever<br/>Similarity search<br/>[retriever.py]"]
  node_corpus["Local corpus<br/>Text files"]
  node_indexed_chunks["Chunks<br/>Indexed units"]
end

subgraph group_abstractions["Abstractions"]
  node_models["Models<br/>Domain schema<br/>[models.py]"]
  node_contracts["Contracts<br/>Interfaces<br/>[contracts.py]"]
end

subgraph group_support["Support"]
  node_logging["Logging<br/>Core infra<br/>[logging.py]"]
  node_tests["Tests<br/>Integration test<br/>[test_pipeline.py]"]
end

node_corpus -->|"load"| node_text_ingestor
node_text_ingestor -->|"normalize"| node_chunker
node_chunker -->|"produce"| node_indexed_chunks
node_indexed_chunks -->|"encode"| node_embedder
node_embedder -->|"index"| node_faiss_store
node_api_main -->|"delegate"| node_pipeline_main
node_pipeline_main -->|"query"| node_retriever
node_query -->|"submit"| node_api_main
node_retriever -->|"embed query"| node_embedder
node_retriever -->|"search"| node_faiss_store
node_retriever -->|"shape hits"| node_models
node_pipeline_main -->|"use"| node_models
node_pipeline_main -->|"depend on"| node_contracts
node_api_main -->|"serialize"| node_models
node_logging -.->|"trace"| node_pipeline_main
node_tests -.->|"verify"| node_pipeline_main
node_tests -.->|"verify"| node_api_main

click node_api_main "https://github.com/youcef3939/godspeed/blob/main/godspeed/api/main.py"
click node_pipeline_main "https://github.com/youcef3939/godspeed/blob/main/godspeed/pipeline.py"
click node_text_ingestor "https://github.com/youcef3939/godspeed/blob/main/godspeed/ingestion/text_ingestor.py"
click node_chunker "https://github.com/youcef3939/godspeed/blob/main/godspeed/processing/chunker.py"
click node_embedder "https://github.com/youcef3939/godspeed/blob/main/godspeed/embedding/sentence_transformer.py"
click node_faiss_store "https://github.com/youcef3939/godspeed/blob/main/godspeed/storage/faiss_store.py"
click node_retriever "https://github.com/youcef3939/godspeed/blob/main/godspeed/retrieval/retriever.py"
click node_models "https://github.com/youcef3939/godspeed/blob/main/godspeed/domain/models.py"
click node_contracts "https://github.com/youcef3939/godspeed/blob/main/godspeed/interfaces/contracts.py"
click node_logging "https://github.com/youcef3939/godspeed/blob/main/godspeed/core/logging.py"
click node_tests "https://github.com/youcef3939/godspeed/blob/main/tests/test_pipeline.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api_main,node_query,node_ranked_results toneBlue
class node_pipeline_main,node_text_ingestor,node_chunker,node_embedder,node_faiss_store,node_retriever,node_corpus,node_indexed_chunks toneAmber
class node_models,node_contracts toneMint
class node_logging,node_tests toneRose
```

## Components

### Runtime

Python application and HTTP entrypoint

- [API](https://github.com/youcef3939/godspeed/blob/main/godspeed/api/main.py): FastAPI entrypoint · Serves health and query endpoints over HTTP · `godspeed/api/main.py`
- **Query**: Request · User search input handled by the API
- **Results**: Response payload · Ranked hits with scores, chunk IDs, document IDs, and citations

### Pipeline

End-to-end retrieval flow from corpus to ranked results

- [Pipeline](https://github.com/youcef3939/godspeed/blob/main/godspeed/pipeline.py): Orchestrator · Coordinates ingestion, embedding, storage, and retrieval · `godspeed/pipeline.py`
- [Ingestor](https://github.com/youcef3939/godspeed/blob/main/godspeed/ingestion/text_ingestor.py): Text loader · Reads local .txt corpus and preserves source metadata · `godspeed/ingestion/text_ingestor.py`
- [Chunker](https://github.com/youcef3939/godspeed/blob/main/godspeed/processing/chunker.py): Text processor · Splits documents into retrieval-ready chunks · `godspeed/processing/chunker.py`
- [Embedder](https://github.com/youcef3939/godspeed/blob/main/godspeed/embedding/sentence_transformer.py): SentenceTransformer · Converts chunks and queries into vectors using a local model · `godspeed/embedding/sentence_transformer.py`
- [Vector store](https://github.com/youcef3939/godspeed/blob/main/godspeed/storage/faiss_store.py): FAISS index · Stores and searches dense vectors locally · `godspeed/storage/faiss_store.py`
- [Retriever](https://github.com/youcef3939/godspeed/blob/main/godspeed/retrieval/retriever.py): Similarity search · Ranks nearest chunks and returns scored hits with citations · `godspeed/retrieval/retriever.py`
- **Local corpus**: Text files · Source directory of raw .txt documents
- **Chunks**: Indexed units · Chunked document segments ready for embedding and citation

### Abstractions

Shared domain models and contracts

- [Models](https://github.com/youcef3939/godspeed/blob/main/godspeed/domain/models.py): Domain schema · Defines documents, chunks, hits, queries, and results · `godspeed/domain/models.py`
- [Contracts](https://github.com/youcef3939/godspeed/blob/main/godspeed/interfaces/contracts.py): Interfaces · Abstract boundaries for swappable ingestion, embedding, storage, and retrieval · `godspeed/interfaces/contracts.py`

### Support

Cross-cutting infrastructure

- [Logging](https://github.com/youcef3939/godspeed/blob/main/godspeed/core/logging.py): Core infra · Centralized logging and traceability support · `godspeed/core/logging.py`
- [Tests](https://github.com/youcef3939/godspeed/blob/main/tests/test_pipeline.py): Integration test · Validates the integrated pipeline behavior end to end · `tests/test_pipeline.py`

## Connections

- Local corpus → Ingestor: load · Read local text documents and retain source metadata
- Ingestor → Chunker: normalize · Pass ingested documents into chunk preparation
- Chunker → Chunks: produce · Create chunked retrieval units
- Chunks → Embedder: encode · Turn chunks into vector embeddings
- Embedder → Vector store: index · Persist embeddings into the local FAISS store
- API → Pipeline: delegate · Forward query handling to the orchestration layer
- Pipeline → Retriever: query · Run retrieval through the search component
- Query → API: submit · Incoming HTTP query request
- Retriever → Embedder: embed query · Convert the search text into a query vector
- Retriever → Vector store: search · Perform nearest-neighbor lookup over the vector index
- Retriever → Models: shape hits · Materialize scored result objects using domain schemas
- Pipeline → Models: use · Share domain entities across stages
- Pipeline → Contracts: depend on · Keep pipeline components swappable behind abstractions
- API → Models: serialize · Return structured results and health responses
- Logging → Pipeline: trace · Provide cross-stage execution logging
- Tests → Pipeline: verify · Exercise the end-to-end orchestration
- Tests → API: verify · Check exposed query behavior
